2024-09-21T21:51:51.758 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-21T21:51:51.773 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-21T21:51:51.852 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-09-21_21:16:02-rbd-squid-distro-default-smithi/7914449 branch: squid description: rbd/device/{base/install cluster/{fixed-3 openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-zlib supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/rbd_nbd} email: ceph-qa@ceph.com first_in_suite: false job_id: '7914449' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: teuthology-2024-09-21_21:16:02-rbd-squid-distro-default-smithi no_nested_subset: false openstack: - machine: cpus: 1 disk: 40 ram: 8000 volumes: count: 4 size: 30 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: squid ceph: conf: global: mon client directed command retry: 5 mon warn on pool no app: false ms inject socket failures: 5000 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bluestore block size: 96636764160 bluestore compression algorithm: zlib bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd failsafe full ratio: 0.95 osd max markdown count: 1000 osd objectstore: bluestore flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - mons down - mon down - out of quorum - noscrub - nodeep-scrub sha1: 4e45e6b490e6cbe082cdf155fea780438ec53fc7 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: extra_packages: - rbd-nbd flavor: default sha1: 4e45e6b490e6cbe082cdf155fea780438ec53fc7 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: 4e45e6b490e6cbe082cdf155fea780438ec53fc7 owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mon.c - osd.0 - osd.1 - osd.2 - - mon.b - mgr.x - osd.3 - osd.4 - osd.5 - - client.0 seed: 1478 sha1: 4e45e6b490e6cbe082cdf155fea780438ec53fc7 sleep_before_teardown: 0 subset: 24/128 suite: rbd suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_4e45e6b490e6cbe082cdf155fea780438ec53fc7/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 4e45e6b490e6cbe082cdf155fea780438ec53fc7 targets: smithi039.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDyux0JgXffDOQ+nsfEFEOKhwO/X/Bt8Gb5yN2jFdTItcwW1aLPNohY6/z+PMzIFLYvjZDYQH/kYCusuWbvP+D4= smithi064.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBF6erKmGp6kmZeoAG9Ue6BM47Vn1pF+/v2hXFcXpv+3xBTC7fMspop7VDOdLmI8TlUIiP0sazN9pCrUaICT4WBo= smithi157.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCM/xq8VesaVmUNWpxZDaK/NfCdrVkcERr1dVpooeRzYtZcb11Pn1TycotbeDAvEhSjn0DJD0j/kfDXkMuIOxRY= tasks: - install: null - ceph: null - thrashosds: timeout: 1200 - workunit: clients: client.0: - rbd/rbd-nbd.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-21_21:16:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2902 2024-09-21T21:51:51.852 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_4e45e6b490e6cbe082cdf155fea780438ec53fc7/qa; will attempt to use it 2024-09-21T21:51:51.853 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_4e45e6b490e6cbe082cdf155fea780438ec53fc7/qa/tasks 2024-09-21T21:51:51.853 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-21T21:51:51.855 INFO:teuthology.task.internal:Checking packages... 2024-09-21T21:51:51.874 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '4e45e6b490e6cbe082cdf155fea780438ec53fc7' 2024-09-21T21:51:51.874 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-21T21:51:51.874 INFO:teuthology.packaging:ref: None 2024-09-21T21:51:51.874 INFO:teuthology.packaging:tag: None 2024-09-21T21:51:51.874 INFO:teuthology.packaging:branch: squid 2024-09-21T21:51:51.874 INFO:teuthology.packaging:sha1: 4e45e6b490e6cbe082cdf155fea780438ec53fc7 2024-09-21T21:51:51.874 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=squid 2024-09-21T21:51:52.095 INFO:teuthology.task.internal:Found packages for ceph version 19.1.1-373-g4e45e6b4-1jammy 2024-09-21T21:51:52.096 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-21T21:51:52.108 INFO:teuthology.task.internal:no buildpackages task found 2024-09-21T21:51:52.109 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-21T21:51:52.126 INFO:teuthology.task.internal:Saving configuration 2024-09-21T21:51:52.138 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-21T21:51:52.151 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-21T21:51:52.178 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi039.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-21_21:16:02-rbd-squid-distro-default-smithi/7914449', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-21 21:45:15.066603', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAWkDJyAaT9Q7/wiyOjL7cYve2uZjqUjXFcWCrd0nSR2'} 2024-09-21T21:51:52.202 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi064.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-21_21:16:02-rbd-squid-distro-default-smithi/7914449', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-21 21:45:15.068536', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGBzkEEMK3debYQq/gfwPfWdss2JlX46bdDcA3ffx0rd'} 2024-09-21T21:51:52.224 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi157.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-21_21:16:02-rbd-squid-distro-default-smithi/7914449', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-21 21:45:15.070186', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFMSerA8zU57uq+OgOHwcqDp76YlNLuFbgHM+uFTGLs6'} 2024-09-21T21:51:52.224 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-21T21:51:52.231 INFO:teuthology.task.internal:roles: ubuntu@smithi039.front.sepia.ceph.com - ['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'] 2024-09-21T21:51:52.231 INFO:teuthology.task.internal:roles: ubuntu@smithi064.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5'] 2024-09-21T21:51:52.231 INFO:teuthology.task.internal:roles: ubuntu@smithi157.front.sepia.ceph.com - ['client.0'] 2024-09-21T21:51:52.231 INFO:teuthology.run_tasks:Running task console_log... 2024-09-21T21:51:52.315 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fe8aebf7490>, signals=[15]) 2024-09-21T21:51:52.315 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-21T21:51:52.321 INFO:teuthology.task.internal:Opening connections... 2024-09-21T21:51:52.321 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi039.front.sepia.ceph.com 2024-09-21T21:51:52.322 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T21:51:52.401 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi064.front.sepia.ceph.com 2024-09-21T21:51:52.402 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T21:51:52.479 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi157.front.sepia.ceph.com 2024-09-21T21:51:52.480 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T21:51:52.558 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-21T21:51:52.565 DEBUG:teuthology.orchestra.run.smithi039:> uname -m 2024-09-21T21:51:52.572 INFO:teuthology.orchestra.run.smithi039.stdout:x86_64 2024-09-21T21:51:52.572 DEBUG:teuthology.orchestra.run.smithi039:> cat /etc/os-release 2024-09-21T21:51:52.620 INFO:teuthology.orchestra.run.smithi039.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-09-21T21:51:52.621 INFO:teuthology.orchestra.run.smithi039.stdout:NAME="Ubuntu" 2024-09-21T21:51:52.621 INFO:teuthology.orchestra.run.smithi039.stdout:VERSION_ID="22.04" 2024-09-21T21:51:52.621 INFO:teuthology.orchestra.run.smithi039.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-09-21T21:51:52.621 INFO:teuthology.orchestra.run.smithi039.stdout:VERSION_CODENAME=jammy 2024-09-21T21:51:52.621 INFO:teuthology.orchestra.run.smithi039.stdout:ID=ubuntu 2024-09-21T21:51:52.621 INFO:teuthology.orchestra.run.smithi039.stdout:ID_LIKE=debian 2024-09-21T21:51:52.621 INFO:teuthology.orchestra.run.smithi039.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-21T21:51:52.621 INFO:teuthology.orchestra.run.smithi039.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-21T21:51:52.621 INFO:teuthology.orchestra.run.smithi039.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-21T21:51:52.621 INFO:teuthology.orchestra.run.smithi039.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-21T21:51:52.621 INFO:teuthology.orchestra.run.smithi039.stdout:UBUNTU_CODENAME=jammy 2024-09-21T21:51:52.622 INFO:teuthology.lock.ops:Updating smithi039.front.sepia.ceph.com on lock server 2024-09-21T21:51:52.638 DEBUG:teuthology.orchestra.run.smithi064:> uname -m 2024-09-21T21:51:52.644 INFO:teuthology.orchestra.run.smithi064.stdout:x86_64 2024-09-21T21:51:52.644 DEBUG:teuthology.orchestra.run.smithi064:> cat /etc/os-release 2024-09-21T21:51:52.692 INFO:teuthology.orchestra.run.smithi064.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-09-21T21:51:52.692 INFO:teuthology.orchestra.run.smithi064.stdout:NAME="Ubuntu" 2024-09-21T21:51:52.692 INFO:teuthology.orchestra.run.smithi064.stdout:VERSION_ID="22.04" 2024-09-21T21:51:52.692 INFO:teuthology.orchestra.run.smithi064.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-09-21T21:51:52.692 INFO:teuthology.orchestra.run.smithi064.stdout:VERSION_CODENAME=jammy 2024-09-21T21:51:52.692 INFO:teuthology.orchestra.run.smithi064.stdout:ID=ubuntu 2024-09-21T21:51:52.693 INFO:teuthology.orchestra.run.smithi064.stdout:ID_LIKE=debian 2024-09-21T21:51:52.693 INFO:teuthology.orchestra.run.smithi064.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-21T21:51:52.693 INFO:teuthology.orchestra.run.smithi064.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-21T21:51:52.693 INFO:teuthology.orchestra.run.smithi064.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-21T21:51:52.693 INFO:teuthology.orchestra.run.smithi064.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-21T21:51:52.693 INFO:teuthology.orchestra.run.smithi064.stdout:UBUNTU_CODENAME=jammy 2024-09-21T21:51:52.693 INFO:teuthology.lock.ops:Updating smithi064.front.sepia.ceph.com on lock server 2024-09-21T21:51:52.713 DEBUG:teuthology.orchestra.run.smithi157:> uname -m 2024-09-21T21:51:52.718 INFO:teuthology.orchestra.run.smithi157.stdout:x86_64 2024-09-21T21:51:52.719 DEBUG:teuthology.orchestra.run.smithi157:> cat /etc/os-release 2024-09-21T21:51:52.764 INFO:teuthology.orchestra.run.smithi157.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-09-21T21:51:52.764 INFO:teuthology.orchestra.run.smithi157.stdout:NAME="Ubuntu" 2024-09-21T21:51:52.764 INFO:teuthology.orchestra.run.smithi157.stdout:VERSION_ID="22.04" 2024-09-21T21:51:52.764 INFO:teuthology.orchestra.run.smithi157.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-09-21T21:51:52.764 INFO:teuthology.orchestra.run.smithi157.stdout:VERSION_CODENAME=jammy 2024-09-21T21:51:52.764 INFO:teuthology.orchestra.run.smithi157.stdout:ID=ubuntu 2024-09-21T21:51:52.764 INFO:teuthology.orchestra.run.smithi157.stdout:ID_LIKE=debian 2024-09-21T21:51:52.764 INFO:teuthology.orchestra.run.smithi157.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-21T21:51:52.764 INFO:teuthology.orchestra.run.smithi157.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-21T21:51:52.764 INFO:teuthology.orchestra.run.smithi157.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-21T21:51:52.765 INFO:teuthology.orchestra.run.smithi157.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-21T21:51:52.765 INFO:teuthology.orchestra.run.smithi157.stdout:UBUNTU_CODENAME=jammy 2024-09-21T21:51:52.765 INFO:teuthology.lock.ops:Updating smithi157.front.sepia.ceph.com on lock server 2024-09-21T21:51:52.788 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-21T21:51:52.796 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-21T21:51:52.803 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-21T21:51:52.803 DEBUG:teuthology.orchestra.run.smithi039:> test '!' -e /home/ubuntu/cephtest 2024-09-21T21:51:52.805 DEBUG:teuthology.orchestra.run.smithi064:> test '!' -e /home/ubuntu/cephtest 2024-09-21T21:51:52.808 DEBUG:teuthology.orchestra.run.smithi157:> test '!' -e /home/ubuntu/cephtest 2024-09-21T21:51:52.813 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-21T21:51:52.820 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-21T21:51:52.820 DEBUG:teuthology.orchestra.run.smithi039:> test -z $(ls -A /var/lib/ceph) 2024-09-21T21:51:52.854 DEBUG:teuthology.orchestra.run.smithi064:> test -z $(ls -A /var/lib/ceph) 2024-09-21T21:51:52.856 DEBUG:teuthology.orchestra.run.smithi157:> test -z $(ls -A /var/lib/ceph) 2024-09-21T21:51:52.866 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-21T21:51:52.945 INFO:teuthology.run_tasks:Running task kernel... 2024-09-21T21:51:52.960 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-21T21:51:52.960 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}, 'client.0': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-21T21:51:52.961 DEBUG:teuthology.orchestra.run.smithi039:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-21T21:51:52.962 DEBUG:teuthology.orchestra.run.smithi064:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-21T21:51:52.962 DEBUG:teuthology.orchestra.run.smithi157:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-21T21:51:52.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T21:51:52.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T21:51:52.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T21:51:52.971 DEBUG:teuthology.orchestra.run.smithi064:> uname -r 2024-09-21T21:51:52.971 DEBUG:teuthology.orchestra.run.smithi157:> uname -r 2024-09-21T21:51:52.972 DEBUG:teuthology.orchestra.run.smithi039:> uname -r 2024-09-21T21:51:53.013 INFO:teuthology.orchestra.run.smithi039.stdout:5.15.0-56-generic 2024-09-21T21:51:53.013 INFO:teuthology.task.kernel:Running kernel on smithi039: 5.15.0-56-generic 2024-09-21T21:51:53.013 DEBUG:teuthology.orchestra.run.smithi039:> sudo apt-get clean 2024-09-21T21:51:53.016 INFO:teuthology.orchestra.run.smithi157.stdout:5.15.0-56-generic 2024-09-21T21:51:53.016 INFO:teuthology.task.kernel:Running kernel on smithi157: 5.15.0-56-generic 2024-09-21T21:51:53.016 DEBUG:teuthology.orchestra.run.smithi157:> sudo apt-get clean 2024-09-21T21:51:53.017 INFO:teuthology.orchestra.run.smithi064.stdout:5.15.0-56-generic 2024-09-21T21:51:53.017 INFO:teuthology.task.kernel:Running kernel on smithi064: 5.15.0-56-generic 2024-09-21T21:51:53.017 DEBUG:teuthology.orchestra.run.smithi064:> sudo apt-get clean 2024-09-21T21:51:53.130 DEBUG:teuthology.orchestra.run.smithi157:> sudo apt-get update 2024-09-21T21:51:53.150 DEBUG:teuthology.orchestra.run.smithi064:> sudo apt-get update 2024-09-21T21:51:53.153 DEBUG:teuthology.orchestra.run.smithi039:> sudo apt-get update 2024-09-21T21:51:53.269 INFO:teuthology.orchestra.run.smithi157.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-21T21:51:53.292 INFO:teuthology.orchestra.run.smithi157.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2024-09-21T21:51:53.299 INFO:teuthology.orchestra.run.smithi039.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-21T21:51:53.299 INFO:teuthology.orchestra.run.smithi039.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2024-09-21T21:51:53.299 INFO:teuthology.orchestra.run.smithi039.stdout:Get:3 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2024-09-21T21:51:53.310 INFO:teuthology.orchestra.run.smithi064.stdout:Get:1 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2024-09-21T21:51:53.358 INFO:teuthology.orchestra.run.smithi157.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2024-09-21T21:51:53.371 INFO:teuthology.orchestra.run.smithi039.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2024-09-21T21:51:53.385 INFO:teuthology.orchestra.run.smithi157.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2024-09-21T21:51:53.416 INFO:teuthology.orchestra.run.smithi064.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-21T21:51:53.498 INFO:teuthology.orchestra.run.smithi064.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2024-09-21T21:51:53.504 INFO:teuthology.orchestra.run.smithi157.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [703 kB] 2024-09-21T21:51:53.525 INFO:teuthology.orchestra.run.smithi064.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [544 kB] 2024-09-21T21:51:53.527 INFO:teuthology.orchestra.run.smithi039.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,839 kB] 2024-09-21T21:51:53.577 INFO:teuthology.orchestra.run.smithi157.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [2,062 kB] 2024-09-21T21:51:53.610 INFO:teuthology.orchestra.run.smithi039.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [2,062 kB] 2024-09-21T21:51:53.613 INFO:teuthology.orchestra.run.smithi157.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [356 kB] 2024-09-21T21:51:53.616 INFO:teuthology.orchestra.run.smithi157.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [17.8 kB] 2024-09-21T21:51:53.616 INFO:teuthology.orchestra.run.smithi157.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [39.3 kB] 2024-09-21T21:51:53.617 INFO:teuthology.orchestra.run.smithi157.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [2,499 kB] 2024-09-21T21:51:53.621 INFO:teuthology.orchestra.run.smithi064.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,839 kB] 2024-09-21T21:51:53.628 INFO:teuthology.orchestra.run.smithi039.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [544 kB] 2024-09-21T21:51:53.632 INFO:teuthology.orchestra.run.smithi039.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [298 kB] 2024-09-21T21:51:53.633 INFO:teuthology.orchestra.run.smithi157.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [430 kB] 2024-09-21T21:51:53.635 INFO:teuthology.orchestra.run.smithi039.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [13.3 kB] 2024-09-21T21:51:53.635 INFO:teuthology.orchestra.run.smithi039.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [2,431 kB] 2024-09-21T21:51:53.636 INFO:teuthology.orchestra.run.smithi157.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [616 B] 2024-09-21T21:51:53.636 INFO:teuthology.orchestra.run.smithi157.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,125 kB] 2024-09-21T21:51:53.645 INFO:teuthology.orchestra.run.smithi157.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [733 kB] 2024-09-21T21:51:53.650 INFO:teuthology.orchestra.run.smithi157.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [261 kB] 2024-09-21T21:51:53.653 INFO:teuthology.orchestra.run.smithi157.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [26.1 kB] 2024-09-21T21:51:53.653 INFO:teuthology.orchestra.run.smithi157.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,752 B] 2024-09-21T21:51:53.654 INFO:teuthology.orchestra.run.smithi039.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [37.7 kB] 2024-09-21T21:51:53.655 INFO:teuthology.orchestra.run.smithi039.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [418 kB] 2024-09-21T21:51:53.658 INFO:teuthology.orchestra.run.smithi039.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [584 B] 2024-09-21T21:51:53.658 INFO:teuthology.orchestra.run.smithi039.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [629 kB] 2024-09-21T21:51:53.659 INFO:teuthology.orchestra.run.smithi064.stdout:Get:6 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [298 kB] 2024-09-21T21:51:53.661 INFO:teuthology.orchestra.run.smithi064.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [13.3 kB] 2024-09-21T21:51:53.662 INFO:teuthology.orchestra.run.smithi064.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [37.7 kB] 2024-09-21T21:51:53.665 INFO:teuthology.orchestra.run.smithi064.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [2,431 kB] 2024-09-21T21:51:53.679 INFO:teuthology.orchestra.run.smithi157.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [43.3 kB] 2024-09-21T21:51:53.680 INFO:teuthology.orchestra.run.smithi157.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.8 kB] 2024-09-21T21:51:53.680 INFO:teuthology.orchestra.run.smithi157.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [444 B] 2024-09-21T21:51:53.680 INFO:teuthology.orchestra.run.smithi157.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.9 kB] 2024-09-21T21:51:53.681 INFO:teuthology.orchestra.run.smithi039.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [905 kB] 2024-09-21T21:51:53.681 INFO:teuthology.orchestra.run.smithi157.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.8 kB] 2024-09-21T21:51:53.682 INFO:teuthology.orchestra.run.smithi157.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.1 kB] 2024-09-21T21:51:53.684 INFO:teuthology.orchestra.run.smithi157.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-09-21T21:51:53.685 INFO:teuthology.orchestra.run.smithi064.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [418 kB] 2024-09-21T21:51:53.685 INFO:teuthology.orchestra.run.smithi157.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.8 kB] 2024-09-21T21:51:53.687 INFO:teuthology.orchestra.run.smithi039.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [177 kB] 2024-09-21T21:51:53.689 INFO:teuthology.orchestra.run.smithi064.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [584 B] 2024-09-21T21:51:53.689 INFO:teuthology.orchestra.run.smithi064.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [905 kB] 2024-09-21T21:51:53.696 INFO:teuthology.orchestra.run.smithi064.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [629 kB] 2024-09-21T21:51:53.699 INFO:teuthology.orchestra.run.smithi039.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [703 kB] 2024-09-21T21:51:53.701 INFO:teuthology.orchestra.run.smithi064.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [177 kB] 2024-09-21T21:51:53.702 INFO:teuthology.orchestra.run.smithi157.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.3 kB] 2024-09-21T21:51:53.702 INFO:teuthology.orchestra.run.smithi157.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-09-21T21:51:53.703 INFO:teuthology.orchestra.run.smithi157.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [672 B] 2024-09-21T21:51:53.703 INFO:teuthology.orchestra.run.smithi064.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [19.3 kB] 2024-09-21T21:51:53.703 INFO:teuthology.orchestra.run.smithi064.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-09-21T21:51:53.703 INFO:teuthology.orchestra.run.smithi064.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-09-21T21:51:53.703 INFO:teuthology.orchestra.run.smithi064.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-09-21T21:51:53.713 INFO:teuthology.orchestra.run.smithi064.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [228 B] 2024-09-21T21:51:53.724 INFO:teuthology.orchestra.run.smithi039.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [356 kB] 2024-09-21T21:51:53.724 INFO:teuthology.orchestra.run.smithi039.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [19.3 kB] 2024-09-21T21:51:53.724 INFO:teuthology.orchestra.run.smithi039.stdout:Get:20 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-09-21T21:51:53.765 INFO:teuthology.orchestra.run.smithi039.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [17.8 kB] 2024-09-21T21:51:53.765 INFO:teuthology.orchestra.run.smithi039.stdout:Get:22 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-09-21T21:51:53.765 INFO:teuthology.orchestra.run.smithi039.stdout:Get:23 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-09-21T21:51:53.766 INFO:teuthology.orchestra.run.smithi039.stdout:Get:24 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [228 B] 2024-09-21T21:51:53.766 INFO:teuthology.orchestra.run.smithi039.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [2,499 kB] 2024-09-21T21:51:53.808 INFO:teuthology.orchestra.run.smithi039.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [39.3 kB] 2024-09-21T21:51:53.809 INFO:teuthology.orchestra.run.smithi039.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [430 kB] 2024-09-21T21:51:53.815 INFO:teuthology.orchestra.run.smithi039.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [616 B] 2024-09-21T21:51:53.816 INFO:teuthology.orchestra.run.smithi039.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,125 kB] 2024-09-21T21:51:53.829 INFO:teuthology.orchestra.run.smithi039.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [733 kB] 2024-09-21T21:51:53.832 INFO:teuthology.orchestra.run.smithi157.stdout:Get:29 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,839 kB] 2024-09-21T21:51:53.836 INFO:teuthology.orchestra.run.smithi039.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [261 kB] 2024-09-21T21:51:53.839 INFO:teuthology.orchestra.run.smithi039.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [26.1 kB] 2024-09-21T21:51:53.839 INFO:teuthology.orchestra.run.smithi039.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,752 B] 2024-09-21T21:51:53.839 INFO:teuthology.orchestra.run.smithi039.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [43.3 kB] 2024-09-21T21:51:53.840 INFO:teuthology.orchestra.run.smithi039.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.8 kB] 2024-09-21T21:51:53.845 INFO:teuthology.orchestra.run.smithi039.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [444 B] 2024-09-21T21:51:53.846 INFO:teuthology.orchestra.run.smithi039.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.9 kB] 2024-09-21T21:51:53.846 INFO:teuthology.orchestra.run.smithi039.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.8 kB] 2024-09-21T21:51:53.853 INFO:teuthology.orchestra.run.smithi064.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2024-09-21T21:51:53.858 INFO:teuthology.orchestra.run.smithi039.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.1 kB] 2024-09-21T21:51:53.859 INFO:teuthology.orchestra.run.smithi039.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-09-21T21:51:53.859 INFO:teuthology.orchestra.run.smithi039.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.8 kB] 2024-09-21T21:51:53.859 INFO:teuthology.orchestra.run.smithi039.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.3 kB] 2024-09-21T21:51:53.859 INFO:teuthology.orchestra.run.smithi039.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-09-21T21:51:53.862 INFO:teuthology.orchestra.run.smithi039.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [672 B] 2024-09-21T21:51:53.991 INFO:teuthology.orchestra.run.smithi064.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [703 kB] 2024-09-21T21:51:54.185 INFO:teuthology.orchestra.run.smithi064.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [2,062 kB] 2024-09-21T21:51:54.254 INFO:teuthology.orchestra.run.smithi157.stdout:Get:30 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [544 kB] 2024-09-21T21:51:54.270 INFO:teuthology.orchestra.run.smithi157.stdout:Get:31 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [298 kB] 2024-09-21T21:51:54.279 INFO:teuthology.orchestra.run.smithi157.stdout:Get:32 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [13.3 kB] 2024-09-21T21:51:54.279 INFO:teuthology.orchestra.run.smithi157.stdout:Get:33 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [37.7 kB] 2024-09-21T21:51:54.280 INFO:teuthology.orchestra.run.smithi157.stdout:Get:34 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [2,431 kB] 2024-09-21T21:51:54.345 INFO:teuthology.orchestra.run.smithi064.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [356 kB] 2024-09-21T21:51:54.359 INFO:teuthology.orchestra.run.smithi064.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [17.8 kB] 2024-09-21T21:51:54.359 INFO:teuthology.orchestra.run.smithi064.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [2,499 kB] 2024-09-21T21:51:54.376 INFO:teuthology.orchestra.run.smithi157.stdout:Get:35 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [418 kB] 2024-09-21T21:51:54.388 INFO:teuthology.orchestra.run.smithi157.stdout:Get:36 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [584 B] 2024-09-21T21:51:54.388 INFO:teuthology.orchestra.run.smithi157.stdout:Get:37 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [905 kB] 2024-09-21T21:51:54.411 INFO:teuthology.orchestra.run.smithi157.stdout:Get:38 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [629 kB] 2024-09-21T21:51:54.429 INFO:teuthology.orchestra.run.smithi064.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [39.3 kB] 2024-09-21T21:51:54.432 INFO:teuthology.orchestra.run.smithi157.stdout:Get:39 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [177 kB] 2024-09-21T21:51:54.433 INFO:teuthology.orchestra.run.smithi064.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [430 kB] 2024-09-21T21:51:54.439 INFO:teuthology.orchestra.run.smithi157.stdout:Get:40 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [19.3 kB] 2024-09-21T21:51:54.439 INFO:teuthology.orchestra.run.smithi157.stdout:Get:41 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-09-21T21:51:54.440 INFO:teuthology.orchestra.run.smithi157.stdout:Get:42 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-09-21T21:51:54.440 INFO:teuthology.orchestra.run.smithi157.stdout:Get:43 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-09-21T21:51:54.449 INFO:teuthology.orchestra.run.smithi064.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [616 B] 2024-09-21T21:51:54.449 INFO:teuthology.orchestra.run.smithi064.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [733 kB] 2024-09-21T21:51:54.467 INFO:teuthology.orchestra.run.smithi064.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,125 kB] 2024-09-21T21:51:54.498 INFO:teuthology.orchestra.run.smithi064.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [261 kB] 2024-09-21T21:51:54.503 INFO:teuthology.orchestra.run.smithi064.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [26.1 kB] 2024-09-21T21:51:54.509 INFO:teuthology.orchestra.run.smithi064.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,752 B] 2024-09-21T21:51:54.509 INFO:teuthology.orchestra.run.smithi064.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [43.3 kB] 2024-09-21T21:51:54.515 INFO:teuthology.orchestra.run.smithi157.stdout:Get:44 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [228 B] 2024-09-21T21:51:54.581 INFO:teuthology.orchestra.run.smithi064.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.8 kB] 2024-09-21T21:51:54.583 INFO:teuthology.orchestra.run.smithi064.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [444 B] 2024-09-21T21:51:54.593 INFO:teuthology.orchestra.run.smithi064.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.9 kB] 2024-09-21T21:51:54.593 INFO:teuthology.orchestra.run.smithi064.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.8 kB] 2024-09-21T21:51:54.595 INFO:teuthology.orchestra.run.smithi064.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.1 kB] 2024-09-21T21:51:54.595 INFO:teuthology.orchestra.run.smithi064.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-09-21T21:51:54.595 INFO:teuthology.orchestra.run.smithi064.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.3 kB] 2024-09-21T21:51:54.595 INFO:teuthology.orchestra.run.smithi064.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.8 kB] 2024-09-21T21:51:54.671 INFO:teuthology.orchestra.run.smithi064.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-09-21T21:51:54.672 INFO:teuthology.orchestra.run.smithi064.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [672 B] 2024-09-21T21:52:00.093 INFO:teuthology.orchestra.run.smithi157.stdout:Fetched 16.3 MB in 2s (7,417 kB/s) 2024-09-21T21:52:00.292 INFO:teuthology.orchestra.run.smithi064.stdout:Fetched 16.3 MB in 2s (7,233 kB/s) 2024-09-21T21:52:00.549 INFO:teuthology.orchestra.run.smithi039.stdout:Fetched 16.3 MB in 2s (7,175 kB/s) 2024-09-21T21:52:01.655 INFO:teuthology.orchestra.run.smithi157.stdout:Reading package lists... 2024-09-21T21:52:01.673 DEBUG:teuthology.orchestra.run.smithi157:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-21T21:52:01.737 INFO:teuthology.orchestra.run.smithi157.stdout:Reading package lists... 2024-09-21T21:52:01.842 INFO:teuthology.orchestra.run.smithi064.stdout:Reading package lists... 2024-09-21T21:52:01.860 DEBUG:teuthology.orchestra.run.smithi064:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-21T21:52:01.926 INFO:teuthology.orchestra.run.smithi064.stdout:Reading package lists... 2024-09-21T21:52:01.929 INFO:teuthology.orchestra.run.smithi039.stdout:Reading package lists... 2024-09-21T21:52:01.947 DEBUG:teuthology.orchestra.run.smithi039:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-21T21:52:01.956 INFO:teuthology.orchestra.run.smithi157.stdout:Building dependency tree... 2024-09-21T21:52:01.956 INFO:teuthology.orchestra.run.smithi157.stdout:Reading state information... 2024-09-21T21:52:02.016 INFO:teuthology.orchestra.run.smithi039.stdout:Reading package lists... 2024-09-21T21:52:02.123 INFO:teuthology.orchestra.run.smithi064.stdout:Building dependency tree... 2024-09-21T21:52:02.123 INFO:teuthology.orchestra.run.smithi064.stdout:Reading state information... 2024-09-21T21:52:02.145 INFO:teuthology.orchestra.run.smithi157.stdout:The following packages were automatically installed and are no longer required: 2024-09-21T21:52:02.145 INFO:teuthology.orchestra.run.smithi157.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-21T21:52:02.145 INFO:teuthology.orchestra.run.smithi157.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-21T21:52:02.146 INFO:teuthology.orchestra.run.smithi157.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-21T21:52:02.146 INFO:teuthology.orchestra.run.smithi157.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-09-21T21:52:02.146 INFO:teuthology.orchestra.run.smithi157.stdout: python2.7-minimal 2024-09-21T21:52:02.146 INFO:teuthology.orchestra.run.smithi157.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-21T21:52:02.147 INFO:teuthology.orchestra.run.smithi157.stdout:The following additional packages will be installed: 2024-09-21T21:52:02.147 INFO:teuthology.orchestra.run.smithi157.stdout: linux-generic linux-headers-5.15.0-122 linux-headers-5.15.0-122-generic 2024-09-21T21:52:02.147 INFO:teuthology.orchestra.run.smithi157.stdout: linux-headers-generic linux-image-5.15.0-122-generic 2024-09-21T21:52:02.148 INFO:teuthology.orchestra.run.smithi157.stdout: linux-modules-5.15.0-122-generic linux-modules-extra-5.15.0-122-generic 2024-09-21T21:52:02.149 INFO:teuthology.orchestra.run.smithi157.stdout:Suggested packages: 2024-09-21T21:52:02.149 INFO:teuthology.orchestra.run.smithi157.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-09-21T21:52:02.149 INFO:teuthology.orchestra.run.smithi157.stdout:Recommended packages: 2024-09-21T21:52:02.149 INFO:teuthology.orchestra.run.smithi157.stdout: thermald 2024-09-21T21:52:02.197 INFO:teuthology.orchestra.run.smithi157.stdout:The following NEW packages will be installed: 2024-09-21T21:52:02.198 INFO:teuthology.orchestra.run.smithi157.stdout: linux-headers-5.15.0-122 linux-headers-5.15.0-122-generic 2024-09-21T21:52:02.198 INFO:teuthology.orchestra.run.smithi157.stdout: linux-image-5.15.0-122-generic linux-modules-5.15.0-122-generic 2024-09-21T21:52:02.198 INFO:teuthology.orchestra.run.smithi157.stdout: linux-modules-extra-5.15.0-122-generic 2024-09-21T21:52:02.199 INFO:teuthology.orchestra.run.smithi157.stdout:The following packages will be upgraded: 2024-09-21T21:52:02.200 INFO:teuthology.orchestra.run.smithi157.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-21T21:52:02.220 INFO:teuthology.orchestra.run.smithi039.stdout:Building dependency tree... 2024-09-21T21:52:02.220 INFO:teuthology.orchestra.run.smithi039.stdout:Reading state information... 2024-09-21T21:52:02.251 INFO:teuthology.orchestra.run.smithi157.stdout:3 upgraded, 5 newly installed, 0 to remove and 345 not upgraded. 2024-09-21T21:52:02.251 INFO:teuthology.orchestra.run.smithi157.stdout:Need to get 113 MB of archives. 2024-09-21T21:52:02.251 INFO:teuthology.orchestra.run.smithi157.stdout:After this operation, 584 MB of additional disk space will be used. 2024-09-21T21:52:02.251 INFO:teuthology.orchestra.run.smithi157.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-122-generic amd64 5.15.0-122.132 [22.7 MB] 2024-09-21T21:52:02.300 INFO:teuthology.orchestra.run.smithi064.stdout:The following packages were automatically installed and are no longer required: 2024-09-21T21:52:02.300 INFO:teuthology.orchestra.run.smithi064.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-21T21:52:02.300 INFO:teuthology.orchestra.run.smithi064.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-21T21:52:02.301 INFO:teuthology.orchestra.run.smithi064.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-21T21:52:02.301 INFO:teuthology.orchestra.run.smithi064.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-09-21T21:52:02.301 INFO:teuthology.orchestra.run.smithi064.stdout: python2.7-minimal 2024-09-21T21:52:02.301 INFO:teuthology.orchestra.run.smithi064.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-21T21:52:02.302 INFO:teuthology.orchestra.run.smithi064.stdout:The following additional packages will be installed: 2024-09-21T21:52:02.303 INFO:teuthology.orchestra.run.smithi064.stdout: linux-generic linux-headers-5.15.0-122 linux-headers-5.15.0-122-generic 2024-09-21T21:52:02.303 INFO:teuthology.orchestra.run.smithi064.stdout: linux-headers-generic linux-image-5.15.0-122-generic 2024-09-21T21:52:02.303 INFO:teuthology.orchestra.run.smithi064.stdout: linux-modules-5.15.0-122-generic linux-modules-extra-5.15.0-122-generic 2024-09-21T21:52:02.304 INFO:teuthology.orchestra.run.smithi064.stdout:Suggested packages: 2024-09-21T21:52:02.304 INFO:teuthology.orchestra.run.smithi064.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-09-21T21:52:02.304 INFO:teuthology.orchestra.run.smithi064.stdout:Recommended packages: 2024-09-21T21:52:02.305 INFO:teuthology.orchestra.run.smithi064.stdout: thermald 2024-09-21T21:52:02.358 INFO:teuthology.orchestra.run.smithi064.stdout:The following NEW packages will be installed: 2024-09-21T21:52:02.359 INFO:teuthology.orchestra.run.smithi064.stdout: linux-headers-5.15.0-122 linux-headers-5.15.0-122-generic 2024-09-21T21:52:02.359 INFO:teuthology.orchestra.run.smithi064.stdout: linux-image-5.15.0-122-generic linux-modules-5.15.0-122-generic 2024-09-21T21:52:02.359 INFO:teuthology.orchestra.run.smithi064.stdout: linux-modules-extra-5.15.0-122-generic 2024-09-21T21:52:02.360 INFO:teuthology.orchestra.run.smithi064.stdout:The following packages will be upgraded: 2024-09-21T21:52:02.360 INFO:teuthology.orchestra.run.smithi064.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-21T21:52:02.394 INFO:teuthology.orchestra.run.smithi039.stdout:The following packages were automatically installed and are no longer required: 2024-09-21T21:52:02.394 INFO:teuthology.orchestra.run.smithi039.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-21T21:52:02.394 INFO:teuthology.orchestra.run.smithi039.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-21T21:52:02.394 INFO:teuthology.orchestra.run.smithi039.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-21T21:52:02.394 INFO:teuthology.orchestra.run.smithi039.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-09-21T21:52:02.394 INFO:teuthology.orchestra.run.smithi039.stdout: python2.7-minimal 2024-09-21T21:52:02.394 INFO:teuthology.orchestra.run.smithi039.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-21T21:52:02.396 INFO:teuthology.orchestra.run.smithi039.stdout:The following additional packages will be installed: 2024-09-21T21:52:02.396 INFO:teuthology.orchestra.run.smithi039.stdout: linux-generic linux-headers-5.15.0-122 linux-headers-5.15.0-122-generic 2024-09-21T21:52:02.396 INFO:teuthology.orchestra.run.smithi039.stdout: linux-headers-generic linux-image-5.15.0-122-generic 2024-09-21T21:52:02.396 INFO:teuthology.orchestra.run.smithi039.stdout: linux-modules-5.15.0-122-generic linux-modules-extra-5.15.0-122-generic 2024-09-21T21:52:02.397 INFO:teuthology.orchestra.run.smithi039.stdout:Suggested packages: 2024-09-21T21:52:02.398 INFO:teuthology.orchestra.run.smithi039.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-09-21T21:52:02.398 INFO:teuthology.orchestra.run.smithi039.stdout:Recommended packages: 2024-09-21T21:52:02.398 INFO:teuthology.orchestra.run.smithi039.stdout: thermald 2024-09-21T21:52:02.412 INFO:teuthology.orchestra.run.smithi064.stdout:3 upgraded, 5 newly installed, 0 to remove and 345 not upgraded. 2024-09-21T21:52:02.412 INFO:teuthology.orchestra.run.smithi064.stdout:Need to get 113 MB of archives. 2024-09-21T21:52:02.413 INFO:teuthology.orchestra.run.smithi064.stdout:After this operation, 584 MB of additional disk space will be used. 2024-09-21T21:52:02.413 INFO:teuthology.orchestra.run.smithi064.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-122-generic amd64 5.15.0-122.132 [22.7 MB] 2024-09-21T21:52:02.445 INFO:teuthology.orchestra.run.smithi039.stdout:The following NEW packages will be installed: 2024-09-21T21:52:02.446 INFO:teuthology.orchestra.run.smithi039.stdout: linux-headers-5.15.0-122 linux-headers-5.15.0-122-generic 2024-09-21T21:52:02.446 INFO:teuthology.orchestra.run.smithi039.stdout: linux-image-5.15.0-122-generic linux-modules-5.15.0-122-generic 2024-09-21T21:52:02.446 INFO:teuthology.orchestra.run.smithi039.stdout: linux-modules-extra-5.15.0-122-generic 2024-09-21T21:52:02.447 INFO:teuthology.orchestra.run.smithi039.stdout:The following packages will be upgraded: 2024-09-21T21:52:02.448 INFO:teuthology.orchestra.run.smithi039.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-21T21:52:02.498 INFO:teuthology.orchestra.run.smithi039.stdout:3 upgraded, 5 newly installed, 0 to remove and 345 not upgraded. 2024-09-21T21:52:02.498 INFO:teuthology.orchestra.run.smithi039.stdout:Need to get 113 MB of archives. 2024-09-21T21:52:02.498 INFO:teuthology.orchestra.run.smithi039.stdout:After this operation, 584 MB of additional disk space will be used. 2024-09-21T21:52:02.498 INFO:teuthology.orchestra.run.smithi039.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-122-generic amd64 5.15.0-122.132 [22.7 MB] 2024-09-21T21:52:02.690 INFO:teuthology.orchestra.run.smithi157.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-122-generic amd64 5.15.0-122.132 [11.5 MB] 2024-09-21T21:52:02.779 INFO:teuthology.orchestra.run.smithi157.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-122-generic amd64 5.15.0-122.132 [63.9 MB] 2024-09-21T21:52:02.852 INFO:teuthology.orchestra.run.smithi064.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-122-generic amd64 5.15.0-122.132 [11.5 MB] 2024-09-21T21:52:02.934 INFO:teuthology.orchestra.run.smithi039.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-122-generic amd64 5.15.0-122.132 [11.5 MB] 2024-09-21T21:52:02.959 INFO:teuthology.orchestra.run.smithi064.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-122-generic amd64 5.15.0-122.132 [63.9 MB] 2024-09-21T21:52:03.031 INFO:teuthology.orchestra.run.smithi039.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-122-generic amd64 5.15.0-122.132 [63.9 MB] 2024-09-21T21:52:03.542 INFO:teuthology.orchestra.run.smithi157.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.122.122 [1,692 B] 2024-09-21T21:52:03.542 INFO:teuthology.orchestra.run.smithi157.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.122.122 [2,436 B] 2024-09-21T21:52:03.542 INFO:teuthology.orchestra.run.smithi157.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-122 all 5.15.0-122.132 [12.3 MB] 2024-09-21T21:52:03.663 INFO:teuthology.orchestra.run.smithi157.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-122-generic amd64 5.15.0-122.132 [2,827 kB] 2024-09-21T21:52:03.739 INFO:teuthology.orchestra.run.smithi064.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.122.122 [1,692 B] 2024-09-21T21:52:03.740 INFO:teuthology.orchestra.run.smithi064.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.122.122 [2,436 B] 2024-09-21T21:52:03.740 INFO:teuthology.orchestra.run.smithi064.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-122 all 5.15.0-122.132 [12.3 MB] 2024-09-21T21:52:03.767 INFO:teuthology.orchestra.run.smithi157.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.122.122 [2,294 B] 2024-09-21T21:52:03.845 INFO:teuthology.orchestra.run.smithi039.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.122.122 [1,692 B] 2024-09-21T21:52:03.845 INFO:teuthology.orchestra.run.smithi039.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.122.122 [2,436 B] 2024-09-21T21:52:03.845 INFO:teuthology.orchestra.run.smithi039.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-122 all 5.15.0-122.132 [12.3 MB] 2024-09-21T21:52:03.863 INFO:teuthology.orchestra.run.smithi064.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-122-generic amd64 5.15.0-122.132 [2,827 kB] 2024-09-21T21:52:03.885 INFO:teuthology.orchestra.run.smithi064.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.122.122 [2,294 B] 2024-09-21T21:52:03.973 INFO:teuthology.orchestra.run.smithi039.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-122-generic amd64 5.15.0-122.132 [2,827 kB] 2024-09-21T21:52:03.995 INFO:teuthology.orchestra.run.smithi039.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.122.122 [2,294 B] 2024-09-21T21:52:04.180 INFO:teuthology.orchestra.run.smithi157.stdout:Fetched 113 MB in 2s (72.6 MB/s) 2024-09-21T21:52:04.342 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package linux-modules-5.15.0-122-generic. 2024-09-21T21:52:04.369 INFO:teuthology.orchestra.run.smithi064.stdout:Fetched 113 MB in 2s (71.3 MB/s) 2024-09-21T21:52:04.417 INFO:teuthology.orchestra.run.smithi039.stdout:Fetched 113 MB in 2s (73.6 MB/s) 2024-09-21T21:52:04.552 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package linux-modules-5.15.0-122-generic. 2024-09-21T21:52:04.565 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package linux-modules-5.15.0-122-generic. 2024-09-21T21:52:06.347 INFO:teuthology.orchestra.run.smithi157.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-09-21T21:52:06.352 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../0-linux-modules-5.15.0-122-generic_5.15.0-122.132_amd64.deb ... 2024-09-21T21:52:06.546 INFO:teuthology.orchestra.run.smithi064.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-09-21T21:52:06.548 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking linux-modules-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-21T21:52:06.550 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../0-linux-modules-5.15.0-122-generic_5.15.0-122.132_amd64.deb ... 2024-09-21T21:52:06.606 INFO:teuthology.orchestra.run.smithi039.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-09-21T21:52:06.610 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../0-linux-modules-5.15.0-122-generic_5.15.0-122.132_amd64.deb ... 2024-09-21T21:52:06.741 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking linux-modules-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-21T21:52:06.845 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking linux-modules-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-21T21:52:08.708 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package linux-image-5.15.0-122-generic. 2024-09-21T21:52:08.729 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../1-linux-image-5.15.0-122-generic_5.15.0-122.132_amd64.deb ... 2024-09-21T21:52:08.814 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking linux-image-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-21T21:52:08.954 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package linux-image-5.15.0-122-generic. 2024-09-21T21:52:08.975 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../1-linux-image-5.15.0-122-generic_5.15.0-122.132_amd64.deb ... 2024-09-21T21:52:09.055 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package linux-image-5.15.0-122-generic. 2024-09-21T21:52:09.073 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking linux-image-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-21T21:52:09.077 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../1-linux-image-5.15.0-122-generic_5.15.0-122.132_amd64.deb ... 2024-09-21T21:52:09.164 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking linux-image-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-21T21:52:09.310 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-122-generic. 2024-09-21T21:52:09.331 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-122-generic_5.15.0-122.132_amd64.deb ... 2024-09-21T21:52:09.369 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking linux-modules-extra-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-21T21:52:09.548 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-122-generic. 2024-09-21T21:52:09.569 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-122-generic_5.15.0-122.132_amd64.deb ... 2024-09-21T21:52:09.606 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking linux-modules-extra-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-21T21:52:09.661 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-122-generic. 2024-09-21T21:52:09.682 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-122-generic_5.15.0-122.132_amd64.deb ... 2024-09-21T21:52:09.881 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking linux-modules-extra-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-21T21:52:15.388 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../3-linux-generic_5.15.0.122.122_amd64.deb ... 2024-09-21T21:52:15.497 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking linux-generic (5.15.0.122.122) over (5.15.0.56.54) ... 2024-09-21T21:52:15.797 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.122.122_amd64.deb ... 2024-09-21T21:52:15.917 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking linux-image-generic (5.15.0.122.122) over (5.15.0.56.54) ... 2024-09-21T21:52:16.158 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../3-linux-generic_5.15.0.122.122_amd64.deb ... 2024-09-21T21:52:16.220 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package linux-headers-5.15.0-122. 2024-09-21T21:52:16.232 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../5-linux-headers-5.15.0-122_5.15.0-122.132_all.deb ... 2024-09-21T21:52:16.270 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking linux-headers-5.15.0-122 (5.15.0-122.132) ... 2024-09-21T21:52:16.280 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking linux-generic (5.15.0.122.122) over (5.15.0.56.54) ... 2024-09-21T21:52:16.322 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../3-linux-generic_5.15.0.122.122_amd64.deb ... 2024-09-21T21:52:16.430 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking linux-generic (5.15.0.122.122) over (5.15.0.56.54) ... 2024-09-21T21:52:16.596 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.122.122_amd64.deb ... 2024-09-21T21:52:16.717 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking linux-image-generic (5.15.0.122.122) over (5.15.0.56.54) ... 2024-09-21T21:52:16.731 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.122.122_amd64.deb ... 2024-09-21T21:52:16.883 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking linux-image-generic (5.15.0.122.122) over (5.15.0.56.54) ... 2024-09-21T21:52:17.037 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package linux-headers-5.15.0-122. 2024-09-21T21:52:17.059 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../5-linux-headers-5.15.0-122_5.15.0-122.132_all.deb ... 2024-09-21T21:52:17.103 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking linux-headers-5.15.0-122 (5.15.0-122.132) ... 2024-09-21T21:52:17.195 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package linux-headers-5.15.0-122. 2024-09-21T21:52:17.217 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../5-linux-headers-5.15.0-122_5.15.0-122.132_all.deb ... 2024-09-21T21:52:17.261 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking linux-headers-5.15.0-122 (5.15.0-122.132) ... 2024-09-21T21:52:22.183 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package linux-headers-5.15.0-122-generic. 2024-09-21T21:52:22.220 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../6-linux-headers-5.15.0-122-generic_5.15.0-122.132_amd64.deb ... 2024-09-21T21:52:22.250 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking linux-headers-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-21T21:52:22.466 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package linux-headers-5.15.0-122-generic. 2024-09-21T21:52:22.503 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../6-linux-headers-5.15.0-122-generic_5.15.0-122.132_amd64.deb ... 2024-09-21T21:52:22.533 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking linux-headers-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-21T21:52:23.482 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package linux-headers-5.15.0-122-generic. 2024-09-21T21:52:23.519 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../6-linux-headers-5.15.0-122-generic_5.15.0-122.132_amd64.deb ... 2024-09-21T21:52:23.574 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking linux-headers-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-21T21:52:25.033 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.122.122_amd64.deb ... 2024-09-21T21:52:25.155 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking linux-headers-generic (5.15.0.122.122) over (5.15.0.56.54) ... 2024-09-21T21:52:25.312 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.122.122_amd64.deb ... 2024-09-21T21:52:25.434 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking linux-headers-generic (5.15.0.122.122) over (5.15.0.56.54) ... 2024-09-21T21:52:25.489 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up linux-headers-5.15.0-122 (5.15.0-122.132) ... 2024-09-21T21:52:25.618 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up linux-headers-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-21T21:52:25.744 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up linux-headers-generic (5.15.0.122.122) ... 2024-09-21T21:52:25.767 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up linux-headers-5.15.0-122 (5.15.0-122.132) ... 2024-09-21T21:52:25.873 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up linux-image-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-21T21:52:25.880 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up linux-headers-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-21T21:52:25.981 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up linux-headers-generic (5.15.0.122.122) ... 2024-09-21T21:52:26.094 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up linux-image-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-21T21:52:26.312 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.122.122_amd64.deb ... 2024-09-21T21:52:26.437 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking linux-headers-generic (5.15.0.122.122) over (5.15.0.56.54) ... 2024-09-21T21:52:26.788 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up linux-headers-5.15.0-122 (5.15.0-122.132) ... 2024-09-21T21:52:26.908 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up linux-headers-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-21T21:52:27.034 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up linux-headers-generic (5.15.0.122.122) ... 2024-09-21T21:52:27.188 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up linux-image-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-21T21:52:27.228 INFO:teuthology.orchestra.run.smithi039.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-09-21T21:52:27.228 INFO:teuthology.orchestra.run.smithi039.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-09-21T21:52:27.228 INFO:teuthology.orchestra.run.smithi039.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-122-generic 2024-09-21T21:52:27.229 INFO:teuthology.orchestra.run.smithi039.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-122-generic 2024-09-21T21:52:27.392 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up linux-modules-extra-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-21T21:52:27.424 INFO:teuthology.orchestra.run.smithi157.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-09-21T21:52:27.424 INFO:teuthology.orchestra.run.smithi157.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-09-21T21:52:27.424 INFO:teuthology.orchestra.run.smithi157.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-122-generic 2024-09-21T21:52:27.424 INFO:teuthology.orchestra.run.smithi157.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-122-generic 2024-09-21T21:52:27.586 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up linux-modules-extra-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-21T21:52:28.548 INFO:teuthology.orchestra.run.smithi064.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-09-21T21:52:28.548 INFO:teuthology.orchestra.run.smithi064.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-09-21T21:52:28.548 INFO:teuthology.orchestra.run.smithi064.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-122-generic 2024-09-21T21:52:28.548 INFO:teuthology.orchestra.run.smithi064.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-122-generic 2024-09-21T21:52:29.397 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up linux-image-generic (5.15.0.122.122) ... 2024-09-21T21:52:29.671 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up linux-image-generic (5.15.0.122.122) ... 2024-09-21T21:52:29.716 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up linux-modules-extra-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-21T21:52:29.762 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up linux-generic (5.15.0.122.122) ... 2024-09-21T21:52:29.797 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up linux-generic (5.15.0.122.122) ... 2024-09-21T21:52:29.888 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up linux-modules-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-21T21:52:29.897 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up linux-modules-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-21T21:52:31.192 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up linux-image-generic (5.15.0.122.122) ... 2024-09-21T21:52:31.283 INFO:teuthology.orchestra.run.smithi157.stdout:Processing triggers for linux-image-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-21T21:52:31.323 INFO:teuthology.orchestra.run.smithi039.stdout:Processing triggers for linux-image-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-21T21:52:31.326 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up linux-generic (5.15.0.122.122) ... 2024-09-21T21:52:31.348 INFO:teuthology.orchestra.run.smithi157.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-21T21:52:31.348 INFO:teuthology.orchestra.run.smithi157.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-122-generic 2024-09-21T21:52:31.402 INFO:teuthology.orchestra.run.smithi039.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-21T21:52:31.402 INFO:teuthology.orchestra.run.smithi039.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-122-generic 2024-09-21T21:52:31.477 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up linux-modules-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-21T21:52:32.946 INFO:teuthology.orchestra.run.smithi064.stdout:Processing triggers for linux-image-5.15.0-122-generic (5.15.0-122.132) ... 2024-09-21T21:52:33.018 INFO:teuthology.orchestra.run.smithi064.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-21T21:52:33.018 INFO:teuthology.orchestra.run.smithi064.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-122-generic 2024-09-21T21:52:45.764 INFO:teuthology.orchestra.run.smithi039.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-21T21:52:45.764 INFO:teuthology.orchestra.run.smithi039.stdout:Sourcing file `/etc/default/grub' 2024-09-21T21:52:45.786 INFO:teuthology.orchestra.run.smithi039.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-21T21:52:45.810 INFO:teuthology.orchestra.run.smithi039.stdout:Generating grub configuration file ... 2024-09-21T21:52:46.000 INFO:teuthology.orchestra.run.smithi157.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-21T21:52:46.001 INFO:teuthology.orchestra.run.smithi157.stdout:Sourcing file `/etc/default/grub' 2024-09-21T21:52:46.031 INFO:teuthology.orchestra.run.smithi157.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-21T21:52:46.058 INFO:teuthology.orchestra.run.smithi157.stdout:Generating grub configuration file ... 2024-09-21T21:52:46.067 INFO:teuthology.orchestra.run.smithi039.stdout:Found linux image: /boot/vmlinuz-5.15.0-122-generic 2024-09-21T21:52:46.093 INFO:teuthology.orchestra.run.smithi039.stdout:Found initrd image: /boot/initrd.img-5.15.0-122-generic 2024-09-21T21:52:46.294 INFO:teuthology.orchestra.run.smithi157.stdout:Found linux image: /boot/vmlinuz-5.15.0-122-generic 2024-09-21T21:52:46.318 INFO:teuthology.orchestra.run.smithi157.stdout:Found initrd image: /boot/initrd.img-5.15.0-122-generic 2024-09-21T21:52:46.397 INFO:teuthology.orchestra.run.smithi039.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-21T21:52:46.403 INFO:teuthology.orchestra.run.smithi039.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-21T21:52:46.472 INFO:teuthology.orchestra.run.smithi039.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-21T21:52:46.477 INFO:teuthology.orchestra.run.smithi039.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-21T21:52:46.605 INFO:teuthology.orchestra.run.smithi157.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-21T21:52:46.614 INFO:teuthology.orchestra.run.smithi157.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-21T21:52:46.646 INFO:teuthology.orchestra.run.smithi039.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-21T21:52:46.646 INFO:teuthology.orchestra.run.smithi039.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-09-21T21:52:46.646 INFO:teuthology.orchestra.run.smithi039.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-21T21:52:46.685 INFO:teuthology.orchestra.run.smithi157.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-21T21:52:46.690 INFO:teuthology.orchestra.run.smithi157.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-21T21:52:46.693 INFO:teuthology.orchestra.run.smithi039.stdout:done 2024-09-21T21:52:46.846 INFO:teuthology.orchestra.run.smithi157.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-21T21:52:46.846 INFO:teuthology.orchestra.run.smithi157.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-09-21T21:52:46.846 INFO:teuthology.orchestra.run.smithi157.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-21T21:52:46.880 INFO:teuthology.orchestra.run.smithi157.stdout:done 2024-09-21T21:52:47.171 DEBUG:teuthology.orchestra.run.smithi039:> dpkg -s linux-image-generic 2024-09-21T21:52:47.191 INFO:teuthology.orchestra.run.smithi039.stdout:Package: linux-image-generic 2024-09-21T21:52:47.191 INFO:teuthology.orchestra.run.smithi039.stdout:Status: install ok installed 2024-09-21T21:52:47.191 INFO:teuthology.orchestra.run.smithi039.stdout:Priority: optional 2024-09-21T21:52:47.191 INFO:teuthology.orchestra.run.smithi039.stdout:Section: kernel 2024-09-21T21:52:47.192 INFO:teuthology.orchestra.run.smithi039.stdout:Installed-Size: 21 2024-09-21T21:52:47.192 INFO:teuthology.orchestra.run.smithi039.stdout:Maintainer: Ubuntu Kernel Team 2024-09-21T21:52:47.192 INFO:teuthology.orchestra.run.smithi039.stdout:Architecture: amd64 2024-09-21T21:52:47.192 INFO:teuthology.orchestra.run.smithi039.stdout:Source: linux-meta 2024-09-21T21:52:47.192 INFO:teuthology.orchestra.run.smithi039.stdout:Version: 5.15.0.122.122 2024-09-21T21:52:47.192 INFO:teuthology.orchestra.run.smithi039.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-122), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-21T21:52:47.192 INFO:teuthology.orchestra.run.smithi039.stdout:Depends: linux-image-5.15.0-122-generic, linux-modules-extra-5.15.0-122-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-21T21:52:47.192 INFO:teuthology.orchestra.run.smithi039.stdout:Recommends: thermald 2024-09-21T21:52:47.192 INFO:teuthology.orchestra.run.smithi039.stdout:Description: Generic Linux kernel image 2024-09-21T21:52:47.192 INFO:teuthology.orchestra.run.smithi039.stdout: This package will always depend on the latest generic kernel image 2024-09-21T21:52:47.192 INFO:teuthology.orchestra.run.smithi039.stdout: available. 2024-09-21T21:52:47.192 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-122-generic 2024-09-21T21:52:47.192 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-21T21:52:47.193 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-21T21:52:47.193 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-21T21:52:47.193 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi039.front.sepia.ceph.com, path=None, version=distro) 2024-09-21T21:52:47.193 DEBUG:teuthology.orchestra.run.smithi039:> sudo apt-get clean 2024-09-21T21:52:47.296 INFO:teuthology.orchestra.run.smithi064.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-21T21:52:47.296 INFO:teuthology.orchestra.run.smithi064.stdout:Sourcing file `/etc/default/grub' 2024-09-21T21:52:47.322 INFO:teuthology.orchestra.run.smithi064.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-21T21:52:47.341 DEBUG:teuthology.orchestra.run.smithi157:> dpkg -s linux-image-generic 2024-09-21T21:52:47.341 INFO:teuthology.orchestra.run.smithi064.stdout:Generating grub configuration file ... 2024-09-21T21:52:47.358 INFO:teuthology.orchestra.run.smithi157.stdout:Package: linux-image-generic 2024-09-21T21:52:47.358 INFO:teuthology.orchestra.run.smithi157.stdout:Status: install ok installed 2024-09-21T21:52:47.358 INFO:teuthology.orchestra.run.smithi157.stdout:Priority: optional 2024-09-21T21:52:47.358 INFO:teuthology.orchestra.run.smithi157.stdout:Section: kernel 2024-09-21T21:52:47.358 INFO:teuthology.orchestra.run.smithi157.stdout:Installed-Size: 21 2024-09-21T21:52:47.358 INFO:teuthology.orchestra.run.smithi157.stdout:Maintainer: Ubuntu Kernel Team 2024-09-21T21:52:47.358 INFO:teuthology.orchestra.run.smithi157.stdout:Architecture: amd64 2024-09-21T21:52:47.359 INFO:teuthology.orchestra.run.smithi157.stdout:Source: linux-meta 2024-09-21T21:52:47.359 INFO:teuthology.orchestra.run.smithi157.stdout:Version: 5.15.0.122.122 2024-09-21T21:52:47.359 INFO:teuthology.orchestra.run.smithi157.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-122), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-21T21:52:47.359 INFO:teuthology.orchestra.run.smithi157.stdout:Depends: linux-image-5.15.0-122-generic, linux-modules-extra-5.15.0-122-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-21T21:52:47.359 INFO:teuthology.orchestra.run.smithi157.stdout:Recommends: thermald 2024-09-21T21:52:47.359 INFO:teuthology.orchestra.run.smithi157.stdout:Description: Generic Linux kernel image 2024-09-21T21:52:47.359 INFO:teuthology.orchestra.run.smithi157.stdout: This package will always depend on the latest generic kernel image 2024-09-21T21:52:47.359 INFO:teuthology.orchestra.run.smithi157.stdout: available. 2024-09-21T21:52:47.359 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-122-generic 2024-09-21T21:52:47.359 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-21T21:52:47.359 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-21T21:52:47.359 INFO:teuthology.task.kernel:Installing distro kernel on client.0... 2024-09-21T21:52:47.360 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi157.front.sepia.ceph.com, path=None, version=distro) 2024-09-21T21:52:47.360 DEBUG:teuthology.orchestra.run.smithi157:> sudo apt-get clean 2024-09-21T21:52:47.385 DEBUG:teuthology.orchestra.run.smithi039:> sudo apt-get update 2024-09-21T21:52:47.504 DEBUG:teuthology.orchestra.run.smithi157:> sudo apt-get update 2024-09-21T21:52:47.624 INFO:teuthology.orchestra.run.smithi064.stdout:Found linux image: /boot/vmlinuz-5.15.0-122-generic 2024-09-21T21:52:47.640 INFO:teuthology.orchestra.run.smithi039.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-21T21:52:47.644 INFO:teuthology.orchestra.run.smithi064.stdout:Found initrd image: /boot/initrd.img-5.15.0-122-generic 2024-09-21T21:52:47.651 INFO:teuthology.orchestra.run.smithi039.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-21T21:52:47.713 INFO:teuthology.orchestra.run.smithi039.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-21T21:52:47.750 INFO:teuthology.orchestra.run.smithi157.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-21T21:52:47.751 INFO:teuthology.orchestra.run.smithi157.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-21T21:52:47.794 INFO:teuthology.orchestra.run.smithi039.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-21T21:52:47.821 INFO:teuthology.orchestra.run.smithi157.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-21T21:52:47.903 INFO:teuthology.orchestra.run.smithi157.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-21T21:52:47.957 INFO:teuthology.orchestra.run.smithi064.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-21T21:52:47.962 INFO:teuthology.orchestra.run.smithi064.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-21T21:52:48.026 INFO:teuthology.orchestra.run.smithi064.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-21T21:52:48.032 INFO:teuthology.orchestra.run.smithi064.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-21T21:52:48.211 INFO:teuthology.orchestra.run.smithi064.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-21T21:52:48.211 INFO:teuthology.orchestra.run.smithi064.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-09-21T21:52:48.211 INFO:teuthology.orchestra.run.smithi064.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-21T21:52:48.261 INFO:teuthology.orchestra.run.smithi064.stdout:done 2024-09-21T21:52:48.770 DEBUG:teuthology.orchestra.run.smithi064:> dpkg -s linux-image-generic 2024-09-21T21:52:48.798 INFO:teuthology.orchestra.run.smithi064.stdout:Package: linux-image-generic 2024-09-21T21:52:48.798 INFO:teuthology.orchestra.run.smithi064.stdout:Status: install ok installed 2024-09-21T21:52:48.798 INFO:teuthology.orchestra.run.smithi064.stdout:Priority: optional 2024-09-21T21:52:48.798 INFO:teuthology.orchestra.run.smithi064.stdout:Section: kernel 2024-09-21T21:52:48.798 INFO:teuthology.orchestra.run.smithi064.stdout:Installed-Size: 21 2024-09-21T21:52:48.798 INFO:teuthology.orchestra.run.smithi064.stdout:Maintainer: Ubuntu Kernel Team 2024-09-21T21:52:48.798 INFO:teuthology.orchestra.run.smithi064.stdout:Architecture: amd64 2024-09-21T21:52:48.798 INFO:teuthology.orchestra.run.smithi064.stdout:Source: linux-meta 2024-09-21T21:52:48.798 INFO:teuthology.orchestra.run.smithi064.stdout:Version: 5.15.0.122.122 2024-09-21T21:52:48.798 INFO:teuthology.orchestra.run.smithi064.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-122), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-21T21:52:48.799 INFO:teuthology.orchestra.run.smithi064.stdout:Depends: linux-image-5.15.0-122-generic, linux-modules-extra-5.15.0-122-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-21T21:52:48.799 INFO:teuthology.orchestra.run.smithi064.stdout:Recommends: thermald 2024-09-21T21:52:48.799 INFO:teuthology.orchestra.run.smithi064.stdout:Description: Generic Linux kernel image 2024-09-21T21:52:48.799 INFO:teuthology.orchestra.run.smithi064.stdout: This package will always depend on the latest generic kernel image 2024-09-21T21:52:48.799 INFO:teuthology.orchestra.run.smithi064.stdout: available. 2024-09-21T21:52:48.799 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-122-generic 2024-09-21T21:52:48.799 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-21T21:52:48.799 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-21T21:52:48.799 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-09-21T21:52:48.799 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi064.front.sepia.ceph.com, path=None, version=distro) 2024-09-21T21:52:48.800 DEBUG:teuthology.orchestra.run.smithi064:> sudo apt-get clean 2024-09-21T21:52:49.001 DEBUG:teuthology.orchestra.run.smithi064:> sudo apt-get update 2024-09-21T21:52:49.116 INFO:teuthology.orchestra.run.smithi064.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-21T21:52:49.122 INFO:teuthology.orchestra.run.smithi039.stdout:Reading package lists... 2024-09-21T21:52:49.140 DEBUG:teuthology.orchestra.run.smithi039:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-21T21:52:49.205 INFO:teuthology.orchestra.run.smithi039.stdout:Reading package lists... 2024-09-21T21:52:49.237 INFO:teuthology.orchestra.run.smithi064.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-21T21:52:49.242 INFO:teuthology.orchestra.run.smithi157.stdout:Reading package lists... 2024-09-21T21:52:49.261 DEBUG:teuthology.orchestra.run.smithi157:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-21T21:52:49.324 INFO:teuthology.orchestra.run.smithi157.stdout:Reading package lists... 2024-09-21T21:52:49.333 INFO:teuthology.orchestra.run.smithi064.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-21T21:52:49.399 INFO:teuthology.orchestra.run.smithi064.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-21T21:52:49.419 INFO:teuthology.orchestra.run.smithi039.stdout:Building dependency tree... 2024-09-21T21:52:49.420 INFO:teuthology.orchestra.run.smithi039.stdout:Reading state information... 2024-09-21T21:52:49.535 INFO:teuthology.orchestra.run.smithi157.stdout:Building dependency tree... 2024-09-21T21:52:49.535 INFO:teuthology.orchestra.run.smithi157.stdout:Reading state information... 2024-09-21T21:52:49.620 INFO:teuthology.orchestra.run.smithi039.stdout:linux-image-generic is already the newest version (5.15.0.122.122). 2024-09-21T21:52:49.620 INFO:teuthology.orchestra.run.smithi039.stdout:The following packages were automatically installed and are no longer required: 2024-09-21T21:52:49.620 INFO:teuthology.orchestra.run.smithi039.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-21T21:52:49.620 INFO:teuthology.orchestra.run.smithi039.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-21T21:52:49.620 INFO:teuthology.orchestra.run.smithi039.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-21T21:52:49.620 INFO:teuthology.orchestra.run.smithi039.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-21T21:52:49.620 INFO:teuthology.orchestra.run.smithi039.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-21T21:52:49.620 INFO:teuthology.orchestra.run.smithi039.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-21T21:52:49.666 INFO:teuthology.orchestra.run.smithi039.stdout:0 upgraded, 0 newly installed, 0 to remove and 345 not upgraded. 2024-09-21T21:52:49.668 DEBUG:teuthology.orchestra.run.smithi039:> dpkg -s linux-image-generic 2024-09-21T21:52:49.684 INFO:teuthology.orchestra.run.smithi039.stdout:Package: linux-image-generic 2024-09-21T21:52:49.684 INFO:teuthology.orchestra.run.smithi039.stdout:Status: install ok installed 2024-09-21T21:52:49.684 INFO:teuthology.orchestra.run.smithi039.stdout:Priority: optional 2024-09-21T21:52:49.684 INFO:teuthology.orchestra.run.smithi039.stdout:Section: kernel 2024-09-21T21:52:49.684 INFO:teuthology.orchestra.run.smithi039.stdout:Installed-Size: 21 2024-09-21T21:52:49.684 INFO:teuthology.orchestra.run.smithi039.stdout:Maintainer: Ubuntu Kernel Team 2024-09-21T21:52:49.684 INFO:teuthology.orchestra.run.smithi039.stdout:Architecture: amd64 2024-09-21T21:52:49.684 INFO:teuthology.orchestra.run.smithi039.stdout:Source: linux-meta 2024-09-21T21:52:49.684 INFO:teuthology.orchestra.run.smithi039.stdout:Version: 5.15.0.122.122 2024-09-21T21:52:49.684 INFO:teuthology.orchestra.run.smithi039.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-122), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-21T21:52:49.685 INFO:teuthology.orchestra.run.smithi039.stdout:Depends: linux-image-5.15.0-122-generic, linux-modules-extra-5.15.0-122-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-21T21:52:49.685 INFO:teuthology.orchestra.run.smithi039.stdout:Recommends: thermald 2024-09-21T21:52:49.685 INFO:teuthology.orchestra.run.smithi039.stdout:Description: Generic Linux kernel image 2024-09-21T21:52:49.685 INFO:teuthology.orchestra.run.smithi039.stdout: This package will always depend on the latest generic kernel image 2024-09-21T21:52:49.685 INFO:teuthology.orchestra.run.smithi039.stdout: available. 2024-09-21T21:52:49.686 DEBUG:teuthology.orchestra.run.smithi039:> mktemp 2024-09-21T21:52:49.713 INFO:teuthology.orchestra.run.smithi157.stdout:linux-image-generic is already the newest version (5.15.0.122.122). 2024-09-21T21:52:49.713 INFO:teuthology.orchestra.run.smithi157.stdout:The following packages were automatically installed and are no longer required: 2024-09-21T21:52:49.713 INFO:teuthology.orchestra.run.smithi157.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-21T21:52:49.713 INFO:teuthology.orchestra.run.smithi157.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-21T21:52:49.713 INFO:teuthology.orchestra.run.smithi157.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-21T21:52:49.714 INFO:teuthology.orchestra.run.smithi157.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-21T21:52:49.714 INFO:teuthology.orchestra.run.smithi157.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-21T21:52:49.714 INFO:teuthology.orchestra.run.smithi157.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-21T21:52:49.732 INFO:teuthology.orchestra.run.smithi039.stdout:/tmp/tmp.0YaQbgnkjF 2024-09-21T21:52:49.732 DEBUG:teuthology.orchestra.run.smithi039:> sudo cp /boot/grub/grub.cfg /tmp/tmp.0YaQbgnkjF 2024-09-21T21:52:49.754 INFO:teuthology.orchestra.run.smithi157.stdout:0 upgraded, 0 newly installed, 0 to remove and 345 not upgraded. 2024-09-21T21:52:49.755 DEBUG:teuthology.orchestra.run.smithi157:> dpkg -s linux-image-generic 2024-09-21T21:52:49.770 INFO:teuthology.orchestra.run.smithi157.stdout:Package: linux-image-generic 2024-09-21T21:52:49.770 INFO:teuthology.orchestra.run.smithi157.stdout:Status: install ok installed 2024-09-21T21:52:49.771 INFO:teuthology.orchestra.run.smithi157.stdout:Priority: optional 2024-09-21T21:52:49.771 INFO:teuthology.orchestra.run.smithi157.stdout:Section: kernel 2024-09-21T21:52:49.771 INFO:teuthology.orchestra.run.smithi157.stdout:Installed-Size: 21 2024-09-21T21:52:49.771 INFO:teuthology.orchestra.run.smithi157.stdout:Maintainer: Ubuntu Kernel Team 2024-09-21T21:52:49.771 INFO:teuthology.orchestra.run.smithi157.stdout:Architecture: amd64 2024-09-21T21:52:49.771 INFO:teuthology.orchestra.run.smithi157.stdout:Source: linux-meta 2024-09-21T21:52:49.771 INFO:teuthology.orchestra.run.smithi157.stdout:Version: 5.15.0.122.122 2024-09-21T21:52:49.771 INFO:teuthology.orchestra.run.smithi157.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-122), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-21T21:52:49.771 INFO:teuthology.orchestra.run.smithi157.stdout:Depends: linux-image-5.15.0-122-generic, linux-modules-extra-5.15.0-122-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-21T21:52:49.771 INFO:teuthology.orchestra.run.smithi157.stdout:Recommends: thermald 2024-09-21T21:52:49.771 INFO:teuthology.orchestra.run.smithi157.stdout:Description: Generic Linux kernel image 2024-09-21T21:52:49.771 INFO:teuthology.orchestra.run.smithi157.stdout: This package will always depend on the latest generic kernel image 2024-09-21T21:52:49.771 INFO:teuthology.orchestra.run.smithi157.stdout: available. 2024-09-21T21:52:49.772 DEBUG:teuthology.orchestra.run.smithi157:> mktemp 2024-09-21T21:52:49.788 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod 0666 /tmp/tmp.0YaQbgnkjF 2024-09-21T21:52:49.814 INFO:teuthology.orchestra.run.smithi157.stdout:/tmp/tmp.fmVlckswA1 2024-09-21T21:52:49.814 DEBUG:teuthology.orchestra.run.smithi157:> sudo cp /boot/grub/grub.cfg /tmp/tmp.fmVlckswA1 2024-09-21T21:52:49.864 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod 0666 /tmp/tmp.fmVlckswA1 2024-09-21T21:52:49.898 DEBUG:teuthology.orchestra.remote:smithi039:/tmp/tmp.0YaQbgnkjF is 10KB 2024-09-21T21:52:49.911 DEBUG:teuthology.orchestra.run.smithi039:> rm -fr /tmp/tmp.0YaQbgnkjF 2024-09-21T21:52:49.917 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-21T21:52:49.977 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-21T21:52:49.977 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-21T21:52:49.978 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-21T21:52:49.987 DEBUG:teuthology.orchestra.remote:smithi157:/tmp/tmp.fmVlckswA1 is 10KB 2024-09-21T21:52:49.999 DEBUG:teuthology.orchestra.run.smithi157:> rm -fr /tmp/tmp.fmVlckswA1 2024-09-21T21:52:50.004 DEBUG:teuthology.orchestra.run.smithi157:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-21T21:52:50.044 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-122-generic 2024-09-21T21:52:50.044 DEBUG:teuthology.orchestra.run.smithi039:> sudo update-grub 2024-09-21T21:52:50.058 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-09-21T21:52:50.058 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-21T21:52:50.058 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-21T21:52:50.120 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-122-generic 2024-09-21T21:52:50.120 DEBUG:teuthology.orchestra.run.smithi157:> sudo update-grub 2024-09-21T21:52:50.633 INFO:teuthology.orchestra.run.smithi064.stdout:Reading package lists... 2024-09-21T21:52:50.652 DEBUG:teuthology.orchestra.run.smithi064:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-21T21:52:50.717 INFO:teuthology.orchestra.run.smithi064.stdout:Reading package lists... 2024-09-21T21:52:50.936 INFO:teuthology.orchestra.run.smithi064.stdout:Building dependency tree... 2024-09-21T21:52:50.937 INFO:teuthology.orchestra.run.smithi064.stdout:Reading state information... 2024-09-21T21:52:51.135 INFO:teuthology.orchestra.run.smithi064.stdout:linux-image-generic is already the newest version (5.15.0.122.122). 2024-09-21T21:52:51.135 INFO:teuthology.orchestra.run.smithi064.stdout:The following packages were automatically installed and are no longer required: 2024-09-21T21:52:51.135 INFO:teuthology.orchestra.run.smithi064.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-21T21:52:51.135 INFO:teuthology.orchestra.run.smithi064.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-21T21:52:51.135 INFO:teuthology.orchestra.run.smithi064.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-21T21:52:51.136 INFO:teuthology.orchestra.run.smithi064.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-21T21:52:51.136 INFO:teuthology.orchestra.run.smithi064.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-21T21:52:51.136 INFO:teuthology.orchestra.run.smithi064.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-21T21:52:51.183 INFO:teuthology.orchestra.run.smithi064.stdout:0 upgraded, 0 newly installed, 0 to remove and 345 not upgraded. 2024-09-21T21:52:51.185 DEBUG:teuthology.orchestra.run.smithi064:> dpkg -s linux-image-generic 2024-09-21T21:52:51.197 INFO:teuthology.orchestra.run.smithi064.stdout:Package: linux-image-generic 2024-09-21T21:52:51.198 INFO:teuthology.orchestra.run.smithi064.stdout:Status: install ok installed 2024-09-21T21:52:51.198 INFO:teuthology.orchestra.run.smithi064.stdout:Priority: optional 2024-09-21T21:52:51.198 INFO:teuthology.orchestra.run.smithi064.stdout:Section: kernel 2024-09-21T21:52:51.198 INFO:teuthology.orchestra.run.smithi064.stdout:Installed-Size: 21 2024-09-21T21:52:51.198 INFO:teuthology.orchestra.run.smithi064.stdout:Maintainer: Ubuntu Kernel Team 2024-09-21T21:52:51.198 INFO:teuthology.orchestra.run.smithi064.stdout:Architecture: amd64 2024-09-21T21:52:51.198 INFO:teuthology.orchestra.run.smithi064.stdout:Source: linux-meta 2024-09-21T21:52:51.198 INFO:teuthology.orchestra.run.smithi064.stdout:Version: 5.15.0.122.122 2024-09-21T21:52:51.198 INFO:teuthology.orchestra.run.smithi064.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-122), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-21T21:52:51.198 INFO:teuthology.orchestra.run.smithi064.stdout:Depends: linux-image-5.15.0-122-generic, linux-modules-extra-5.15.0-122-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-21T21:52:51.198 INFO:teuthology.orchestra.run.smithi064.stdout:Recommends: thermald 2024-09-21T21:52:51.198 INFO:teuthology.orchestra.run.smithi064.stdout:Description: Generic Linux kernel image 2024-09-21T21:52:51.198 INFO:teuthology.orchestra.run.smithi064.stdout: This package will always depend on the latest generic kernel image 2024-09-21T21:52:51.199 INFO:teuthology.orchestra.run.smithi064.stdout: available. 2024-09-21T21:52:51.199 DEBUG:teuthology.orchestra.run.smithi064:> mktemp 2024-09-21T21:52:51.200 INFO:teuthology.orchestra.run.smithi039.stderr:Sourcing file `/etc/default/grub' 2024-09-21T21:52:51.202 INFO:teuthology.orchestra.run.smithi039.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-21T21:52:51.204 INFO:teuthology.orchestra.run.smithi039.stderr:Generating grub configuration file ... 2024-09-21T21:52:51.242 INFO:teuthology.orchestra.run.smithi064.stdout:/tmp/tmp.0A3Poh11m6 2024-09-21T21:52:51.243 DEBUG:teuthology.orchestra.run.smithi064:> sudo cp /boot/grub/grub.cfg /tmp/tmp.0A3Poh11m6 2024-09-21T21:52:51.272 INFO:teuthology.orchestra.run.smithi157.stderr:Sourcing file `/etc/default/grub' 2024-09-21T21:52:51.274 INFO:teuthology.orchestra.run.smithi157.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-21T21:52:51.276 INFO:teuthology.orchestra.run.smithi157.stderr:Generating grub configuration file ... 2024-09-21T21:52:51.300 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 0666 /tmp/tmp.0A3Poh11m6 2024-09-21T21:52:51.377 INFO:teuthology.orchestra.run.smithi039.stderr:Found linux image: /boot/vmlinuz-5.15.0-122-generic 2024-09-21T21:52:51.382 INFO:teuthology.orchestra.run.smithi039.stderr:Found initrd image: /boot/initrd.img-5.15.0-122-generic 2024-09-21T21:52:51.424 DEBUG:teuthology.orchestra.remote:smithi064:/tmp/tmp.0A3Poh11m6 is 10KB 2024-09-21T21:52:51.436 DEBUG:teuthology.orchestra.run.smithi064:> rm -fr /tmp/tmp.0A3Poh11m6 2024-09-21T21:52:51.441 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-21T21:52:51.442 INFO:teuthology.orchestra.run.smithi157.stderr:Found linux image: /boot/vmlinuz-5.15.0-122-generic 2024-09-21T21:52:51.448 INFO:teuthology.orchestra.run.smithi157.stderr:Found initrd image: /boot/initrd.img-5.15.0-122-generic 2024-09-21T21:52:51.499 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-09-21T21:52:51.499 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-21T21:52:51.499 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-21T21:52:51.589 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-122-generic 2024-09-21T21:52:51.589 DEBUG:teuthology.orchestra.run.smithi064:> sudo update-grub 2024-09-21T21:52:51.597 INFO:teuthology.orchestra.run.smithi039.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-21T21:52:51.602 INFO:teuthology.orchestra.run.smithi039.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-21T21:52:51.643 INFO:teuthology.orchestra.run.smithi039.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-21T21:52:51.649 INFO:teuthology.orchestra.run.smithi039.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-21T21:52:51.706 INFO:teuthology.orchestra.run.smithi157.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-21T21:52:51.711 INFO:teuthology.orchestra.run.smithi157.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-21T21:52:51.753 INFO:teuthology.orchestra.run.smithi157.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-21T21:52:51.758 INFO:teuthology.orchestra.run.smithi157.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-21T21:52:51.778 INFO:teuthology.orchestra.run.smithi039.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-21T21:52:51.778 INFO:teuthology.orchestra.run.smithi039.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-09-21T21:52:51.778 INFO:teuthology.orchestra.run.smithi039.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-21T21:52:51.804 INFO:teuthology.orchestra.run.smithi039.stderr:done 2024-09-21T21:52:51.813 DEBUG:teuthology.orchestra.run.smithi039:> sudo shutdown -r now 2024-09-21T21:52:51.913 INFO:teuthology.orchestra.run.smithi157.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-21T21:52:51.913 INFO:teuthology.orchestra.run.smithi157.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-09-21T21:52:51.914 INFO:teuthology.orchestra.run.smithi157.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-21T21:52:51.925 INFO:teuthology.orchestra.run.smithi157.stderr:done 2024-09-21T21:52:51.927 DEBUG:teuthology.orchestra.run.smithi157:> sudo shutdown -r now 2024-09-21T21:52:52.870 INFO:teuthology.orchestra.run.smithi064.stderr:Sourcing file `/etc/default/grub' 2024-09-21T21:52:52.872 INFO:teuthology.orchestra.run.smithi064.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-21T21:52:52.873 INFO:teuthology.orchestra.run.smithi064.stderr:Generating grub configuration file ... 2024-09-21T21:52:53.112 INFO:teuthology.orchestra.run.smithi064.stderr:Found linux image: /boot/vmlinuz-5.15.0-122-generic 2024-09-21T21:52:53.119 INFO:teuthology.orchestra.run.smithi064.stderr:Found initrd image: /boot/initrd.img-5.15.0-122-generic 2024-09-21T21:52:53.351 INFO:teuthology.orchestra.run.smithi064.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-21T21:52:53.356 INFO:teuthology.orchestra.run.smithi064.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-21T21:52:53.398 INFO:teuthology.orchestra.run.smithi064.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-21T21:52:53.403 INFO:teuthology.orchestra.run.smithi064.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-21T21:52:53.570 INFO:teuthology.orchestra.run.smithi064.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-21T21:52:53.570 INFO:teuthology.orchestra.run.smithi064.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-09-21T21:52:53.570 INFO:teuthology.orchestra.run.smithi064.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-21T21:52:53.580 INFO:teuthology.orchestra.run.smithi064.stderr:done 2024-09-21T21:52:53.582 DEBUG:teuthology.orchestra.run.smithi064:> sudo shutdown -r now 2024-09-21T21:53:21.842 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-21T21:53:21.843 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2024-09-21T21:53:21.844 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T21:53:21.931 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-09-21T21:53:21.932 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com' 2024-09-21T21:53:21.932 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T21:53:23.586 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-21T21:53:23.587 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-09-21T21:53:23.587 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T21:53:40.335 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.39 2024-09-21T21:53:40.335 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.157 2024-09-21T21:53:42.127 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.64 2024-09-21T21:53:49.343 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2024-09-21T21:53:49.344 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T21:53:49.345 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com' 2024-09-21T21:53:49.346 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T21:53:51.129 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-09-21T21:53:51.129 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T21:53:52.399 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.39 2024-09-21T21:53:52.399 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.157 2024-09-21T21:53:54.191 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.64 2024-09-21T21:54:04.408 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2024-09-21T21:54:04.408 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T21:54:04.409 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com' 2024-09-21T21:54:04.410 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T21:54:06.194 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-09-21T21:54:06.194 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T21:54:07.471 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.39 2024-09-21T21:54:07.471 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.157 2024-09-21T21:54:22.484 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2024-09-21T21:54:22.485 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T21:54:22.486 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com' 2024-09-21T21:54:22.487 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T21:54:24.627 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.64 2024-09-21T21:54:25.551 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.39 2024-09-21T21:54:39.642 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-09-21T21:54:39.643 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T21:54:39.644 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.64 2024-09-21T21:54:43.554 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2024-09-21T21:54:43.555 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T21:54:54.860 DEBUG:teuthology.orchestra.run.smithi157:> true 2024-09-21T21:54:56.410 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi157.front.sepia.ceph.com' 2024-09-21T21:54:56.410 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "5.15.0-122-generic"... 2024-09-21T21:54:56.410 DEBUG:teuthology.orchestra.run.smithi157:> uname -r 2024-09-21T21:54:56.457 INFO:teuthology.orchestra.run.smithi157.stdout:5.15.0-122-generic 2024-09-21T21:54:56.457 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-122-generic vs 5.15.0-122-generic 2024-09-21T21:54:56.457 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-21T21:54:56.458 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-21T21:54:57.458 INFO:teuthology.task.kernel:Enabling kdb on client.0... 2024-09-21T21:54:57.458 DEBUG:teuthology.orchestra.run.smithi157:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-21T21:54:57.605 INFO:teuthology.orchestra.run.smithi157.stdout:ttyS1 2024-09-21T21:54:57.632 DEBUG:teuthology.parallel:result is None 2024-09-21T21:54:57.645 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-09-21T21:54:57.646 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T21:54:57.858 DEBUG:teuthology.orchestra.run.smithi064:> true 2024-09-21T21:54:58.395 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-09-21T21:54:58.395 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.15.0-122-generic"... 2024-09-21T21:54:58.395 DEBUG:teuthology.orchestra.run.smithi064:> uname -r 2024-09-21T21:54:58.441 INFO:teuthology.orchestra.run.smithi064.stdout:5.15.0-122-generic 2024-09-21T21:54:58.441 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-122-generic vs 5.15.0-122-generic 2024-09-21T21:54:58.441 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-21T21:54:58.441 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-21T21:54:59.442 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-21T21:54:59.442 DEBUG:teuthology.orchestra.run.smithi064:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-21T21:54:59.511 INFO:teuthology.orchestra.run.smithi064.stdout:ttyS1 2024-09-21T21:54:59.535 DEBUG:teuthology.parallel:result is None 2024-09-21T21:55:15.058 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.39 2024-09-21T21:55:36.059 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2024-09-21T21:55:36.060 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T21:55:36.331 DEBUG:teuthology.orchestra.run.smithi039:> true 2024-09-21T21:55:36.872 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi039.front.sepia.ceph.com' 2024-09-21T21:55:36.872 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-122-generic"... 2024-09-21T21:55:36.873 DEBUG:teuthology.orchestra.run.smithi039:> uname -r 2024-09-21T21:55:36.920 INFO:teuthology.orchestra.run.smithi039.stdout:5.15.0-122-generic 2024-09-21T21:55:36.921 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-122-generic vs 5.15.0-122-generic 2024-09-21T21:55:36.921 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-21T21:55:36.921 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-21T21:55:37.922 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-21T21:55:37.922 DEBUG:teuthology.orchestra.run.smithi039:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-21T21:55:37.999 INFO:teuthology.orchestra.run.smithi039.stdout:ttyS1 2024-09-21T21:55:38.019 DEBUG:teuthology.parallel:result is None 2024-09-21T21:55:38.019 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-21T21:55:38.028 INFO:teuthology.task.internal:Creating test directory... 2024-09-21T21:55:38.029 DEBUG:teuthology.orchestra.run.smithi039:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-21T21:55:38.031 DEBUG:teuthology.orchestra.run.smithi064:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-21T21:55:38.033 DEBUG:teuthology.orchestra.run.smithi157:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-21T21:55:38.039 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-21T21:55:38.045 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-21T21:55:38.055 INFO:teuthology.task.internal:Creating archive directory... 2024-09-21T21:55:38.055 DEBUG:teuthology.orchestra.run.smithi039:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-21T21:55:38.078 DEBUG:teuthology.orchestra.run.smithi064:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-21T21:55:38.081 DEBUG:teuthology.orchestra.run.smithi157:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-21T21:55:38.105 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-21T21:55:38.111 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-21T21:55:38.112 DEBUG:teuthology.orchestra.run.smithi039:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-21T21:55:38.137 DEBUG:teuthology.orchestra.run.smithi064:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-21T21:55:38.140 DEBUG:teuthology.orchestra.run.smithi157:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-21T21:55:38.155 INFO:teuthology.orchestra.run.smithi039.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T21:55:38.162 INFO:teuthology.orchestra.run.smithi064.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T21:55:38.164 INFO:teuthology.orchestra.run.smithi039.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T21:55:38.170 INFO:teuthology.orchestra.run.smithi064.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T21:55:38.211 INFO:teuthology.orchestra.run.smithi157.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T21:55:38.219 INFO:teuthology.orchestra.run.smithi157.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T21:55:38.221 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-21T21:55:38.228 INFO:teuthology.task.internal:Configuring sudo... 2024-09-21T21:55:38.228 DEBUG:teuthology.orchestra.run.smithi039:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-21T21:55:38.230 DEBUG:teuthology.orchestra.run.smithi064:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-21T21:55:38.233 DEBUG:teuthology.orchestra.run.smithi157:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-21T21:55:38.245 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-21T21:55:38.253 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-21T21:55:38.253 DEBUG:teuthology.orchestra.run.smithi039:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-21T21:55:38.290 DEBUG:teuthology.orchestra.run.smithi064:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-21T21:55:38.292 DEBUG:teuthology.orchestra.run.smithi157:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-21T21:55:38.297 DEBUG:teuthology.orchestra.run.smithi039:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-21T21:55:38.341 DEBUG:teuthology.orchestra.run.smithi039:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-21T21:55:38.389 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-21T21:55:38.389 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-21T21:55:38.449 DEBUG:teuthology.orchestra.run.smithi064:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-21T21:55:38.454 DEBUG:teuthology.orchestra.run.smithi064:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-21T21:55:38.501 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-09-21T21:55:38.501 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-21T21:55:38.560 DEBUG:teuthology.orchestra.run.smithi157:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-21T21:55:38.566 DEBUG:teuthology.orchestra.run.smithi157:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-21T21:55:38.614 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-09-21T21:55:38.614 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-21T21:55:38.682 DEBUG:teuthology.orchestra.run.smithi039:> sudo service rsyslog restart 2024-09-21T21:55:38.685 DEBUG:teuthology.orchestra.run.smithi064:> sudo service rsyslog restart 2024-09-21T21:55:38.687 DEBUG:teuthology.orchestra.run.smithi157:> sudo service rsyslog restart 2024-09-21T21:55:38.804 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-21T21:55:38.812 INFO:teuthology.task.internal:Starting timer... 2024-09-21T21:55:38.812 INFO:teuthology.run_tasks:Running task pcp... 2024-09-21T21:55:38.828 INFO:teuthology.run_tasks:Running task selinux... 2024-09-21T21:55:38.847 DEBUG:teuthology.task.selinux:Excluding smithi039: OS 'ubuntu' does not support SELinux 2024-09-21T21:55:38.848 DEBUG:teuthology.task.selinux:Excluding smithi064: OS 'ubuntu' does not support SELinux 2024-09-21T21:55:38.848 DEBUG:teuthology.task.selinux:Excluding smithi157: OS 'ubuntu' does not support SELinux 2024-09-21T21:55:38.848 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-21T21:55:38.848 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-09-21T21:55:38.848 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-21T21:55:38.848 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-21T21:55:38.926 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-21T21:55:38.937 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-21T21:55:38.937 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi039.front.sepia.ceph.com,smithi064.front.sepia.ceph.com,smithi157.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-21T22:03:11.899 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi039.front.sepia.ceph.com'), Remote(name='ubuntu@smithi064.front.sepia.ceph.com'), Remote(name='ubuntu@smithi157.front.sepia.ceph.com')] 2024-09-21T22:03:11.900 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2024-09-21T22:03:11.901 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T22:03:11.979 DEBUG:teuthology.orchestra.run.smithi039:> true 2024-09-21T22:03:12.059 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi039.front.sepia.ceph.com' 2024-09-21T22:03:12.112 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-09-21T22:03:12.112 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T22:03:12.191 DEBUG:teuthology.orchestra.run.smithi064:> true 2024-09-21T22:03:12.266 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-09-21T22:03:12.267 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi157.front.sepia.ceph.com' 2024-09-21T22:03:12.267 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi157.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T22:03:12.347 DEBUG:teuthology.orchestra.run.smithi157:> true 2024-09-21T22:03:12.426 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi157.front.sepia.ceph.com' 2024-09-21T22:03:12.427 INFO:teuthology.run_tasks:Running task clock... 2024-09-21T22:03:12.437 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-21T22:03:12.437 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-21T22:03:12.437 DEBUG:teuthology.orchestra.run.smithi039:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T22:03:12.440 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-21T22:03:12.440 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T22:03:12.442 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-21T22:03:12.442 DEBUG:teuthology.orchestra.run.smithi157:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T22:03:12.467 INFO:teuthology.orchestra.run.smithi039.stdout:21 Sep 22:03:12 ntpd[18413]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-09-21T22:03:12.467 INFO:teuthology.orchestra.run.smithi039.stdout:21 Sep 22:03:12 ntpd[18413]: Command line: ntpd -gq 2024-09-21T22:03:12.468 INFO:teuthology.orchestra.run.smithi039.stdout:21 Sep 22:03:12 ntpd[18413]: ---------------------------------------------------- 2024-09-21T22:03:12.468 INFO:teuthology.orchestra.run.smithi039.stdout:21 Sep 22:03:12 ntpd[18413]: ntp-4 is maintained by Network Time Foundation, 2024-09-21T22:03:12.468 INFO:teuthology.orchestra.run.smithi039.stdout:21 Sep 22:03:12 ntpd[18413]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-09-21T22:03:12.468 INFO:teuthology.orchestra.run.smithi039.stdout:21 Sep 22:03:12 ntpd[18413]: corporation. Support and training for ntp-4 are 2024-09-21T22:03:12.468 INFO:teuthology.orchestra.run.smithi039.stdout:21 Sep 22:03:12 ntpd[18413]: available at https://www.nwtime.org/support 2024-09-21T22:03:12.468 INFO:teuthology.orchestra.run.smithi039.stdout:21 Sep 22:03:12 ntpd[18413]: ---------------------------------------------------- 2024-09-21T22:03:12.468 INFO:teuthology.orchestra.run.smithi039.stdout:21 Sep 22:03:12 ntpd[18413]: proto: precision = 0.047 usec (-24) 2024-09-21T22:03:12.469 INFO:teuthology.orchestra.run.smithi039.stdout:21 Sep 22:03:12 ntpd[18413]: basedate set to 2022-02-04 2024-09-21T22:03:12.469 INFO:teuthology.orchestra.run.smithi039.stdout:21 Sep 22:03:12 ntpd[18413]: gps base set to 2022-02-06 (week 2196) 2024-09-21T22:03:12.469 INFO:teuthology.orchestra.run.smithi039.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-21T22:03:12.469 INFO:teuthology.orchestra.run.smithi039.stdout:21 Sep 22:03:12 ntpd[18413]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-21T22:03:12.469 INFO:teuthology.orchestra.run.smithi039.stdout:21 Sep 22:03:12 ntpd[18413]: restrict ::: KOD does nothing without LIMITED. 2024-09-21T22:03:12.469 INFO:teuthology.orchestra.run.smithi039.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-21T22:03:12.470 INFO:teuthology.orchestra.run.smithi039.stdout:21 Sep 22:03:12 ntpd[18413]: Listen and drop on 0 v6wildcard [::]:123 2024-09-21T22:03:12.470 INFO:teuthology.orchestra.run.smithi039.stdout:21 Sep 22:03:12 ntpd[18413]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-21T22:03:12.471 INFO:teuthology.orchestra.run.smithi039.stdout:21 Sep 22:03:12 ntpd[18413]: Listen normally on 2 lo 127.0.0.1:123 2024-09-21T22:03:12.471 INFO:teuthology.orchestra.run.smithi039.stdout:21 Sep 22:03:12 ntpd[18413]: Listen normally on 3 ens1f0 172.21.15.39:123 2024-09-21T22:03:12.471 INFO:teuthology.orchestra.run.smithi039.stdout:21 Sep 22:03:12 ntpd[18413]: Listen normally on 4 lo [::1]:123 2024-09-21T22:03:12.471 INFO:teuthology.orchestra.run.smithi039.stdout:21 Sep 22:03:12 ntpd[18413]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:1370%4]:123 2024-09-21T22:03:12.471 INFO:teuthology.orchestra.run.smithi039.stdout:21 Sep 22:03:12 ntpd[18413]: Listening on routing socket on fd #22 for interface updates 2024-09-21T22:03:12.472 INFO:teuthology.orchestra.run.smithi064.stdout:21 Sep 22:03:12 ntpd[18490]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-09-21T22:03:12.472 INFO:teuthology.orchestra.run.smithi064.stdout:21 Sep 22:03:12 ntpd[18490]: Command line: ntpd -gq 2024-09-21T22:03:12.472 INFO:teuthology.orchestra.run.smithi064.stdout:21 Sep 22:03:12 ntpd[18490]: ---------------------------------------------------- 2024-09-21T22:03:12.472 INFO:teuthology.orchestra.run.smithi064.stdout:21 Sep 22:03:12 ntpd[18490]: ntp-4 is maintained by Network Time Foundation, 2024-09-21T22:03:12.472 INFO:teuthology.orchestra.run.smithi064.stdout:21 Sep 22:03:12 ntpd[18490]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-09-21T22:03:12.472 INFO:teuthology.orchestra.run.smithi064.stdout:21 Sep 22:03:12 ntpd[18490]: corporation. Support and training for ntp-4 are 2024-09-21T22:03:12.472 INFO:teuthology.orchestra.run.smithi064.stdout:21 Sep 22:03:12 ntpd[18490]: available at https://www.nwtime.org/support 2024-09-21T22:03:12.472 INFO:teuthology.orchestra.run.smithi064.stdout:21 Sep 22:03:12 ntpd[18490]: ---------------------------------------------------- 2024-09-21T22:03:12.472 INFO:teuthology.orchestra.run.smithi064.stdout:21 Sep 22:03:12 ntpd[18490]: proto: precision = 0.069 usec (-24) 2024-09-21T22:03:12.473 INFO:teuthology.orchestra.run.smithi064.stdout:21 Sep 22:03:12 ntpd[18490]: basedate set to 2022-02-04 2024-09-21T22:03:12.473 INFO:teuthology.orchestra.run.smithi064.stdout:21 Sep 22:03:12 ntpd[18490]: gps base set to 2022-02-06 (week 2196) 2024-09-21T22:03:12.473 INFO:teuthology.orchestra.run.smithi064.stdout:21 Sep 22:03:12 ntpd[18490]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-21T22:03:12.473 INFO:teuthology.orchestra.run.smithi064.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-21T22:03:12.473 INFO:teuthology.orchestra.run.smithi064.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-21T22:03:12.474 INFO:teuthology.orchestra.run.smithi064.stdout:21 Sep 22:03:12 ntpd[18490]: restrict ::: KOD does nothing without LIMITED. 2024-09-21T22:03:12.474 INFO:teuthology.orchestra.run.smithi064.stdout:21 Sep 22:03:12 ntpd[18490]: Listen and drop on 0 v6wildcard [::]:123 2024-09-21T22:03:12.474 INFO:teuthology.orchestra.run.smithi064.stdout:21 Sep 22:03:12 ntpd[18490]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-21T22:03:12.474 INFO:teuthology.orchestra.run.smithi064.stdout:21 Sep 22:03:12 ntpd[18490]: Listen normally on 2 lo 127.0.0.1:123 2024-09-21T22:03:12.474 INFO:teuthology.orchestra.run.smithi064.stdout:21 Sep 22:03:12 ntpd[18490]: Listen normally on 3 enp3s0f1 172.21.15.64:123 2024-09-21T22:03:12.474 INFO:teuthology.orchestra.run.smithi064.stdout:21 Sep 22:03:12 ntpd[18490]: Listen normally on 4 lo [::1]:123 2024-09-21T22:03:12.474 INFO:teuthology.orchestra.run.smithi064.stdout:21 Sep 22:03:12 ntpd[18490]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:feb6:f8df%5]:123 2024-09-21T22:03:12.475 INFO:teuthology.orchestra.run.smithi064.stdout:21 Sep 22:03:12 ntpd[18490]: Listening on routing socket on fd #22 for interface updates 2024-09-21T22:03:12.495 INFO:teuthology.orchestra.run.smithi157.stdout:21 Sep 22:03:12 ntpd[18456]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-09-21T22:03:12.495 INFO:teuthology.orchestra.run.smithi157.stdout:21 Sep 22:03:12 ntpd[18456]: Command line: ntpd -gq 2024-09-21T22:03:12.495 INFO:teuthology.orchestra.run.smithi157.stdout:21 Sep 22:03:12 ntpd[18456]: ---------------------------------------------------- 2024-09-21T22:03:12.495 INFO:teuthology.orchestra.run.smithi157.stdout:21 Sep 22:03:12 ntpd[18456]: ntp-4 is maintained by Network Time Foundation, 2024-09-21T22:03:12.495 INFO:teuthology.orchestra.run.smithi157.stdout:21 Sep 22:03:12 ntpd[18456]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-09-21T22:03:12.495 INFO:teuthology.orchestra.run.smithi157.stdout:21 Sep 22:03:12 ntpd[18456]: corporation. Support and training for ntp-4 are 2024-09-21T22:03:12.495 INFO:teuthology.orchestra.run.smithi157.stdout:21 Sep 22:03:12 ntpd[18456]: available at https://www.nwtime.org/support 2024-09-21T22:03:12.495 INFO:teuthology.orchestra.run.smithi157.stdout:21 Sep 22:03:12 ntpd[18456]: ---------------------------------------------------- 2024-09-21T22:03:12.496 INFO:teuthology.orchestra.run.smithi157.stdout:21 Sep 22:03:12 ntpd[18456]: proto: precision = 0.044 usec (-24) 2024-09-21T22:03:12.496 INFO:teuthology.orchestra.run.smithi157.stdout:21 Sep 22:03:12 ntpd[18456]: basedate set to 2022-02-04 2024-09-21T22:03:12.496 INFO:teuthology.orchestra.run.smithi157.stdout:21 Sep 22:03:12 ntpd[18456]: gps base set to 2022-02-06 (week 2196) 2024-09-21T22:03:12.496 INFO:teuthology.orchestra.run.smithi157.stdout:21 Sep 22:03:12 ntpd[18456]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-21T22:03:12.496 INFO:teuthology.orchestra.run.smithi157.stdout:21 Sep 22:03:12 ntpd[18456]: restrict ::: KOD does nothing without LIMITED. 2024-09-21T22:03:12.496 INFO:teuthology.orchestra.run.smithi157.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-21T22:03:12.497 INFO:teuthology.orchestra.run.smithi157.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-21T22:03:12.497 INFO:teuthology.orchestra.run.smithi157.stdout:21 Sep 22:03:12 ntpd[18456]: Listen and drop on 0 v6wildcard [::]:123 2024-09-21T22:03:12.497 INFO:teuthology.orchestra.run.smithi157.stdout:21 Sep 22:03:12 ntpd[18456]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-21T22:03:12.497 INFO:teuthology.orchestra.run.smithi157.stdout:21 Sep 22:03:12 ntpd[18456]: Listen normally on 2 lo 127.0.0.1:123 2024-09-21T22:03:12.497 INFO:teuthology.orchestra.run.smithi157.stdout:21 Sep 22:03:12 ntpd[18456]: Listen normally on 3 enp3s0f1 172.21.15.157:123 2024-09-21T22:03:12.498 INFO:teuthology.orchestra.run.smithi157.stdout:21 Sep 22:03:12 ntpd[18456]: Listen normally on 4 lo [::1]:123 2024-09-21T22:03:12.498 INFO:teuthology.orchestra.run.smithi157.stdout:21 Sep 22:03:12 ntpd[18456]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:7ea1%5]:123 2024-09-21T22:03:12.498 INFO:teuthology.orchestra.run.smithi157.stdout:21 Sep 22:03:12 ntpd[18456]: Listening on routing socket on fd #22 for interface updates 2024-09-21T22:03:13.471 INFO:teuthology.orchestra.run.smithi064.stderr:21 Sep 22:03:13 ntpd[18490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T22:03:13.471 INFO:teuthology.orchestra.run.smithi064.stderr:21 Sep 22:03:13 ntpd[18490]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T22:03:13.471 INFO:teuthology.orchestra.run.smithi064.stderr:21 Sep 22:03:13 ntpd[18490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T22:03:13.471 INFO:teuthology.orchestra.run.smithi064.stderr:21 Sep 22:03:13 ntpd[18490]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T22:03:13.471 INFO:teuthology.orchestra.run.smithi064.stderr:21 Sep 22:03:13 ntpd[18490]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T22:03:13.471 INFO:teuthology.orchestra.run.smithi064.stderr:21 Sep 22:03:13 ntpd[18490]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T22:03:13.496 INFO:teuthology.orchestra.run.smithi157.stderr:21 Sep 22:03:13 ntpd[18456]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T22:03:13.496 INFO:teuthology.orchestra.run.smithi157.stderr:21 Sep 22:03:13 ntpd[18456]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T22:03:13.496 INFO:teuthology.orchestra.run.smithi157.stderr:21 Sep 22:03:13 ntpd[18456]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T22:03:13.496 INFO:teuthology.orchestra.run.smithi157.stderr:21 Sep 22:03:13 ntpd[18456]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T22:03:13.496 INFO:teuthology.orchestra.run.smithi157.stderr:21 Sep 22:03:13 ntpd[18456]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T22:03:13.496 INFO:teuthology.orchestra.run.smithi157.stderr:21 Sep 22:03:13 ntpd[18456]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T22:03:13.496 INFO:teuthology.orchestra.run.smithi157.stderr:21 Sep 22:03:13 ntpd[18456]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T22:03:13.496 INFO:teuthology.orchestra.run.smithi157.stderr:21 Sep 22:03:13 ntpd[18456]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T22:03:14.471 INFO:teuthology.orchestra.run.smithi064.stderr:21 Sep 22:03:14 ntpd[18490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T22:03:14.471 INFO:teuthology.orchestra.run.smithi064.stderr:21 Sep 22:03:14 ntpd[18490]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T22:03:14.495 INFO:teuthology.orchestra.run.smithi157.stderr:21 Sep 22:03:14 ntpd[18456]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T22:03:14.495 INFO:teuthology.orchestra.run.smithi157.stderr:21 Sep 22:03:14 ntpd[18456]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T22:03:14.495 INFO:teuthology.orchestra.run.smithi157.stderr:21 Sep 22:03:14 ntpd[18456]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T22:03:14.496 INFO:teuthology.orchestra.run.smithi157.stderr:21 Sep 22:03:14 ntpd[18456]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T22:03:14.496 INFO:teuthology.orchestra.run.smithi157.stderr:21 Sep 22:03:14 ntpd[18456]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T22:03:14.496 INFO:teuthology.orchestra.run.smithi157.stderr:21 Sep 22:03:14 ntpd[18456]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T22:03:15.468 INFO:teuthology.orchestra.run.smithi039.stderr:21 Sep 22:03:15 ntpd[18413]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T22:03:15.468 INFO:teuthology.orchestra.run.smithi039.stderr:21 Sep 22:03:15 ntpd[18413]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T22:03:15.468 INFO:teuthology.orchestra.run.smithi039.stderr:21 Sep 22:03:15 ntpd[18413]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T22:03:15.469 INFO:teuthology.orchestra.run.smithi039.stderr:21 Sep 22:03:15 ntpd[18413]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T22:03:15.469 INFO:teuthology.orchestra.run.smithi039.stderr:21 Sep 22:03:15 ntpd[18413]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T22:03:15.469 INFO:teuthology.orchestra.run.smithi039.stderr:21 Sep 22:03:15 ntpd[18413]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T22:03:15.471 INFO:teuthology.orchestra.run.smithi064.stderr:21 Sep 22:03:15 ntpd[18490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T22:03:15.471 INFO:teuthology.orchestra.run.smithi064.stderr:21 Sep 22:03:15 ntpd[18490]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T22:03:15.471 INFO:teuthology.orchestra.run.smithi064.stderr:21 Sep 22:03:15 ntpd[18490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T22:03:15.471 INFO:teuthology.orchestra.run.smithi064.stderr:21 Sep 22:03:15 ntpd[18490]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T22:03:15.471 INFO:teuthology.orchestra.run.smithi064.stderr:21 Sep 22:03:15 ntpd[18490]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T22:03:15.471 INFO:teuthology.orchestra.run.smithi064.stderr:21 Sep 22:03:15 ntpd[18490]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T22:03:15.495 INFO:teuthology.orchestra.run.smithi157.stderr:21 Sep 22:03:15 ntpd[18456]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T22:03:15.496 INFO:teuthology.orchestra.run.smithi157.stderr:21 Sep 22:03:15 ntpd[18456]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T22:03:15.496 INFO:teuthology.orchestra.run.smithi157.stderr:21 Sep 22:03:15 ntpd[18456]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T22:03:15.496 INFO:teuthology.orchestra.run.smithi157.stderr:21 Sep 22:03:15 ntpd[18456]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T22:03:15.496 INFO:teuthology.orchestra.run.smithi157.stderr:21 Sep 22:03:15 ntpd[18456]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T22:03:15.496 INFO:teuthology.orchestra.run.smithi157.stderr:21 Sep 22:03:15 ntpd[18456]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T22:03:16.468 INFO:teuthology.orchestra.run.smithi039.stderr:21 Sep 22:03:16 ntpd[18413]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T22:03:16.469 INFO:teuthology.orchestra.run.smithi039.stderr:21 Sep 22:03:16 ntpd[18413]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T22:03:16.469 INFO:teuthology.orchestra.run.smithi039.stderr:21 Sep 22:03:16 ntpd[18413]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T22:03:16.469 INFO:teuthology.orchestra.run.smithi039.stderr:21 Sep 22:03:16 ntpd[18413]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T22:03:16.469 INFO:teuthology.orchestra.run.smithi039.stderr:21 Sep 22:03:16 ntpd[18413]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T22:03:16.469 INFO:teuthology.orchestra.run.smithi039.stderr:21 Sep 22:03:16 ntpd[18413]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T22:03:16.469 INFO:teuthology.orchestra.run.smithi039.stderr:21 Sep 22:03:16 ntpd[18413]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T22:03:16.469 INFO:teuthology.orchestra.run.smithi039.stderr:21 Sep 22:03:16 ntpd[18413]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T22:03:16.469 INFO:teuthology.orchestra.run.smithi039.stderr:21 Sep 22:03:16 ntpd[18413]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T22:03:16.469 INFO:teuthology.orchestra.run.smithi039.stderr:21 Sep 22:03:16 ntpd[18413]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T22:03:16.469 INFO:teuthology.orchestra.run.smithi039.stderr:21 Sep 22:03:16 ntpd[18413]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T22:03:16.469 INFO:teuthology.orchestra.run.smithi039.stderr:21 Sep 22:03:16 ntpd[18413]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T22:03:16.471 INFO:teuthology.orchestra.run.smithi064.stderr:21 Sep 22:03:16 ntpd[18490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T22:03:16.471 INFO:teuthology.orchestra.run.smithi064.stderr:21 Sep 22:03:16 ntpd[18490]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T22:03:16.471 INFO:teuthology.orchestra.run.smithi064.stderr:21 Sep 22:03:16 ntpd[18490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T22:03:16.471 INFO:teuthology.orchestra.run.smithi064.stderr:21 Sep 22:03:16 ntpd[18490]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T22:03:16.471 INFO:teuthology.orchestra.run.smithi064.stderr:21 Sep 22:03:16 ntpd[18490]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T22:03:16.471 INFO:teuthology.orchestra.run.smithi064.stderr:21 Sep 22:03:16 ntpd[18490]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T22:03:16.495 INFO:teuthology.orchestra.run.smithi157.stderr:21 Sep 22:03:16 ntpd[18456]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T22:03:16.495 INFO:teuthology.orchestra.run.smithi157.stderr:21 Sep 22:03:16 ntpd[18456]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T22:03:16.495 INFO:teuthology.orchestra.run.smithi157.stderr:21 Sep 22:03:16 ntpd[18456]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T22:03:16.496 INFO:teuthology.orchestra.run.smithi157.stderr:21 Sep 22:03:16 ntpd[18456]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T22:03:16.496 INFO:teuthology.orchestra.run.smithi157.stderr:21 Sep 22:03:16 ntpd[18456]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T22:03:16.496 INFO:teuthology.orchestra.run.smithi157.stderr:21 Sep 22:03:16 ntpd[18456]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T22:03:17.468 INFO:teuthology.orchestra.run.smithi039.stderr:21 Sep 22:03:17 ntpd[18413]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T22:03:17.468 INFO:teuthology.orchestra.run.smithi039.stderr:21 Sep 22:03:17 ntpd[18413]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T22:03:17.468 INFO:teuthology.orchestra.run.smithi039.stderr:21 Sep 22:03:17 ntpd[18413]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T22:03:17.468 INFO:teuthology.orchestra.run.smithi039.stderr:21 Sep 22:03:17 ntpd[18413]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T22:03:17.468 INFO:teuthology.orchestra.run.smithi039.stderr:21 Sep 22:03:17 ntpd[18413]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T22:03:17.469 INFO:teuthology.orchestra.run.smithi039.stderr:21 Sep 22:03:17 ntpd[18413]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T22:03:17.469 INFO:teuthology.orchestra.run.smithi039.stderr:21 Sep 22:03:17 ntpd[18413]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T22:03:17.469 INFO:teuthology.orchestra.run.smithi039.stderr:21 Sep 22:03:17 ntpd[18413]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T22:03:17.471 INFO:teuthology.orchestra.run.smithi064.stderr:21 Sep 22:03:17 ntpd[18490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T22:03:17.471 INFO:teuthology.orchestra.run.smithi064.stderr:21 Sep 22:03:17 ntpd[18490]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T22:03:17.471 INFO:teuthology.orchestra.run.smithi064.stderr:21 Sep 22:03:17 ntpd[18490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T22:03:17.472 INFO:teuthology.orchestra.run.smithi064.stderr:21 Sep 22:03:17 ntpd[18490]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T22:03:17.472 INFO:teuthology.orchestra.run.smithi064.stderr:21 Sep 22:03:17 ntpd[18490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T22:03:17.472 INFO:teuthology.orchestra.run.smithi064.stderr:21 Sep 22:03:17 ntpd[18490]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T22:03:17.472 INFO:teuthology.orchestra.run.smithi064.stderr:21 Sep 22:03:17 ntpd[18490]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T22:03:17.472 INFO:teuthology.orchestra.run.smithi064.stderr:21 Sep 22:03:17 ntpd[18490]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T22:03:17.472 INFO:teuthology.orchestra.run.smithi064.stderr:21 Sep 22:03:17 ntpd[18490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T22:03:17.472 INFO:teuthology.orchestra.run.smithi064.stderr:21 Sep 22:03:17 ntpd[18490]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T22:03:17.472 INFO:teuthology.orchestra.run.smithi064.stderr:21 Sep 22:03:17 ntpd[18490]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T22:03:17.472 INFO:teuthology.orchestra.run.smithi064.stderr:21 Sep 22:03:17 ntpd[18490]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T22:03:17.496 INFO:teuthology.orchestra.run.smithi157.stderr:21 Sep 22:03:17 ntpd[18456]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T22:03:17.496 INFO:teuthology.orchestra.run.smithi157.stderr:21 Sep 22:03:17 ntpd[18456]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T22:03:17.496 INFO:teuthology.orchestra.run.smithi157.stderr:21 Sep 22:03:17 ntpd[18456]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T22:03:17.496 INFO:teuthology.orchestra.run.smithi157.stderr:21 Sep 22:03:17 ntpd[18456]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T22:03:17.496 INFO:teuthology.orchestra.run.smithi157.stderr:21 Sep 22:03:17 ntpd[18456]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T22:03:17.497 INFO:teuthology.orchestra.run.smithi157.stderr:21 Sep 22:03:17 ntpd[18456]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T22:03:17.497 INFO:teuthology.orchestra.run.smithi157.stderr:21 Sep 22:03:17 ntpd[18456]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T22:03:17.497 INFO:teuthology.orchestra.run.smithi157.stderr:21 Sep 22:03:17 ntpd[18456]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T22:03:17.497 INFO:teuthology.orchestra.run.smithi157.stderr:21 Sep 22:03:17 ntpd[18456]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T22:03:17.497 INFO:teuthology.orchestra.run.smithi157.stderr:21 Sep 22:03:17 ntpd[18456]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T22:03:17.497 INFO:teuthology.orchestra.run.smithi157.stderr:21 Sep 22:03:17 ntpd[18456]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T22:03:17.497 INFO:teuthology.orchestra.run.smithi157.stderr:21 Sep 22:03:17 ntpd[18456]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T22:03:18.469 INFO:teuthology.orchestra.run.smithi039.stderr:21 Sep 22:03:18 ntpd[18413]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T22:03:18.469 INFO:teuthology.orchestra.run.smithi039.stderr:21 Sep 22:03:18 ntpd[18413]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T22:03:18.469 INFO:teuthology.orchestra.run.smithi039.stderr:21 Sep 22:03:18 ntpd[18413]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T22:03:18.469 INFO:teuthology.orchestra.run.smithi039.stderr:21 Sep 22:03:18 ntpd[18413]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T22:03:18.469 INFO:teuthology.orchestra.run.smithi039.stderr:21 Sep 22:03:18 ntpd[18413]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T22:03:18.469 INFO:teuthology.orchestra.run.smithi039.stderr:21 Sep 22:03:18 ntpd[18413]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T22:03:18.469 INFO:teuthology.orchestra.run.smithi039.stderr:21 Sep 22:03:18 ntpd[18413]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T22:03:18.469 INFO:teuthology.orchestra.run.smithi039.stderr:21 Sep 22:03:18 ntpd[18413]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T22:03:18.470 INFO:teuthology.orchestra.run.smithi039.stderr:21 Sep 22:03:18 ntpd[18413]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T22:03:18.470 INFO:teuthology.orchestra.run.smithi039.stderr:21 Sep 22:03:18 ntpd[18413]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T22:03:18.470 INFO:teuthology.orchestra.run.smithi039.stderr:21 Sep 22:03:18 ntpd[18413]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T22:03:18.470 INFO:teuthology.orchestra.run.smithi039.stderr:21 Sep 22:03:18 ntpd[18413]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T22:03:18.471 INFO:teuthology.orchestra.run.smithi064.stderr:21 Sep 22:03:18 ntpd[18490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T22:03:18.471 INFO:teuthology.orchestra.run.smithi064.stderr:21 Sep 22:03:18 ntpd[18490]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T22:03:18.471 INFO:teuthology.orchestra.run.smithi064.stderr:21 Sep 22:03:18 ntpd[18490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T22:03:18.471 INFO:teuthology.orchestra.run.smithi064.stderr:21 Sep 22:03:18 ntpd[18490]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T22:03:18.471 INFO:teuthology.orchestra.run.smithi064.stderr:21 Sep 22:03:18 ntpd[18490]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T22:03:18.471 INFO:teuthology.orchestra.run.smithi064.stderr:21 Sep 22:03:18 ntpd[18490]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T22:03:18.495 INFO:teuthology.orchestra.run.smithi157.stderr:21 Sep 22:03:18 ntpd[18456]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T22:03:18.495 INFO:teuthology.orchestra.run.smithi157.stderr:21 Sep 22:03:18 ntpd[18456]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T22:03:18.496 INFO:teuthology.orchestra.run.smithi157.stderr:21 Sep 22:03:18 ntpd[18456]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T22:03:18.496 INFO:teuthology.orchestra.run.smithi157.stderr:21 Sep 22:03:18 ntpd[18456]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T22:03:18.496 INFO:teuthology.orchestra.run.smithi157.stderr:21 Sep 22:03:18 ntpd[18456]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T22:03:18.496 INFO:teuthology.orchestra.run.smithi157.stderr:21 Sep 22:03:18 ntpd[18456]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T22:03:19.468 INFO:teuthology.orchestra.run.smithi039.stderr:21 Sep 22:03:19 ntpd[18413]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T22:03:19.468 INFO:teuthology.orchestra.run.smithi039.stderr:21 Sep 22:03:19 ntpd[18413]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T22:03:19.468 INFO:teuthology.orchestra.run.smithi039.stderr:21 Sep 22:03:19 ntpd[18413]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T22:03:19.468 INFO:teuthology.orchestra.run.smithi039.stderr:21 Sep 22:03:19 ntpd[18413]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T22:03:19.469 INFO:teuthology.orchestra.run.smithi039.stderr:21 Sep 22:03:19 ntpd[18413]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T22:03:19.469 INFO:teuthology.orchestra.run.smithi039.stderr:21 Sep 22:03:19 ntpd[18413]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T22:03:19.471 INFO:teuthology.orchestra.run.smithi064.stderr:21 Sep 22:03:19 ntpd[18490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T22:03:19.471 INFO:teuthology.orchestra.run.smithi064.stderr:21 Sep 22:03:19 ntpd[18490]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T22:03:19.471 INFO:teuthology.orchestra.run.smithi064.stderr:21 Sep 22:03:19 ntpd[18490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T22:03:19.471 INFO:teuthology.orchestra.run.smithi064.stderr:21 Sep 22:03:19 ntpd[18490]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T22:03:19.471 INFO:teuthology.orchestra.run.smithi064.stderr:21 Sep 22:03:19 ntpd[18490]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T22:03:19.471 INFO:teuthology.orchestra.run.smithi064.stderr:21 Sep 22:03:19 ntpd[18490]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T22:03:19.471 INFO:teuthology.orchestra.run.smithi064.stderr:21 Sep 22:03:19 ntpd[18490]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T22:03:19.471 INFO:teuthology.orchestra.run.smithi064.stderr:21 Sep 22:03:19 ntpd[18490]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T22:03:19.472 INFO:teuthology.orchestra.run.smithi064.stdout:21 Sep 22:03:19 ntpd[18490]: ntpd: time slew -0.001074 s 2024-09-21T22:03:19.472 INFO:teuthology.orchestra.run.smithi064.stdout:ntpd: time slew -0.001074s 2024-09-21T22:03:19.472 INFO:teuthology.orchestra.run.smithi064.stderr:21 Sep 22:03:19 ntpd[18490]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-21T22:03:19.472 INFO:teuthology.orchestra.run.smithi064.stderr:21 Sep 22:03:19 ntpd[18490]: can't open /var/log/ntpstats/loopstats.20240921: Permission denied 2024-09-21T22:03:19.496 INFO:teuthology.orchestra.run.smithi157.stderr:21 Sep 22:03:19 ntpd[18456]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T22:03:19.496 INFO:teuthology.orchestra.run.smithi157.stderr:21 Sep 22:03:19 ntpd[18456]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T22:03:19.496 INFO:teuthology.orchestra.run.smithi157.stderr:21 Sep 22:03:19 ntpd[18456]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T22:03:19.496 INFO:teuthology.orchestra.run.smithi157.stderr:21 Sep 22:03:19 ntpd[18456]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T22:03:19.496 INFO:teuthology.orchestra.run.smithi157.stderr:21 Sep 22:03:19 ntpd[18456]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T22:03:19.496 INFO:teuthology.orchestra.run.smithi157.stderr:21 Sep 22:03:19 ntpd[18456]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T22:03:19.496 INFO:teuthology.orchestra.run.smithi157.stderr:21 Sep 22:03:19 ntpd[18456]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T22:03:19.496 INFO:teuthology.orchestra.run.smithi157.stderr:21 Sep 22:03:19 ntpd[18456]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T22:03:19.496 INFO:teuthology.orchestra.run.smithi157.stdout:21 Sep 22:03:19 ntpd[18456]: ntpd: time slew +0.002265 s 2024-09-21T22:03:19.496 INFO:teuthology.orchestra.run.smithi157.stdout:ntpd: time slew +0.002265s 2024-09-21T22:03:19.497 INFO:teuthology.orchestra.run.smithi157.stderr:21 Sep 22:03:19 ntpd[18456]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-21T22:03:19.497 INFO:teuthology.orchestra.run.smithi157.stderr:21 Sep 22:03:19 ntpd[18456]: can't open /var/log/ntpstats/loopstats.20240921: Permission denied 2024-09-21T22:03:19.521 INFO:teuthology.orchestra.run.smithi064.stdout: remote refid st t when poll reach delay offset jitter 2024-09-21T22:03:19.521 INFO:teuthology.orchestra.run.smithi064.stdout:============================================================================== 2024-09-21T22:03:19.521 INFO:teuthology.orchestra.run.smithi064.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-21T22:03:19.521 INFO:teuthology.orchestra.run.smithi064.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-21T22:03:19.521 INFO:teuthology.orchestra.run.smithi064.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-21T22:03:19.521 INFO:teuthology.orchestra.run.smithi064.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-21T22:03:19.544 INFO:teuthology.orchestra.run.smithi157.stdout: remote refid st t when poll reach delay offset jitter 2024-09-21T22:03:19.544 INFO:teuthology.orchestra.run.smithi157.stdout:============================================================================== 2024-09-21T22:03:19.544 INFO:teuthology.orchestra.run.smithi157.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-21T22:03:19.545 INFO:teuthology.orchestra.run.smithi157.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-21T22:03:19.545 INFO:teuthology.orchestra.run.smithi157.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-21T22:03:19.545 INFO:teuthology.orchestra.run.smithi157.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-21T22:03:20.469 INFO:teuthology.orchestra.run.smithi039.stderr:21 Sep 22:03:20 ntpd[18413]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T22:03:20.469 INFO:teuthology.orchestra.run.smithi039.stderr:21 Sep 22:03:20 ntpd[18413]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T22:03:20.469 INFO:teuthology.orchestra.run.smithi039.stderr:21 Sep 22:03:20 ntpd[18413]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T22:03:20.469 INFO:teuthology.orchestra.run.smithi039.stderr:21 Sep 22:03:20 ntpd[18413]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T22:03:20.469 INFO:teuthology.orchestra.run.smithi039.stderr:21 Sep 22:03:20 ntpd[18413]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T22:03:20.469 INFO:teuthology.orchestra.run.smithi039.stderr:21 Sep 22:03:20 ntpd[18413]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T22:03:20.469 INFO:teuthology.orchestra.run.smithi039.stderr:21 Sep 22:03:20 ntpd[18413]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T22:03:20.469 INFO:teuthology.orchestra.run.smithi039.stderr:21 Sep 22:03:20 ntpd[18413]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T22:03:20.469 INFO:teuthology.orchestra.run.smithi039.stderr:21 Sep 22:03:20 ntpd[18413]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T22:03:20.469 INFO:teuthology.orchestra.run.smithi039.stderr:21 Sep 22:03:20 ntpd[18413]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T22:03:20.469 INFO:teuthology.orchestra.run.smithi039.stderr:21 Sep 22:03:20 ntpd[18413]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T22:03:20.470 INFO:teuthology.orchestra.run.smithi039.stderr:21 Sep 22:03:20 ntpd[18413]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T22:03:21.468 INFO:teuthology.orchestra.run.smithi039.stderr:21 Sep 22:03:21 ntpd[18413]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T22:03:21.468 INFO:teuthology.orchestra.run.smithi039.stderr:21 Sep 22:03:21 ntpd[18413]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T22:03:21.468 INFO:teuthology.orchestra.run.smithi039.stderr:21 Sep 22:03:21 ntpd[18413]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-21T22:03:21.468 INFO:teuthology.orchestra.run.smithi039.stderr:21 Sep 22:03:21 ntpd[18413]: can't open /var/log/ntpstats/rawstats.20240921: Permission denied 2024-09-21T22:03:21.468 INFO:teuthology.orchestra.run.smithi039.stderr:21 Sep 22:03:21 ntpd[18413]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-21T22:03:21.468 INFO:teuthology.orchestra.run.smithi039.stderr:21 Sep 22:03:21 ntpd[18413]: can't open /var/log/ntpstats/peerstats.20240921: Permission denied 2024-09-21T22:03:21.469 INFO:teuthology.orchestra.run.smithi039.stdout:21 Sep 22:03:21 ntpd[18413]: ntpd: time slew -0.000278 s 2024-09-21T22:03:21.469 INFO:teuthology.orchestra.run.smithi039.stdout:ntpd: time slew -0.000278s 2024-09-21T22:03:21.469 INFO:teuthology.orchestra.run.smithi039.stderr:21 Sep 22:03:21 ntpd[18413]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-21T22:03:21.469 INFO:teuthology.orchestra.run.smithi039.stderr:21 Sep 22:03:21 ntpd[18413]: can't open /var/log/ntpstats/loopstats.20240921: Permission denied 2024-09-21T22:03:21.523 INFO:teuthology.orchestra.run.smithi039.stdout: remote refid st t when poll reach delay offset jitter 2024-09-21T22:03:21.523 INFO:teuthology.orchestra.run.smithi039.stdout:============================================================================== 2024-09-21T22:03:21.523 INFO:teuthology.orchestra.run.smithi039.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-21T22:03:21.524 INFO:teuthology.orchestra.run.smithi039.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-21T22:03:21.524 INFO:teuthology.orchestra.run.smithi039.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-21T22:03:21.524 INFO:teuthology.orchestra.run.smithi039.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-21T22:03:21.524 INFO:teuthology.run_tasks:Running task install... 2024-09-21T22:03:21.533 DEBUG:teuthology.task.install:project ceph 2024-09-21T22:03:21.533 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'extra_packages': ['rbd-nbd'], 'flavor': 'default', 'sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7'}} 2024-09-21T22:03:21.533 DEBUG:teuthology.task.install:config {'extra_packages': ['rbd-nbd'], 'flavor': 'default', 'sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7'} 2024-09-21T22:03:21.533 INFO:teuthology.task.install:Using flavor: default 2024-09-21T22:03:21.538 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-21T22:03:21.538 INFO:teuthology.task.install:extra packages: ['rbd-nbd'] 2024-09-21T22:03:21.538 DEBUG:teuthology.orchestra.run.smithi039:> sudo apt-key list | grep Ceph 2024-09-21T22:03:21.538 DEBUG:teuthology.orchestra.run.smithi064:> sudo apt-key list | grep Ceph 2024-09-21T22:03:21.539 DEBUG:teuthology.orchestra.run.smithi157:> sudo apt-key list | grep Ceph 2024-09-21T22:03:21.598 INFO:teuthology.orchestra.run.smithi157.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-09-21T22:03:21.608 INFO:teuthology.orchestra.run.smithi064.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-09-21T22:03:21.627 INFO:teuthology.orchestra.run.smithi157.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-21T22:03:21.627 INFO:teuthology.orchestra.run.smithi157.stdout:uid [ unknown] Ceph.com (release key) 2024-09-21T22:03:21.628 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, rbd-nbd on remote deb x86_64 2024-09-21T22:03:21.628 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=4e45e6b490e6cbe082cdf155fea780438ec53fc7 2024-09-21T22:03:21.632 INFO:teuthology.orchestra.run.smithi039.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-09-21T22:03:21.640 INFO:teuthology.orchestra.run.smithi064.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-21T22:03:21.640 INFO:teuthology.orchestra.run.smithi064.stdout:uid [ unknown] Ceph.com (release key) 2024-09-21T22:03:21.641 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, rbd-nbd on remote deb x86_64 2024-09-21T22:03:21.641 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=4e45e6b490e6cbe082cdf155fea780438ec53fc7 2024-09-21T22:03:21.660 INFO:teuthology.orchestra.run.smithi039.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-21T22:03:21.660 INFO:teuthology.orchestra.run.smithi039.stdout:uid [ unknown] Ceph.com (release key) 2024-09-21T22:03:21.661 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, rbd-nbd on remote deb x86_64 2024-09-21T22:03:21.661 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=4e45e6b490e6cbe082cdf155fea780438ec53fc7 2024-09-21T22:03:21.777 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default/ 2024-09-21T22:03:21.778 INFO:teuthology.task.install.deb:Package version is 19.1.1-373-g4e45e6b4-1jammy 2024-09-21T22:03:21.780 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default/ 2024-09-21T22:03:21.781 INFO:teuthology.task.install.deb:Package version is 19.1.1-373-g4e45e6b4-1jammy 2024-09-21T22:03:21.799 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default/ 2024-09-21T22:03:21.799 INFO:teuthology.task.install.deb:Package version is 19.1.1-373-g4e45e6b4-1jammy 2024-09-21T22:03:21.887 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-09-21T22:03:21.887 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-21T22:03:21.896 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-09-21T22:03:21.896 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-21T22:03:21.901 DEBUG:teuthology.orchestra.run.smithi157:> sudo apt-get update 2024-09-21T22:03:21.906 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-21T22:03:21.906 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-21T22:03:21.910 DEBUG:teuthology.orchestra.run.smithi064:> sudo apt-get update 2024-09-21T22:03:21.920 DEBUG:teuthology.orchestra.run.smithi039:> sudo apt-get update 2024-09-21T22:03:22.088 INFO:teuthology.orchestra.run.smithi157.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-21T22:03:22.089 INFO:teuthology.orchestra.run.smithi064.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-21T22:03:22.094 INFO:teuthology.orchestra.run.smithi039.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-21T22:03:22.125 INFO:teuthology.orchestra.run.smithi157.stdout:Ign:2 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy InRelease 2024-09-21T22:03:22.139 INFO:teuthology.orchestra.run.smithi064.stdout:Ign:2 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy InRelease 2024-09-21T22:03:22.145 INFO:teuthology.orchestra.run.smithi157.stdout:Get:3 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-09-21T22:03:22.147 INFO:teuthology.orchestra.run.smithi039.stdout:Ign:2 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy InRelease 2024-09-21T22:03:22.160 INFO:teuthology.orchestra.run.smithi064.stdout:Get:3 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-09-21T22:03:22.167 INFO:teuthology.orchestra.run.smithi157.stdout:Ign:4 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy Release.gpg 2024-09-21T22:03:22.170 INFO:teuthology.orchestra.run.smithi039.stdout:Get:3 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-09-21T22:03:22.182 INFO:teuthology.orchestra.run.smithi064.stdout:Ign:4 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy Release.gpg 2024-09-21T22:03:22.189 INFO:teuthology.orchestra.run.smithi157.stdout:Get:5 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.9 kB] 2024-09-21T22:03:22.191 INFO:teuthology.orchestra.run.smithi039.stdout:Ign:4 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy Release.gpg 2024-09-21T22:03:22.195 INFO:teuthology.orchestra.run.smithi157.stdout:Hit:6 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-21T22:03:22.203 INFO:teuthology.orchestra.run.smithi064.stdout:Hit:5 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-21T22:03:22.203 INFO:teuthology.orchestra.run.smithi064.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.9 kB] 2024-09-21T22:03:22.213 INFO:teuthology.orchestra.run.smithi039.stdout:Get:5 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.9 kB] 2024-09-21T22:03:22.217 INFO:teuthology.orchestra.run.smithi157.stdout:Get:7 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,413 B] 2024-09-21T22:03:22.219 INFO:teuthology.orchestra.run.smithi039.stdout:Hit:6 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-21T22:03:22.231 INFO:teuthology.orchestra.run.smithi064.stdout:Get:7 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,413 B] 2024-09-21T22:03:22.241 INFO:teuthology.orchestra.run.smithi039.stdout:Get:7 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,413 B] 2024-09-21T22:03:22.272 INFO:teuthology.orchestra.run.smithi157.stdout:Hit:8 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-21T22:03:22.285 INFO:teuthology.orchestra.run.smithi064.stdout:Hit:8 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-21T22:03:22.299 INFO:teuthology.orchestra.run.smithi039.stdout:Hit:8 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-21T22:03:22.353 INFO:teuthology.orchestra.run.smithi157.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-21T22:03:22.367 INFO:teuthology.orchestra.run.smithi064.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-21T22:03:22.377 INFO:teuthology.orchestra.run.smithi039.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-21T22:03:22.504 INFO:teuthology.orchestra.run.smithi157.stdout:Fetched 28.9 kB in 0s (69.5 kB/s) 2024-09-21T22:03:22.519 INFO:teuthology.orchestra.run.smithi064.stdout:Fetched 28.9 kB in 0s (69.3 kB/s) 2024-09-21T22:03:22.536 INFO:teuthology.orchestra.run.smithi039.stdout:Fetched 28.9 kB in 0s (68.6 kB/s) 2024-09-21T22:03:23.523 INFO:teuthology.orchestra.run.smithi157.stdout:Reading package lists... 2024-09-21T22:03:23.539 DEBUG:teuthology.orchestra.run.smithi157:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.1.1-373-g4e45e6b4-1jammy cephadm=19.1.1-373-g4e45e6b4-1jammy ceph-mds=19.1.1-373-g4e45e6b4-1jammy ceph-mgr=19.1.1-373-g4e45e6b4-1jammy ceph-common=19.1.1-373-g4e45e6b4-1jammy ceph-fuse=19.1.1-373-g4e45e6b4-1jammy ceph-test=19.1.1-373-g4e45e6b4-1jammy ceph-volume=19.1.1-373-g4e45e6b4-1jammy radosgw=19.1.1-373-g4e45e6b4-1jammy python3-rados=19.1.1-373-g4e45e6b4-1jammy python3-rgw=19.1.1-373-g4e45e6b4-1jammy python3-cephfs=19.1.1-373-g4e45e6b4-1jammy python3-rbd=19.1.1-373-g4e45e6b4-1jammy libcephfs2=19.1.1-373-g4e45e6b4-1jammy libcephfs-dev=19.1.1-373-g4e45e6b4-1jammy librados2=19.1.1-373-g4e45e6b4-1jammy librbd1=19.1.1-373-g4e45e6b4-1jammy rbd-fuse=19.1.1-373-g4e45e6b4-1jammy rbd-nbd=19.1.1-373-g4e45e6b4-1jammy 2024-09-21T22:03:23.564 INFO:teuthology.orchestra.run.smithi064.stdout:Reading package lists... 2024-09-21T22:03:23.579 DEBUG:teuthology.orchestra.run.smithi064:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.1.1-373-g4e45e6b4-1jammy cephadm=19.1.1-373-g4e45e6b4-1jammy ceph-mds=19.1.1-373-g4e45e6b4-1jammy ceph-mgr=19.1.1-373-g4e45e6b4-1jammy ceph-common=19.1.1-373-g4e45e6b4-1jammy ceph-fuse=19.1.1-373-g4e45e6b4-1jammy ceph-test=19.1.1-373-g4e45e6b4-1jammy ceph-volume=19.1.1-373-g4e45e6b4-1jammy radosgw=19.1.1-373-g4e45e6b4-1jammy python3-rados=19.1.1-373-g4e45e6b4-1jammy python3-rgw=19.1.1-373-g4e45e6b4-1jammy python3-cephfs=19.1.1-373-g4e45e6b4-1jammy python3-rbd=19.1.1-373-g4e45e6b4-1jammy libcephfs2=19.1.1-373-g4e45e6b4-1jammy libcephfs-dev=19.1.1-373-g4e45e6b4-1jammy librados2=19.1.1-373-g4e45e6b4-1jammy librbd1=19.1.1-373-g4e45e6b4-1jammy rbd-fuse=19.1.1-373-g4e45e6b4-1jammy rbd-nbd=19.1.1-373-g4e45e6b4-1jammy 2024-09-21T22:03:23.599 INFO:teuthology.orchestra.run.smithi039.stdout:Reading package lists... 2024-09-21T22:03:23.606 INFO:teuthology.orchestra.run.smithi157.stdout:Reading package lists... 2024-09-21T22:03:23.615 DEBUG:teuthology.orchestra.run.smithi039:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.1.1-373-g4e45e6b4-1jammy cephadm=19.1.1-373-g4e45e6b4-1jammy ceph-mds=19.1.1-373-g4e45e6b4-1jammy ceph-mgr=19.1.1-373-g4e45e6b4-1jammy ceph-common=19.1.1-373-g4e45e6b4-1jammy ceph-fuse=19.1.1-373-g4e45e6b4-1jammy ceph-test=19.1.1-373-g4e45e6b4-1jammy ceph-volume=19.1.1-373-g4e45e6b4-1jammy radosgw=19.1.1-373-g4e45e6b4-1jammy python3-rados=19.1.1-373-g4e45e6b4-1jammy python3-rgw=19.1.1-373-g4e45e6b4-1jammy python3-cephfs=19.1.1-373-g4e45e6b4-1jammy python3-rbd=19.1.1-373-g4e45e6b4-1jammy libcephfs2=19.1.1-373-g4e45e6b4-1jammy libcephfs-dev=19.1.1-373-g4e45e6b4-1jammy librados2=19.1.1-373-g4e45e6b4-1jammy librbd1=19.1.1-373-g4e45e6b4-1jammy rbd-fuse=19.1.1-373-g4e45e6b4-1jammy rbd-nbd=19.1.1-373-g4e45e6b4-1jammy 2024-09-21T22:03:23.634 INFO:teuthology.orchestra.run.smithi064.stdout:Reading package lists... 2024-09-21T22:03:23.669 INFO:teuthology.orchestra.run.smithi039.stdout:Reading package lists... 2024-09-21T22:03:23.816 INFO:teuthology.orchestra.run.smithi157.stdout:Building dependency tree... 2024-09-21T22:03:23.817 INFO:teuthology.orchestra.run.smithi157.stdout:Reading state information... 2024-09-21T22:03:23.824 INFO:teuthology.orchestra.run.smithi064.stdout:Building dependency tree... 2024-09-21T22:03:23.825 INFO:teuthology.orchestra.run.smithi064.stdout:Reading state information... 2024-09-21T22:03:23.867 INFO:teuthology.orchestra.run.smithi039.stdout:Building dependency tree... 2024-09-21T22:03:23.868 INFO:teuthology.orchestra.run.smithi039.stdout:Reading state information... 2024-09-21T22:03:24.022 INFO:teuthology.orchestra.run.smithi064.stdout:The following packages were automatically installed and are no longer required: 2024-09-21T22:03:24.022 INFO:teuthology.orchestra.run.smithi064.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-21T22:03:24.023 INFO:teuthology.orchestra.run.smithi064.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-21T22:03:24.023 INFO:teuthology.orchestra.run.smithi064.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-21T22:03:24.023 INFO:teuthology.orchestra.run.smithi064.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-21T22:03:24.027 INFO:teuthology.orchestra.run.smithi064.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-21T22:03:24.027 INFO:teuthology.orchestra.run.smithi064.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-21T22:03:24.027 INFO:teuthology.orchestra.run.smithi064.stdout:The following additional packages will be installed: 2024-09-21T22:03:24.027 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-21T22:03:24.027 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-09-21T22:03:24.028 INFO:teuthology.orchestra.run.smithi064.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-09-21T22:03:24.028 INFO:teuthology.orchestra.run.smithi064.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-09-21T22:03:24.028 INFO:teuthology.orchestra.run.smithi064.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-09-21T22:03:24.028 INFO:teuthology.orchestra.run.smithi064.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-09-21T22:03:24.028 INFO:teuthology.orchestra.run.smithi064.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-09-21T22:03:24.028 INFO:teuthology.orchestra.run.smithi064.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-09-21T22:03:24.028 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-09-21T22:03:24.028 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-21T22:03:24.028 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-21T22:03:24.028 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-21T22:03:24.028 INFO:teuthology.orchestra.run.smithi064.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-21T22:03:24.028 INFO:teuthology.orchestra.run.smithi064.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-21T22:03:24.028 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-21T22:03:24.028 INFO:teuthology.orchestra.run.smithi064.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-21T22:03:24.028 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-09-21T22:03:24.029 INFO:teuthology.orchestra.run.smithi064.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-09-21T22:03:24.029 INFO:teuthology.orchestra.run.smithi064.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-21T22:03:24.029 INFO:teuthology.orchestra.run.smithi064.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-09-21T22:03:24.029 INFO:teuthology.orchestra.run.smithi064.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-09-21T22:03:24.029 INFO:teuthology.orchestra.run.smithi064.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-09-21T22:03:24.029 INFO:teuthology.orchestra.run.smithi064.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-09-21T22:03:24.029 INFO:teuthology.orchestra.run.smithi064.stdout: xmlstarlet zip 2024-09-21T22:03:24.029 INFO:teuthology.orchestra.run.smithi064.stdout:Suggested packages: 2024-09-21T22:03:24.029 INFO:teuthology.orchestra.run.smithi064.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-09-21T22:03:24.029 INFO:teuthology.orchestra.run.smithi064.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-09-21T22:03:24.029 INFO:teuthology.orchestra.run.smithi064.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-09-21T22:03:24.029 INFO:teuthology.orchestra.run.smithi064.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-09-21T22:03:24.029 INFO:teuthology.orchestra.run.smithi064.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-09-21T22:03:24.029 INFO:teuthology.orchestra.run.smithi064.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-21T22:03:24.029 INFO:teuthology.orchestra.run.smithi064.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-09-21T22:03:24.029 INFO:teuthology.orchestra.run.smithi064.stdout:Recommended packages: 2024-09-21T22:03:24.029 INFO:teuthology.orchestra.run.smithi064.stdout: btrfs-tools 2024-09-21T22:03:24.030 INFO:teuthology.orchestra.run.smithi157.stdout:The following packages were automatically installed and are no longer required: 2024-09-21T22:03:24.030 INFO:teuthology.orchestra.run.smithi157.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-21T22:03:24.030 INFO:teuthology.orchestra.run.smithi157.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-21T22:03:24.030 INFO:teuthology.orchestra.run.smithi157.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-21T22:03:24.030 INFO:teuthology.orchestra.run.smithi157.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-21T22:03:24.030 INFO:teuthology.orchestra.run.smithi157.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-21T22:03:24.030 INFO:teuthology.orchestra.run.smithi157.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-21T22:03:24.031 INFO:teuthology.orchestra.run.smithi157.stdout:The following additional packages will be installed: 2024-09-21T22:03:24.031 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-21T22:03:24.031 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-09-21T22:03:24.031 INFO:teuthology.orchestra.run.smithi157.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-09-21T22:03:24.031 INFO:teuthology.orchestra.run.smithi157.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-09-21T22:03:24.031 INFO:teuthology.orchestra.run.smithi157.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-09-21T22:03:24.031 INFO:teuthology.orchestra.run.smithi157.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-09-21T22:03:24.031 INFO:teuthology.orchestra.run.smithi157.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-09-21T22:03:24.031 INFO:teuthology.orchestra.run.smithi157.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-09-21T22:03:24.031 INFO:teuthology.orchestra.run.smithi157.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-09-21T22:03:24.031 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-21T22:03:24.031 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-21T22:03:24.031 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-21T22:03:24.031 INFO:teuthology.orchestra.run.smithi157.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-21T22:03:24.031 INFO:teuthology.orchestra.run.smithi157.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-21T22:03:24.032 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-21T22:03:24.032 INFO:teuthology.orchestra.run.smithi157.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-21T22:03:24.032 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-09-21T22:03:24.032 INFO:teuthology.orchestra.run.smithi157.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-09-21T22:03:24.032 INFO:teuthology.orchestra.run.smithi157.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-21T22:03:24.032 INFO:teuthology.orchestra.run.smithi157.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-09-21T22:03:24.032 INFO:teuthology.orchestra.run.smithi157.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-09-21T22:03:24.032 INFO:teuthology.orchestra.run.smithi157.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-09-21T22:03:24.032 INFO:teuthology.orchestra.run.smithi157.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-09-21T22:03:24.032 INFO:teuthology.orchestra.run.smithi157.stdout: xmlstarlet zip 2024-09-21T22:03:24.032 INFO:teuthology.orchestra.run.smithi157.stdout:Suggested packages: 2024-09-21T22:03:24.032 INFO:teuthology.orchestra.run.smithi157.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-09-21T22:03:24.032 INFO:teuthology.orchestra.run.smithi157.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-09-21T22:03:24.032 INFO:teuthology.orchestra.run.smithi157.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-09-21T22:03:24.032 INFO:teuthology.orchestra.run.smithi157.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-09-21T22:03:24.032 INFO:teuthology.orchestra.run.smithi157.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-09-21T22:03:24.033 INFO:teuthology.orchestra.run.smithi157.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-21T22:03:24.033 INFO:teuthology.orchestra.run.smithi157.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-09-21T22:03:24.033 INFO:teuthology.orchestra.run.smithi157.stdout:Recommended packages: 2024-09-21T22:03:24.033 INFO:teuthology.orchestra.run.smithi157.stdout: btrfs-tools 2024-09-21T22:03:24.067 INFO:teuthology.orchestra.run.smithi039.stdout:The following packages were automatically installed and are no longer required: 2024-09-21T22:03:24.067 INFO:teuthology.orchestra.run.smithi039.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-21T22:03:24.068 INFO:teuthology.orchestra.run.smithi039.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-21T22:03:24.068 INFO:teuthology.orchestra.run.smithi039.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-21T22:03:24.068 INFO:teuthology.orchestra.run.smithi039.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-21T22:03:24.069 INFO:teuthology.orchestra.run.smithi039.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-21T22:03:24.069 INFO:teuthology.orchestra.run.smithi039.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-21T22:03:24.070 INFO:teuthology.orchestra.run.smithi039.stdout:The following additional packages will be installed: 2024-09-21T22:03:24.070 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-21T22:03:24.070 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-09-21T22:03:24.070 INFO:teuthology.orchestra.run.smithi039.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-09-21T22:03:24.070 INFO:teuthology.orchestra.run.smithi039.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-09-21T22:03:24.071 INFO:teuthology.orchestra.run.smithi039.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-09-21T22:03:24.071 INFO:teuthology.orchestra.run.smithi039.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-09-21T22:03:24.072 INFO:teuthology.orchestra.run.smithi039.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-09-21T22:03:24.072 INFO:teuthology.orchestra.run.smithi039.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-09-21T22:03:24.072 INFO:teuthology.orchestra.run.smithi039.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-09-21T22:03:24.072 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-21T22:03:24.072 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-21T22:03:24.072 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-21T22:03:24.072 INFO:teuthology.orchestra.run.smithi039.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-21T22:03:24.072 INFO:teuthology.orchestra.run.smithi039.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-21T22:03:24.072 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-21T22:03:24.072 INFO:teuthology.orchestra.run.smithi039.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-21T22:03:24.072 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-09-21T22:03:24.072 INFO:teuthology.orchestra.run.smithi039.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-09-21T22:03:24.072 INFO:teuthology.orchestra.run.smithi039.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-21T22:03:24.072 INFO:teuthology.orchestra.run.smithi039.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-09-21T22:03:24.073 INFO:teuthology.orchestra.run.smithi039.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-09-21T22:03:24.073 INFO:teuthology.orchestra.run.smithi039.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-09-21T22:03:24.073 INFO:teuthology.orchestra.run.smithi039.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-09-21T22:03:24.073 INFO:teuthology.orchestra.run.smithi039.stdout: xmlstarlet zip 2024-09-21T22:03:24.073 INFO:teuthology.orchestra.run.smithi039.stdout:Suggested packages: 2024-09-21T22:03:24.073 INFO:teuthology.orchestra.run.smithi039.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-09-21T22:03:24.073 INFO:teuthology.orchestra.run.smithi039.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-09-21T22:03:24.073 INFO:teuthology.orchestra.run.smithi039.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-09-21T22:03:24.073 INFO:teuthology.orchestra.run.smithi039.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-09-21T22:03:24.073 INFO:teuthology.orchestra.run.smithi039.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-09-21T22:03:24.073 INFO:teuthology.orchestra.run.smithi039.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-21T22:03:24.073 INFO:teuthology.orchestra.run.smithi039.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-09-21T22:03:24.074 INFO:teuthology.orchestra.run.smithi039.stdout:Recommended packages: 2024-09-21T22:03:24.074 INFO:teuthology.orchestra.run.smithi039.stdout: btrfs-tools 2024-09-21T22:03:24.134 INFO:teuthology.orchestra.run.smithi157.stdout:The following NEW packages will be installed: 2024-09-21T22:03:24.134 INFO:teuthology.orchestra.run.smithi157.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-21T22:03:24.134 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-21T22:03:24.134 INFO:teuthology.orchestra.run.smithi157.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-09-21T22:03:24.134 INFO:teuthology.orchestra.run.smithi157.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-09-21T22:03:24.135 INFO:teuthology.orchestra.run.smithi157.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-09-21T22:03:24.135 INFO:teuthology.orchestra.run.smithi157.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-09-21T22:03:24.136 INFO:teuthology.orchestra.run.smithi157.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-09-21T22:03:24.136 INFO:teuthology.orchestra.run.smithi157.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-09-21T22:03:24.136 INFO:teuthology.orchestra.run.smithi157.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-09-21T22:03:24.136 INFO:teuthology.orchestra.run.smithi157.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-09-21T22:03:24.136 INFO:teuthology.orchestra.run.smithi157.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-09-21T22:03:24.136 INFO:teuthology.orchestra.run.smithi157.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-21T22:03:24.136 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-21T22:03:24.137 INFO:teuthology.orchestra.run.smithi157.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-21T22:03:24.137 INFO:teuthology.orchestra.run.smithi157.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-21T22:03:24.137 INFO:teuthology.orchestra.run.smithi157.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-21T22:03:24.137 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-21T22:03:24.137 INFO:teuthology.orchestra.run.smithi157.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-21T22:03:24.137 INFO:teuthology.orchestra.run.smithi157.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-09-21T22:03:24.137 INFO:teuthology.orchestra.run.smithi157.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-09-21T22:03:24.137 INFO:teuthology.orchestra.run.smithi157.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-09-21T22:03:24.137 INFO:teuthology.orchestra.run.smithi157.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-09-21T22:03:24.137 INFO:teuthology.orchestra.run.smithi157.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-09-21T22:03:24.138 INFO:teuthology.orchestra.run.smithi157.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-09-21T22:03:24.138 INFO:teuthology.orchestra.run.smithi157.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-09-21T22:03:24.138 INFO:teuthology.orchestra.run.smithi157.stdout: radosgw rbd-fuse rbd-nbd socat unzip xmlstarlet zip 2024-09-21T22:03:24.139 INFO:teuthology.orchestra.run.smithi064.stdout:The following NEW packages will be installed: 2024-09-21T22:03:24.139 INFO:teuthology.orchestra.run.smithi064.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-21T22:03:24.139 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-21T22:03:24.140 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-09-21T22:03:24.140 INFO:teuthology.orchestra.run.smithi064.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-09-21T22:03:24.140 INFO:teuthology.orchestra.run.smithi064.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-09-21T22:03:24.140 INFO:teuthology.orchestra.run.smithi064.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-09-21T22:03:24.140 INFO:teuthology.orchestra.run.smithi064.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-09-21T22:03:24.140 INFO:teuthology.orchestra.run.smithi064.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-09-21T22:03:24.140 INFO:teuthology.orchestra.run.smithi064.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-09-21T22:03:24.140 INFO:teuthology.orchestra.run.smithi064.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-09-21T22:03:24.140 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-09-21T22:03:24.140 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-21T22:03:24.140 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-21T22:03:24.140 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-21T22:03:24.140 INFO:teuthology.orchestra.run.smithi064.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-21T22:03:24.140 INFO:teuthology.orchestra.run.smithi064.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-21T22:03:24.140 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-21T22:03:24.141 INFO:teuthology.orchestra.run.smithi064.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-21T22:03:24.141 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-09-21T22:03:24.141 INFO:teuthology.orchestra.run.smithi064.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-09-21T22:03:24.141 INFO:teuthology.orchestra.run.smithi064.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-09-21T22:03:24.141 INFO:teuthology.orchestra.run.smithi064.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-09-21T22:03:24.141 INFO:teuthology.orchestra.run.smithi064.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-09-21T22:03:24.141 INFO:teuthology.orchestra.run.smithi064.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-09-21T22:03:24.141 INFO:teuthology.orchestra.run.smithi064.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-09-21T22:03:24.141 INFO:teuthology.orchestra.run.smithi064.stdout: radosgw rbd-fuse rbd-nbd socat unzip xmlstarlet zip 2024-09-21T22:03:24.180 INFO:teuthology.orchestra.run.smithi039.stdout:The following NEW packages will be installed: 2024-09-21T22:03:24.180 INFO:teuthology.orchestra.run.smithi039.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-21T22:03:24.180 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-21T22:03:24.180 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-09-21T22:03:24.180 INFO:teuthology.orchestra.run.smithi039.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-09-21T22:03:24.180 INFO:teuthology.orchestra.run.smithi039.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-09-21T22:03:24.181 INFO:teuthology.orchestra.run.smithi039.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-09-21T22:03:24.181 INFO:teuthology.orchestra.run.smithi039.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-09-21T22:03:24.182 INFO:teuthology.orchestra.run.smithi039.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-09-21T22:03:24.182 INFO:teuthology.orchestra.run.smithi039.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-09-21T22:03:24.182 INFO:teuthology.orchestra.run.smithi039.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-09-21T22:03:24.182 INFO:teuthology.orchestra.run.smithi039.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-09-21T22:03:24.182 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-21T22:03:24.182 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-21T22:03:24.182 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-21T22:03:24.182 INFO:teuthology.orchestra.run.smithi039.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-21T22:03:24.182 INFO:teuthology.orchestra.run.smithi039.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-21T22:03:24.182 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-21T22:03:24.182 INFO:teuthology.orchestra.run.smithi039.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-21T22:03:24.182 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-09-21T22:03:24.182 INFO:teuthology.orchestra.run.smithi039.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-09-21T22:03:24.183 INFO:teuthology.orchestra.run.smithi039.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-09-21T22:03:24.183 INFO:teuthology.orchestra.run.smithi039.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-09-21T22:03:24.183 INFO:teuthology.orchestra.run.smithi039.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-09-21T22:03:24.183 INFO:teuthology.orchestra.run.smithi039.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-09-21T22:03:24.183 INFO:teuthology.orchestra.run.smithi039.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-09-21T22:03:24.183 INFO:teuthology.orchestra.run.smithi039.stdout: radosgw rbd-fuse rbd-nbd socat unzip xmlstarlet zip 2024-09-21T22:03:24.194 INFO:teuthology.orchestra.run.smithi157.stdout:0 upgraded, 119 newly installed, 0 to remove and 332 not upgraded. 2024-09-21T22:03:24.194 INFO:teuthology.orchestra.run.smithi157.stdout:Need to get 181 MB of archives. 2024-09-21T22:03:24.194 INFO:teuthology.orchestra.run.smithi157.stdout:After this operation, 830 MB of additional disk space will be used. 2024-09-21T22:03:24.194 INFO:teuthology.orchestra.run.smithi157.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-09-21T22:03:24.278 INFO:teuthology.orchestra.run.smithi157.stdout:Get:2 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.1.1-373-g4e45e6b4-1jammy [3,592 kB] 2024-09-21T22:03:24.281 INFO:teuthology.orchestra.run.smithi157.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-09-21T22:03:24.282 INFO:teuthology.orchestra.run.smithi064.stdout:0 upgraded, 119 newly installed, 0 to remove and 332 not upgraded. 2024-09-21T22:03:24.282 INFO:teuthology.orchestra.run.smithi064.stdout:Need to get 181 MB of archives. 2024-09-21T22:03:24.282 INFO:teuthology.orchestra.run.smithi064.stdout:After this operation, 830 MB of additional disk space will be used. 2024-09-21T22:03:24.282 INFO:teuthology.orchestra.run.smithi064.stdout:Get:1 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.1.1-373-g4e45e6b4-1jammy [3,592 kB] 2024-09-21T22:03:24.284 INFO:teuthology.orchestra.run.smithi157.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-09-21T22:03:24.301 INFO:teuthology.orchestra.run.smithi157.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-09-21T22:03:24.322 INFO:teuthology.orchestra.run.smithi039.stdout:0 upgraded, 119 newly installed, 0 to remove and 332 not upgraded. 2024-09-21T22:03:24.322 INFO:teuthology.orchestra.run.smithi039.stdout:Need to get 181 MB of archives. 2024-09-21T22:03:24.322 INFO:teuthology.orchestra.run.smithi039.stdout:After this operation, 830 MB of additional disk space will be used. 2024-09-21T22:03:24.322 INFO:teuthology.orchestra.run.smithi039.stdout:Get:1 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.1.1-373-g4e45e6b4-1jammy [3,592 kB] 2024-09-21T22:03:24.327 INFO:teuthology.orchestra.run.smithi064.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-09-21T22:03:24.363 INFO:teuthology.orchestra.run.smithi157.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-09-21T22:03:24.368 INFO:teuthology.orchestra.run.smithi157.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-09-21T22:03:24.369 INFO:teuthology.orchestra.run.smithi039.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-09-21T22:03:24.380 INFO:teuthology.orchestra.run.smithi157.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-09-21T22:03:24.383 INFO:teuthology.orchestra.run.smithi157.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-09-21T22:03:24.384 INFO:teuthology.orchestra.run.smithi157.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-09-21T22:03:24.385 INFO:teuthology.orchestra.run.smithi157.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-09-21T22:03:24.393 INFO:teuthology.orchestra.run.smithi157.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-09-21T22:03:24.395 INFO:teuthology.orchestra.run.smithi157.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-09-21T22:03:24.397 INFO:teuthology.orchestra.run.smithi157.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-09-21T22:03:24.398 INFO:teuthology.orchestra.run.smithi157.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-09-21T22:03:24.401 INFO:teuthology.orchestra.run.smithi157.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-09-21T22:03:24.403 INFO:teuthology.orchestra.run.smithi157.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-09-21T22:03:24.416 INFO:teuthology.orchestra.run.smithi157.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-09-21T22:03:24.418 INFO:teuthology.orchestra.run.smithi157.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-09-21T22:03:24.418 INFO:teuthology.orchestra.run.smithi157.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-09-21T22:03:24.419 INFO:teuthology.orchestra.run.smithi157.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-09-21T22:03:24.420 INFO:teuthology.orchestra.run.smithi157.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-09-21T22:03:24.420 INFO:teuthology.orchestra.run.smithi157.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-09-21T22:03:24.421 INFO:teuthology.orchestra.run.smithi157.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-09-21T22:03:24.421 INFO:teuthology.orchestra.run.smithi157.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-09-21T22:03:24.435 INFO:teuthology.orchestra.run.smithi157.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-09-21T22:03:24.435 INFO:teuthology.orchestra.run.smithi157.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-09-21T22:03:24.452 INFO:teuthology.orchestra.run.smithi157.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-09-21T22:03:24.455 INFO:teuthology.orchestra.run.smithi157.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-09-21T22:03:24.455 INFO:teuthology.orchestra.run.smithi157.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-09-21T22:03:24.455 INFO:teuthology.orchestra.run.smithi157.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-09-21T22:03:24.456 INFO:teuthology.orchestra.run.smithi157.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-09-21T22:03:24.457 INFO:teuthology.orchestra.run.smithi157.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-21T22:03:24.457 INFO:teuthology.orchestra.run.smithi157.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-09-21T22:03:24.458 INFO:teuthology.orchestra.run.smithi157.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2024-09-21T22:03:24.469 INFO:teuthology.orchestra.run.smithi157.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-09-21T22:03:24.470 INFO:teuthology.orchestra.run.smithi157.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-09-21T22:03:24.483 INFO:teuthology.orchestra.run.smithi157.stdout:Get:38 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.1.1-373-g4e45e6b4-1jammy [3,241 kB] 2024-09-21T22:03:24.486 INFO:teuthology.orchestra.run.smithi157.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-09-21T22:03:24.491 INFO:teuthology.orchestra.run.smithi157.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-09-21T22:03:24.491 INFO:teuthology.orchestra.run.smithi157.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-09-21T22:03:24.492 INFO:teuthology.orchestra.run.smithi157.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-09-21T22:03:24.494 INFO:teuthology.orchestra.run.smithi157.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-09-21T22:03:24.495 INFO:teuthology.orchestra.run.smithi157.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-09-21T22:03:24.497 INFO:teuthology.orchestra.run.smithi157.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-idna all 3.3-1ubuntu0.1 [52.1 kB] 2024-09-21T22:03:24.498 INFO:teuthology.orchestra.run.smithi157.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-09-21T22:03:24.503 INFO:teuthology.orchestra.run.smithi157.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-09-21T22:03:24.504 INFO:teuthology.orchestra.run.smithi157.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.2 [181 kB] 2024-09-21T22:03:24.520 INFO:teuthology.orchestra.run.smithi157.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-09-21T22:03:24.520 INFO:teuthology.orchestra.run.smithi157.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-09-21T22:03:24.521 INFO:teuthology.orchestra.run.smithi157.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-09-21T22:03:24.525 INFO:teuthology.orchestra.run.smithi157.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-09-21T22:03:24.534 INFO:teuthology.orchestra.run.smithi064.stdout:Get:3 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.1.1-373-g4e45e6b4-1jammy [3,241 kB] 2024-09-21T22:03:24.567 INFO:teuthology.orchestra.run.smithi157.stdout:Get:53 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.1.1-373-g4e45e6b4-1jammy [719 kB] 2024-09-21T22:03:24.592 INFO:teuthology.orchestra.run.smithi157.stdout:Get:54 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.1.1-373-g4e45e6b4-1jammy [356 kB] 2024-09-21T22:03:24.593 INFO:teuthology.orchestra.run.smithi157.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-09-21T22:03:24.593 INFO:teuthology.orchestra.run.smithi157.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.2 [108 kB] 2024-09-21T22:03:24.595 INFO:teuthology.orchestra.run.smithi157.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-21T22:03:24.595 INFO:teuthology.orchestra.run.smithi157.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-09-21T22:03:24.596 INFO:teuthology.orchestra.run.smithi157.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-09-21T22:03:24.599 INFO:teuthology.orchestra.run.smithi157.stdout:Get:60 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.1.1-373-g4e45e6b4-1jammy [32.8 kB] 2024-09-21T22:03:24.600 INFO:teuthology.orchestra.run.smithi157.stdout:Get:61 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.1.1-373-g4e45e6b4-1jammy [180 kB] 2024-09-21T22:03:24.603 INFO:teuthology.orchestra.run.smithi157.stdout:Get:62 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.1.1-373-g4e45e6b4-1jammy [67.8 kB] 2024-09-21T22:03:24.605 INFO:teuthology.orchestra.run.smithi157.stdout:Get:63 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.1.1-373-g4e45e6b4-1jammy [333 kB] 2024-09-21T22:03:24.611 INFO:teuthology.orchestra.run.smithi157.stdout:Get:64 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.1.1-373-g4e45e6b4-1jammy [6,875 kB] 2024-09-21T22:03:24.614 INFO:teuthology.orchestra.run.smithi157.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-09-21T22:03:24.616 INFO:teuthology.orchestra.run.smithi157.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-09-21T22:03:24.616 INFO:teuthology.orchestra.run.smithi157.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-09-21T22:03:24.631 INFO:teuthology.orchestra.run.smithi157.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-09-21T22:03:24.632 INFO:teuthology.orchestra.run.smithi157.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-09-21T22:03:24.632 INFO:teuthology.orchestra.run.smithi157.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-09-21T22:03:24.633 INFO:teuthology.orchestra.run.smithi157.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-09-21T22:03:24.633 INFO:teuthology.orchestra.run.smithi157.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-09-21T22:03:24.633 INFO:teuthology.orchestra.run.smithi157.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-09-21T22:03:24.645 INFO:teuthology.orchestra.run.smithi157.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-09-21T22:03:24.645 INFO:teuthology.orchestra.run.smithi157.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-09-21T22:03:24.648 INFO:teuthology.orchestra.run.smithi157.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-09-21T22:03:24.662 INFO:teuthology.orchestra.run.smithi157.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-09-21T22:03:24.663 INFO:teuthology.orchestra.run.smithi157.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-09-21T22:03:24.664 INFO:teuthology.orchestra.run.smithi157.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-09-21T22:03:24.667 INFO:teuthology.orchestra.run.smithi157.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-09-21T22:03:24.669 INFO:teuthology.orchestra.run.smithi157.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-09-21T22:03:24.670 INFO:teuthology.orchestra.run.smithi157.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-09-21T22:03:24.670 INFO:teuthology.orchestra.run.smithi157.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.3 [474 kB] 2024-09-21T22:03:24.678 INFO:teuthology.orchestra.run.smithi157.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-09-21T22:03:24.679 INFO:teuthology.orchestra.run.smithi157.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-09-21T22:03:24.682 INFO:teuthology.orchestra.run.smithi157.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-09-21T22:03:24.695 INFO:teuthology.orchestra.run.smithi064.stdout:Get:4 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.1.1-373-g4e45e6b4-1jammy [719 kB] 2024-09-21T22:03:24.696 INFO:teuthology.orchestra.run.smithi157.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-21T22:03:24.696 INFO:teuthology.orchestra.run.smithi157.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-09-21T22:03:24.697 INFO:teuthology.orchestra.run.smithi157.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-09-21T22:03:24.698 INFO:teuthology.orchestra.run.smithi157.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-09-21T22:03:24.699 INFO:teuthology.orchestra.run.smithi157.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-09-21T22:03:24.705 INFO:teuthology.orchestra.run.smithi157.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-09-21T22:03:24.705 INFO:teuthology.orchestra.run.smithi157.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-09-21T22:03:24.713 INFO:teuthology.orchestra.run.smithi157.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-09-21T22:03:24.715 INFO:teuthology.orchestra.run.smithi157.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-09-21T22:03:24.716 INFO:teuthology.orchestra.run.smithi157.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-09-21T22:03:24.750 INFO:teuthology.orchestra.run.smithi064.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-09-21T22:03:24.754 INFO:teuthology.orchestra.run.smithi064.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.1.1-373-g4e45e6b4-1jammy [356 kB] 2024-09-21T22:03:24.762 INFO:teuthology.orchestra.run.smithi064.stdout:Get:7 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.1.1-373-g4e45e6b4-1jammy [32.8 kB] 2024-09-21T22:03:24.762 INFO:teuthology.orchestra.run.smithi064.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-09-21T22:03:24.763 INFO:teuthology.orchestra.run.smithi064.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.1.1-373-g4e45e6b4-1jammy [180 kB] 2024-09-21T22:03:24.767 INFO:teuthology.orchestra.run.smithi064.stdout:Get:10 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.1.1-373-g4e45e6b4-1jammy [67.8 kB] 2024-09-21T22:03:24.769 INFO:teuthology.orchestra.run.smithi064.stdout:Get:11 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.1.1-373-g4e45e6b4-1jammy [333 kB] 2024-09-21T22:03:24.791 INFO:teuthology.orchestra.run.smithi157.stdout:Get:97 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.1.1-373-g4e45e6b4-1jammy [112 kB] 2024-09-21T22:03:24.792 INFO:teuthology.orchestra.run.smithi039.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-09-21T22:03:24.793 INFO:teuthology.orchestra.run.smithi039.stdout:Get:4 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.1.1-373-g4e45e6b4-1jammy [3,241 kB] 2024-09-21T22:03:24.793 INFO:teuthology.orchestra.run.smithi157.stdout:Get:98 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.1.1-373-g4e45e6b4-1jammy [472 kB] 2024-09-21T22:03:24.804 INFO:teuthology.orchestra.run.smithi039.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-09-21T22:03:24.807 INFO:teuthology.orchestra.run.smithi157.stdout:Get:99 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.1.1-373-g4e45e6b4-1jammy [25.9 MB] 2024-09-21T22:03:24.828 INFO:teuthology.orchestra.run.smithi064.stdout:Get:12 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.1.1-373-g4e45e6b4-1jammy [6,875 kB] 2024-09-21T22:03:24.843 INFO:teuthology.orchestra.run.smithi064.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-09-21T22:03:24.885 INFO:teuthology.orchestra.run.smithi039.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-09-21T22:03:25.073 INFO:teuthology.orchestra.run.smithi064.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-09-21T22:03:25.083 INFO:teuthology.orchestra.run.smithi064.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-09-21T22:03:25.112 INFO:teuthology.orchestra.run.smithi039.stdout:Get:7 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.1.1-373-g4e45e6b4-1jammy [719 kB] 2024-09-21T22:03:25.114 INFO:teuthology.orchestra.run.smithi039.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-09-21T22:03:25.114 INFO:teuthology.orchestra.run.smithi064.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-09-21T22:03:25.125 INFO:teuthology.orchestra.run.smithi039.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-09-21T22:03:25.125 INFO:teuthology.orchestra.run.smithi064.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-09-21T22:03:25.126 INFO:teuthology.orchestra.run.smithi064.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-09-21T22:03:25.127 INFO:teuthology.orchestra.run.smithi064.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-09-21T22:03:25.142 INFO:teuthology.orchestra.run.smithi064.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-09-21T22:03:25.146 INFO:teuthology.orchestra.run.smithi064.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-09-21T22:03:25.151 INFO:teuthology.orchestra.run.smithi064.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-09-21T22:03:25.154 INFO:teuthology.orchestra.run.smithi039.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-09-21T22:03:25.155 INFO:teuthology.orchestra.run.smithi064.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-09-21T22:03:25.163 INFO:teuthology.orchestra.run.smithi039.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-09-21T22:03:25.164 INFO:teuthology.orchestra.run.smithi039.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-09-21T22:03:25.165 INFO:teuthology.orchestra.run.smithi039.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-09-21T22:03:25.172 INFO:teuthology.orchestra.run.smithi064.stdout:Get:24 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.1.1-373-g4e45e6b4-1jammy [112 kB] 2024-09-21T22:03:25.174 INFO:teuthology.orchestra.run.smithi064.stdout:Get:25 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.1.1-373-g4e45e6b4-1jammy [472 kB] 2024-09-21T22:03:25.177 INFO:teuthology.orchestra.run.smithi039.stdout:Get:14 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.1.1-373-g4e45e6b4-1jammy [356 kB] 2024-09-21T22:03:25.183 INFO:teuthology.orchestra.run.smithi039.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-09-21T22:03:25.187 INFO:teuthology.orchestra.run.smithi039.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-09-21T22:03:25.192 INFO:teuthology.orchestra.run.smithi039.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-09-21T22:03:25.196 INFO:teuthology.orchestra.run.smithi039.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-09-21T22:03:25.226 INFO:teuthology.orchestra.run.smithi064.stdout:Get:26 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.1.1-373-g4e45e6b4-1jammy [25.9 MB] 2024-09-21T22:03:25.234 INFO:teuthology.orchestra.run.smithi039.stdout:Get:19 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.1.1-373-g4e45e6b4-1jammy [32.8 kB] 2024-09-21T22:03:25.235 INFO:teuthology.orchestra.run.smithi064.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-09-21T22:03:25.236 INFO:teuthology.orchestra.run.smithi039.stdout:Get:20 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.1.1-373-g4e45e6b4-1jammy [180 kB] 2024-09-21T22:03:25.240 INFO:teuthology.orchestra.run.smithi064.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-09-21T22:03:25.240 INFO:teuthology.orchestra.run.smithi039.stdout:Get:21 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.1.1-373-g4e45e6b4-1jammy [67.8 kB] 2024-09-21T22:03:25.241 INFO:teuthology.orchestra.run.smithi039.stdout:Get:22 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.1.1-373-g4e45e6b4-1jammy [333 kB] 2024-09-21T22:03:25.244 INFO:teuthology.orchestra.run.smithi064.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-09-21T22:03:25.247 INFO:teuthology.orchestra.run.smithi064.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-09-21T22:03:25.248 INFO:teuthology.orchestra.run.smithi064.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-09-21T22:03:25.249 INFO:teuthology.orchestra.run.smithi064.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-09-21T22:03:25.251 INFO:teuthology.orchestra.run.smithi064.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-09-21T22:03:25.251 INFO:teuthology.orchestra.run.smithi064.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-09-21T22:03:25.252 INFO:teuthology.orchestra.run.smithi064.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-09-21T22:03:25.276 INFO:teuthology.orchestra.run.smithi039.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-09-21T22:03:25.280 INFO:teuthology.orchestra.run.smithi039.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-09-21T22:03:25.285 INFO:teuthology.orchestra.run.smithi039.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-09-21T22:03:25.288 INFO:teuthology.orchestra.run.smithi039.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-09-21T22:03:25.289 INFO:teuthology.orchestra.run.smithi039.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-09-21T22:03:25.290 INFO:teuthology.orchestra.run.smithi039.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-09-21T22:03:25.292 INFO:teuthology.orchestra.run.smithi039.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-09-21T22:03:25.293 INFO:teuthology.orchestra.run.smithi039.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-09-21T22:03:25.294 INFO:teuthology.orchestra.run.smithi039.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-09-21T22:03:25.298 INFO:teuthology.orchestra.run.smithi039.stdout:Get:32 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.1.1-373-g4e45e6b4-1jammy [6,875 kB] 2024-09-21T22:03:25.318 INFO:teuthology.orchestra.run.smithi064.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-09-21T22:03:25.335 INFO:teuthology.orchestra.run.smithi064.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-09-21T22:03:25.335 INFO:teuthology.orchestra.run.smithi064.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-09-21T22:03:25.336 INFO:teuthology.orchestra.run.smithi064.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-09-21T22:03:25.343 INFO:teuthology.orchestra.run.smithi064.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-09-21T22:03:25.345 INFO:teuthology.orchestra.run.smithi064.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-09-21T22:03:25.358 INFO:teuthology.orchestra.run.smithi039.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-09-21T22:03:25.376 INFO:teuthology.orchestra.run.smithi039.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-09-21T22:03:25.377 INFO:teuthology.orchestra.run.smithi039.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-09-21T22:03:25.377 INFO:teuthology.orchestra.run.smithi039.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-09-21T22:03:25.383 INFO:teuthology.orchestra.run.smithi039.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-09-21T22:03:25.384 INFO:teuthology.orchestra.run.smithi039.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-09-21T22:03:25.399 INFO:teuthology.orchestra.run.smithi064.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-09-21T22:03:25.400 INFO:teuthology.orchestra.run.smithi064.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-09-21T22:03:25.403 INFO:teuthology.orchestra.run.smithi064.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-21T22:03:25.404 INFO:teuthology.orchestra.run.smithi064.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-09-21T22:03:25.440 INFO:teuthology.orchestra.run.smithi039.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-09-21T22:03:25.440 INFO:teuthology.orchestra.run.smithi039.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-09-21T22:03:25.442 INFO:teuthology.orchestra.run.smithi039.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-21T22:03:25.443 INFO:teuthology.orchestra.run.smithi039.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-09-21T22:03:25.480 INFO:teuthology.orchestra.run.smithi064.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2024-09-21T22:03:25.484 INFO:teuthology.orchestra.run.smithi064.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-09-21T22:03:25.485 INFO:teuthology.orchestra.run.smithi064.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-09-21T22:03:25.486 INFO:teuthology.orchestra.run.smithi064.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-09-21T22:03:25.497 INFO:teuthology.orchestra.run.smithi064.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-09-21T22:03:25.520 INFO:teuthology.orchestra.run.smithi039.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2024-09-21T22:03:25.523 INFO:teuthology.orchestra.run.smithi039.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-09-21T22:03:25.524 INFO:teuthology.orchestra.run.smithi039.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-09-21T22:03:25.525 INFO:teuthology.orchestra.run.smithi039.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-09-21T22:03:25.536 INFO:teuthology.orchestra.run.smithi039.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-09-21T22:03:25.562 INFO:teuthology.orchestra.run.smithi064.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-09-21T22:03:25.563 INFO:teuthology.orchestra.run.smithi064.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-09-21T22:03:25.564 INFO:teuthology.orchestra.run.smithi064.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-09-21T22:03:25.566 INFO:teuthology.orchestra.run.smithi064.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-09-21T22:03:25.570 INFO:teuthology.orchestra.run.smithi064.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-idna all 3.3-1ubuntu0.1 [52.1 kB] 2024-09-21T22:03:25.601 INFO:teuthology.orchestra.run.smithi039.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-09-21T22:03:25.603 INFO:teuthology.orchestra.run.smithi039.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-09-21T22:03:25.603 INFO:teuthology.orchestra.run.smithi039.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-09-21T22:03:25.606 INFO:teuthology.orchestra.run.smithi039.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-09-21T22:03:25.610 INFO:teuthology.orchestra.run.smithi039.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-idna all 3.3-1ubuntu0.1 [52.1 kB] 2024-09-21T22:03:25.636 INFO:teuthology.orchestra.run.smithi157.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.1.1-373-g4e45e6b4-1jammy [5,145 kB] 2024-09-21T22:03:25.643 INFO:teuthology.orchestra.run.smithi064.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-09-21T22:03:25.646 INFO:teuthology.orchestra.run.smithi064.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-09-21T22:03:25.647 INFO:teuthology.orchestra.run.smithi064.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.2 [181 kB] 2024-09-21T22:03:25.653 INFO:teuthology.orchestra.run.smithi064.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-09-21T22:03:25.655 INFO:teuthology.orchestra.run.smithi064.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-09-21T22:03:25.683 INFO:teuthology.orchestra.run.smithi039.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-09-21T22:03:25.685 INFO:teuthology.orchestra.run.smithi039.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-09-21T22:03:25.687 INFO:teuthology.orchestra.run.smithi039.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.2 [181 kB] 2024-09-21T22:03:25.692 INFO:teuthology.orchestra.run.smithi039.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-09-21T22:03:25.693 INFO:teuthology.orchestra.run.smithi039.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-09-21T22:03:25.724 INFO:teuthology.orchestra.run.smithi064.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-09-21T22:03:25.732 INFO:teuthology.orchestra.run.smithi064.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-09-21T22:03:25.735 INFO:teuthology.orchestra.run.smithi157.stdout:Get:101 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.1.1-373-g4e45e6b4-1jammy [245 kB] 2024-09-21T22:03:25.739 INFO:teuthology.orchestra.run.smithi157.stdout:Get:102 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.1.1-373-g4e45e6b4-1jammy [124 kB] 2024-09-21T22:03:25.742 INFO:teuthology.orchestra.run.smithi157.stdout:Get:103 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.1.1-373-g4e45e6b4-1jammy [1,555 kB] 2024-09-21T22:03:25.764 INFO:teuthology.orchestra.run.smithi039.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-09-21T22:03:25.768 INFO:teuthology.orchestra.run.smithi039.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-09-21T22:03:25.775 INFO:teuthology.orchestra.run.smithi157.stdout:Get:104 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.1.1-373-g4e45e6b4-1jammy [6,142 kB] 2024-09-21T22:03:25.903 INFO:teuthology.orchestra.run.smithi064.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-09-21T22:03:25.905 INFO:teuthology.orchestra.run.smithi064.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.2 [108 kB] 2024-09-21T22:03:25.909 INFO:teuthology.orchestra.run.smithi064.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-21T22:03:25.909 INFO:teuthology.orchestra.run.smithi064.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-09-21T22:03:25.912 INFO:teuthology.orchestra.run.smithi064.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-09-21T22:03:25.931 INFO:teuthology.orchestra.run.smithi039.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-09-21T22:03:25.933 INFO:teuthology.orchestra.run.smithi039.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.2 [108 kB] 2024-09-21T22:03:25.936 INFO:teuthology.orchestra.run.smithi039.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-21T22:03:25.936 INFO:teuthology.orchestra.run.smithi039.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-09-21T22:03:25.938 INFO:teuthology.orchestra.run.smithi039.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-09-21T22:03:25.949 INFO:teuthology.orchestra.run.smithi064.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-09-21T22:03:25.951 INFO:teuthology.orchestra.run.smithi157.stdout:Get:105 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.1.1-373-g4e45e6b4-1jammy [22.8 MB] 2024-09-21T22:03:25.976 INFO:teuthology.orchestra.run.smithi064.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-09-21T22:03:25.976 INFO:teuthology.orchestra.run.smithi039.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-09-21T22:03:25.976 INFO:teuthology.orchestra.run.smithi064.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-09-21T22:03:25.979 INFO:teuthology.orchestra.run.smithi039.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-09-21T22:03:25.979 INFO:teuthology.orchestra.run.smithi039.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-09-21T22:03:26.036 INFO:teuthology.orchestra.run.smithi039.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-09-21T22:03:26.037 INFO:teuthology.orchestra.run.smithi039.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-09-21T22:03:26.057 INFO:teuthology.orchestra.run.smithi064.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-09-21T22:03:26.057 INFO:teuthology.orchestra.run.smithi064.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-09-21T22:03:26.070 INFO:teuthology.orchestra.run.smithi064.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-09-21T22:03:26.071 INFO:teuthology.orchestra.run.smithi064.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-09-21T22:03:26.072 INFO:teuthology.orchestra.run.smithi064.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-09-21T22:03:26.073 INFO:teuthology.orchestra.run.smithi064.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-09-21T22:03:26.074 INFO:teuthology.orchestra.run.smithi064.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-09-21T22:03:26.075 INFO:teuthology.orchestra.run.smithi064.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-09-21T22:03:26.079 INFO:teuthology.orchestra.run.smithi039.stdout:Get:70 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.1.1-373-g4e45e6b4-1jammy [112 kB] 2024-09-21T22:03:26.083 INFO:teuthology.orchestra.run.smithi064.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-09-21T22:03:26.091 INFO:teuthology.orchestra.run.smithi039.stdout:Get:71 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.1.1-373-g4e45e6b4-1jammy [472 kB] 2024-09-21T22:03:26.097 INFO:teuthology.orchestra.run.smithi039.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-09-21T22:03:26.098 INFO:teuthology.orchestra.run.smithi039.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-09-21T22:03:26.099 INFO:teuthology.orchestra.run.smithi039.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-09-21T22:03:26.101 INFO:teuthology.orchestra.run.smithi039.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-09-21T22:03:26.102 INFO:teuthology.orchestra.run.smithi039.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-09-21T22:03:26.103 INFO:teuthology.orchestra.run.smithi039.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-09-21T22:03:26.111 INFO:teuthology.orchestra.run.smithi039.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-09-21T22:03:26.151 INFO:teuthology.orchestra.run.smithi064.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-09-21T22:03:26.153 INFO:teuthology.orchestra.run.smithi039.stdout:Get:79 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.1.1-373-g4e45e6b4-1jammy [25.9 MB] 2024-09-21T22:03:26.155 INFO:teuthology.orchestra.run.smithi064.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-09-21T22:03:26.156 INFO:teuthology.orchestra.run.smithi064.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-09-21T22:03:26.178 INFO:teuthology.orchestra.run.smithi039.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-09-21T22:03:26.182 INFO:teuthology.orchestra.run.smithi039.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-09-21T22:03:26.184 INFO:teuthology.orchestra.run.smithi039.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-09-21T22:03:26.233 INFO:teuthology.orchestra.run.smithi064.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-09-21T22:03:26.239 INFO:teuthology.orchestra.run.smithi064.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-09-21T22:03:26.241 INFO:teuthology.orchestra.run.smithi064.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-09-21T22:03:26.243 INFO:teuthology.orchestra.run.smithi064.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.3 [474 kB] 2024-09-21T22:03:26.255 INFO:teuthology.orchestra.run.smithi064.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-09-21T22:03:26.256 INFO:teuthology.orchestra.run.smithi064.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-09-21T22:03:26.259 INFO:teuthology.orchestra.run.smithi039.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-09-21T22:03:26.264 INFO:teuthology.orchestra.run.smithi064.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-09-21T22:03:26.266 INFO:teuthology.orchestra.run.smithi039.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-09-21T22:03:26.269 INFO:teuthology.orchestra.run.smithi039.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-09-21T22:03:26.270 INFO:teuthology.orchestra.run.smithi039.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.3 [474 kB] 2024-09-21T22:03:26.281 INFO:teuthology.orchestra.run.smithi039.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-09-21T22:03:26.282 INFO:teuthology.orchestra.run.smithi039.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-09-21T22:03:26.290 INFO:teuthology.orchestra.run.smithi039.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-09-21T22:03:26.314 INFO:teuthology.orchestra.run.smithi064.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-21T22:03:26.316 INFO:teuthology.orchestra.run.smithi064.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-09-21T22:03:26.317 INFO:teuthology.orchestra.run.smithi064.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-09-21T22:03:26.340 INFO:teuthology.orchestra.run.smithi039.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-21T22:03:26.342 INFO:teuthology.orchestra.run.smithi039.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-09-21T22:03:26.343 INFO:teuthology.orchestra.run.smithi039.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-09-21T22:03:26.396 INFO:teuthology.orchestra.run.smithi064.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-09-21T22:03:26.398 INFO:teuthology.orchestra.run.smithi064.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-09-21T22:03:26.415 INFO:teuthology.orchestra.run.smithi064.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-09-21T22:03:26.416 INFO:teuthology.orchestra.run.smithi064.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-09-21T22:03:26.416 INFO:teuthology.orchestra.run.smithi064.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-09-21T22:03:26.422 INFO:teuthology.orchestra.run.smithi064.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-09-21T22:03:26.422 INFO:teuthology.orchestra.run.smithi039.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-09-21T22:03:26.423 INFO:teuthology.orchestra.run.smithi064.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-09-21T22:03:26.424 INFO:teuthology.orchestra.run.smithi039.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-09-21T22:03:26.441 INFO:teuthology.orchestra.run.smithi039.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-09-21T22:03:26.442 INFO:teuthology.orchestra.run.smithi039.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-09-21T22:03:26.443 INFO:teuthology.orchestra.run.smithi039.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-09-21T22:03:26.449 INFO:teuthology.orchestra.run.smithi039.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-09-21T22:03:26.450 INFO:teuthology.orchestra.run.smithi039.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-09-21T22:03:26.680 INFO:teuthology.orchestra.run.smithi157.stdout:Get:106 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.1.1-373-g4e45e6b4-1jammy [14.1 kB] 2024-09-21T22:03:26.681 INFO:teuthology.orchestra.run.smithi157.stdout:Get:107 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.1.1-373-g4e45e6b4-1jammy [877 kB] 2024-09-21T22:03:26.699 INFO:teuthology.orchestra.run.smithi157.stdout:Get:108 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.1.1-373-g4e45e6b4-1jammy [2,462 kB] 2024-09-21T22:03:26.741 INFO:teuthology.orchestra.run.smithi157.stdout:Get:109 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.1.1-373-g4e45e6b4-1jammy [794 kB] 2024-09-21T22:03:26.750 INFO:teuthology.orchestra.run.smithi157.stdout:Get:110 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.1.1-373-g4e45e6b4-1jammy [152 kB] 2024-09-21T22:03:26.752 INFO:teuthology.orchestra.run.smithi157.stdout:Get:111 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.1.1-373-g4e45e6b4-1jammy [2,347 kB] 2024-09-21T22:03:26.788 INFO:teuthology.orchestra.run.smithi064.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.1.1-373-g4e45e6b4-1jammy [5,145 kB] 2024-09-21T22:03:26.805 INFO:teuthology.orchestra.run.smithi157.stdout:Get:112 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.1.1-373-g4e45e6b4-1jammy [8,625 kB] 2024-09-21T22:03:27.037 INFO:teuthology.orchestra.run.smithi157.stdout:Get:113 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.1.1-373-g4e45e6b4-1jammy [14.2 kB] 2024-09-21T22:03:27.037 INFO:teuthology.orchestra.run.smithi157.stdout:Get:114 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.1.1-373-g4e45e6b4-1jammy [51.4 MB] 2024-09-21T22:03:27.042 INFO:teuthology.orchestra.run.smithi064.stdout:Get:101 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.1.1-373-g4e45e6b4-1jammy [245 kB] 2024-09-21T22:03:27.063 INFO:teuthology.orchestra.run.smithi064.stdout:Get:102 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.1.1-373-g4e45e6b4-1jammy [124 kB] 2024-09-21T22:03:27.067 INFO:teuthology.orchestra.run.smithi064.stdout:Get:103 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.1.1-373-g4e45e6b4-1jammy [1,555 kB] 2024-09-21T22:03:27.177 INFO:teuthology.orchestra.run.smithi064.stdout:Get:104 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.1.1-373-g4e45e6b4-1jammy [6,142 kB] 2024-09-21T22:03:27.545 INFO:teuthology.orchestra.run.smithi064.stdout:Get:105 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.1.1-373-g4e45e6b4-1jammy [22.8 MB] 2024-09-21T22:03:28.714 INFO:teuthology.orchestra.run.smithi157.stdout:Get:115 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.1.1-373-g4e45e6b4-1jammy [130 kB] 2024-09-21T22:03:28.716 INFO:teuthology.orchestra.run.smithi157.stdout:Get:116 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.1.1-373-g4e45e6b4-1jammy [39.0 kB] 2024-09-21T22:03:28.718 INFO:teuthology.orchestra.run.smithi157.stdout:Get:117 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.1.1-373-g4e45e6b4-1jammy [13.6 MB] 2024-09-21T22:03:28.805 INFO:teuthology.orchestra.run.smithi064.stdout:Get:106 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.1.1-373-g4e45e6b4-1jammy [14.1 kB] 2024-09-21T22:03:28.806 INFO:teuthology.orchestra.run.smithi064.stdout:Get:107 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.1.1-373-g4e45e6b4-1jammy [877 kB] 2024-09-21T22:03:28.820 INFO:teuthology.orchestra.run.smithi064.stdout:Get:108 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.1.1-373-g4e45e6b4-1jammy [2,462 kB] 2024-09-21T22:03:28.870 INFO:teuthology.orchestra.run.smithi039.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.1.1-373-g4e45e6b4-1jammy [5,145 kB] 2024-09-21T22:03:28.887 INFO:teuthology.orchestra.run.smithi064.stdout:Get:109 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.1.1-373-g4e45e6b4-1jammy [794 kB] 2024-09-21T22:03:28.955 INFO:teuthology.orchestra.run.smithi064.stdout:Get:110 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.1.1-373-g4e45e6b4-1jammy [152 kB] 2024-09-21T22:03:28.959 INFO:teuthology.orchestra.run.smithi064.stdout:Get:111 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.1.1-373-g4e45e6b4-1jammy [2,347 kB] 2024-09-21T22:03:29.020 INFO:teuthology.orchestra.run.smithi157.stdout:Get:118 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.1.1-373-g4e45e6b4-1jammy [91.4 kB] 2024-09-21T22:03:29.022 INFO:teuthology.orchestra.run.smithi157.stdout:Get:119 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 rbd-nbd amd64 19.1.1-373-g4e45e6b4-1jammy [178 kB] 2024-09-21T22:03:29.063 INFO:teuthology.orchestra.run.smithi064.stdout:Get:112 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.1.1-373-g4e45e6b4-1jammy [8,625 kB] 2024-09-21T22:03:29.179 INFO:teuthology.orchestra.run.smithi039.stdout:Get:101 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.1.1-373-g4e45e6b4-1jammy [245 kB] 2024-09-21T22:03:29.184 INFO:teuthology.orchestra.run.smithi039.stdout:Get:102 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.1.1-373-g4e45e6b4-1jammy [124 kB] 2024-09-21T22:03:29.202 INFO:teuthology.orchestra.run.smithi039.stdout:Get:103 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.1.1-373-g4e45e6b4-1jammy [1,555 kB] 2024-09-21T22:03:29.272 INFO:teuthology.orchestra.run.smithi039.stdout:Get:104 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.1.1-373-g4e45e6b4-1jammy [6,142 kB] 2024-09-21T22:03:29.285 INFO:teuthology.orchestra.run.smithi064.stdout:Get:113 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.1.1-373-g4e45e6b4-1jammy [14.2 kB] 2024-09-21T22:03:29.285 INFO:teuthology.orchestra.run.smithi064.stdout:Get:114 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.1.1-373-g4e45e6b4-1jammy [51.4 MB] 2024-09-21T22:03:29.563 INFO:teuthology.orchestra.run.smithi039.stdout:Get:105 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.1.1-373-g4e45e6b4-1jammy [22.8 MB] 2024-09-21T22:03:29.586 INFO:teuthology.orchestra.run.smithi157.stdout:Fetched 181 MB in 5s (37.1 MB/s) 2024-09-21T22:03:29.746 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-09-21T22:03:29.845 INFO:teuthology.orchestra.run.smithi157.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165990 files and directories currently installed.) 2024-09-21T22:03:29.851 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-09-21T22:03:29.936 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-21T22:03:30.281 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-21T22:03:30.307 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-09-21T22:03:30.348 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-21T22:03:30.626 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-21T22:03:30.652 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-09-21T22:03:30.684 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-21T22:03:30.717 INFO:teuthology.orchestra.run.smithi039.stdout:Get:106 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.1.1-373-g4e45e6b4-1jammy [14.1 kB] 2024-09-21T22:03:30.717 INFO:teuthology.orchestra.run.smithi039.stdout:Get:107 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.1.1-373-g4e45e6b4-1jammy [877 kB] 2024-09-21T22:03:30.734 INFO:teuthology.orchestra.run.smithi039.stdout:Get:108 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.1.1-373-g4e45e6b4-1jammy [2,462 kB] 2024-09-21T22:03:30.790 INFO:teuthology.orchestra.run.smithi039.stdout:Get:109 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.1.1-373-g4e45e6b4-1jammy [794 kB] 2024-09-21T22:03:30.812 INFO:teuthology.orchestra.run.smithi039.stdout:Get:110 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.1.1-373-g4e45e6b4-1jammy [152 kB] 2024-09-21T22:03:30.822 INFO:teuthology.orchestra.run.smithi039.stdout:Get:111 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.1.1-373-g4e45e6b4-1jammy [2,347 kB] 2024-09-21T22:03:30.825 INFO:teuthology.orchestra.run.smithi064.stdout:Get:115 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.1.1-373-g4e45e6b4-1jammy [130 kB] 2024-09-21T22:03:30.828 INFO:teuthology.orchestra.run.smithi064.stdout:Get:116 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.1.1-373-g4e45e6b4-1jammy [39.0 kB] 2024-09-21T22:03:30.829 INFO:teuthology.orchestra.run.smithi064.stdout:Get:117 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.1.1-373-g4e45e6b4-1jammy [13.6 MB] 2024-09-21T22:03:30.893 INFO:teuthology.orchestra.run.smithi039.stdout:Get:112 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.1.1-373-g4e45e6b4-1jammy [8,625 kB] 2024-09-21T22:03:31.091 INFO:teuthology.orchestra.run.smithi064.stdout:Get:118 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.1.1-373-g4e45e6b4-1jammy [91.4 kB] 2024-09-21T22:03:31.093 INFO:teuthology.orchestra.run.smithi064.stdout:Get:119 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 rbd-nbd amd64 19.1.1-373-g4e45e6b4-1jammy [178 kB] 2024-09-21T22:03:31.121 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-21T22:03:31.147 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-21T22:03:31.196 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-21T22:03:31.223 INFO:teuthology.orchestra.run.smithi039.stdout:Get:113 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.1.1-373-g4e45e6b4-1jammy [14.2 kB] 2024-09-21T22:03:31.224 INFO:teuthology.orchestra.run.smithi039.stdout:Get:114 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.1.1-373-g4e45e6b4-1jammy [51.4 MB] 2024-09-21T22:03:31.583 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-21T22:03:31.608 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-21T22:03:31.649 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-21T22:03:31.692 INFO:teuthology.orchestra.run.smithi064.stdout:Fetched 181 MB in 7s (26.1 MB/s) 2024-09-21T22:03:31.850 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-09-21T22:03:31.952 INFO:teuthology.orchestra.run.smithi064.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165990 files and directories currently installed.) 2024-09-21T22:03:31.957 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-09-21T22:03:31.969 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-21T22:03:31.994 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-21T22:03:32.035 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-21T22:03:32.040 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-21T22:03:32.380 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-09-21T22:03:32.407 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-09-21T22:03:32.410 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-21T22:03:32.436 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-09-21T22:03:32.446 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-21T22:03:32.476 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-21T22:03:32.725 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package librados2. 2024-09-21T22:03:32.751 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../007-librados2_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T22:03:32.791 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking librados2 (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:03:32.796 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-21T22:03:32.823 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-09-21T22:03:32.863 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-21T22:03:32.980 INFO:teuthology.orchestra.run.smithi039.stdout:Get:115 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.1.1-373-g4e45e6b4-1jammy [130 kB] 2024-09-21T22:03:32.983 INFO:teuthology.orchestra.run.smithi039.stdout:Get:116 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.1.1-373-g4e45e6b4-1jammy [39.0 kB] 2024-09-21T22:03:32.984 INFO:teuthology.orchestra.run.smithi039.stdout:Get:117 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.1.1-373-g4e45e6b4-1jammy [13.6 MB] 2024-09-21T22:03:33.160 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package librbd1. 2024-09-21T22:03:33.176 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../008-librbd1_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T22:03:33.216 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-21T22:03:33.219 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking librbd1 (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:03:33.243 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-21T22:03:33.283 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-21T22:03:33.309 INFO:teuthology.orchestra.run.smithi039.stdout:Get:118 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.1.1-373-g4e45e6b4-1jammy [91.4 kB] 2024-09-21T22:03:33.311 INFO:teuthology.orchestra.run.smithi039.stdout:Get:119 https://3.chacra.ceph.com/r/ceph/squid/4e45e6b490e6cbe082cdf155fea780438ec53fc7/ubuntu/jammy/flavors/default jammy/main amd64 rbd-nbd amd64 19.1.1-373-g4e45e6b4-1jammy [178 kB] 2024-09-21T22:03:33.655 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package libcephfs2. 2024-09-21T22:03:33.681 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../009-libcephfs2_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T22:03:33.703 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-21T22:03:33.721 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking libcephfs2 (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:03:33.729 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-21T22:03:33.769 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-21T22:03:33.917 INFO:teuthology.orchestra.run.smithi039.stdout:Fetched 181 MB in 9s (19.9 MB/s) 2024-09-21T22:03:34.008 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-rados. 2024-09-21T22:03:34.034 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../010-python3-rados_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T22:03:34.074 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-rados (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:03:34.079 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-09-21T22:03:34.105 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-21T22:03:34.132 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-21T22:03:34.172 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-21T22:03:34.182 INFO:teuthology.orchestra.run.smithi039.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165990 files and directories currently installed.) 2024-09-21T22:03:34.188 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-09-21T22:03:34.277 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-21T22:03:34.352 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-21T22:03:34.367 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.1.1-373-g4e45e6b4-1jammy_all.deb ... 2024-09-21T22:03:34.402 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-ceph-argparse (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:03:34.614 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-21T22:03:34.621 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-cephfs. 2024-09-21T22:03:34.641 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-09-21T22:03:34.647 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../012-python3-cephfs_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T22:03:34.680 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-21T22:03:34.683 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-09-21T22:03:34.687 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-cephfs (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:03:34.710 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-09-21T22:03:34.741 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-21T22:03:34.975 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-21T22:03:35.002 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-09-21T22:03:35.033 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-21T22:03:35.199 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-21T22:03:35.203 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package librados2. 2024-09-21T22:03:35.226 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../013-python3-ceph-common_19.1.1-373-g4e45e6b4-1jammy_all.deb ... 2024-09-21T22:03:35.229 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../007-librados2_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T22:03:35.265 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-ceph-common (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:03:35.269 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking librados2 (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:03:35.593 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-21T22:03:35.619 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-09-21T22:03:35.660 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-21T22:03:35.687 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-21T22:03:35.714 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package librbd1. 2024-09-21T22:03:35.714 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-21T22:03:35.740 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../008-librbd1_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T22:03:35.762 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-21T22:03:35.780 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking librbd1 (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:03:35.954 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-prettytable. 2024-09-21T22:03:35.981 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-09-21T22:03:36.012 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-09-21T22:03:36.190 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-21T22:03:36.208 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package libcephfs2. 2024-09-21T22:03:36.217 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-21T22:03:36.224 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../009-libcephfs2_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T22:03:36.249 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-rbd. 2024-09-21T22:03:36.257 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-21T22:03:36.258 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking libcephfs2 (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:03:36.275 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../016-python3-rbd_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T22:03:36.315 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-rbd (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:03:36.485 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-rados. 2024-09-21T22:03:36.501 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../010-python3-rados_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T22:03:36.543 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-21T22:03:36.544 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-rados (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:03:36.570 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-21T22:03:36.601 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-21T22:03:36.660 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-21T22:03:36.687 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-09-21T22:03:36.726 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-21T22:03:36.855 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-21T22:03:36.881 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.1.1-373-g4e45e6b4-1jammy_all.deb ... 2024-09-21T22:03:36.921 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-ceph-argparse (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:03:36.937 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-09-21T22:03:36.964 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-09-21T22:03:36.979 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-09-21T22:03:36.994 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-09-21T22:03:37.004 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-21T22:03:37.029 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-09-21T22:03:37.156 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-cephfs. 2024-09-21T22:03:37.183 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../012-python3-cephfs_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T22:03:37.223 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-cephfs (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:03:37.324 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package librados2. 2024-09-21T22:03:37.325 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-09-21T22:03:37.351 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../007-librados2_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T22:03:37.351 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-09-21T22:03:37.392 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-21T22:03:37.407 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking librados2 (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:03:37.534 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-21T22:03:37.561 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../013-python3-ceph-common_19.1.1-373-g4e45e6b4-1jammy_all.deb ... 2024-09-21T22:03:37.592 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-ceph-common (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:03:37.695 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package lua5.1. 2024-09-21T22:03:37.721 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-09-21T22:03:37.762 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-09-21T22:03:37.826 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package librbd1. 2024-09-21T22:03:37.842 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../008-librbd1_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T22:03:37.876 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking librbd1 (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:03:37.879 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-21T22:03:37.906 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-09-21T22:03:37.937 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-21T22:03:38.065 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package lua-any. 2024-09-21T22:03:38.080 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-09-21T22:03:38.123 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking lua-any (27ubuntu1) ... 2024-09-21T22:03:38.240 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-prettytable. 2024-09-21T22:03:38.266 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-09-21T22:03:38.304 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package libcephfs2. 2024-09-21T22:03:38.306 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-09-21T22:03:38.331 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../009-libcephfs2_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T22:03:38.370 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking libcephfs2 (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:03:38.376 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package zip. 2024-09-21T22:03:38.402 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-09-21T22:03:38.442 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking zip (3.0-12build2) ... 2024-09-21T22:03:38.567 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-rbd. 2024-09-21T22:03:38.583 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../016-python3-rbd_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T22:03:38.625 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-rbd (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:03:38.631 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-rados. 2024-09-21T22:03:38.647 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../010-python3-rados_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T22:03:38.690 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-rados (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:03:38.754 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package unzip. 2024-09-21T22:03:38.780 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-09-21T22:03:38.821 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-09-21T22:03:38.987 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-21T22:03:39.014 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-09-21T22:03:39.017 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-21T22:03:39.044 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.1.1-373-g4e45e6b4-1jammy_all.deb ... 2024-09-21T22:03:39.053 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-21T22:03:39.085 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-ceph-argparse (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:03:39.191 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package luarocks. 2024-09-21T22:03:39.217 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-09-21T22:03:39.249 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-09-21T22:03:39.314 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-09-21T22:03:39.319 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-cephfs. 2024-09-21T22:03:39.336 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../012-python3-cephfs_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T22:03:39.341 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-09-21T22:03:39.369 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-cephfs (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:03:39.372 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-09-21T22:03:39.632 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package librgw2. 2024-09-21T22:03:39.642 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-09-21T22:03:39.647 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-21T22:03:39.658 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-09-21T22:03:39.658 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../025-librgw2_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T22:03:39.663 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../013-python3-ceph-common_19.1.1-373-g4e45e6b4-1jammy_all.deb ... 2024-09-21T22:03:39.698 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking librgw2 (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:03:39.700 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-21T22:03:39.705 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-ceph-common (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:03:40.025 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-21T22:03:40.053 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-09-21T22:03:40.087 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package lua5.1. 2024-09-21T22:03:40.092 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-21T22:03:40.114 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-09-21T22:03:40.154 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-09-21T22:03:40.251 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-rgw. 2024-09-21T22:03:40.277 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../026-python3-rgw_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T22:03:40.397 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-rgw (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:03:40.411 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-prettytable. 2024-09-21T22:03:40.438 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-09-21T22:03:40.478 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-09-21T22:03:40.573 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package lua-any. 2024-09-21T22:03:40.600 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-09-21T22:03:40.640 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking lua-any (27ubuntu1) ... 2024-09-21T22:03:40.762 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package liboath0:amd64. 2024-09-21T22:03:40.789 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-09-21T22:03:40.829 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-09-21T22:03:40.847 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-rbd. 2024-09-21T22:03:40.859 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package zip. 2024-09-21T22:03:40.875 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../016-python3-rbd_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T22:03:40.886 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-09-21T22:03:40.914 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-rbd (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:03:40.926 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking zip (3.0-12build2) ... 2024-09-21T22:03:41.140 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package libradosstriper1. 2024-09-21T22:03:41.166 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../028-libradosstriper1_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T22:03:41.206 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking libradosstriper1 (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:03:41.209 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-21T22:03:41.236 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-09-21T22:03:41.254 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package unzip. 2024-09-21T22:03:41.275 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-21T22:03:41.281 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-09-21T22:03:41.321 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-09-21T22:03:41.553 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-09-21T22:03:41.569 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-09-21T22:03:41.611 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-09-21T22:03:41.660 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package ceph-common. 2024-09-21T22:03:41.687 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../029-ceph-common_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T22:03:41.708 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package luarocks. 2024-09-21T22:03:41.726 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking ceph-common (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:03:41.735 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-09-21T22:03:41.774 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-09-21T22:03:41.914 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-09-21T22:03:41.930 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-09-21T22:03:41.972 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-21T22:03:42.132 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package librgw2. 2024-09-21T22:03:42.159 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../025-librgw2_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T22:03:42.190 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking librgw2 (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:03:42.300 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package lua5.1. 2024-09-21T22:03:42.327 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-09-21T22:03:42.367 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-09-21T22:03:42.645 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package lua-any. 2024-09-21T22:03:42.661 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-09-21T22:03:42.695 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking lua-any (27ubuntu1) ... 2024-09-21T22:03:42.734 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-rgw. 2024-09-21T22:03:42.750 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../026-python3-rgw_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T22:03:42.792 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-rgw (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:03:42.931 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package zip. 2024-09-21T22:03:42.947 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-09-21T22:03:42.989 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking zip (3.0-12build2) ... 2024-09-21T22:03:43.065 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package ceph-base. 2024-09-21T22:03:43.091 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../030-ceph-base_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T22:03:43.112 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package liboath0:amd64. 2024-09-21T22:03:43.139 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-09-21T22:03:43.141 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking ceph-base (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:03:43.179 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-09-21T22:03:43.309 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package unzip. 2024-09-21T22:03:43.336 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-09-21T22:03:43.375 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-09-21T22:03:43.498 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package libradosstriper1. 2024-09-21T22:03:43.525 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../028-libradosstriper1_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T22:03:43.565 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking libradosstriper1 (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:03:43.728 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-09-21T22:03:43.738 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package luarocks. 2024-09-21T22:03:43.754 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-09-21T22:03:43.764 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-09-21T22:03:43.794 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-09-21T22:03:43.804 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-09-21T22:03:44.018 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package ceph-common. 2024-09-21T22:03:44.045 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../029-ceph-common_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T22:03:44.084 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking ceph-common (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:03:44.097 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-cheroot. 2024-09-21T22:03:44.124 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-09-21T22:03:44.153 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package librgw2. 2024-09-21T22:03:44.163 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-21T22:03:44.169 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../025-librgw2_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T22:03:44.203 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking librgw2 (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:03:44.475 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-09-21T22:03:44.501 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-09-21T22:03:44.541 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-09-21T22:03:44.764 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-rgw. 2024-09-21T22:03:44.791 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../026-python3-rgw_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T22:03:44.819 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-jaraco.text. 2024-09-21T22:03:44.830 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-rgw (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:03:44.834 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-09-21T22:03:44.869 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-09-21T22:03:45.164 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-09-21T22:03:45.175 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package liboath0:amd64. 2024-09-21T22:03:45.190 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-09-21T22:03:45.202 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-09-21T22:03:45.230 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-09-21T22:03:45.242 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-09-21T22:03:45.465 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package ceph-base. 2024-09-21T22:03:45.492 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../030-ceph-base_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T22:03:45.511 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package libradosstriper1. 2024-09-21T22:03:45.533 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking ceph-base (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:03:45.533 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-tempora. 2024-09-21T22:03:45.538 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../028-libradosstriper1_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T22:03:45.560 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-09-21T22:03:45.569 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking libradosstriper1 (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:03:45.600 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-09-21T22:03:45.878 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-portend. 2024-09-21T22:03:45.904 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-09-21T22:03:45.914 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package ceph-common. 2024-09-21T22:03:45.941 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../029-ceph-common_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T22:03:45.944 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-09-21T22:03:45.972 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking ceph-common (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:03:46.119 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-09-21T22:03:46.146 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-09-21T22:03:46.185 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-09-21T22:03:46.222 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-09-21T22:03:46.249 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-09-21T22:03:46.289 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-09-21T22:03:46.488 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-cheroot. 2024-09-21T22:03:46.515 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-09-21T22:03:46.546 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-21T22:03:46.617 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-21T22:03:46.643 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-09-21T22:03:46.683 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-09-21T22:03:46.824 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-09-21T22:03:46.851 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-09-21T22:03:46.891 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-09-21T22:03:47.012 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-natsort. 2024-09-21T22:03:47.038 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-09-21T22:03:47.078 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-09-21T22:03:47.210 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-jaraco.text. 2024-09-21T22:03:47.237 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-09-21T22:03:47.277 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-09-21T22:03:47.378 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package ceph-base. 2024-09-21T22:03:47.381 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-logutils. 2024-09-21T22:03:47.405 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../030-ceph-base_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T22:03:47.408 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-09-21T22:03:47.448 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-09-21T22:03:47.454 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking ceph-base (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:03:47.563 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-09-21T22:03:47.590 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-09-21T22:03:47.630 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-09-21T22:03:47.759 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-21T22:03:47.786 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-09-21T22:03:47.826 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-09-21T22:03:47.916 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-tempora. 2024-09-21T22:03:47.943 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-09-21T22:03:47.974 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-09-21T22:03:48.024 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-09-21T22:03:48.051 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-09-21T22:03:48.090 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-09-21T22:03:48.095 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-mako. 2024-09-21T22:03:48.122 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-09-21T22:03:48.154 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-21T22:03:48.235 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-portend. 2024-09-21T22:03:48.262 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-09-21T22:03:48.302 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-09-21T22:03:48.418 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-cheroot. 2024-09-21T22:03:48.445 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-09-21T22:03:48.448 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-21T22:03:48.475 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-21T22:03:48.484 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-21T22:03:48.515 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-21T22:03:48.613 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-09-21T22:03:48.640 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-09-21T22:03:48.680 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-09-21T22:03:48.771 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-09-21T22:03:48.798 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-09-21T22:03:48.818 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-21T22:03:48.829 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-09-21T22:03:48.845 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-09-21T22:03:48.885 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-09-21T22:03:48.982 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-21T22:03:49.009 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-09-21T22:03:49.048 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-jaraco.text. 2024-09-21T22:03:49.049 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-09-21T22:03:49.076 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-09-21T22:03:49.115 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-09-21T22:03:49.187 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-webob. 2024-09-21T22:03:49.214 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2024-09-21T22:03:49.254 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-21T22:03:49.411 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-natsort. 2024-09-21T22:03:49.418 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-09-21T22:03:49.438 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-09-21T22:03:49.445 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-09-21T22:03:49.477 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-09-21T22:03:49.485 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-09-21T22:03:49.598 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-waitress. 2024-09-21T22:03:49.625 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-09-21T22:03:49.668 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-21T22:03:49.787 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-tempora. 2024-09-21T22:03:49.789 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-logutils. 2024-09-21T22:03:49.814 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-09-21T22:03:49.816 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-09-21T22:03:49.854 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-09-21T22:03:49.855 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-09-21T22:03:49.968 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-tempita. 2024-09-21T22:03:49.995 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-09-21T22:03:50.034 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-09-21T22:03:50.115 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-portend. 2024-09-21T22:03:50.131 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-09-21T22:03:50.133 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-21T22:03:50.160 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-09-21T22:03:50.165 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-09-21T22:03:50.191 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-09-21T22:03:50.321 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-paste. 2024-09-21T22:03:50.336 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-09-21T22:03:50.370 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-09-21T22:03:50.461 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-mako. 2024-09-21T22:03:50.470 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-09-21T22:03:50.488 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-09-21T22:03:50.497 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-09-21T22:03:50.527 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-21T22:03:50.536 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-09-21T22:03:50.691 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-21T22:03:50.718 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-09-21T22:03:50.757 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-09-21T22:03:50.815 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-21T22:03:50.842 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-09-21T22:03:50.856 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-21T22:03:50.881 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-09-21T22:03:50.883 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-21T22:03:50.922 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-21T22:03:51.060 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-21T22:03:51.087 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-09-21T22:03:51.126 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-09-21T22:03:51.194 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-natsort. 2024-09-21T22:03:51.221 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-09-21T22:03:51.225 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-21T22:03:51.252 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-09-21T22:03:51.260 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-09-21T22:03:51.291 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-09-21T22:03:51.396 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-webtest. 2024-09-21T22:03:51.423 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-09-21T22:03:51.454 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-09-21T22:03:51.572 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-logutils. 2024-09-21T22:03:51.586 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-webob. 2024-09-21T22:03:51.599 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-09-21T22:03:51.613 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2024-09-21T22:03:51.638 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-09-21T22:03:51.652 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-21T22:03:51.749 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-pecan. 2024-09-21T22:03:51.775 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-09-21T22:03:51.815 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-09-21T22:03:51.941 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-21T22:03:51.968 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-09-21T22:03:52.007 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-09-21T22:03:52.014 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-waitress. 2024-09-21T22:03:52.041 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-09-21T22:03:52.084 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-21T22:03:52.170 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-certifi. 2024-09-21T22:03:52.197 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-09-21T22:03:52.237 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-09-21T22:03:52.358 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-tempita. 2024-09-21T22:03:52.385 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-09-21T22:03:52.402 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-mako. 2024-09-21T22:03:52.424 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-09-21T22:03:52.430 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-09-21T22:03:52.460 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-21T22:03:52.556 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-idna. 2024-09-21T22:03:52.572 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../055-python3-idna_3.3-1ubuntu0.1_all.deb ... 2024-09-21T22:03:52.615 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-idna (3.3-1ubuntu0.1) ... 2024-09-21T22:03:52.728 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-paste. 2024-09-21T22:03:52.755 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-09-21T22:03:52.772 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-21T22:03:52.794 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-09-21T22:03:52.800 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-21T22:03:52.839 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-21T22:03:52.901 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-urllib3. 2024-09-21T22:03:52.916 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-09-21T22:03:52.951 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-21T22:03:53.064 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-21T22:03:53.080 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-09-21T22:03:53.114 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-09-21T22:03:53.142 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-21T22:03:53.169 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-09-21T22:03:53.208 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-09-21T22:03:53.246 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-requests. 2024-09-21T22:03:53.261 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-09-21T22:03:53.295 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-21T22:03:53.408 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-21T22:03:53.435 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-09-21T22:03:53.475 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-09-21T22:03:53.520 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-webob. 2024-09-21T22:03:53.547 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2024-09-21T22:03:53.565 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-21T22:03:53.586 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-21T22:03:53.592 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.2_all.deb ... 2024-09-21T22:03:53.623 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-21T22:03:53.778 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-webtest. 2024-09-21T22:03:53.805 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-09-21T22:03:53.844 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-09-21T22:03:53.889 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-waitress. 2024-09-21T22:03:53.916 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-09-21T22:03:53.935 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-21T22:03:53.950 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.1.1-373-g4e45e6b4-1jammy_all.deb ... 2024-09-21T22:03:53.958 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-21T22:03:53.985 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking ceph-mgr-modules-core (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:03:54.130 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-pecan. 2024-09-21T22:03:54.158 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-09-21T22:03:54.188 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-09-21T22:03:54.234 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-tempita. 2024-09-21T22:03:54.261 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-09-21T22:03:54.292 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-09-21T22:03:54.356 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-21T22:03:54.383 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T22:03:54.423 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking libsqlite3-mod-ceph (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:03:54.553 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-certifi. 2024-09-21T22:03:54.579 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-09-21T22:03:54.595 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-paste. 2024-09-21T22:03:54.618 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-09-21T22:03:54.623 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-09-21T22:03:54.662 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-09-21T22:03:54.734 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-21T22:03:54.760 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-09-21T22:03:54.800 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-09-21T22:03:54.938 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-idna. 2024-09-21T22:03:54.940 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-21T22:03:54.957 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-09-21T22:03:54.965 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../055-python3-idna_3.3-1ubuntu0.1_all.deb ... 2024-09-21T22:03:54.999 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-09-21T22:03:55.004 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-idna (3.3-1ubuntu0.1) ... 2024-09-21T22:03:55.053 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-openssl. 2024-09-21T22:03:55.080 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-09-21T22:03:55.111 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-09-21T22:03:55.299 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-urllib3. 2024-09-21T22:03:55.310 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-21T22:03:55.315 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-09-21T22:03:55.337 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-09-21T22:03:55.376 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-09-21T22:03:55.409 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-21T22:03:55.663 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-webtest. 2024-09-21T22:03:55.690 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-09-21T22:03:55.715 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package ceph-mgr. 2024-09-21T22:03:55.721 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-09-21T22:03:55.742 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../063-ceph-mgr_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T22:03:56.024 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-pecan. 2024-09-21T22:03:56.051 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-09-21T22:03:56.091 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-09-21T22:03:56.106 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking ceph-mgr (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:03:56.454 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-certifi. 2024-09-21T22:03:56.481 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-09-21T22:03:56.521 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-09-21T22:03:56.661 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-requests. 2024-09-21T22:03:56.668 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package ceph-mon. 2024-09-21T22:03:56.688 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-09-21T22:03:56.695 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../064-ceph-mon_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T22:03:56.718 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-21T22:03:56.734 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking ceph-mon (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:03:56.832 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-idna. 2024-09-21T22:03:56.859 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../055-python3-idna_3.3-1ubuntu0.1_all.deb ... 2024-09-21T22:03:56.898 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-idna (3.3-1ubuntu0.1) ... 2024-09-21T22:03:57.013 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-21T22:03:57.040 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.2_all.deb ... 2024-09-21T22:03:57.080 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-21T22:03:57.227 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-urllib3. 2024-09-21T22:03:57.254 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-09-21T22:03:57.288 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package ceph-osd. 2024-09-21T22:03:57.293 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-21T22:03:57.315 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../065-ceph-osd_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T22:03:57.346 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking ceph-osd (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:03:57.349 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-21T22:03:57.365 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.1.1-373-g4e45e6b4-1jammy_all.deb ... 2024-09-21T22:03:57.399 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking ceph-mgr-modules-core (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:03:57.821 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-21T22:03:57.848 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T22:03:57.887 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking libsqlite3-mod-ceph (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:03:58.165 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-21T22:03:58.193 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-09-21T22:03:58.223 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-09-21T22:03:58.399 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package ceph. 2024-09-21T22:03:58.415 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../066-ceph_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T22:03:58.457 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking ceph (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:03:58.518 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-openssl. 2024-09-21T22:03:58.545 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-09-21T22:03:58.580 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-requests. 2024-09-21T22:03:58.584 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-09-21T22:03:58.607 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-09-21T22:03:58.638 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-21T22:03:58.786 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package ceph-fuse. 2024-09-21T22:03:58.812 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../067-ceph-fuse_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T22:03:58.852 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking ceph-fuse (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:03:58.904 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package ceph-mgr. 2024-09-21T22:03:58.931 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../063-ceph-mgr_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T22:03:58.941 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-21T22:03:58.969 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.2_all.deb ... 2024-09-21T22:03:58.970 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking ceph-mgr (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:03:59.016 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-21T22:03:59.188 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package ceph-mds. 2024-09-21T22:03:59.215 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../068-ceph-mds_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T22:03:59.246 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking ceph-mds (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:03:59.286 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-21T22:03:59.303 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.1.1-373-g4e45e6b4-1jammy_all.deb ... 2024-09-21T22:03:59.336 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking ceph-mgr-modules-core (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:03:59.349 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package ceph-mon. 2024-09-21T22:03:59.376 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../064-ceph-mon_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T22:03:59.407 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking ceph-mon (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:03:59.599 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package cephadm. 2024-09-21T22:03:59.626 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../069-cephadm_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T22:03:59.666 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking cephadm (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:03:59.725 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-21T22:03:59.752 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T22:03:59.791 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking libsqlite3-mod-ceph (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:03:59.986 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-asyncssh. 2024-09-21T22:04:00.010 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package ceph-osd. 2024-09-21T22:04:00.012 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-09-21T22:04:00.038 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../065-ceph-osd_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T22:04:00.052 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-09-21T22:04:00.052 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-21T22:04:00.069 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking ceph-osd (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:00.080 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-09-21T22:04:00.111 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-09-21T22:04:00.354 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python-babel-localedata. 2024-09-21T22:04:00.370 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-09-21T22:04:00.413 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-21T22:04:00.414 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-openssl. 2024-09-21T22:04:00.441 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-09-21T22:04:00.481 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-09-21T22:04:00.817 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package ceph-mgr. 2024-09-21T22:04:00.844 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../063-ceph-mgr_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T22:04:00.883 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking ceph-mgr (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:01.080 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package ceph. 2024-09-21T22:04:01.096 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../066-ceph_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T22:04:01.138 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking ceph (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:01.245 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package ceph-mon. 2024-09-21T22:04:01.273 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../064-ceph-mon_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T22:04:01.303 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking ceph-mon (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:01.458 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package ceph-fuse. 2024-09-21T22:04:01.485 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../067-ceph-fuse_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T22:04:01.532 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking ceph-fuse (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:01.710 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-babel. 2024-09-21T22:04:01.738 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-09-21T22:04:01.776 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-09-21T22:04:01.890 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package ceph-osd. 2024-09-21T22:04:01.911 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package ceph-mds. 2024-09-21T22:04:01.918 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../065-ceph-osd_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T22:04:01.938 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../068-ceph-mds_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T22:04:01.957 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking ceph-osd (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:01.977 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking ceph-mds (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:02.104 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-jinja2. 2024-09-21T22:04:02.128 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.2_all.deb ... 2024-09-21T22:04:02.162 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-21T22:04:02.380 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package cephadm. 2024-09-21T22:04:02.397 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../069-cephadm_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T22:04:02.438 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking cephadm (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:02.482 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-21T22:04:02.509 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.1.1-373-g4e45e6b4-1jammy_all.deb ... 2024-09-21T22:04:02.548 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking ceph-mgr-cephadm (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:02.758 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-asyncssh. 2024-09-21T22:04:02.785 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-09-21T22:04:02.824 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-09-21T22:04:02.877 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-21T22:04:02.905 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-09-21T22:04:02.943 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-21T22:04:03.043 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package ceph. 2024-09-21T22:04:03.060 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../066-ceph_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T22:04:03.110 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking ceph (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:03.160 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python-babel-localedata. 2024-09-21T22:04:03.187 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-09-21T22:04:03.196 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-routes. 2024-09-21T22:04:03.219 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-21T22:04:03.227 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-09-21T22:04:03.254 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-09-21T22:04:03.380 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package ceph-fuse. 2024-09-21T22:04:03.407 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../067-ceph-fuse_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T22:04:03.438 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking ceph-fuse (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:03.566 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-21T22:04:03.593 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.1.1-373-g4e45e6b4-1jammy_all.deb ... 2024-09-21T22:04:03.624 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking ceph-mgr-dashboard (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:03.783 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package ceph-mds. 2024-09-21T22:04:03.811 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../068-ceph-mds_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T22:04:03.849 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking ceph-mds (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:04.253 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package cephadm. 2024-09-21T22:04:04.265 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-babel. 2024-09-21T22:04:04.280 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../069-cephadm_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T22:04:04.293 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-09-21T22:04:04.319 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking cephadm (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:04.332 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-09-21T22:04:04.614 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-asyncssh. 2024-09-21T22:04:04.641 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-09-21T22:04:04.668 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-jinja2. 2024-09-21T22:04:04.680 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-09-21T22:04:04.696 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.2_all.deb ... 2024-09-21T22:04:04.734 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-21T22:04:04.887 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-09-21T22:04:04.903 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-09-21T22:04:04.946 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-21T22:04:04.991 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python-babel-localedata. 2024-09-21T22:04:05.007 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-09-21T22:04:05.049 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-21T22:04:05.062 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-21T22:04:05.090 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.1.1-373-g4e45e6b4-1jammy_all.deb ... 2024-09-21T22:04:05.128 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking ceph-mgr-cephadm (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:05.432 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-joblib. 2024-09-21T22:04:05.448 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-09-21T22:04:05.476 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-21T22:04:05.490 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-09-21T22:04:05.503 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-09-21T22:04:05.542 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-21T22:04:05.812 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-routes. 2024-09-21T22:04:05.827 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-09-21T22:04:05.829 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-09-21T22:04:05.855 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-09-21T22:04:05.894 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-09-21T22:04:06.016 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-09-21T22:04:06.105 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-babel. 2024-09-21T22:04:06.132 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-09-21T22:04:06.171 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-09-21T22:04:06.197 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-sklearn. 2024-09-21T22:04:06.224 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-09-21T22:04:06.263 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-21T22:04:06.381 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-21T22:04:06.408 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.1.1-373-g4e45e6b4-1jammy_all.deb ... 2024-09-21T22:04:06.447 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking ceph-mgr-dashboard (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:06.516 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-jinja2. 2024-09-21T22:04:06.543 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.2_all.deb ... 2024-09-21T22:04:06.582 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-21T22:04:06.911 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-21T22:04:06.930 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-21T22:04:06.938 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.1.1-373-g4e45e6b4-1jammy_all.deb ... 2024-09-21T22:04:06.958 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.1.1-373-g4e45e6b4-1jammy_all.deb ... 2024-09-21T22:04:06.977 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking ceph-mgr-cephadm (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:07.082 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking ceph-mgr-diskprediction-local (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:07.272 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-21T22:04:07.300 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-09-21T22:04:07.339 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-21T22:04:07.642 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-routes. 2024-09-21T22:04:07.669 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-09-21T22:04:07.708 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-09-21T22:04:07.769 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-09-21T22:04:07.786 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-09-21T22:04:07.827 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-21T22:04:08.037 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-21T22:04:08.064 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.1.1-373-g4e45e6b4-1jammy_all.deb ... 2024-09-21T22:04:08.077 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-cachetools. 2024-09-21T22:04:08.105 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-09-21T22:04:08.135 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-09-21T22:04:08.278 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking ceph-mgr-dashboard (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:08.356 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-joblib. 2024-09-21T22:04:08.383 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-09-21T22:04:08.422 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-09-21T22:04:08.446 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-21T22:04:08.474 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-09-21T22:04:08.513 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-09-21T22:04:08.784 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-09-21T22:04:08.812 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-09-21T22:04:08.841 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-21T22:04:08.850 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-09-21T22:04:08.869 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-09-21T22:04:08.908 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-09-21T22:04:09.153 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-sklearn. 2024-09-21T22:04:09.181 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-09-21T22:04:09.219 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-rsa. 2024-09-21T22:04:09.220 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-21T22:04:09.247 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-09-21T22:04:09.286 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-rsa (4.8-1) ... 2024-09-21T22:04:09.475 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-09-21T22:04:09.503 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-09-21T22:04:09.542 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-21T22:04:09.573 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-google-auth. 2024-09-21T22:04:09.600 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-09-21T22:04:09.639 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-09-21T22:04:09.895 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-21T22:04:09.923 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.1.1-373-g4e45e6b4-1jammy_all.deb ... 2024-09-21T22:04:09.959 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-21T22:04:09.986 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-09-21T22:04:10.025 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-21T22:04:10.038 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking ceph-mgr-diskprediction-local (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:10.046 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-joblib. 2024-09-21T22:04:10.073 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-09-21T22:04:10.112 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-09-21T22:04:10.312 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-websocket. 2024-09-21T22:04:10.328 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-09-21T22:04:10.370 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-09-21T22:04:10.466 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-09-21T22:04:10.493 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-09-21T22:04:10.532 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-09-21T22:04:10.632 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-21T22:04:10.659 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-09-21T22:04:10.698 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-21T22:04:10.835 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-sklearn. 2024-09-21T22:04:10.863 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-09-21T22:04:10.901 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-21T22:04:11.009 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-cachetools. 2024-09-21T22:04:11.036 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-09-21T22:04:11.075 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-09-21T22:04:11.361 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-21T22:04:11.378 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-09-21T22:04:11.379 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-21T22:04:11.407 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.1.1-373-g4e45e6b4-1jammy_all.deb ... 2024-09-21T22:04:11.419 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-09-21T22:04:11.445 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking ceph-mgr-k8sevents (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:11.577 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-21T22:04:11.606 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.1.1-373-g4e45e6b4-1jammy_all.deb ... 2024-09-21T22:04:11.729 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking ceph-mgr-diskprediction-local (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:11.748 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-21T22:04:11.765 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-21T22:04:11.775 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-09-21T22:04:11.792 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-09-21T22:04:11.814 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-09-21T22:04:11.856 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-21T22:04:12.118 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-rsa. 2024-09-21T22:04:12.145 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-09-21T22:04:12.159 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-21T22:04:12.184 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-rsa (4.8-1) ... 2024-09-21T22:04:12.187 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-09-21T22:04:12.218 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-21T22:04:12.471 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package jq. 2024-09-21T22:04:12.498 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-09-21T22:04:12.504 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-google-auth. 2024-09-21T22:04:12.531 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-09-21T22:04:12.537 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-09-21T22:04:12.570 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-09-21T22:04:12.608 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-cachetools. 2024-09-21T22:04:12.635 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-09-21T22:04:12.666 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-09-21T22:04:12.789 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package socat. 2024-09-21T22:04:12.805 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-09-21T22:04:12.847 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-09-21T22:04:12.890 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-21T22:04:12.918 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-09-21T22:04:12.944 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-21T22:04:12.957 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-21T22:04:12.972 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-09-21T22:04:13.011 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-09-21T22:04:13.159 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package xmlstarlet. 2024-09-21T22:04:13.175 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-09-21T22:04:13.217 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-09-21T22:04:13.243 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-websocket. 2024-09-21T22:04:13.271 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-09-21T22:04:13.301 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-09-21T22:04:13.348 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-21T22:04:13.375 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-09-21T22:04:13.414 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-09-21T22:04:13.564 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package ceph-test. 2024-09-21T22:04:13.579 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-21T22:04:13.591 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../097-ceph-test_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T22:04:13.607 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-09-21T22:04:13.630 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking ceph-test (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:13.637 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-21T22:04:13.717 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-rsa. 2024-09-21T22:04:13.745 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-09-21T22:04:13.783 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-rsa (4.8-1) ... 2024-09-21T22:04:14.113 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-google-auth. 2024-09-21T22:04:14.129 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-09-21T22:04:14.171 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-09-21T22:04:14.326 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-21T22:04:14.343 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.1.1-373-g4e45e6b4-1jammy_all.deb ... 2024-09-21T22:04:14.384 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking ceph-mgr-k8sevents (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:14.497 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-21T22:04:14.524 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-09-21T22:04:14.565 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-21T22:04:14.737 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-21T22:04:14.765 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-09-21T22:04:14.803 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-21T22:04:14.819 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-websocket. 2024-09-21T22:04:14.847 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-09-21T22:04:14.885 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-09-21T22:04:15.081 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-21T22:04:15.106 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-09-21T22:04:15.139 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-21T22:04:15.206 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-21T22:04:15.234 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-09-21T22:04:15.272 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-21T22:04:15.385 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package jq. 2024-09-21T22:04:15.401 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-09-21T22:04:15.442 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-09-21T22:04:15.695 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package socat. 2024-09-21T22:04:15.711 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-09-21T22:04:15.753 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-09-21T22:04:15.868 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package ceph-volume. 2024-09-21T22:04:15.895 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../098-ceph-volume_19.1.1-373-g4e45e6b4-1jammy_all.deb ... 2024-09-21T22:04:15.934 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking ceph-volume (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:15.987 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-21T22:04:16.015 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.1.1-373-g4e45e6b4-1jammy_all.deb ... 2024-09-21T22:04:16.053 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking ceph-mgr-k8sevents (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:16.064 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package xmlstarlet. 2024-09-21T22:04:16.081 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-09-21T22:04:16.122 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-09-21T22:04:16.265 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-21T22:04:16.281 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../099-libcephfs-dev_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T22:04:16.323 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking libcephfs-dev (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:16.347 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-21T22:04:16.364 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-09-21T22:04:16.405 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-21T22:04:16.584 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package lua-socket:amd64. 2024-09-21T22:04:16.600 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-09-21T22:04:16.634 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-21T22:04:16.734 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-21T22:04:16.762 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-09-21T22:04:16.785 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package ceph-test. 2024-09-21T22:04:16.813 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../097-ceph-test_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T22:04:16.817 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-21T22:04:16.851 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking ceph-test (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:17.087 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package jq. 2024-09-21T22:04:17.115 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-09-21T22:04:17.153 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-09-21T22:04:17.305 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package lua-sec:amd64. 2024-09-21T22:04:17.333 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-09-21T22:04:17.371 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-09-21T22:04:17.406 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package socat. 2024-09-21T22:04:17.434 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-09-21T22:04:17.472 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-09-21T22:04:17.734 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package nvme-cli. 2024-09-21T22:04:17.762 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.3_amd64.deb ... 2024-09-21T22:04:17.800 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking nvme-cli (1.16-3ubuntu0.3) ... 2024-09-21T22:04:17.818 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package xmlstarlet. 2024-09-21T22:04:17.845 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-09-21T22:04:17.884 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-09-21T22:04:18.222 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package pkg-config. 2024-09-21T22:04:18.250 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-09-21T22:04:18.255 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package ceph-test. 2024-09-21T22:04:18.272 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../097-ceph-test_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T22:04:18.288 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-09-21T22:04:18.313 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking ceph-test (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:18.516 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-09-21T22:04:18.533 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-09-21T22:04:18.566 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-09-21T22:04:19.011 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-iniconfig. 2024-09-21T22:04:19.039 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-09-21T22:04:19.077 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-09-21T22:04:19.239 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package ceph-volume. 2024-09-21T22:04:19.267 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../098-ceph-volume_19.1.1-373-g4e45e6b4-1jammy_all.deb ... 2024-09-21T22:04:19.314 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking ceph-volume (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:19.389 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-pastescript. 2024-09-21T22:04:19.416 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-09-21T22:04:19.455 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-21T22:04:19.634 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-21T22:04:19.663 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../099-libcephfs-dev_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T22:04:19.701 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking libcephfs-dev (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:19.775 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-pluggy. 2024-09-21T22:04:19.803 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-09-21T22:04:19.842 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-09-21T22:04:20.079 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package lua-socket:amd64. 2024-09-21T22:04:20.107 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-09-21T22:04:20.120 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-psutil. 2024-09-21T22:04:20.135 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-09-21T22:04:20.153 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-21T22:04:20.169 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-09-21T22:04:20.497 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-py. 2024-09-21T22:04:20.514 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-09-21T22:04:20.533 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package lua-sec:amd64. 2024-09-21T22:04:20.547 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-py (1.10.0-1) ... 2024-09-21T22:04:20.551 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package ceph-volume. 2024-09-21T22:04:20.561 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-09-21T22:04:20.580 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../098-ceph-volume_19.1.1-373-g4e45e6b4-1jammy_all.deb ... 2024-09-21T22:04:20.599 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-09-21T22:04:20.618 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking ceph-volume (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:20.868 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-pygments. 2024-09-21T22:04:20.896 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-09-21T22:04:20.922 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-21T22:04:20.934 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-09-21T22:04:20.939 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../099-libcephfs-dev_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T22:04:20.970 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package nvme-cli. 2024-09-21T22:04:20.980 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking libcephfs-dev (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:20.998 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.3_amd64.deb ... 2024-09-21T22:04:21.036 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking nvme-cli (1.16-3ubuntu0.3) ... 2024-09-21T22:04:21.301 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package lua-socket:amd64. 2024-09-21T22:04:21.328 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-09-21T22:04:21.366 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-21T22:04:21.422 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-21T22:04:21.441 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package pkg-config. 2024-09-21T22:04:21.450 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-09-21T22:04:21.458 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-09-21T22:04:21.488 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-09-21T22:04:21.491 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-09-21T22:04:21.713 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package lua-sec:amd64. 2024-09-21T22:04:21.728 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-09-21T22:04:21.741 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-09-21T22:04:21.756 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-09-21T22:04:21.766 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-toml. 2024-09-21T22:04:21.779 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-09-21T22:04:21.794 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-09-21T22:04:21.794 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-09-21T22:04:21.824 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-09-21T22:04:22.109 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package nvme-cli. 2024-09-21T22:04:22.137 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.3_amd64.deb ... 2024-09-21T22:04:22.160 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-pytest. 2024-09-21T22:04:22.172 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-iniconfig. 2024-09-21T22:04:22.175 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking nvme-cli (1.16-3ubuntu0.3) ... 2024-09-21T22:04:22.188 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-09-21T22:04:22.200 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-09-21T22:04:22.235 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-09-21T22:04:22.246 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-09-21T22:04:22.558 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-pastescript. 2024-09-21T22:04:22.564 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package pkg-config. 2024-09-21T22:04:22.586 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-09-21T22:04:22.592 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-09-21T22:04:22.597 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package python3-simplejson. 2024-09-21T22:04:22.624 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-21T22:04:22.625 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-09-21T22:04:22.630 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-09-21T22:04:22.664 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-09-21T22:04:22.892 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-09-21T22:04:22.908 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-09-21T22:04:22.933 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-21T22:04:22.944 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-pluggy. 2024-09-21T22:04:22.950 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-09-21T22:04:22.961 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-09-21T22:04:22.973 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-09-21T22:04:22.999 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-09-21T22:04:23.011 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-09-21T22:04:23.322 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-psutil. 2024-09-21T22:04:23.350 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-09-21T22:04:23.388 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-09-21T22:04:23.403 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-iniconfig. 2024-09-21T22:04:23.431 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-09-21T22:04:23.469 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-09-21T22:04:23.733 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-py. 2024-09-21T22:04:23.737 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package radosgw. 2024-09-21T22:04:23.761 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-09-21T22:04:23.765 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../116-radosgw_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T22:04:23.772 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-pastescript. 2024-09-21T22:04:23.795 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking radosgw (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:23.800 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-09-21T22:04:23.808 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-py (1.10.0-1) ... 2024-09-21T22:04:23.839 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-21T22:04:24.128 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-pygments. 2024-09-21T22:04:24.156 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-09-21T22:04:24.167 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-pluggy. 2024-09-21T22:04:24.194 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-09-21T22:04:24.195 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-09-21T22:04:24.234 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-09-21T22:04:24.520 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-psutil. 2024-09-21T22:04:24.548 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-09-21T22:04:24.587 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-09-21T22:04:24.590 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package rbd-fuse. 2024-09-21T22:04:24.618 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../117-rbd-fuse_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T22:04:24.631 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-21T22:04:24.656 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking rbd-fuse (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:24.660 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-09-21T22:04:24.698 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-09-21T22:04:24.907 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-py. 2024-09-21T22:04:24.917 INFO:teuthology.orchestra.run.smithi157.stdout:Selecting previously unselected package rbd-nbd. 2024-09-21T22:04:24.935 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-09-21T22:04:24.945 INFO:teuthology.orchestra.run.smithi157.stdout:Preparing to unpack .../118-rbd-nbd_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T22:04:24.973 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-py (1.10.0-1) ... 2024-09-21T22:04:24.984 INFO:teuthology.orchestra.run.smithi157.stdout:Unpacking rbd-nbd (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:25.001 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-toml. 2024-09-21T22:04:25.029 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-09-21T22:04:25.059 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-09-21T22:04:25.297 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-09-21T22:04:25.302 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-pygments. 2024-09-21T22:04:25.311 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-pytest. 2024-09-21T22:04:25.330 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-09-21T22:04:25.340 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-09-21T22:04:25.368 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-09-21T22:04:25.378 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-09-21T22:04:25.547 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-21T22:04:25.673 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up nvme-cli (1.16-3ubuntu0.3) ... 2024-09-21T22:04:25.698 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package python3-simplejson. 2024-09-21T22:04:25.726 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-09-21T22:04:25.764 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-09-21T22:04:25.781 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-21T22:04:25.807 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-09-21T22:04:25.839 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-09-21T22:04:25.939 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-09-21T22:04:26.025 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-21T22:04:26.042 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-09-21T22:04:26.100 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-09-21T22:04:26.167 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-toml. 2024-09-21T22:04:26.195 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-09-21T22:04:26.233 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-09-21T22:04:26.278 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-09-21T22:04:26.536 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-pytest. 2024-09-21T22:04:26.565 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-09-21T22:04:26.603 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-09-21T22:04:26.715 INFO:teuthology.orchestra.run.smithi157.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-09-21T22:04:26.804 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package radosgw. 2024-09-21T22:04:26.825 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up cephadm (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:26.832 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../116-radosgw_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T22:04:26.870 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking radosgw (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:26.948 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package python3-simplejson. 2024-09-21T22:04:26.977 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-09-21T22:04:27.015 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-09-21T22:04:27.284 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-21T22:04:27.309 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-09-21T22:04:27.351 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-09-21T22:04:27.367 INFO:teuthology.orchestra.run.smithi157.stdout:Adding system user cephadm....done 2024-09-21T22:04:27.532 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-21T22:04:27.707 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package rbd-fuse. 2024-09-21T22:04:27.723 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../117-rbd-fuse_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T22:04:27.765 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking rbd-fuse (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:28.026 INFO:teuthology.orchestra.run.smithi064.stdout:Selecting previously unselected package rbd-nbd. 2024-09-21T22:04:28.036 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-09-21T22:04:28.052 INFO:teuthology.orchestra.run.smithi064.stdout:Preparing to unpack .../118-rbd-nbd_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T22:04:28.065 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package radosgw. 2024-09-21T22:04:28.092 INFO:teuthology.orchestra.run.smithi064.stdout:Unpacking rbd-nbd (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:28.094 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../116-radosgw_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T22:04:28.131 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking radosgw (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:28.294 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-09-21T22:04:28.389 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-09-21T22:04:28.411 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-09-21T22:04:28.647 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-21T22:04:28.654 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-21T22:04:28.773 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up nvme-cli (1.16-3ubuntu0.3) ... 2024-09-21T22:04:28.889 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-09-21T22:04:28.935 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package rbd-fuse. 2024-09-21T22:04:28.952 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../117-rbd-fuse_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T22:04:28.992 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking rbd-fuse (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:29.015 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-py (1.10.0-1) ... 2024-09-21T22:04:29.048 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-09-21T22:04:29.253 INFO:teuthology.orchestra.run.smithi039.stdout:Selecting previously unselected package rbd-nbd. 2024-09-21T22:04:29.270 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing to unpack .../118-rbd-nbd_19.1.1-373-g4e45e6b4-1jammy_amd64.deb ... 2024-09-21T22:04:29.311 INFO:teuthology.orchestra.run.smithi039.stdout:Unpacking rbd-nbd (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:29.325 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-09-21T22:04:29.387 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-09-21T22:04:29.625 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-09-21T22:04:29.702 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-09-21T22:04:29.801 INFO:teuthology.orchestra.run.smithi064.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-09-21T22:04:29.866 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-21T22:04:29.867 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up cephadm (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:29.978 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-09-21T22:04:29.992 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up nvme-cli (1.16-3ubuntu0.3) ... 2024-09-21T22:04:30.237 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-09-21T22:04:30.257 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-09-21T22:04:30.467 INFO:teuthology.orchestra.run.smithi064.stdout:Adding system user cephadm....done 2024-09-21T22:04:30.481 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-09-21T22:04:30.599 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-09-21T22:04:30.614 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-21T22:04:30.740 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-09-21T22:04:30.907 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-09-21T22:04:30.999 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-09-21T22:04:31.033 INFO:teuthology.orchestra.run.smithi039.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-09-21T22:04:31.102 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up cephadm (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:31.175 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-09-21T22:04:31.301 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-09-21T22:04:31.359 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-ceph-argparse (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:31.561 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-21T22:04:31.618 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-21T22:04:31.636 INFO:teuthology.orchestra.run.smithi039.stdout:Adding system user cephadm....done 2024-09-21T22:04:31.744 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-21T22:04:31.791 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-21T22:04:31.812 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-09-21T22:04:31.862 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-09-21T22:04:31.929 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-py (1.10.0-1) ... 2024-09-21T22:04:31.980 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-09-21T22:04:32.068 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-09-21T22:04:32.239 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-09-21T22:04:32.336 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-09-21T22:04:32.461 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-09-21T22:04:32.564 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-09-21T22:04:32.721 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-21T22:04:32.874 INFO:teuthology.orchestra.run.smithi157.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-09-21T22:04:32.933 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-09-21T22:04:32.980 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-09-21T22:04:32.988 INFO:teuthology.orchestra.run.smithi157.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-09-21T22:04:33.051 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-21T22:04:33.107 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-py (1.10.0-1) ... 2024-09-21T22:04:33.176 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-09-21T22:04:33.192 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-09-21T22:04:33.400 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-09-21T22:04:33.469 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-09-21T22:04:33.553 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-21T22:04:33.662 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-09-21T22:04:33.695 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-09-21T22:04:33.752 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-09-21T22:04:33.913 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-09-21T22:04:33.963 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-09-21T22:04:34.011 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-09-21T22:04:34.172 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-09-21T22:04:34.231 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-09-21T22:04:34.287 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-09-21T22:04:34.490 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up zip (3.0-12build2) ... 2024-09-21T22:04:34.572 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-09-21T22:04:34.616 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-09-21T22:04:34.632 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-ceph-argparse (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:34.900 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-21T22:04:34.990 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-09-21T22:04:35.026 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-21T22:04:35.152 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-09-21T22:04:35.241 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-09-21T22:04:35.278 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-09-21T22:04:35.310 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-09-21T22:04:35.492 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-ceph-argparse (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:35.537 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-09-21T22:04:35.578 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-09-21T22:04:35.679 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-09-21T22:04:35.735 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-21T22:04:35.780 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-21T22:04:35.861 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-21T22:04:35.880 INFO:teuthology.orchestra.run.smithi064.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-09-21T22:04:35.910 INFO:teuthology.orchestra.run.smithi064.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-09-21T22:04:35.973 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-21T22:04:35.987 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-09-21T22:04:36.081 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-09-21T22:04:36.108 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-09-21T22:04:36.113 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-09-21T22:04:36.372 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-09-21T22:04:36.474 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-09-21T22:04:36.476 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-21T22:04:36.602 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-09-21T22:04:36.725 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-09-21T22:04:36.774 INFO:teuthology.orchestra.run.smithi039.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-09-21T22:04:36.804 INFO:teuthology.orchestra.run.smithi039.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-09-21T22:04:36.861 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-09-21T22:04:36.875 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-21T22:04:36.977 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-09-21T22:04:37.101 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-21T22:04:37.120 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-09-21T22:04:37.344 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-21T22:04:37.419 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-21T22:04:37.470 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-09-21T22:04:37.471 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up zip (3.0-12build2) ... 2024-09-21T22:04:37.581 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-09-21T22:04:37.738 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-09-21T22:04:37.762 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-09-21T22:04:38.005 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-09-21T22:04:38.006 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-09-21T22:04:38.131 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-ceph-common (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:38.258 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-09-21T22:04:38.340 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up zip (3.0-12build2) ... 2024-09-21T22:04:38.424 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-idna (3.3-1ubuntu0.1) ... 2024-09-21T22:04:38.450 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-09-21T22:04:38.517 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-09-21T22:04:38.643 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-09-21T22:04:38.742 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-21T22:04:38.769 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-21T22:04:39.070 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-09-21T22:04:39.151 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-09-21T22:04:39.411 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-09-21T22:04:39.496 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-09-21T22:04:39.537 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-09-21T22:04:39.663 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-21T22:04:39.747 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-09-21T22:04:39.847 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-09-21T22:04:39.973 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-09-21T22:04:40.091 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-21T22:04:40.149 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-21T22:04:40.242 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-toml (0.10.2-1) ... 2024-09-21T22:04:40.365 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-09-21T22:04:40.459 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-21T22:04:40.535 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-21T22:04:40.660 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-09-21T22:04:40.675 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-09-21T22:04:40.778 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-21T22:04:40.810 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-09-21T22:04:41.055 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-09-21T22:04:41.070 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-09-21T22:04:41.076 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-21T22:04:41.195 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-ceph-common (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:41.306 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-09-21T22:04:41.369 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-21T22:04:41.497 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-idna (3.3-1ubuntu0.1) ... 2024-09-21T22:04:41.548 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-21T22:04:41.674 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-09-21T22:04:41.728 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-09-21T22:04:41.831 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-21T22:04:41.976 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-09-21T22:04:41.988 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-09-21T22:04:42.123 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-ceph-common (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:42.227 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-09-21T22:04:42.491 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-idna (3.3-1ubuntu0.1) ... 2024-09-21T22:04:42.495 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-09-21T22:04:42.738 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-21T22:04:42.800 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-21T22:04:42.980 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-09-21T22:04:43.003 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-09-21T22:04:43.238 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-21T22:04:43.248 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-21T22:04:43.372 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-toml (0.10.2-1) ... 2024-09-21T22:04:43.374 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-09-21T22:04:43.640 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-21T22:04:43.658 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-21T22:04:43.766 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-09-21T22:04:43.784 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-21T22:04:43.892 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-21T22:04:44.039 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-09-21T22:04:44.094 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-21T22:04:44.194 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-09-21T22:04:44.282 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-21T22:04:44.379 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-09-21T22:04:44.391 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-toml (0.10.2-1) ... 2024-09-21T22:04:44.453 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-09-21T22:04:44.621 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-21T22:04:44.642 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-21T22:04:44.737 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-21T22:04:44.747 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-09-21T22:04:44.768 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-09-21T22:04:44.863 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-09-21T22:04:44.873 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-09-21T22:04:44.894 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-21T22:04:44.961 INFO:teuthology.orchestra.run.smithi157.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-09-21T22:04:45.182 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-09-21T22:04:45.191 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-21T22:04:45.221 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-09-21T22:04:45.317 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-09-21T22:04:45.458 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-09-21T22:04:45.480 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-09-21T22:04:45.702 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-09-21T22:04:45.717 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-09-21T22:04:45.748 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-21T22:04:45.865 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-09-21T22:04:45.962 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up lua-any (27ubuntu1) ... 2024-09-21T22:04:45.976 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-21T22:04:46.071 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-portend (3.0.0-1) ... 2024-09-21T22:04:46.150 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-09-21T22:04:46.219 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-09-21T22:04:46.322 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-21T22:04:46.409 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-09-21T22:04:46.448 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-21T22:04:46.512 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-21T22:04:46.638 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-09-21T22:04:46.686 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-09-21T22:04:46.774 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-09-21T22:04:46.914 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-21T22:04:46.945 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-21T22:04:47.040 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-21T22:04:47.083 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-21T22:04:47.204 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-09-21T22:04:47.341 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-21T22:04:47.368 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-09-21T22:04:47.472 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-21T22:04:47.485 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-rsa (4.8-1) ... 2024-09-21T22:04:47.598 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-09-21T22:04:47.737 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-09-21T22:04:47.791 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-09-21T22:04:47.874 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-21T22:04:47.975 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-21T22:04:48.013 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-21T22:04:48.051 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-21T22:04:48.178 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-09-21T22:04:48.285 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-21T22:04:48.304 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-09-21T22:04:48.358 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-09-21T22:04:48.393 INFO:teuthology.orchestra.run.smithi064.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-09-21T22:04:48.594 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-09-21T22:04:48.639 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-21T22:04:48.734 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-21T22:04:48.764 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-09-21T22:04:48.853 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-21T22:04:48.979 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-09-21T22:04:49.019 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-09-21T22:04:49.105 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-09-21T22:04:49.174 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-09-21T22:04:49.196 INFO:teuthology.orchestra.run.smithi039.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-09-21T22:04:49.339 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-21T22:04:49.432 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-21T22:04:49.442 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up lua-any (27ubuntu1) ... 2024-09-21T22:04:49.465 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up librados2 (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:49.533 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-09-21T22:04:49.568 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-portend (3.0.0-1) ... 2024-09-21T22:04:49.591 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up libsqlite3-mod-ceph (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:49.717 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-09-21T22:04:49.827 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-21T22:04:49.918 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-09-21T22:04:49.929 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-21T22:04:50.044 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up libcephfs2 (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:50.162 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up libradosstriper1 (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:50.185 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up lua-any (27ubuntu1) ... 2024-09-21T22:04:50.246 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-09-21T22:04:50.289 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-09-21T22:04:50.312 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-portend (3.0.0-1) ... 2024-09-21T22:04:50.539 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-21T22:04:50.565 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up librbd1 (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:50.571 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-21T22:04:50.691 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up ceph-mgr-modules-core (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:50.705 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-21T22:04:50.815 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-09-21T22:04:50.817 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up ceph-fuse (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:50.943 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-rsa (4.8-1) ... 2024-09-21T22:04:51.031 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-21T22:04:51.031 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-21T22:04:51.040 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-09-21T22:04:51.227 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-09-21T22:04:51.341 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-21T22:04:51.478 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-21T22:04:51.518 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up libcephfs-dev (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:51.600 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-09-21T22:04:51.628 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-rados (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:51.726 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-rsa (4.8-1) ... 2024-09-21T22:04:51.737 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up librgw2 (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:51.831 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-09-21T22:04:51.863 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-21T22:04:52.010 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-09-21T22:04:52.225 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-21T22:04:52.295 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-21T22:04:52.510 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-09-21T22:04:52.682 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-09-21T22:04:52.837 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-21T22:04:52.971 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up librados2 (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:53.000 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-rbd (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:53.050 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-21T22:04:53.089 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up libsqlite3-mod-ceph (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:53.160 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up rbd-fuse (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:53.198 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-09-21T22:04:53.269 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up rbd-nbd (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:53.343 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-09-21T22:04:53.387 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-rgw (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:53.496 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up python3-cephfs (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:53.585 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up libcephfs2 (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:53.622 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up ceph-common (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:53.695 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-21T22:04:53.735 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up libradosstriper1 (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:53.821 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up librados2 (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:53.878 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-09-21T22:04:53.956 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up libsqlite3-mod-ceph (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:54.073 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-09-21T22:04:54.131 INFO:teuthology.orchestra.run.smithi157.stdout:Adding group ceph....done 2024-09-21T22:04:54.163 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up librbd1 (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:54.289 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up ceph-mgr-modules-core (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:54.376 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up libcephfs2 (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:54.415 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up ceph-fuse (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:54.466 INFO:teuthology.orchestra.run.smithi157.stdout:Adding system user ceph....done 2024-09-21T22:04:54.502 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up libradosstriper1 (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:54.628 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-09-21T22:04:54.635 INFO:teuthology.orchestra.run.smithi157.stdout:Setting system user ceph properties....done 2024-09-21T22:04:54.638 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-21T22:04:54.638 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-21T22:04:54.647 INFO:teuthology.orchestra.run.smithi157.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-21T22:04:54.779 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-21T22:04:54.904 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up librbd1 (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:55.030 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up ceph-mgr-modules-core (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:55.113 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-21T22:04:55.143 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up libcephfs-dev (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:55.165 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up ceph-fuse (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:55.334 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-rados (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:55.384 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-21T22:04:55.385 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-21T22:04:55.510 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up librgw2 (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:55.565 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up ceph-test (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:55.661 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-21T22:04:55.690 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up radosgw (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:55.857 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up libcephfs-dev (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:55.958 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-rados (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:56.068 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up librgw2 (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:56.154 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-21T22:04:56.154 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-21T22:04:56.194 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-21T22:04:56.634 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up ceph-base (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:56.966 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-rbd (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:56.979 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-21T22:04:57.108 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up rbd-fuse (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:57.260 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up rbd-nbd (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:57.411 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-rgw (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:57.436 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up ceph-mds (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:57.497 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-rbd (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:57.553 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up python3-cephfs (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:57.623 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up rbd-fuse (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:57.654 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-21T22:04:57.655 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-21T22:04:57.679 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up ceph-common (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:57.741 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up rbd-nbd (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:57.841 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-rgw (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:57.942 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up python3-cephfs (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:58.094 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up ceph-common (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:58.154 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up ceph-mgr (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:58.372 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-21T22:04:58.372 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-21T22:04:58.434 INFO:teuthology.orchestra.run.smithi064.stdout:Adding group ceph....done 2024-09-21T22:04:58.645 INFO:teuthology.orchestra.run.smithi039.stdout:Adding group ceph....done 2024-09-21T22:04:58.864 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up ceph-osd (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:58.902 INFO:teuthology.orchestra.run.smithi064.stdout:Adding system user ceph....done 2024-09-21T22:04:59.013 INFO:teuthology.orchestra.run.smithi039.stdout:Adding system user ceph....done 2024-09-21T22:04:59.130 INFO:teuthology.orchestra.run.smithi064.stdout:Setting system user ceph properties....done 2024-09-21T22:04:59.145 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-21T22:04:59.145 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-21T22:04:59.147 INFO:teuthology.orchestra.run.smithi064.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-21T22:04:59.147 INFO:teuthology.orchestra.run.smithi039.stdout:Setting system user ceph properties....done 2024-09-21T22:04:59.161 INFO:teuthology.orchestra.run.smithi039.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-21T22:04:59.298 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-21T22:04:59.305 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-21T22:04:59.618 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-21T22:04:59.649 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up ceph-mgr-k8sevents (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:59.660 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-21T22:04:59.775 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up ceph-mgr-diskprediction-local (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:04:59.926 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up ceph-mon (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:05:00.101 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up ceph-test (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:05:00.132 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-21T22:05:00.132 INFO:teuthology.orchestra.run.smithi157.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-21T22:05:00.144 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up ceph-test (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:05:00.227 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up radosgw (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:05:00.329 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up radosgw (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:05:00.644 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up ceph-mgr-cephadm (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:05:00.728 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-21T22:05:00.729 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-21T22:05:00.770 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up ceph (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:05:00.818 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-21T22:05:00.819 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-21T22:05:00.879 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up ceph-mgr-dashboard (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:05:01.030 INFO:teuthology.orchestra.run.smithi157.stdout:Setting up ceph-volume (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:05:01.208 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up ceph-base (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:05:01.299 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up ceph-base (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:05:01.391 INFO:teuthology.orchestra.run.smithi157.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-09-21T22:05:01.500 INFO:teuthology.orchestra.run.smithi157.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-09-21T22:05:01.640 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-21T22:05:01.653 INFO:teuthology.orchestra.run.smithi157.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-09-21T22:05:01.711 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-21T22:05:02.103 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up ceph-mds (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:05:02.215 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up ceph-mds (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:05:02.451 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-21T22:05:02.451 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-21T22:05:02.460 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-21T22:05:02.460 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-21T22:05:02.812 INFO:teuthology.orchestra.run.smithi157.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-09-21T22:05:02.920 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up ceph-mgr (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:05:02.951 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up ceph-mgr (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:05:03.142 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-21T22:05:03.143 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-21T22:05:03.181 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-21T22:05:03.181 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-21T22:05:03.479 INFO:teuthology.orchestra.run.smithi157.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-21T22:05:03.484 DEBUG:teuthology.parallel:result is None 2024-09-21T22:05:03.645 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up ceph-osd (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:05:03.671 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up ceph-osd (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:05:03.938 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-21T22:05:03.938 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-21T22:05:03.960 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-21T22:05:03.960 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-21T22:05:04.414 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up ceph-mgr-k8sevents (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:05:04.456 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up ceph-mgr-k8sevents (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:05:04.540 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up ceph-mgr-diskprediction-local (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:05:04.599 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up ceph-mgr-diskprediction-local (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:05:04.691 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up ceph-mon (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:05:04.758 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up ceph-mon (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:05:04.898 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-21T22:05:04.898 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-21T22:05:04.991 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-21T22:05:04.992 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-21T22:05:05.359 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up ceph-mgr-cephadm (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:05:05.442 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up ceph-mgr-cephadm (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:05:05.619 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up ceph (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:05:05.745 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up ceph-mgr-dashboard (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:05:05.877 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up ceph (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:05:05.896 INFO:teuthology.orchestra.run.smithi064.stdout:Setting up ceph-volume (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:05:06.004 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up ceph-mgr-dashboard (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:05:06.130 INFO:teuthology.orchestra.run.smithi039.stdout:Setting up ceph-volume (19.1.1-373-g4e45e6b4-1jammy) ... 2024-09-21T22:05:06.257 INFO:teuthology.orchestra.run.smithi064.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-09-21T22:05:06.366 INFO:teuthology.orchestra.run.smithi064.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-09-21T22:05:06.500 INFO:teuthology.orchestra.run.smithi039.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-09-21T22:05:06.527 INFO:teuthology.orchestra.run.smithi064.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-09-21T22:05:06.600 INFO:teuthology.orchestra.run.smithi039.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-09-21T22:05:06.761 INFO:teuthology.orchestra.run.smithi039.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-09-21T22:05:07.326 INFO:teuthology.orchestra.run.smithi064.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-09-21T22:05:07.477 INFO:teuthology.orchestra.run.smithi039.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-09-21T22:05:08.021 INFO:teuthology.orchestra.run.smithi064.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-21T22:05:08.025 DEBUG:teuthology.parallel:result is None 2024-09-21T22:05:08.121 INFO:teuthology.orchestra.run.smithi039.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-21T22:05:08.126 DEBUG:teuthology.parallel:result is None 2024-09-21T22:05:08.126 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=4e45e6b490e6cbe082cdf155fea780438ec53fc7 2024-09-21T22:05:08.272 DEBUG:teuthology.orchestra.run.smithi039:> dpkg-query -W -f '${Version}' ceph 2024-09-21T22:05:08.292 INFO:teuthology.orchestra.run.smithi039.stdout:19.1.1-373-g4e45e6b4-1jammy 2024-09-21T22:05:08.292 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-373-g4e45e6b4-1jammy 2024-09-21T22:05:08.292 INFO:teuthology.task.install:The correct ceph version 19.1.1-373-g4e45e6b4-1jammy is installed. 2024-09-21T22:05:08.294 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=4e45e6b490e6cbe082cdf155fea780438ec53fc7 2024-09-21T22:05:08.426 DEBUG:teuthology.orchestra.run.smithi064:> dpkg-query -W -f '${Version}' ceph 2024-09-21T22:05:08.446 INFO:teuthology.orchestra.run.smithi064.stdout:19.1.1-373-g4e45e6b4-1jammy 2024-09-21T22:05:08.446 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-373-g4e45e6b4-1jammy 2024-09-21T22:05:08.446 INFO:teuthology.task.install:The correct ceph version 19.1.1-373-g4e45e6b4-1jammy is installed. 2024-09-21T22:05:08.447 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=4e45e6b490e6cbe082cdf155fea780438ec53fc7 2024-09-21T22:05:08.576 DEBUG:teuthology.orchestra.run.smithi157:> dpkg-query -W -f '${Version}' ceph 2024-09-21T22:05:08.595 INFO:teuthology.orchestra.run.smithi157.stdout:19.1.1-373-g4e45e6b4-1jammy 2024-09-21T22:05:08.595 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-373-g4e45e6b4-1jammy 2024-09-21T22:05:08.596 INFO:teuthology.task.install:The correct ceph version 19.1.1-373-g4e45e6b4-1jammy is installed. 2024-09-21T22:05:08.597 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-21T22:05:08.597 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-21T22:05:08.597 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-21T22:05:08.612 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-09-21T22:05:08.612 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-21T22:05:08.626 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-09-21T22:05:08.626 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-21T22:05:08.652 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-21T22:05:08.652 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-21T22:05:08.653 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/usr/bin/daemon-helper 2024-09-21T22:05:08.671 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-21T22:05:08.731 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-09-21T22:05:08.731 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/usr/bin/daemon-helper 2024-09-21T22:05:08.745 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-21T22:05:08.800 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-09-21T22:05:08.800 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/usr/bin/daemon-helper 2024-09-21T22:05:08.813 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-21T22:05:08.866 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-21T22:05:08.866 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-21T22:05:08.867 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-21T22:05:08.882 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-21T22:05:08.935 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-09-21T22:05:08.935 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-21T22:05:08.949 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-21T22:05:09.005 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-09-21T22:05:09.005 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-21T22:05:09.020 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-21T22:05:09.074 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-21T22:05:09.074 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-21T22:05:09.074 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/usr/bin/stdin-killer 2024-09-21T22:05:09.088 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-21T22:05:09.143 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-09-21T22:05:09.143 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/usr/bin/stdin-killer 2024-09-21T22:05:09.157 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-21T22:05:09.212 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-09-21T22:05:09.212 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/usr/bin/stdin-killer 2024-09-21T22:05:09.225 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-21T22:05:09.279 INFO:teuthology.run_tasks:Running task ceph... 2024-09-21T22:05:09.346 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-21T22:05:09.347 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod 777 /var/log/ceph 2024-09-21T22:05:09.349 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 777 /var/log/ceph 2024-09-21T22:05:09.352 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod 777 /var/log/ceph 2024-09-21T22:05:09.363 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-21T22:05:09.364 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-21T22:05:09.404 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-21T22:05:09.407 DEBUG:teuthology.orchestra.run.smithi157:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-21T22:05:09.418 INFO:tasks.ceph:Creating extra log directories... 2024-09-21T22:05:09.418 DEBUG:teuthology.orchestra.run.smithi039:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-21T22:05:09.460 DEBUG:teuthology.orchestra.run.smithi064:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-21T22:05:09.463 DEBUG:teuthology.orchestra.run.smithi157:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-21T22:05:09.484 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-21T22:05:09.484 INFO:tasks.ceph:config {'conf': {'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-21T22:05:09.484 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-09-21_21:16:02-rbd-squid-distro-default-smithi/7914449', 'branch': 'squid', 'description': 'rbd/device/{base/install cluster/{fixed-3 openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-zlib supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/rbd_nbd}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7914449', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'teuthology-2024-09-21_21:16:02-rbd-squid-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'machine': {'cpus': 1, 'disk': 40, 'ram': 8000}, 'volumes': {'count': 4, 'size': 30}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'squid'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub'], 'sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'extra_packages': ['rbd-nbd'], 'flavor': 'default', 'sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'squid', 'sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 100, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'], ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5'], ['client.0']], 'seed': 1478, 'sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7', 'sleep_before_teardown': 0, 'subset': '24/128', 'suite': 'rbd', 'suite_branch': 'squid', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_4e45e6b490e6cbe082cdf155fea780438ec53fc7/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7', 'targets': {'smithi039.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDyux0JgXffDOQ+nsfEFEOKhwO/X/Bt8Gb5yN2jFdTItcwW1aLPNohY6/z+PMzIFLYvjZDYQH/kYCusuWbvP+D4=', 'smithi064.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBF6erKmGp6kmZeoAG9Ue6BM47Vn1pF+/v2hXFcXpv+3xBTC7fMspop7VDOdLmI8TlUIiP0sazN9pCrUaICT4WBo=', 'smithi157.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCM/xq8VesaVmUNWpxZDaK/NfCdrVkcERr1dVpooeRzYtZcb11Pn1TycotbeDAvEhSjn0DJD0j/kfDXkMuIOxRY='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'thrashosds': {'timeout': 1200}}, {'workunit': {'clients': {'client.0': ['rbd/rbd-nbd.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-21_21:16:02', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2902'} 2024-09-21T22:05:09.485 DEBUG:teuthology.orchestra.run.smithi039:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-21T22:05:09.517 DEBUG:teuthology.orchestra.run.smithi064:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-21T22:05:09.519 DEBUG:teuthology.orchestra.run.smithi157:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-21T22:05:09.525 DEBUG:teuthology.orchestra.run.smithi039:> sudo install -d -m0777 -- /var/run/ceph 2024-09-21T22:05:09.564 DEBUG:teuthology.orchestra.run.smithi064:> sudo install -d -m0777 -- /var/run/ceph 2024-09-21T22:05:09.567 DEBUG:teuthology.orchestra.run.smithi157:> sudo install -d -m0777 -- /var/run/ceph 2024-09-21T22:05:09.580 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-21T22:05:09.580 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/scratch_devs of=/dev/stdout 2024-09-21T22:05:09.623 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-21T22:05:09.623 DEBUG:teuthology.orchestra.run.smithi039:> stat /dev/vg_nvme/lv_1 2024-09-21T22:05:09.671 INFO:teuthology.orchestra.run.smithi039.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-21T22:05:09.671 INFO:teuthology.orchestra.run.smithi039.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T22:05:09.672 INFO:teuthology.orchestra.run.smithi039.stdout:Device: 5h/5d Inode: 851 Links: 1 2024-09-21T22:05:09.672 INFO:teuthology.orchestra.run.smithi039.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T22:05:09.672 INFO:teuthology.orchestra.run.smithi039.stdout:Access: 2024-09-21 21:59:37.654066332 +0000 2024-09-21T22:05:09.672 INFO:teuthology.orchestra.run.smithi039.stdout:Modify: 2024-09-21 21:59:37.414068477 +0000 2024-09-21T22:05:09.672 INFO:teuthology.orchestra.run.smithi039.stdout:Change: 2024-09-21 21:59:37.414068477 +0000 2024-09-21T22:05:09.672 INFO:teuthology.orchestra.run.smithi039.stdout: Birth: - 2024-09-21T22:05:09.672 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-21T22:05:09.726 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records in 2024-09-21T22:05:09.726 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records out 2024-09-21T22:05:09.727 INFO:teuthology.orchestra.run.smithi039.stderr:512 bytes copied, 0.000553956 s, 924 kB/s 2024-09-21T22:05:09.728 DEBUG:teuthology.orchestra.run.smithi039:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-21T22:05:09.777 DEBUG:teuthology.orchestra.run.smithi039:> stat /dev/vg_nvme/lv_2 2024-09-21T22:05:09.823 INFO:teuthology.orchestra.run.smithi039.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-21T22:05:09.824 INFO:teuthology.orchestra.run.smithi039.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T22:05:09.824 INFO:teuthology.orchestra.run.smithi039.stdout:Device: 5h/5d Inode: 869 Links: 1 2024-09-21T22:05:09.824 INFO:teuthology.orchestra.run.smithi039.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T22:05:09.824 INFO:teuthology.orchestra.run.smithi039.stdout:Access: 2024-09-21 21:59:38.186061577 +0000 2024-09-21T22:05:09.824 INFO:teuthology.orchestra.run.smithi039.stdout:Modify: 2024-09-21 21:59:37.910064044 +0000 2024-09-21T22:05:09.824 INFO:teuthology.orchestra.run.smithi039.stdout:Change: 2024-09-21 21:59:37.910064044 +0000 2024-09-21T22:05:09.824 INFO:teuthology.orchestra.run.smithi039.stdout: Birth: - 2024-09-21T22:05:09.824 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-21T22:05:09.878 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records in 2024-09-21T22:05:09.879 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records out 2024-09-21T22:05:09.879 INFO:teuthology.orchestra.run.smithi039.stderr:512 bytes copied, 0.00047577 s, 1.1 MB/s 2024-09-21T22:05:09.880 DEBUG:teuthology.orchestra.run.smithi039:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-21T22:05:09.929 DEBUG:teuthology.orchestra.run.smithi039:> stat /dev/vg_nvme/lv_3 2024-09-21T22:05:09.976 INFO:teuthology.orchestra.run.smithi039.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-21T22:05:09.976 INFO:teuthology.orchestra.run.smithi039.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T22:05:09.976 INFO:teuthology.orchestra.run.smithi039.stdout:Device: 5h/5d Inode: 892 Links: 1 2024-09-21T22:05:09.976 INFO:teuthology.orchestra.run.smithi039.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T22:05:09.976 INFO:teuthology.orchestra.run.smithi039.stdout:Access: 2024-09-21 21:59:38.710056894 +0000 2024-09-21T22:05:09.976 INFO:teuthology.orchestra.run.smithi039.stdout:Modify: 2024-09-21 21:59:38.462059110 +0000 2024-09-21T22:05:09.977 INFO:teuthology.orchestra.run.smithi039.stdout:Change: 2024-09-21 21:59:38.462059110 +0000 2024-09-21T22:05:09.977 INFO:teuthology.orchestra.run.smithi039.stdout: Birth: - 2024-09-21T22:05:09.977 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-21T22:05:10.031 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records in 2024-09-21T22:05:10.031 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records out 2024-09-21T22:05:10.031 INFO:teuthology.orchestra.run.smithi039.stderr:512 bytes copied, 0.000548241 s, 934 kB/s 2024-09-21T22:05:10.032 DEBUG:teuthology.orchestra.run.smithi039:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-21T22:05:10.081 DEBUG:teuthology.orchestra.run.smithi039:> stat /dev/vg_nvme/lv_4 2024-09-21T22:05:10.128 INFO:teuthology.orchestra.run.smithi039.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-21T22:05:10.128 INFO:teuthology.orchestra.run.smithi039.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T22:05:10.128 INFO:teuthology.orchestra.run.smithi039.stdout:Device: 5h/5d Inode: 909 Links: 1 2024-09-21T22:05:10.128 INFO:teuthology.orchestra.run.smithi039.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T22:05:10.128 INFO:teuthology.orchestra.run.smithi039.stdout:Access: 2024-09-21 21:59:39.234052210 +0000 2024-09-21T22:05:10.128 INFO:teuthology.orchestra.run.smithi039.stdout:Modify: 2024-09-21 21:59:38.958054677 +0000 2024-09-21T22:05:10.128 INFO:teuthology.orchestra.run.smithi039.stdout:Change: 2024-09-21 21:59:38.958054677 +0000 2024-09-21T22:05:10.128 INFO:teuthology.orchestra.run.smithi039.stdout: Birth: - 2024-09-21T22:05:10.128 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-21T22:05:10.182 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records in 2024-09-21T22:05:10.183 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records out 2024-09-21T22:05:10.183 INFO:teuthology.orchestra.run.smithi039.stderr:512 bytes copied, 0.00044626 s, 1.1 MB/s 2024-09-21T22:05:10.184 DEBUG:teuthology.orchestra.run.smithi039:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-21T22:05:10.233 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-21T22:05:10.233 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-09-21T22:05:10.233 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/scratch_devs of=/dev/stdout 2024-09-21T22:05:10.239 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-21T22:05:10.239 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_1 2024-09-21T22:05:10.285 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-21T22:05:10.285 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T22:05:10.285 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 5h/5d Inode: 883 Links: 1 2024-09-21T22:05:10.285 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T22:05:10.285 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2024-09-21 21:59:37.330366522 +0000 2024-09-21T22:05:10.286 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2024-09-21 21:59:37.078368773 +0000 2024-09-21T22:05:10.286 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2024-09-21 21:59:37.078368773 +0000 2024-09-21T22:05:10.286 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: - 2024-09-21T22:05:10.286 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-21T22:05:10.339 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2024-09-21T22:05:10.340 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2024-09-21T22:05:10.340 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000422084 s, 1.2 MB/s 2024-09-21T22:05:10.341 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-21T22:05:10.390 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_2 2024-09-21T22:05:10.437 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-21T22:05:10.437 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T22:05:10.437 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 5h/5d Inode: 897 Links: 1 2024-09-21T22:05:10.437 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T22:05:10.437 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2024-09-21 21:59:37.562364450 +0000 2024-09-21T22:05:10.437 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2024-09-21 21:59:37.558364486 +0000 2024-09-21T22:05:10.437 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2024-09-21 21:59:37.558364486 +0000 2024-09-21T22:05:10.437 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: - 2024-09-21T22:05:10.437 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-21T22:05:10.491 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2024-09-21T22:05:10.492 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2024-09-21T22:05:10.492 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000418261 s, 1.2 MB/s 2024-09-21T22:05:10.493 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-21T22:05:10.542 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_3 2024-09-21T22:05:10.589 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-21T22:05:10.589 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T22:05:10.589 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 5h/5d Inode: 915 Links: 1 2024-09-21T22:05:10.589 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T22:05:10.589 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2024-09-21 21:59:38.254358269 +0000 2024-09-21T22:05:10.589 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2024-09-21 21:59:38.026360306 +0000 2024-09-21T22:05:10.589 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2024-09-21 21:59:38.026360306 +0000 2024-09-21T22:05:10.590 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: - 2024-09-21T22:05:10.590 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-21T22:05:10.643 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2024-09-21T22:05:10.643 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2024-09-21T22:05:10.644 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000410083 s, 1.2 MB/s 2024-09-21T22:05:10.645 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-21T22:05:10.694 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_4 2024-09-21T22:05:10.741 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-21T22:05:10.741 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T22:05:10.741 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 5h/5d Inode: 936 Links: 1 2024-09-21T22:05:10.741 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T22:05:10.741 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2024-09-21 21:59:38.710354196 +0000 2024-09-21T22:05:10.741 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2024-09-21 21:59:38.486356197 +0000 2024-09-21T22:05:10.741 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2024-09-21 21:59:38.486356197 +0000 2024-09-21T22:05:10.741 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: - 2024-09-21T22:05:10.742 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-21T22:05:10.795 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2024-09-21T22:05:10.795 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2024-09-21T22:05:10.795 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000337022 s, 1.5 MB/s 2024-09-21T22:05:10.796 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-21T22:05:10.845 INFO:tasks.ceph:osd dev map: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-21T22:05:10.846 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi039.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi064.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2024-09-21T22:05:10.846 INFO:tasks.ceph:Generating config... 2024-09-21T22:05:10.847 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-21T22:05:10.847 INFO:tasks.ceph:[global] mon warn on pool no app = False 2024-09-21T22:05:10.847 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-09-21T22:05:10.847 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-21T22:05:10.847 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-21T22:05:10.847 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-21T22:05:10.847 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-21T22:05:10.847 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-21T22:05:10.848 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-21T22:05:10.848 INFO:tasks.ceph:[osd] bluestore compression algorithm = zlib 2024-09-21T22:05:10.848 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-09-21T22:05:10.848 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-21T22:05:10.848 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-09-21T22:05:10.848 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-09-21T22:05:10.848 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-21T22:05:10.848 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-21T22:05:10.848 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-09-21T22:05:10.848 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-21T22:05:10.848 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-21T22:05:10.848 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-21T22:05:10.849 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-09-21T22:05:10.849 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-21T22:05:10.849 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-09-21T22:05:10.849 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-21T22:05:10.849 INFO:tasks.ceph:Setting up mon.a... 2024-09-21T22:05:10.849 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-21T22:05:10.885 INFO:teuthology.orchestra.run.smithi039.stdout:creating /etc/ceph/ceph.keyring 2024-09-21T22:05:10.899 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-21T22:05:10.981 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-21T22:05:11.035 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.39'), ('mon.c', '[v2:172.21.15.39:3301,v1:172.21.15.39:6790]'), ('mon.b', '172.21.15.64')] 2024-09-21T22:05:11.035 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.39,[v2:172.21.15.39:3301,v1:172.21.15.39:6790],172.21.15.64', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-09-21T22:05:11.036 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-21T22:05:11.036 DEBUG:teuthology.orchestra.run.smithi039:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-21T22:05:11.087 DEBUG:teuthology.orchestra.run.smithi039:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.39 --addv c '[v2:172.21.15.39:3301,v1:172.21.15.39:6790]' --add b 172.21.15.64 --print /home/ubuntu/cephtest/ceph.monmap 2024-09-21T22:05:11.174 INFO:teuthology.orchestra.run.smithi039.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-21T22:05:11.174 INFO:teuthology.orchestra.run.smithi039.stdout:monmaptool: generated fsid 00bb09a3-e410-4e18-a2b2-08eaf61978e6 2024-09-21T22:05:11.175 INFO:teuthology.orchestra.run.smithi039.stdout:setting min_mon_release = quincy 2024-09-21T22:05:11.175 INFO:teuthology.orchestra.run.smithi039.stdout:epoch 0 2024-09-21T22:05:11.175 INFO:teuthology.orchestra.run.smithi039.stdout:fsid 00bb09a3-e410-4e18-a2b2-08eaf61978e6 2024-09-21T22:05:11.175 INFO:teuthology.orchestra.run.smithi039.stdout:last_changed 2024-09-21T22:05:11.174825+0000 2024-09-21T22:05:11.175 INFO:teuthology.orchestra.run.smithi039.stdout:created 2024-09-21T22:05:11.174825+0000 2024-09-21T22:05:11.175 INFO:teuthology.orchestra.run.smithi039.stdout:min_mon_release 17 (quincy) 2024-09-21T22:05:11.175 INFO:teuthology.orchestra.run.smithi039.stdout:election_strategy: 1 2024-09-21T22:05:11.175 INFO:teuthology.orchestra.run.smithi039.stdout:0: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.a 2024-09-21T22:05:11.175 INFO:teuthology.orchestra.run.smithi039.stdout:1: [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon.b 2024-09-21T22:05:11.175 INFO:teuthology.orchestra.run.smithi039.stdout:2: [v2:172.21.15.39:3301/0,v1:172.21.15.39:6790/0] mon.c 2024-09-21T22:05:11.175 INFO:teuthology.orchestra.run.smithi039.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-09-21T22:05:11.176 DEBUG:teuthology.orchestra.run.smithi039:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-21T22:05:11.224 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 00bb09a3-e410-4e18-a2b2-08eaf61978e6... 2024-09-21T22:05:11.224 DEBUG:teuthology.orchestra.run.smithi039:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-21T22:05:11.269 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-21T22:05:11.271 DEBUG:teuthology.orchestra.run.smithi157:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-21T22:05:11.298 INFO:teuthology.orchestra.run.smithi039.stdout:[global] 2024-09-21T22:05:11.298 INFO:teuthology.orchestra.run.smithi039.stdout: chdir = "" 2024-09-21T22:05:11.298 INFO:teuthology.orchestra.run.smithi039.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-21T22:05:11.298 INFO:teuthology.orchestra.run.smithi039.stdout: auth supported = cephx 2024-09-21T22:05:11.298 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T22:05:11.298 INFO:teuthology.orchestra.run.smithi039.stdout: filestore xattr use omap = true 2024-09-21T22:05:11.298 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T22:05:11.298 INFO:teuthology.orchestra.run.smithi039.stdout: mon clock drift allowed = 1.000 2024-09-21T22:05:11.299 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T22:05:11.299 INFO:teuthology.orchestra.run.smithi039.stdout: osd crush chooseleaf type = 0 2024-09-21T22:05:11.299 INFO:teuthology.orchestra.run.smithi039.stdout: auth debug = true 2024-09-21T22:05:11.299 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T22:05:11.299 INFO:teuthology.orchestra.run.smithi039.stdout: ms die on old message = true 2024-09-21T22:05:11.299 INFO:teuthology.orchestra.run.smithi039.stdout: ms die on bug = true 2024-09-21T22:05:11.299 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T22:05:11.299 INFO:teuthology.orchestra.run.smithi039.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-21T22:05:11.299 INFO:teuthology.orchestra.run.smithi039.stdout: mon pg warn max object skew = 0 2024-09-21T22:05:11.299 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T22:05:11.299 INFO:teuthology.orchestra.run.smithi039.stdout: # disable pg_autoscaler by default for new pools 2024-09-21T22:05:11.299 INFO:teuthology.orchestra.run.smithi039.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-21T22:05:11.299 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T22:05:11.299 INFO:teuthology.orchestra.run.smithi039.stdout: osd pool default size = 2 2024-09-21T22:05:11.300 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T22:05:11.300 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd allow primary affinity = true 2024-09-21T22:05:11.300 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd allow pg remap = true 2024-09-21T22:05:11.300 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on legacy crush tunables = false 2024-09-21T22:05:11.300 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on crush straw calc version zero = false 2024-09-21T22:05:11.300 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on no sortbitwise = false 2024-09-21T22:05:11.300 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on osd down out interval zero = false 2024-09-21T22:05:11.300 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on too few osds = false 2024-09-21T22:05:11.300 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-21T22:05:11.300 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-21T22:05:11.300 INFO:teuthology.orchestra.run.smithi039.stdout: mon_allow_pool_size_one = true 2024-09-21T22:05:11.300 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T22:05:11.300 INFO:teuthology.orchestra.run.smithi039.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-21T22:05:11.300 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T22:05:11.300 INFO:teuthology.orchestra.run.smithi039.stdout: osd default data pool replay window = 5 2024-09-21T22:05:11.300 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T22:05:11.301 INFO:teuthology.orchestra.run.smithi039.stdout: mon allow pool delete = true 2024-09-21T22:05:11.301 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T22:05:11.301 INFO:teuthology.orchestra.run.smithi039.stdout: mon cluster log file level = debug 2024-09-21T22:05:11.301 INFO:teuthology.orchestra.run.smithi039.stdout: debug asserts on shutdown = true 2024-09-21T22:05:11.301 INFO:teuthology.orchestra.run.smithi039.stdout: mon health detail to clog = false 2024-09-21T22:05:11.301 INFO:teuthology.orchestra.run.smithi039.stdout: mon host = "172.21.15.39,[v2:172.21.15.39:3301,v1:172.21.15.39:6790],172.21.15.64" 2024-09-21T22:05:11.301 INFO:teuthology.orchestra.run.smithi039.stdout: mon client directed command retry = 5 2024-09-21T22:05:11.301 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on pool no app = False 2024-09-21T22:05:11.301 INFO:teuthology.orchestra.run.smithi039.stdout: ms inject socket failures = 5000 2024-09-21T22:05:11.301 INFO:teuthology.orchestra.run.smithi039.stdout: fsid = 00bb09a3-e410-4e18-a2b2-08eaf61978e6 2024-09-21T22:05:11.301 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T22:05:11.301 INFO:teuthology.orchestra.run.smithi039.stdout:[osd] 2024-09-21T22:05:11.301 INFO:teuthology.orchestra.run.smithi039.stdout: osd journal size = 100 2024-09-21T22:05:11.301 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T22:05:11.302 INFO:teuthology.orchestra.run.smithi039.stdout: osd scrub load threshold = 5.0 2024-09-21T22:05:11.302 INFO:teuthology.orchestra.run.smithi039.stdout: osd scrub max interval = 600 2024-09-21T22:05:11.302 INFO:teuthology.orchestra.run.smithi039.stdout: osd mclock profile = high_recovery_ops 2024-09-21T22:05:11.302 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T22:05:11.302 INFO:teuthology.orchestra.run.smithi039.stdout: osd recover clone overlap = true 2024-09-21T22:05:11.302 INFO:teuthology.orchestra.run.smithi039.stdout: osd recovery max chunk = 1048576 2024-09-21T22:05:11.302 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T22:05:11.302 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug shutdown = true 2024-09-21T22:05:11.302 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug op order = true 2024-09-21T22:05:11.302 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug verify stray on activate = true 2024-09-21T22:05:11.302 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug trim objects = true 2024-09-21T22:05:11.302 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T22:05:11.302 INFO:teuthology.orchestra.run.smithi039.stdout: osd open classes on start = true 2024-09-21T22:05:11.302 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug pg log writeout = true 2024-09-21T22:05:11.302 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T22:05:11.303 INFO:teuthology.orchestra.run.smithi039.stdout: osd deep scrub update digest min age = 30 2024-09-21T22:05:11.303 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T22:05:11.303 INFO:teuthology.orchestra.run.smithi039.stdout: osd map max advance = 10 2024-09-21T22:05:11.303 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T22:05:11.303 INFO:teuthology.orchestra.run.smithi039.stdout: journal zero on create = true 2024-09-21T22:05:11.303 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T22:05:11.303 INFO:teuthology.orchestra.run.smithi039.stdout: filestore ondisk finisher threads = 3 2024-09-21T22:05:11.303 INFO:teuthology.orchestra.run.smithi039.stdout: filestore apply finisher threads = 3 2024-09-21T22:05:11.303 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T22:05:11.303 INFO:teuthology.orchestra.run.smithi039.stdout: bdev debug aio = true 2024-09-21T22:05:11.303 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug misdirected ops = true 2024-09-21T22:05:11.303 INFO:teuthology.orchestra.run.smithi039.stdout: bluestore block size = 96636764160 2024-09-21T22:05:11.303 INFO:teuthology.orchestra.run.smithi039.stdout: bluestore compression algorithm = zlib 2024-09-21T22:05:11.303 INFO:teuthology.orchestra.run.smithi039.stdout: bluestore compression mode = aggressive 2024-09-21T22:05:11.303 INFO:teuthology.orchestra.run.smithi039.stdout: bluestore fsck on mount = True 2024-09-21T22:05:11.303 INFO:teuthology.orchestra.run.smithi039.stdout: debug bluefs = 1/20 2024-09-21T22:05:11.304 INFO:teuthology.orchestra.run.smithi039.stdout: debug bluestore = 1/20 2024-09-21T22:05:11.304 INFO:teuthology.orchestra.run.smithi039.stdout: debug ms = 1 2024-09-21T22:05:11.304 INFO:teuthology.orchestra.run.smithi039.stdout: debug osd = 20 2024-09-21T22:05:11.304 INFO:teuthology.orchestra.run.smithi039.stdout: debug rocksdb = 4/10 2024-09-21T22:05:11.304 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd backfillfull_ratio = 0.85 2024-09-21T22:05:11.304 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd full ratio = 0.9 2024-09-21T22:05:11.304 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd nearfull ratio = 0.8 2024-09-21T22:05:11.304 INFO:teuthology.orchestra.run.smithi039.stdout: osd blocked scrub grace period = 3600 2024-09-21T22:05:11.304 INFO:teuthology.orchestra.run.smithi039.stdout: osd failsafe full ratio = 0.95 2024-09-21T22:05:11.304 INFO:teuthology.orchestra.run.smithi039.stdout: osd max markdown count = 1000 2024-09-21T22:05:11.304 INFO:teuthology.orchestra.run.smithi039.stdout: osd objectstore = bluestore 2024-09-21T22:05:11.304 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T22:05:11.304 INFO:teuthology.orchestra.run.smithi039.stdout:[mgr] 2024-09-21T22:05:11.304 INFO:teuthology.orchestra.run.smithi039.stdout: debug ms = 1 2024-09-21T22:05:11.304 INFO:teuthology.orchestra.run.smithi039.stdout: debug mgr = 20 2024-09-21T22:05:11.304 INFO:teuthology.orchestra.run.smithi039.stdout: debug mon = 20 2024-09-21T22:05:11.305 INFO:teuthology.orchestra.run.smithi039.stdout: debug auth = 20 2024-09-21T22:05:11.305 INFO:teuthology.orchestra.run.smithi039.stdout: mon reweight min pgs per osd = 4 2024-09-21T22:05:11.305 INFO:teuthology.orchestra.run.smithi039.stdout: mon reweight min bytes per osd = 10 2024-09-21T22:05:11.305 INFO:teuthology.orchestra.run.smithi039.stdout: mgr/telemetry/nag = false 2024-09-21T22:05:11.305 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T22:05:11.305 INFO:teuthology.orchestra.run.smithi039.stdout:[mon] 2024-09-21T22:05:11.305 INFO:teuthology.orchestra.run.smithi039.stdout: debug ms = 1 2024-09-21T22:05:11.305 INFO:teuthology.orchestra.run.smithi039.stdout: debug mon = 20 2024-09-21T22:05:11.305 INFO:teuthology.orchestra.run.smithi039.stdout: debug paxos = 20 2024-09-21T22:05:11.305 INFO:teuthology.orchestra.run.smithi039.stdout: debug auth = 20 2024-09-21T22:05:11.305 INFO:teuthology.orchestra.run.smithi039.stdout: mon data avail warn = 5 2024-09-21T22:05:11.305 INFO:teuthology.orchestra.run.smithi039.stdout: mon mgr mkfs grace = 240 2024-09-21T22:05:11.305 INFO:teuthology.orchestra.run.smithi039.stdout: mon reweight min pgs per osd = 4 2024-09-21T22:05:11.305 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd reporter subtree level = osd 2024-09-21T22:05:11.305 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd prime pg temp = true 2024-09-21T22:05:11.306 INFO:teuthology.orchestra.run.smithi039.stdout: mon reweight min bytes per osd = 10 2024-09-21T22:05:11.306 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T22:05:11.306 INFO:teuthology.orchestra.run.smithi039.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-21T22:05:11.306 INFO:teuthology.orchestra.run.smithi039.stdout: auth mon ticket ttl = 660 # 11m 2024-09-21T22:05:11.306 INFO:teuthology.orchestra.run.smithi039.stdout: auth service ticket ttl = 240 # 4m 2024-09-21T22:05:11.306 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T22:05:11.306 INFO:teuthology.orchestra.run.smithi039.stdout: # don't complain about insecure global_id in the test suite 2024-09-21T22:05:11.306 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-21T22:05:11.306 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-21T22:05:11.306 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T22:05:11.306 INFO:teuthology.orchestra.run.smithi039.stdout: # 1m isn't quite enough 2024-09-21T22:05:11.306 INFO:teuthology.orchestra.run.smithi039.stdout: mon_down_mkfs_grace = 2m 2024-09-21T22:05:11.306 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T22:05:11.306 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_filestore_osds = false 2024-09-21T22:05:11.306 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T22:05:11.306 INFO:teuthology.orchestra.run.smithi039.stdout:[client] 2024-09-21T22:05:11.307 INFO:teuthology.orchestra.run.smithi039.stdout: rgw cache enabled = true 2024-09-21T22:05:11.307 INFO:teuthology.orchestra.run.smithi039.stdout: rgw enable ops log = true 2024-09-21T22:05:11.307 INFO:teuthology.orchestra.run.smithi039.stdout: rgw enable usage log = true 2024-09-21T22:05:11.307 INFO:teuthology.orchestra.run.smithi039.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-21T22:05:11.307 INFO:teuthology.orchestra.run.smithi039.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-21T22:05:11.307 INFO:teuthology.orchestra.run.smithi039.stdout:[mon.a] 2024-09-21T22:05:11.307 INFO:teuthology.orchestra.run.smithi039.stdout:[mon.c] 2024-09-21T22:05:11.307 INFO:teuthology.orchestra.run.smithi039.stdout:[mon.b] 2024-09-21T22:05:11.308 INFO:teuthology.orchestra.run.smithi157.stdout:[global] 2024-09-21T22:05:11.308 INFO:teuthology.orchestra.run.smithi157.stdout: chdir = "" 2024-09-21T22:05:11.308 INFO:teuthology.orchestra.run.smithi157.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-21T22:05:11.308 INFO:teuthology.orchestra.run.smithi157.stdout: auth supported = cephx 2024-09-21T22:05:11.308 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-21T22:05:11.308 INFO:teuthology.orchestra.run.smithi157.stdout: filestore xattr use omap = true 2024-09-21T22:05:11.308 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-21T22:05:11.308 INFO:teuthology.orchestra.run.smithi157.stdout: mon clock drift allowed = 1.000 2024-09-21T22:05:11.308 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-21T22:05:11.308 INFO:teuthology.orchestra.run.smithi157.stdout: osd crush chooseleaf type = 0 2024-09-21T22:05:11.308 INFO:teuthology.orchestra.run.smithi157.stdout: auth debug = true 2024-09-21T22:05:11.309 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-21T22:05:11.309 INFO:teuthology.orchestra.run.smithi157.stdout: ms die on old message = true 2024-09-21T22:05:11.309 INFO:teuthology.orchestra.run.smithi157.stdout: ms die on bug = true 2024-09-21T22:05:11.309 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-21T22:05:11.309 INFO:teuthology.orchestra.run.smithi157.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-21T22:05:11.309 INFO:teuthology.orchestra.run.smithi157.stdout: mon pg warn max object skew = 0 2024-09-21T22:05:11.309 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-21T22:05:11.309 INFO:teuthology.orchestra.run.smithi157.stdout: # disable pg_autoscaler by default for new pools 2024-09-21T22:05:11.309 INFO:teuthology.orchestra.run.smithi157.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-21T22:05:11.309 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-21T22:05:11.309 INFO:teuthology.orchestra.run.smithi157.stdout: osd pool default size = 2 2024-09-21T22:05:11.309 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-21T22:05:11.309 INFO:teuthology.orchestra.run.smithi157.stdout: mon osd allow primary affinity = true 2024-09-21T22:05:11.309 INFO:teuthology.orchestra.run.smithi157.stdout: mon osd allow pg remap = true 2024-09-21T22:05:11.310 INFO:teuthology.orchestra.run.smithi157.stdout: mon warn on legacy crush tunables = false 2024-09-21T22:05:11.310 INFO:teuthology.orchestra.run.smithi157.stdout: mon warn on crush straw calc version zero = false 2024-09-21T22:05:11.310 INFO:teuthology.orchestra.run.smithi157.stdout: mon warn on no sortbitwise = false 2024-09-21T22:05:11.310 INFO:teuthology.orchestra.run.smithi157.stdout: mon warn on osd down out interval zero = false 2024-09-21T22:05:11.310 INFO:teuthology.orchestra.run.smithi157.stdout: mon warn on too few osds = false 2024-09-21T22:05:11.310 INFO:teuthology.orchestra.run.smithi157.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-21T22:05:11.310 INFO:teuthology.orchestra.run.smithi157.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-21T22:05:11.310 INFO:teuthology.orchestra.run.smithi157.stdout: mon_allow_pool_size_one = true 2024-09-21T22:05:11.310 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-21T22:05:11.310 INFO:teuthology.orchestra.run.smithi157.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-21T22:05:11.310 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-21T22:05:11.310 INFO:teuthology.orchestra.run.smithi157.stdout: osd default data pool replay window = 5 2024-09-21T22:05:11.310 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-21T22:05:11.311 INFO:teuthology.orchestra.run.smithi157.stdout: mon allow pool delete = true 2024-09-21T22:05:11.311 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-21T22:05:11.311 INFO:teuthology.orchestra.run.smithi157.stdout: mon cluster log file level = debug 2024-09-21T22:05:11.311 INFO:teuthology.orchestra.run.smithi157.stdout: debug asserts on shutdown = true 2024-09-21T22:05:11.311 INFO:teuthology.orchestra.run.smithi157.stdout: mon health detail to clog = false 2024-09-21T22:05:11.311 INFO:teuthology.orchestra.run.smithi157.stdout: mon host = "172.21.15.39,[v2:172.21.15.39:3301,v1:172.21.15.39:6790],172.21.15.64" 2024-09-21T22:05:11.311 INFO:teuthology.orchestra.run.smithi157.stdout: mon client directed command retry = 5 2024-09-21T22:05:11.311 INFO:teuthology.orchestra.run.smithi157.stdout: mon warn on pool no app = False 2024-09-21T22:05:11.311 INFO:teuthology.orchestra.run.smithi157.stdout: ms inject socket failures = 5000 2024-09-21T22:05:11.311 INFO:teuthology.orchestra.run.smithi157.stdout: fsid = 00bb09a3-e410-4e18-a2b2-08eaf61978e6 2024-09-21T22:05:11.311 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-21T22:05:11.311 INFO:teuthology.orchestra.run.smithi157.stdout:[osd] 2024-09-21T22:05:11.311 INFO:teuthology.orchestra.run.smithi157.stdout: osd journal size = 100 2024-09-21T22:05:11.311 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-21T22:05:11.311 INFO:teuthology.orchestra.run.smithi157.stdout: osd scrub load threshold = 5.0 2024-09-21T22:05:11.311 INFO:teuthology.orchestra.run.smithi157.stdout: osd scrub max interval = 600 2024-09-21T22:05:11.312 INFO:teuthology.orchestra.run.smithi157.stdout: osd mclock profile = high_recovery_ops 2024-09-21T22:05:11.312 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-21T22:05:11.312 INFO:teuthology.orchestra.run.smithi157.stdout: osd recover clone overlap = true 2024-09-21T22:05:11.312 INFO:teuthology.orchestra.run.smithi157.stdout: osd recovery max chunk = 1048576 2024-09-21T22:05:11.312 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-21T22:05:11.312 INFO:teuthology.orchestra.run.smithi157.stdout: osd debug shutdown = true 2024-09-21T22:05:11.312 INFO:teuthology.orchestra.run.smithi157.stdout: osd debug op order = true 2024-09-21T22:05:11.312 INFO:teuthology.orchestra.run.smithi157.stdout: osd debug verify stray on activate = true 2024-09-21T22:05:11.312 INFO:teuthology.orchestra.run.smithi157.stdout: osd debug trim objects = true 2024-09-21T22:05:11.312 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-21T22:05:11.312 INFO:teuthology.orchestra.run.smithi157.stdout: osd open classes on start = true 2024-09-21T22:05:11.312 INFO:teuthology.orchestra.run.smithi157.stdout: osd debug pg log writeout = true 2024-09-21T22:05:11.312 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-21T22:05:11.312 INFO:teuthology.orchestra.run.smithi157.stdout: osd deep scrub update digest min age = 30 2024-09-21T22:05:11.312 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-21T22:05:11.313 INFO:teuthology.orchestra.run.smithi157.stdout: osd map max advance = 10 2024-09-21T22:05:11.313 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-21T22:05:11.313 INFO:teuthology.orchestra.run.smithi157.stdout: journal zero on create = true 2024-09-21T22:05:11.313 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-21T22:05:11.313 INFO:teuthology.orchestra.run.smithi157.stdout: filestore ondisk finisher threads = 3 2024-09-21T22:05:11.313 INFO:teuthology.orchestra.run.smithi157.stdout: filestore apply finisher threads = 3 2024-09-21T22:05:11.313 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-21T22:05:11.313 INFO:teuthology.orchestra.run.smithi157.stdout: bdev debug aio = true 2024-09-21T22:05:11.313 INFO:teuthology.orchestra.run.smithi157.stdout: osd debug misdirected ops = true 2024-09-21T22:05:11.313 INFO:teuthology.orchestra.run.smithi157.stdout: bluestore block size = 96636764160 2024-09-21T22:05:11.313 INFO:teuthology.orchestra.run.smithi157.stdout: bluestore compression algorithm = zlib 2024-09-21T22:05:11.313 INFO:teuthology.orchestra.run.smithi157.stdout: bluestore compression mode = aggressive 2024-09-21T22:05:11.313 INFO:teuthology.orchestra.run.smithi157.stdout: bluestore fsck on mount = True 2024-09-21T22:05:11.313 INFO:teuthology.orchestra.run.smithi157.stdout: debug bluefs = 1/20 2024-09-21T22:05:11.313 INFO:teuthology.orchestra.run.smithi157.stdout: debug bluestore = 1/20 2024-09-21T22:05:11.313 INFO:teuthology.orchestra.run.smithi157.stdout: debug ms = 1 2024-09-21T22:05:11.314 INFO:teuthology.orchestra.run.smithi157.stdout: debug osd = 20 2024-09-21T22:05:11.314 INFO:teuthology.orchestra.run.smithi157.stdout: debug rocksdb = 4/10 2024-09-21T22:05:11.314 INFO:teuthology.orchestra.run.smithi157.stdout: mon osd backfillfull_ratio = 0.85 2024-09-21T22:05:11.314 INFO:teuthology.orchestra.run.smithi157.stdout: mon osd full ratio = 0.9 2024-09-21T22:05:11.314 INFO:teuthology.orchestra.run.smithi157.stdout: mon osd nearfull ratio = 0.8 2024-09-21T22:05:11.314 INFO:teuthology.orchestra.run.smithi157.stdout: osd blocked scrub grace period = 3600 2024-09-21T22:05:11.314 INFO:teuthology.orchestra.run.smithi157.stdout: osd failsafe full ratio = 0.95 2024-09-21T22:05:11.314 INFO:teuthology.orchestra.run.smithi157.stdout: osd max markdown count = 1000 2024-09-21T22:05:11.314 INFO:teuthology.orchestra.run.smithi157.stdout: osd objectstore = bluestore 2024-09-21T22:05:11.314 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-21T22:05:11.314 INFO:teuthology.orchestra.run.smithi157.stdout:[mgr] 2024-09-21T22:05:11.314 INFO:teuthology.orchestra.run.smithi157.stdout: debug ms = 1 2024-09-21T22:05:11.314 INFO:teuthology.orchestra.run.smithi157.stdout: debug mgr = 20 2024-09-21T22:05:11.314 INFO:teuthology.orchestra.run.smithi157.stdout: debug mon = 20 2024-09-21T22:05:11.315 INFO:teuthology.orchestra.run.smithi157.stdout: debug auth = 20 2024-09-21T22:05:11.315 INFO:teuthology.orchestra.run.smithi157.stdout: mon reweight min pgs per osd = 4 2024-09-21T22:05:11.315 INFO:teuthology.orchestra.run.smithi157.stdout: mon reweight min bytes per osd = 10 2024-09-21T22:05:11.315 INFO:teuthology.orchestra.run.smithi157.stdout: mgr/telemetry/nag = false 2024-09-21T22:05:11.315 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-21T22:05:11.315 INFO:teuthology.orchestra.run.smithi157.stdout:[mon] 2024-09-21T22:05:11.315 INFO:teuthology.orchestra.run.smithi157.stdout: debug ms = 1 2024-09-21T22:05:11.315 INFO:teuthology.orchestra.run.smithi157.stdout: debug mon = 20 2024-09-21T22:05:11.315 INFO:teuthology.orchestra.run.smithi157.stdout: debug paxos = 20 2024-09-21T22:05:11.315 INFO:teuthology.orchestra.run.smithi157.stdout: debug auth = 20 2024-09-21T22:05:11.315 INFO:teuthology.orchestra.run.smithi157.stdout: mon data avail warn = 5 2024-09-21T22:05:11.315 INFO:teuthology.orchestra.run.smithi157.stdout: mon mgr mkfs grace = 240 2024-09-21T22:05:11.315 INFO:teuthology.orchestra.run.smithi157.stdout: mon reweight min pgs per osd = 4 2024-09-21T22:05:11.315 INFO:teuthology.orchestra.run.smithi157.stdout: mon osd reporter subtree level = osd 2024-09-21T22:05:11.315 INFO:teuthology.orchestra.run.smithi157.stdout: mon osd prime pg temp = true 2024-09-21T22:05:11.315 INFO:teuthology.orchestra.run.smithi157.stdout: mon reweight min bytes per osd = 10 2024-09-21T22:05:11.316 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-21T22:05:11.316 INFO:teuthology.orchestra.run.smithi157.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-21T22:05:11.316 INFO:teuthology.orchestra.run.smithi157.stdout: auth mon ticket ttl = 660 # 11m 2024-09-21T22:05:11.316 INFO:teuthology.orchestra.run.smithi157.stdout: auth service ticket ttl = 240 # 4m 2024-09-21T22:05:11.316 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-21T22:05:11.316 INFO:teuthology.orchestra.run.smithi157.stdout: # don't complain about insecure global_id in the test suite 2024-09-21T22:05:11.316 INFO:teuthology.orchestra.run.smithi157.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-21T22:05:11.316 INFO:teuthology.orchestra.run.smithi157.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-21T22:05:11.316 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-21T22:05:11.316 INFO:teuthology.orchestra.run.smithi157.stdout: # 1m isn't quite enough 2024-09-21T22:05:11.316 INFO:teuthology.orchestra.run.smithi157.stdout: mon_down_mkfs_grace = 2m 2024-09-21T22:05:11.316 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-21T22:05:11.316 INFO:teuthology.orchestra.run.smithi157.stdout: mon_warn_on_filestore_osds = false 2024-09-21T22:05:11.316 INFO:teuthology.orchestra.run.smithi157.stdout: 2024-09-21T22:05:11.316 INFO:teuthology.orchestra.run.smithi157.stdout:[client] 2024-09-21T22:05:11.317 INFO:teuthology.orchestra.run.smithi157.stdout: rgw cache enabled = true 2024-09-21T22:05:11.317 INFO:teuthology.orchestra.run.smithi157.stdout: rgw enable ops log = true 2024-09-21T22:05:11.317 INFO:teuthology.orchestra.run.smithi157.stdout: rgw enable usage log = true 2024-09-21T22:05:11.317 INFO:teuthology.orchestra.run.smithi157.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-21T22:05:11.317 INFO:teuthology.orchestra.run.smithi157.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-21T22:05:11.317 INFO:teuthology.orchestra.run.smithi157.stdout:[mon.a] 2024-09-21T22:05:11.317 INFO:teuthology.orchestra.run.smithi157.stdout:[mon.c] 2024-09-21T22:05:11.317 INFO:teuthology.orchestra.run.smithi157.stdout:[mon.b] 2024-09-21T22:05:11.317 INFO:teuthology.orchestra.run.smithi064.stdout:[global] 2024-09-21T22:05:11.317 INFO:teuthology.orchestra.run.smithi064.stdout: chdir = "" 2024-09-21T22:05:11.317 INFO:teuthology.orchestra.run.smithi064.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-21T22:05:11.317 INFO:teuthology.orchestra.run.smithi064.stdout: auth supported = cephx 2024-09-21T22:05:11.317 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-21T22:05:11.317 INFO:teuthology.orchestra.run.smithi064.stdout: filestore xattr use omap = true 2024-09-21T22:05:11.317 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-21T22:05:11.318 INFO:teuthology.orchestra.run.smithi064.stdout: mon clock drift allowed = 1.000 2024-09-21T22:05:11.318 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-21T22:05:11.318 INFO:teuthology.orchestra.run.smithi064.stdout: osd crush chooseleaf type = 0 2024-09-21T22:05:11.318 INFO:teuthology.orchestra.run.smithi064.stdout: auth debug = true 2024-09-21T22:05:11.318 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-21T22:05:11.318 INFO:teuthology.orchestra.run.smithi064.stdout: ms die on old message = true 2024-09-21T22:05:11.318 INFO:teuthology.orchestra.run.smithi064.stdout: ms die on bug = true 2024-09-21T22:05:11.318 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-21T22:05:11.318 INFO:teuthology.orchestra.run.smithi064.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-21T22:05:11.318 INFO:teuthology.orchestra.run.smithi064.stdout: mon pg warn max object skew = 0 2024-09-21T22:05:11.318 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-21T22:05:11.318 INFO:teuthology.orchestra.run.smithi064.stdout: # disable pg_autoscaler by default for new pools 2024-09-21T22:05:11.318 INFO:teuthology.orchestra.run.smithi064.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-21T22:05:11.318 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-21T22:05:11.318 INFO:teuthology.orchestra.run.smithi064.stdout: osd pool default size = 2 2024-09-21T22:05:11.319 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-21T22:05:11.319 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd allow primary affinity = true 2024-09-21T22:05:11.319 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd allow pg remap = true 2024-09-21T22:05:11.319 INFO:teuthology.orchestra.run.smithi064.stdout: mon warn on legacy crush tunables = false 2024-09-21T22:05:11.319 INFO:teuthology.orchestra.run.smithi064.stdout: mon warn on crush straw calc version zero = false 2024-09-21T22:05:11.319 INFO:teuthology.orchestra.run.smithi064.stdout: mon warn on no sortbitwise = false 2024-09-21T22:05:11.319 INFO:teuthology.orchestra.run.smithi064.stdout: mon warn on osd down out interval zero = false 2024-09-21T22:05:11.319 INFO:teuthology.orchestra.run.smithi064.stdout: mon warn on too few osds = false 2024-09-21T22:05:11.319 INFO:teuthology.orchestra.run.smithi064.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-21T22:05:11.319 INFO:teuthology.orchestra.run.smithi064.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-21T22:05:11.319 INFO:teuthology.orchestra.run.smithi064.stdout: mon_allow_pool_size_one = true 2024-09-21T22:05:11.319 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-21T22:05:11.319 INFO:teuthology.orchestra.run.smithi064.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-21T22:05:11.319 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-21T22:05:11.319 INFO:teuthology.orchestra.run.smithi064.stdout: osd default data pool replay window = 5 2024-09-21T22:05:11.320 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-21T22:05:11.320 INFO:teuthology.orchestra.run.smithi064.stdout: mon allow pool delete = true 2024-09-21T22:05:11.320 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-21T22:05:11.320 INFO:teuthology.orchestra.run.smithi064.stdout: mon cluster log file level = debug 2024-09-21T22:05:11.320 INFO:teuthology.orchestra.run.smithi064.stdout: debug asserts on shutdown = true 2024-09-21T22:05:11.320 INFO:teuthology.orchestra.run.smithi064.stdout: mon health detail to clog = false 2024-09-21T22:05:11.320 INFO:teuthology.orchestra.run.smithi064.stdout: mon host = "172.21.15.39,[v2:172.21.15.39:3301,v1:172.21.15.39:6790],172.21.15.64" 2024-09-21T22:05:11.320 INFO:teuthology.orchestra.run.smithi064.stdout: mon client directed command retry = 5 2024-09-21T22:05:11.320 INFO:teuthology.orchestra.run.smithi064.stdout: mon warn on pool no app = False 2024-09-21T22:05:11.320 INFO:teuthology.orchestra.run.smithi064.stdout: ms inject socket failures = 5000 2024-09-21T22:05:11.320 INFO:teuthology.orchestra.run.smithi064.stdout: fsid = 00bb09a3-e410-4e18-a2b2-08eaf61978e6 2024-09-21T22:05:11.320 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-21T22:05:11.320 INFO:teuthology.orchestra.run.smithi064.stdout:[osd] 2024-09-21T22:05:11.320 INFO:teuthology.orchestra.run.smithi064.stdout: osd journal size = 100 2024-09-21T22:05:11.320 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-21T22:05:11.321 INFO:teuthology.orchestra.run.smithi064.stdout: osd scrub load threshold = 5.0 2024-09-21T22:05:11.321 INFO:teuthology.orchestra.run.smithi064.stdout: osd scrub max interval = 600 2024-09-21T22:05:11.321 INFO:teuthology.orchestra.run.smithi064.stdout: osd mclock profile = high_recovery_ops 2024-09-21T22:05:11.321 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-21T22:05:11.321 INFO:teuthology.orchestra.run.smithi064.stdout: osd recover clone overlap = true 2024-09-21T22:05:11.321 INFO:teuthology.orchestra.run.smithi064.stdout: osd recovery max chunk = 1048576 2024-09-21T22:05:11.321 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-21T22:05:11.321 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug shutdown = true 2024-09-21T22:05:11.321 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug op order = true 2024-09-21T22:05:11.321 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug verify stray on activate = true 2024-09-21T22:05:11.321 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug trim objects = true 2024-09-21T22:05:11.321 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-21T22:05:11.321 INFO:teuthology.orchestra.run.smithi064.stdout: osd open classes on start = true 2024-09-21T22:05:11.321 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug pg log writeout = true 2024-09-21T22:05:11.321 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-21T22:05:11.321 INFO:teuthology.orchestra.run.smithi064.stdout: osd deep scrub update digest min age = 30 2024-09-21T22:05:11.322 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-21T22:05:11.322 INFO:teuthology.orchestra.run.smithi064.stdout: osd map max advance = 10 2024-09-21T22:05:11.322 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-21T22:05:11.322 INFO:teuthology.orchestra.run.smithi064.stdout: journal zero on create = true 2024-09-21T22:05:11.322 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-21T22:05:11.322 INFO:teuthology.orchestra.run.smithi064.stdout: filestore ondisk finisher threads = 3 2024-09-21T22:05:11.322 INFO:teuthology.orchestra.run.smithi064.stdout: filestore apply finisher threads = 3 2024-09-21T22:05:11.322 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-21T22:05:11.322 INFO:teuthology.orchestra.run.smithi064.stdout: bdev debug aio = true 2024-09-21T22:05:11.322 INFO:teuthology.orchestra.run.smithi064.stdout: osd debug misdirected ops = true 2024-09-21T22:05:11.322 INFO:teuthology.orchestra.run.smithi064.stdout: bluestore block size = 96636764160 2024-09-21T22:05:11.322 INFO:teuthology.orchestra.run.smithi064.stdout: bluestore compression algorithm = zlib 2024-09-21T22:05:11.322 INFO:teuthology.orchestra.run.smithi064.stdout: bluestore compression mode = aggressive 2024-09-21T22:05:11.322 INFO:teuthology.orchestra.run.smithi064.stdout: bluestore fsck on mount = True 2024-09-21T22:05:11.322 INFO:teuthology.orchestra.run.smithi064.stdout: debug bluefs = 1/20 2024-09-21T22:05:11.322 INFO:teuthology.orchestra.run.smithi064.stdout: debug bluestore = 1/20 2024-09-21T22:05:11.323 INFO:teuthology.orchestra.run.smithi064.stdout: debug ms = 1 2024-09-21T22:05:11.323 INFO:teuthology.orchestra.run.smithi064.stdout: debug osd = 20 2024-09-21T22:05:11.323 INFO:teuthology.orchestra.run.smithi064.stdout: debug rocksdb = 4/10 2024-09-21T22:05:11.323 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd backfillfull_ratio = 0.85 2024-09-21T22:05:11.323 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd full ratio = 0.9 2024-09-21T22:05:11.323 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd nearfull ratio = 0.8 2024-09-21T22:05:11.323 INFO:teuthology.orchestra.run.smithi064.stdout: osd blocked scrub grace period = 3600 2024-09-21T22:05:11.323 INFO:teuthology.orchestra.run.smithi064.stdout: osd failsafe full ratio = 0.95 2024-09-21T22:05:11.323 INFO:teuthology.orchestra.run.smithi064.stdout: osd max markdown count = 1000 2024-09-21T22:05:11.323 INFO:teuthology.orchestra.run.smithi064.stdout: osd objectstore = bluestore 2024-09-21T22:05:11.323 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-21T22:05:11.323 INFO:teuthology.orchestra.run.smithi064.stdout:[mgr] 2024-09-21T22:05:11.323 INFO:teuthology.orchestra.run.smithi064.stdout: debug ms = 1 2024-09-21T22:05:11.323 INFO:teuthology.orchestra.run.smithi064.stdout: debug mgr = 20 2024-09-21T22:05:11.323 INFO:teuthology.orchestra.run.smithi064.stdout: debug mon = 20 2024-09-21T22:05:11.324 INFO:teuthology.orchestra.run.smithi064.stdout: debug auth = 20 2024-09-21T22:05:11.324 INFO:teuthology.orchestra.run.smithi064.stdout: mon reweight min pgs per osd = 4 2024-09-21T22:05:11.324 INFO:teuthology.orchestra.run.smithi064.stdout: mon reweight min bytes per osd = 10 2024-09-21T22:05:11.324 INFO:teuthology.orchestra.run.smithi064.stdout: mgr/telemetry/nag = false 2024-09-21T22:05:11.324 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-21T22:05:11.324 INFO:teuthology.orchestra.run.smithi064.stdout:[mon] 2024-09-21T22:05:11.324 INFO:teuthology.orchestra.run.smithi064.stdout: debug ms = 1 2024-09-21T22:05:11.324 INFO:teuthology.orchestra.run.smithi064.stdout: debug mon = 20 2024-09-21T22:05:11.324 INFO:teuthology.orchestra.run.smithi064.stdout: debug paxos = 20 2024-09-21T22:05:11.324 INFO:teuthology.orchestra.run.smithi064.stdout: debug auth = 20 2024-09-21T22:05:11.324 INFO:teuthology.orchestra.run.smithi064.stdout: mon data avail warn = 5 2024-09-21T22:05:11.324 INFO:teuthology.orchestra.run.smithi064.stdout: mon mgr mkfs grace = 240 2024-09-21T22:05:11.324 INFO:teuthology.orchestra.run.smithi064.stdout: mon reweight min pgs per osd = 4 2024-09-21T22:05:11.324 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd reporter subtree level = osd 2024-09-21T22:05:11.324 INFO:teuthology.orchestra.run.smithi064.stdout: mon osd prime pg temp = true 2024-09-21T22:05:11.324 INFO:teuthology.orchestra.run.smithi064.stdout: mon reweight min bytes per osd = 10 2024-09-21T22:05:11.325 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-21T22:05:11.325 INFO:teuthology.orchestra.run.smithi064.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-21T22:05:11.325 INFO:teuthology.orchestra.run.smithi064.stdout: auth mon ticket ttl = 660 # 11m 2024-09-21T22:05:11.325 INFO:teuthology.orchestra.run.smithi064.stdout: auth service ticket ttl = 240 # 4m 2024-09-21T22:05:11.325 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-21T22:05:11.325 INFO:teuthology.orchestra.run.smithi064.stdout: # don't complain about insecure global_id in the test suite 2024-09-21T22:05:11.325 INFO:teuthology.orchestra.run.smithi064.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-21T22:05:11.325 INFO:teuthology.orchestra.run.smithi064.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-21T22:05:11.325 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-21T22:05:11.325 INFO:teuthology.orchestra.run.smithi064.stdout: # 1m isn't quite enough 2024-09-21T22:05:11.325 INFO:teuthology.orchestra.run.smithi064.stdout: mon_down_mkfs_grace = 2m 2024-09-21T22:05:11.325 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-21T22:05:11.325 INFO:teuthology.orchestra.run.smithi064.stdout: mon_warn_on_filestore_osds = false 2024-09-21T22:05:11.325 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-09-21T22:05:11.325 INFO:teuthology.orchestra.run.smithi064.stdout:[client] 2024-09-21T22:05:11.325 INFO:teuthology.orchestra.run.smithi064.stdout: rgw cache enabled = true 2024-09-21T22:05:11.326 INFO:teuthology.orchestra.run.smithi064.stdout: rgw enable ops log = true 2024-09-21T22:05:11.326 INFO:teuthology.orchestra.run.smithi064.stdout: rgw enable usage log = true 2024-09-21T22:05:11.326 INFO:teuthology.orchestra.run.smithi064.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-21T22:05:11.326 INFO:teuthology.orchestra.run.smithi064.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-21T22:05:11.326 INFO:teuthology.orchestra.run.smithi064.stdout:[mon.a] 2024-09-21T22:05:11.326 INFO:teuthology.orchestra.run.smithi064.stdout:[mon.c] 2024-09-21T22:05:11.326 INFO:teuthology.orchestra.run.smithi064.stdout:[mon.b] 2024-09-21T22:05:11.327 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-21T22:05:11.328 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-21T22:05:11.420 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-21T22:05:11.420 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-21T22:05:11.421 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-21T22:05:11.426 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-21T22:05:11.427 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-21T22:05:11.479 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi039.front.sepia.ceph.com 2024-09-21T22:05:11.479 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-21T22:05:11.479 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-21T22:05:11.480 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-21T22:05:11.547 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-21T22:05:11.547 DEBUG:teuthology.orchestra.run.smithi039:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-21T22:05:11.600 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi064.front.sepia.ceph.com 2024-09-21T22:05:11.600 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-09-21T22:05:11.600 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-21T22:05:11.600 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-21T22:05:11.624 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-09-21T22:05:11.624 DEBUG:teuthology.orchestra.run.smithi064:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-21T22:05:11.673 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi157.front.sepia.ceph.com 2024-09-21T22:05:11.673 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-09-21T22:05:11.673 DEBUG:teuthology.orchestra.run.smithi157:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-21T22:05:11.673 DEBUG:teuthology.orchestra.run.smithi157:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-21T22:05:11.696 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-09-21T22:05:11.696 DEBUG:teuthology.orchestra.run.smithi157:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-21T22:05:11.744 INFO:tasks.ceph:Setting up mon nodes... 2024-09-21T22:05:11.744 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-21T22:05:11.744 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-21T22:05:11.790 INFO:teuthology.orchestra.run.smithi064.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-21T22:05:11.802 INFO:tasks.ceph:Setting up mds nodes... 2024-09-21T22:05:11.802 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-21T22:05:11.802 DEBUG:teuthology.orchestra.run.smithi157:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-21T22:05:11.841 INFO:teuthology.orchestra.run.smithi157.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-09-21T22:05:11.862 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-21T22:05:11.862 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi039.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi064.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2024-09-21T22:05:11.862 DEBUG:teuthology.orchestra.run.smithi039:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-21T22:05:11.876 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-21T22:05:11.876 INFO:tasks.ceph:role: osd.0 2024-09-21T22:05:11.876 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi039.front.sepia.ceph.com 2024-09-21T22:05:11.876 DEBUG:teuthology.orchestra.run.smithi039:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-21T22:05:11.959 INFO:teuthology.orchestra.run.smithi039.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-21T22:05:11.959 INFO:teuthology.orchestra.run.smithi039.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-21T22:05:11.959 INFO:teuthology.orchestra.run.smithi039.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-21T22:05:11.959 INFO:teuthology.orchestra.run.smithi039.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-21T22:05:11.960 INFO:teuthology.orchestra.run.smithi039.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-21T22:05:11.960 INFO:teuthology.orchestra.run.smithi039.stdout: = sunit=0 swidth=0 blks 2024-09-21T22:05:11.960 INFO:teuthology.orchestra.run.smithi039.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-21T22:05:11.960 INFO:teuthology.orchestra.run.smithi039.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-21T22:05:11.960 INFO:teuthology.orchestra.run.smithi039.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-21T22:05:11.960 INFO:teuthology.orchestra.run.smithi039.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-21T22:05:13.275 INFO:teuthology.orchestra.run.smithi039.stdout:Discarding blocks...Done. 2024-09-21T22:05:13.277 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi039.front.sepia.ceph.com -o noatime 2024-09-21T22:05:13.277 DEBUG:teuthology.orchestra.run.smithi039:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-21T22:05:13.338 DEBUG:teuthology.orchestra.run.smithi039:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-21T22:05:13.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T22:05:13.388 INFO:teuthology.orchestra.run.smithi039.stderr:sudo: /sbin/restorecon: command not found 2024-09-21T22:05:13.388 DEBUG:teuthology.orchestra.run.smithi039:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-21T22:05:13.445 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-21T22:05:13.445 INFO:tasks.ceph:role: osd.1 2024-09-21T22:05:13.445 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi039.front.sepia.ceph.com 2024-09-21T22:05:13.446 DEBUG:teuthology.orchestra.run.smithi039:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-21T22:05:13.532 INFO:teuthology.orchestra.run.smithi039.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-21T22:05:13.532 INFO:teuthology.orchestra.run.smithi039.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-21T22:05:13.532 INFO:teuthology.orchestra.run.smithi039.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-21T22:05:13.532 INFO:teuthology.orchestra.run.smithi039.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-21T22:05:13.532 INFO:teuthology.orchestra.run.smithi039.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-21T22:05:13.532 INFO:teuthology.orchestra.run.smithi039.stdout: = sunit=0 swidth=0 blks 2024-09-21T22:05:13.532 INFO:teuthology.orchestra.run.smithi039.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-21T22:05:13.532 INFO:teuthology.orchestra.run.smithi039.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-21T22:05:13.532 INFO:teuthology.orchestra.run.smithi039.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-21T22:05:13.532 INFO:teuthology.orchestra.run.smithi039.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-21T22:05:14.851 INFO:teuthology.orchestra.run.smithi039.stdout:Discarding blocks...Done. 2024-09-21T22:05:14.853 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi039.front.sepia.ceph.com -o noatime 2024-09-21T22:05:14.854 DEBUG:teuthology.orchestra.run.smithi039:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-09-21T22:05:14.915 DEBUG:teuthology.orchestra.run.smithi039:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-21T22:05:14.967 INFO:teuthology.orchestra.run.smithi039.stderr:sudo: /sbin/restorecon: command not found 2024-09-21T22:05:14.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T22:05:14.968 DEBUG:teuthology.orchestra.run.smithi039:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-21T22:05:15.021 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-21T22:05:15.021 INFO:tasks.ceph:role: osd.2 2024-09-21T22:05:15.021 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi039.front.sepia.ceph.com 2024-09-21T22:05:15.021 DEBUG:teuthology.orchestra.run.smithi039:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-21T22:05:15.103 INFO:teuthology.orchestra.run.smithi039.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-21T22:05:15.103 INFO:teuthology.orchestra.run.smithi039.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-21T22:05:15.103 INFO:teuthology.orchestra.run.smithi039.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-21T22:05:15.103 INFO:teuthology.orchestra.run.smithi039.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-21T22:05:15.103 INFO:teuthology.orchestra.run.smithi039.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-21T22:05:15.104 INFO:teuthology.orchestra.run.smithi039.stdout: = sunit=0 swidth=0 blks 2024-09-21T22:05:15.104 INFO:teuthology.orchestra.run.smithi039.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-21T22:05:15.104 INFO:teuthology.orchestra.run.smithi039.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-21T22:05:15.104 INFO:teuthology.orchestra.run.smithi039.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-21T22:05:15.104 INFO:teuthology.orchestra.run.smithi039.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-21T22:05:16.422 INFO:teuthology.orchestra.run.smithi039.stdout:Discarding blocks...Done. 2024-09-21T22:05:16.425 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi039.front.sepia.ceph.com -o noatime 2024-09-21T22:05:16.425 DEBUG:teuthology.orchestra.run.smithi039:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-09-21T22:05:16.487 DEBUG:teuthology.orchestra.run.smithi039:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-21T22:05:16.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T22:05:16.539 INFO:teuthology.orchestra.run.smithi039.stderr:sudo: /sbin/restorecon: command not found 2024-09-21T22:05:16.540 DEBUG:teuthology.orchestra.run.smithi039:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-21T22:05:16.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T22:05:16.650+0000 7f6e66b4a7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-09-21T22:05:16.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T22:05:16.650+0000 7f6e66b4a7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-09-21T22:05:16.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T22:05:16.650+0000 7f6e66b4a7c0 -1 bdev(0x55af8a489800 /var/lib/ceph/osd/ceph-0/block) open stat got: (1) Operation not permitted 2024-09-21T22:05:16.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T22:05:16.650+0000 7f6e66b4a7c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-09-21T22:05:19.814 DEBUG:teuthology.orchestra.run.smithi039:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-21T22:05:19.828 DEBUG:teuthology.orchestra.run.smithi039:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-21T22:05:19.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T22:05:19.910+0000 7f6e2ac447c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-09-21T22:05:19.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T22:05:19.910+0000 7f6e2ac447c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-09-21T22:05:19.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T22:05:19.910+0000 7f6e2ac447c0 -1 bdev(0x55ba1cc9f800 /var/lib/ceph/osd/ceph-1/block) open stat got: (1) Operation not permitted 2024-09-21T22:05:19.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T22:05:19.910+0000 7f6e2ac447c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-09-21T22:05:23.038 DEBUG:teuthology.orchestra.run.smithi039:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-21T22:05:23.096 DEBUG:teuthology.orchestra.run.smithi039:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-21T22:05:23.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T22:05:23.174+0000 7fd35813c7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-09-21T22:05:23.179 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T22:05:23.178+0000 7fd35813c7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-09-21T22:05:23.179 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T22:05:23.178+0000 7fd35813c7c0 -1 bdev(0x558668f49800 /var/lib/ceph/osd/ceph-2/block) open stat got: (1) Operation not permitted 2024-09-21T22:05:23.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T22:05:23.178+0000 7fd35813c7c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-09-21T22:05:26.315 DEBUG:teuthology.orchestra.run.smithi039:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-21T22:05:26.372 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-09-21T22:05:26.387 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-21T22:05:26.387 INFO:tasks.ceph:role: osd.3 2024-09-21T22:05:26.387 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi064.front.sepia.ceph.com 2024-09-21T22:05:26.387 DEBUG:teuthology.orchestra.run.smithi064:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-21T22:05:26.448 INFO:teuthology.orchestra.run.smithi064.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-21T22:05:26.448 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-21T22:05:26.448 INFO:teuthology.orchestra.run.smithi064.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-21T22:05:26.448 INFO:teuthology.orchestra.run.smithi064.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-21T22:05:26.448 INFO:teuthology.orchestra.run.smithi064.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-21T22:05:26.449 INFO:teuthology.orchestra.run.smithi064.stdout: = sunit=0 swidth=0 blks 2024-09-21T22:05:26.449 INFO:teuthology.orchestra.run.smithi064.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-21T22:05:26.449 INFO:teuthology.orchestra.run.smithi064.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-21T22:05:26.449 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-21T22:05:26.449 INFO:teuthology.orchestra.run.smithi064.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-21T22:05:26.905 INFO:teuthology.orchestra.run.smithi064.stdout:Discarding blocks...Done. 2024-09-21T22:05:26.907 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi064.front.sepia.ceph.com -o noatime 2024-09-21T22:05:26.907 DEBUG:teuthology.orchestra.run.smithi064:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3 2024-09-21T22:05:27.004 DEBUG:teuthology.orchestra.run.smithi064:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-09-21T22:05:27.014 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: /sbin/restorecon: command not found 2024-09-21T22:05:27.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T22:05:27.015 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-09-21T22:05:27.065 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-21T22:05:27.065 INFO:tasks.ceph:role: osd.4 2024-09-21T22:05:27.066 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi064.front.sepia.ceph.com 2024-09-21T22:05:27.066 DEBUG:teuthology.orchestra.run.smithi064:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-21T22:05:27.128 INFO:teuthology.orchestra.run.smithi064.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-21T22:05:27.128 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-21T22:05:27.128 INFO:teuthology.orchestra.run.smithi064.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-21T22:05:27.128 INFO:teuthology.orchestra.run.smithi064.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-21T22:05:27.128 INFO:teuthology.orchestra.run.smithi064.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-21T22:05:27.128 INFO:teuthology.orchestra.run.smithi064.stdout: = sunit=0 swidth=0 blks 2024-09-21T22:05:27.128 INFO:teuthology.orchestra.run.smithi064.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-21T22:05:27.128 INFO:teuthology.orchestra.run.smithi064.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-21T22:05:27.129 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-21T22:05:27.129 INFO:teuthology.orchestra.run.smithi064.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-21T22:05:27.588 INFO:teuthology.orchestra.run.smithi064.stdout:Discarding blocks...Done. 2024-09-21T22:05:27.591 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi064.front.sepia.ceph.com -o noatime 2024-09-21T22:05:27.591 DEBUG:teuthology.orchestra.run.smithi064:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-4 2024-09-21T22:05:27.727 DEBUG:teuthology.orchestra.run.smithi064:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-09-21T22:05:27.739 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: /sbin/restorecon: command not found 2024-09-21T22:05:27.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T22:05:27.740 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-09-21T22:05:27.797 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-21T22:05:27.797 INFO:tasks.ceph:role: osd.5 2024-09-21T22:05:27.797 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi064.front.sepia.ceph.com 2024-09-21T22:05:27.797 DEBUG:teuthology.orchestra.run.smithi064:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-21T22:05:27.860 INFO:teuthology.orchestra.run.smithi064.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-21T22:05:27.860 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-21T22:05:27.860 INFO:teuthology.orchestra.run.smithi064.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-21T22:05:27.860 INFO:teuthology.orchestra.run.smithi064.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-21T22:05:27.860 INFO:teuthology.orchestra.run.smithi064.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-21T22:05:27.860 INFO:teuthology.orchestra.run.smithi064.stdout: = sunit=0 swidth=0 blks 2024-09-21T22:05:27.860 INFO:teuthology.orchestra.run.smithi064.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-21T22:05:27.860 INFO:teuthology.orchestra.run.smithi064.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-21T22:05:27.860 INFO:teuthology.orchestra.run.smithi064.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-21T22:05:27.860 INFO:teuthology.orchestra.run.smithi064.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-21T22:05:28.357 INFO:teuthology.orchestra.run.smithi064.stdout:Discarding blocks...Done. 2024-09-21T22:05:28.358 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi064.front.sepia.ceph.com -o noatime 2024-09-21T22:05:28.358 DEBUG:teuthology.orchestra.run.smithi064:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-5 2024-09-21T22:05:28.498 DEBUG:teuthology.orchestra.run.smithi064:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-09-21T22:05:28.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T22:05:28.511 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: /sbin/restorecon: command not found 2024-09-21T22:05:28.511 DEBUG:teuthology.orchestra.run.smithi064:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-21T22:05:28.705 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-21T22:05:28.701+0000 7f7427ef37c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-09-21T22:05:28.706 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-21T22:05:28.701+0000 7f7427ef37c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-09-21T22:05:28.706 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-21T22:05:28.701+0000 7f7427ef37c0 -1 bdev(0x55c6baa13800 /var/lib/ceph/osd/ceph-3/block) open stat got: (1) Operation not permitted 2024-09-21T22:05:28.706 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-21T22:05:28.701+0000 7f7427ef37c0 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-09-21T22:05:31.836 DEBUG:teuthology.orchestra.run.smithi064:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-21T22:05:31.850 DEBUG:teuthology.orchestra.run.smithi064:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-21T22:05:31.931 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-21T22:05:31.925+0000 7ff1a82d37c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-09-21T22:05:31.932 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-21T22:05:31.929+0000 7ff1a82d37c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-09-21T22:05:31.932 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-21T22:05:31.929+0000 7ff1a82d37c0 -1 bdev(0x563fd3811800 /var/lib/ceph/osd/ceph-4/block) open stat got: (1) Operation not permitted 2024-09-21T22:05:31.932 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-21T22:05:31.929+0000 7ff1a82d37c0 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-09-21T22:05:35.085 DEBUG:teuthology.orchestra.run.smithi064:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-21T22:05:35.141 DEBUG:teuthology.orchestra.run.smithi064:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-21T22:05:35.224 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-21T22:05:35.221+0000 7f7f63e437c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-09-21T22:05:35.224 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-21T22:05:35.221+0000 7f7f63e437c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-09-21T22:05:35.225 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-21T22:05:35.221+0000 7f7f63e437c0 -1 bdev(0x561983f77800 /var/lib/ceph/osd/ceph-5/block) open stat got: (1) Operation not permitted 2024-09-21T22:05:35.225 INFO:teuthology.orchestra.run.smithi064.stderr:2024-09-21T22:05:35.221+0000 7f7f63e437c0 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-09-21T22:05:38.351 DEBUG:teuthology.orchestra.run.smithi064:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-21T22:05:38.409 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-21T22:05:38.409 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-21T22:05:38.409 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-21T22:05:38.424 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-21T22:05:38.424 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-21T22:05:38.480 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-21T22:05:38.480 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-21T22:05:38.535 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-09-21T22:05:38.535 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-21T22:05:38.549 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-09-21T22:05:38.549 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-09-21T22:05:38.603 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-09-21T22:05:38.603 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-09-21T22:05:38.659 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-09-21T22:05:38.659 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-09-21T22:05:38.716 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-09-21T22:05:38.716 DEBUG:teuthology.orchestra.run.smithi157:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-09-21T22:05:38.722 INFO:tasks.ceph:Adding keys to all mons... 2024-09-21T22:05:38.722 DEBUG:teuthology.orchestra.run.smithi039:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-21T22:05:38.725 DEBUG:teuthology.orchestra.run.smithi064:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-21T22:05:38.762 INFO:teuthology.orchestra.run.smithi039.stdout:[osd.0] 2024-09-21T22:05:38.762 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQAcQ+9m9RzjJhAAkSJ2lyil6+5Zeqhi9+Bp/Q== 2024-09-21T22:05:38.762 INFO:teuthology.orchestra.run.smithi039.stdout:[osd.1] 2024-09-21T22:05:38.762 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQAfQ+9m6r1ZNhAAEu5cX4SiEyRJKCYnikCJXQ== 2024-09-21T22:05:38.762 INFO:teuthology.orchestra.run.smithi039.stdout:[osd.2] 2024-09-21T22:05:38.762 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQAjQ+9m24eoChAAcS15KQE2Ath9SQEPOI5iEQ== 2024-09-21T22:05:38.762 INFO:teuthology.orchestra.run.smithi039.stdout:[mgr.x] 2024-09-21T22:05:38.762 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQAXQ+9m5SobLxAA4c6ml8efYXbJUvFmok/GmQ== 2024-09-21T22:05:38.762 INFO:teuthology.orchestra.run.smithi039.stdout:[osd.3] 2024-09-21T22:05:38.762 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQAoQ+9mssQEKhAAcMB/wcfC8VZq2ETkvpS5GQ== 2024-09-21T22:05:38.763 INFO:teuthology.orchestra.run.smithi039.stdout:[osd.4] 2024-09-21T22:05:38.763 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQArQ+9mJP6ANxAAq+c7EcvpxOlE+4iIePut0A== 2024-09-21T22:05:38.763 INFO:teuthology.orchestra.run.smithi039.stdout:[osd.5] 2024-09-21T22:05:38.763 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQAvQ+9mHFVZDRAAC99eqfWYCgAgkY9y4tFARA== 2024-09-21T22:05:38.763 INFO:teuthology.orchestra.run.smithi039.stdout:[client.0] 2024-09-21T22:05:38.763 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQAXQ+9my1I2MhAAhPjL0gSMK9qS6iJV90j1EA== 2024-09-21T22:05:38.770 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.0] 2024-09-21T22:05:38.770 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQAcQ+9m9RzjJhAAkSJ2lyil6+5Zeqhi9+Bp/Q== 2024-09-21T22:05:38.770 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.1] 2024-09-21T22:05:38.770 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQAfQ+9m6r1ZNhAAEu5cX4SiEyRJKCYnikCJXQ== 2024-09-21T22:05:38.770 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.2] 2024-09-21T22:05:38.770 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQAjQ+9m24eoChAAcS15KQE2Ath9SQEPOI5iEQ== 2024-09-21T22:05:38.770 INFO:teuthology.orchestra.run.smithi064.stdout:[mgr.x] 2024-09-21T22:05:38.770 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQAXQ+9m5SobLxAA4c6ml8efYXbJUvFmok/GmQ== 2024-09-21T22:05:38.771 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.3] 2024-09-21T22:05:38.771 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQAoQ+9mssQEKhAAcMB/wcfC8VZq2ETkvpS5GQ== 2024-09-21T22:05:38.771 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.4] 2024-09-21T22:05:38.771 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQArQ+9mJP6ANxAAq+c7EcvpxOlE+4iIePut0A== 2024-09-21T22:05:38.771 INFO:teuthology.orchestra.run.smithi064.stdout:[osd.5] 2024-09-21T22:05:38.771 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQAvQ+9mHFVZDRAAC99eqfWYCgAgkY9y4tFARA== 2024-09-21T22:05:38.771 INFO:teuthology.orchestra.run.smithi064.stdout:[client.0] 2024-09-21T22:05:38.771 INFO:teuthology.orchestra.run.smithi064.stdout: key = AQAXQ+9my1I2MhAAhPjL0gSMK9qS6iJV90j1EA== 2024-09-21T22:05:38.772 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T22:05:38.808 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T22:05:38.862 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T22:05:38.864 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T22:05:38.908 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T22:05:38.948 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T22:05:38.995 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-21T22:05:39.032 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-21T22:05:39.087 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T22:05:39.116 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T22:05:39.160 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T22:05:39.162 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T22:05:39.240 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T22:05:39.244 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T22:05:39.390 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-21T22:05:39.393 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-21T22:05:39.451 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-21T22:05:39.451 DEBUG:teuthology.orchestra.run.smithi039:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-21T22:05:39.464 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-21T22:05:39.577 DEBUG:teuthology.orchestra.run.smithi039:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-21T22:05:39.589 DEBUG:teuthology.orchestra.run.smithi039:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-09-21T22:05:39.641 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-21T22:05:39.737 DEBUG:teuthology.orchestra.run.smithi039:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-09-21T22:05:39.748 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-09-21T22:05:39.763 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-21T22:05:39.856 DEBUG:teuthology.orchestra.run.smithi064:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-09-21T22:05:39.868 DEBUG:teuthology.orchestra.run.smithi039:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-21T22:05:39.871 DEBUG:teuthology.orchestra.run.smithi064:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-21T22:05:39.915 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-21T22:05:39.915 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-21T22:05:39.915 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-21T22:05:39.918 INFO:tasks.ceph.mon.a:Started 2024-09-21T22:05:39.918 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-21T22:05:39.918 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-21T22:05:39.920 INFO:tasks.ceph.mon.c:Started 2024-09-21T22:05:39.920 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-21T22:05:39.920 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-21T22:05:39.961 INFO:tasks.ceph.mon.b:Started 2024-09-21T22:05:39.961 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-21T22:05:39.961 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-21T22:05:39.961 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-21T22:05:39.964 INFO:tasks.ceph.mgr.x:Started 2024-09-21T22:05:39.964 INFO:tasks.ceph:Setting crush tunables to default 2024-09-21T22:05:39.964 DEBUG:teuthology.orchestra.run.smithi039:> sudo ceph --cluster ceph osd crush tunables default 2024-09-21T22:05:46.269 INFO:tasks.ceph.mgr.x.smithi064.stderr:2024-09-21T22:05:46.265+0000 7f65c356c180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-21T22:05:46.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-09-21T22:05:46.293+0000 7f53dcb0f640 0 --2- 172.21.15.39:0/3695549029 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f53c400c1b0 0x7f53d8121e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-21T22:05:46.372 INFO:teuthology.orchestra.run.smithi039.stderr:adjusted tunables profile to default 2024-09-21T22:05:46.392 INFO:tasks.ceph:check_enable_crimson: False 2024-09-21T22:05:46.393 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-21T22:05:46.393 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-21T22:05:46.393 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-21T22:05:46.404 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-21T22:05:46.404 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-21T22:05:46.458 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-09-21T22:05:46.458 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-21T22:05:46.515 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-09-21T22:05:46.515 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-09-21T22:05:46.529 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-09-21T22:05:46.529 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-09-21T22:05:46.583 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-09-21T22:05:46.583 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-09-21T22:05:46.639 DEBUG:teuthology.orchestra.run.smithi064:> sudo ceph --cluster ceph osd new 8336c397-2018-4c75-ab31-2e5f6ad564b5 0 2024-09-21T22:05:46.766 INFO:tasks.ceph.mgr.x.smithi064.stderr:2024-09-21T22:05:46.761+0000 7f65c356c180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-21T22:05:46.911 INFO:teuthology.orchestra.run.smithi064.stdout:0 2024-09-21T22:05:46.927 DEBUG:teuthology.orchestra.run.smithi064:> sudo ceph --cluster ceph osd new bca64af6-7427-4c5d-97a4-a183b1f3a7d9 1 2024-09-21T22:05:46.954 INFO:tasks.ceph.mgr.x.smithi064.stderr:2024-09-21T22:05:46.949+0000 7f65c356c180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-21T22:05:47.126 INFO:teuthology.orchestra.run.smithi064.stdout:1 2024-09-21T22:05:47.143 DEBUG:teuthology.orchestra.run.smithi064:> sudo ceph --cluster ceph osd new 5f2678ed-6a23-4a67-96a4-c0ca91254bd7 2 2024-09-21T22:05:47.395 INFO:teuthology.orchestra.run.smithi064.stdout:2 2024-09-21T22:05:47.411 DEBUG:teuthology.orchestra.run.smithi064:> sudo ceph --cluster ceph osd new aba30555-ef9e-47c9-a38b-ee1a4b8092f5 3 2024-09-21T22:05:47.553 INFO:tasks.ceph.mgr.x.smithi064.stderr:2024-09-21T22:05:47.549+0000 7f65c356c180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-21T22:05:47.610 INFO:tasks.ceph.mgr.x.smithi064.stderr:2024-09-21T22:05:47.605+0000 7f65c356c180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-21T22:05:47.621 INFO:teuthology.orchestra.run.smithi064.stdout:3 2024-09-21T22:05:47.638 DEBUG:teuthology.orchestra.run.smithi064:> sudo ceph --cluster ceph osd new 2a1288d7-618d-4f44-8346-930cf56d4e80 4 2024-09-21T22:05:47.668 INFO:tasks.ceph.mgr.x.smithi064.stderr:2024-09-21T22:05:47.665+0000 7f65c356c180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-21T22:05:47.751 INFO:tasks.ceph.mgr.x.smithi064.stderr:2024-09-21T22:05:47.745+0000 7f65c356c180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-21T22:05:47.808 INFO:tasks.ceph.mgr.x.smithi064.stderr:2024-09-21T22:05:47.805+0000 7f65c356c180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-21T22:05:47.849 INFO:teuthology.orchestra.run.smithi064.stdout:4 2024-09-21T22:05:47.865 DEBUG:teuthology.orchestra.run.smithi064:> sudo ceph --cluster ceph osd new 3ce719a8-34e9-4890-bd40-d22201e7bad8 5 2024-09-21T22:05:47.890 INFO:tasks.ceph.mgr.x.smithi064.stderr:2024-09-21T22:05:47.885+0000 7f65c356c180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-21T22:05:47.948 INFO:tasks.ceph.mgr.x.smithi064.stderr:2024-09-21T22:05:47.945+0000 7f65c356c180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-21T22:05:48.094 INFO:teuthology.orchestra.run.smithi064.stdout:5 2024-09-21T22:05:48.110 INFO:tasks.ceph.mgr.x.smithi064.stderr:2024-09-21T22:05:48.105+0000 7f65c356c180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-21T22:05:48.111 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-21T22:05:48.111 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-09-21T22:05:48.114 INFO:tasks.ceph.osd.0:Started 2024-09-21T22:05:48.114 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-21T22:05:48.114 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-21T22:05:48.116 INFO:tasks.ceph.osd.1:Started 2024-09-21T22:05:48.116 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-21T22:05:48.117 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-09-21T22:05:48.120 INFO:tasks.ceph.osd.2:Started 2024-09-21T22:05:48.120 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-21T22:05:48.120 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-09-21T22:05:48.122 INFO:tasks.ceph.osd.3:Started 2024-09-21T22:05:48.122 INFO:tasks.ceph.osd.4:Restarting daemon 2024-09-21T22:05:48.122 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-09-21T22:05:48.124 INFO:tasks.ceph.osd.4:Started 2024-09-21T22:05:48.124 INFO:tasks.ceph.osd.5:Restarting daemon 2024-09-21T22:05:48.124 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-09-21T22:05:48.126 INFO:tasks.ceph.osd.5:Started 2024-09-21T22:05:48.126 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-21T22:05:48.179 INFO:tasks.ceph.mgr.x.smithi064.stderr:2024-09-21T22:05:48.173+0000 7f65c356c180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-21T22:05:48.325 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T22:05:48.325 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":8,"fsid":"00bb09a3-e410-4e18-a2b2-08eaf61978e6","created":"2024-09-21T22:05:45.303341+0000","modified":"2024-09-21T22:05:48.083837+0000","last_up_change":"0.000000","last_in_change":"2024-09-21T22:05:48.083837+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8336c397-2018-4c75-ab31-2e5f6ad564b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bca64af6-7427-4c5d-97a4-a183b1f3a7d9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5f2678ed-6a23-4a67-96a4-c0ca91254bd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aba30555-ef9e-47c9-a38b-ee1a4b8092f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2a1288d7-618d-4f44-8346-930cf56d4e80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3ce719a8-34e9-4890-bd40-d22201e7bad8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T22:05:48.343 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-21T22:05:48.343 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-21T22:05:48.821 INFO:tasks.ceph.mgr.x.smithi064.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-09-21T22:05:48.821 INFO:tasks.ceph.mgr.x.smithi064.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-09-21T22:05:48.821 INFO:tasks.ceph.mgr.x.smithi064.stderr: from numpy import show_config as show_numpy_config 2024-09-21T22:05:48.863 INFO:tasks.ceph.mgr.x.smithi064.stderr:2024-09-21T22:05:48.857+0000 7f65c356c180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-21T22:05:48.948 DEBUG:teuthology.orchestra.run.smithi039:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-21T22:05:49.009 INFO:tasks.ceph.mgr.x.smithi064.stderr:2024-09-21T22:05:49.005+0000 7f65c356c180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-21T22:05:49.144 INFO:teuthology.misc.health.smithi039.stdout: 2024-09-21T22:05:49.144 INFO:teuthology.misc.health.smithi039.stdout:{"epoch":8,"fsid":"00bb09a3-e410-4e18-a2b2-08eaf61978e6","created":"2024-09-21T22:05:45.303341+0000","modified":"2024-09-21T22:05:48.083837+0000","last_up_change":"0.000000","last_in_change":"2024-09-21T22:05:48.083837+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8336c397-2018-4c75-ab31-2e5f6ad564b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bca64af6-7427-4c5d-97a4-a183b1f3a7d9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5f2678ed-6a23-4a67-96a4-c0ca91254bd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aba30555-ef9e-47c9-a38b-ee1a4b8092f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2a1288d7-618d-4f44-8346-930cf56d4e80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3ce719a8-34e9-4890-bd40-d22201e7bad8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T22:05:49.160 DEBUG:teuthology.misc:0 of 6 OSDs are up 2024-09-21T22:05:49.179 INFO:tasks.ceph.mgr.x.smithi064.stderr:2024-09-21T22:05:49.173+0000 7f65c356c180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-21T22:05:49.236 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:05:49.233+0000 7fa1f3c027c0 -1 Falling back to public interface 2024-09-21T22:05:49.236 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:05:49.233+0000 7fb13b9057c0 -1 Falling back to public interface 2024-09-21T22:05:49.246 INFO:tasks.ceph.mgr.x.smithi064.stderr:2024-09-21T22:05:49.241+0000 7f65c356c180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-21T22:05:49.291 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:05:49.289+0000 7fb10d1027c0 -1 Falling back to public interface 2024-09-21T22:05:49.317 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:05:49.313+0000 7fcc60b397c0 -1 Falling back to public interface 2024-09-21T22:05:49.320 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:05:49.317+0000 7f6e7f4ee7c0 -1 Falling back to public interface 2024-09-21T22:05:49.389 INFO:tasks.ceph.mgr.x.smithi064.stderr:2024-09-21T22:05:49.385+0000 7f65c356c180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-21T22:05:49.749 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:05:49.745+0000 7fc8d6a3b7c0 -1 Falling back to public interface 2024-09-21T22:05:50.024 INFO:tasks.ceph.mgr.x.smithi064.stderr:2024-09-21T22:05:50.021+0000 7f65c356c180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-21T22:05:50.103 INFO:tasks.ceph.mgr.x.smithi064.stderr:2024-09-21T22:05:50.097+0000 7f65c356c180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-21T22:05:50.295 INFO:tasks.ceph.mgr.x.smithi064.stderr:2024-09-21T22:05:50.289+0000 7f65c356c180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-21T22:05:50.372 INFO:tasks.ceph.mgr.x.smithi064.stderr:2024-09-21T22:05:50.369+0000 7f65c356c180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-21T22:05:50.680 INFO:tasks.ceph.mgr.x.smithi064.stderr:2024-09-21T22:05:50.677+0000 7f65c356c180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-21T22:05:50.745 INFO:tasks.ceph.mgr.x.smithi064.stderr:2024-09-21T22:05:50.741+0000 7f65c356c180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-21T22:05:51.801 INFO:tasks.ceph.mgr.x.smithi064.stderr:2024-09-21T22:05:51.797+0000 7f656a0a0640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-21T22:05:51.802 INFO:tasks.ceph.mgr.x.smithi064.stderr:2024-09-21T22:05:51.797+0000 7f656a0a0640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-21T22:05:53.207 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:05:53.205+0000 7fa1f3c027c0 -1 osd.0 0 log_to_monitors true 2024-09-21T22:05:53.300 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:05:53.297+0000 7fb13b9057c0 -1 osd.2 0 log_to_monitors true 2024-09-21T22:05:53.302 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:05:53.297+0000 7f6e7f4ee7c0 -1 osd.4 0 log_to_monitors true 2024-09-21T22:05:53.318 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:05:53.313+0000 7fcc60b397c0 -1 osd.3 0 log_to_monitors true 2024-09-21T22:05:53.617 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:05:53.613+0000 7fb10d1027c0 -1 osd.1 0 log_to_monitors true 2024-09-21T22:05:53.728 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:05:53.725+0000 7fc8d6a3b7c0 -1 osd.5 0 log_to_monitors true 2024-09-21T22:05:55.766 DEBUG:teuthology.orchestra.run.smithi039:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-21T22:05:56.066 INFO:teuthology.misc.health.smithi039.stdout: 2024-09-21T22:05:56.067 INFO:teuthology.misc.health.smithi039.stdout:{"epoch":10,"fsid":"00bb09a3-e410-4e18-a2b2-08eaf61978e6","created":"2024-09-21T22:05:45.303341+0000","modified":"2024-09-21T22:05:54.833473+0000","last_up_change":"0.000000","last_in_change":"2024-09-21T22:05:48.083837+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"8336c397-2018-4c75-ab31-2e5f6ad564b5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"bca64af6-7427-4c5d-97a4-a183b1f3a7d9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"5f2678ed-6a23-4a67-96a4-c0ca91254bd7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"aba30555-ef9e-47c9-a38b-ee1a4b8092f5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2a1288d7-618d-4f44-8346-930cf56d4e80","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3ce719a8-34e9-4890-bd40-d22201e7bad8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T22:05:56.094 DEBUG:teuthology.misc:0 of 6 OSDs are up 2024-09-21T22:05:57.924 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:05:57.921+0000 7f6e7b497640 -1 osd.4 0 waiting for initial osdmap 2024-09-21T22:05:57.948 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:05:57.945+0000 7f6e732cc640 -1 osd.4 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T22:05:57.955 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:05:57.953+0000 7fb1380c0640 -1 osd.2 0 waiting for initial osdmap 2024-09-21T22:05:57.974 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:05:57.969+0000 7fb1090ab640 -1 osd.1 0 waiting for initial osdmap 2024-09-21T22:05:57.987 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:05:57.985+0000 7fb12f6e3640 -1 osd.2 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T22:05:57.995 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:05:57.993+0000 7fb100ee0640 -1 osd.1 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T22:05:58.010 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:05:58.009+0000 7fa1efbab640 -1 osd.0 0 waiting for initial osdmap 2024-09-21T22:05:58.043 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:05:58.041+0000 7fa1e79e0640 -1 osd.0 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T22:05:58.048 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:05:58.045+0000 7fc8d29e4640 -1 osd.5 0 waiting for initial osdmap 2024-09-21T22:05:58.076 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:05:58.073+0000 7fc8ca819640 -1 osd.5 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T22:05:58.085 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:05:58.081+0000 7fcc5d2f4640 -1 osd.3 0 waiting for initial osdmap 2024-09-21T22:05:58.109 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:05:58.105+0000 7fcc54917640 -1 osd.3 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T22:06:02.755 DEBUG:teuthology.orchestra.run.smithi039:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-21T22:06:03.063 INFO:teuthology.misc.health.smithi039.stdout: 2024-09-21T22:06:03.063 INFO:teuthology.misc.health.smithi039.stdout:{"epoch":15,"fsid":"00bb09a3-e410-4e18-a2b2-08eaf61978e6","created":"2024-09-21T22:05:45.303341+0000","modified":"2024-09-21T22:06:03.027560+0000","last_up_change":"2024-09-21T22:05:58.958630+0000","last_in_change":"2024-09-21T22:05:48.083837+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T22:06:00.868591+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8336c397-2018-4c75-ab31-2e5f6ad564b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":3054061468},{"type":"v1","addr":"172.21.15.39:6801","nonce":3054061468}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":3054061468},{"type":"v1","addr":"172.21.15.39:6803","nonce":3054061468}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":3054061468},{"type":"v1","addr":"172.21.15.39:6810","nonce":3054061468}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":3054061468},{"type":"v1","addr":"172.21.15.39:6806","nonce":3054061468}]},"public_addr":"172.21.15.39:6801/3054061468","cluster_addr":"172.21.15.39:6803/3054061468","heartbeat_back_addr":"172.21.15.39:6810/3054061468","heartbeat_front_addr":"172.21.15.39:6806/3054061468","state":["exists","up"]},{"osd":1,"uuid":"bca64af6-7427-4c5d-97a4-a183b1f3a7d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":2876495139},{"type":"v1","addr":"172.21.15.39:6817","nonce":2876495139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":2876495139},{"type":"v1","addr":"172.21.15.39:6819","nonce":2876495139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":2876495139},{"type":"v1","addr":"172.21.15.39:6823","nonce":2876495139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":2876495139},{"type":"v1","addr":"172.21.15.39:6821","nonce":2876495139}]},"public_addr":"172.21.15.39:6817/2876495139","cluster_addr":"172.21.15.39:6819/2876495139","heartbeat_back_addr":"172.21.15.39:6823/2876495139","heartbeat_front_addr":"172.21.15.39:6821/2876495139","state":["exists","up"]},{"osd":2,"uuid":"5f2678ed-6a23-4a67-96a4-c0ca91254bd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6805","nonce":1262038957},{"type":"v1","addr":"172.21.15.39:6807","nonce":1262038957}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6809","nonce":1262038957},{"type":"v1","addr":"172.21.15.39:6811","nonce":1262038957}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":1262038957},{"type":"v1","addr":"172.21.15.39:6815","nonce":1262038957}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":1262038957},{"type":"v1","addr":"172.21.15.39:6813","nonce":1262038957}]},"public_addr":"172.21.15.39:6807/1262038957","cluster_addr":"172.21.15.39:6811/1262038957","heartbeat_back_addr":"172.21.15.39:6815/1262038957","heartbeat_front_addr":"172.21.15.39:6813/1262038957","state":["exists","up"]},{"osd":3,"uuid":"aba30555-ef9e-47c9-a38b-ee1a4b8092f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":1132198485},{"type":"v1","addr":"172.21.15.64:6801","nonce":1132198485}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":1132198485},{"type":"v1","addr":"172.21.15.64:6803","nonce":1132198485}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":1132198485},{"type":"v1","addr":"172.21.15.64:6807","nonce":1132198485}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":1132198485},{"type":"v1","addr":"172.21.15.64:6805","nonce":1132198485}]},"public_addr":"172.21.15.64:6801/1132198485","cluster_addr":"172.21.15.64:6803/1132198485","heartbeat_back_addr":"172.21.15.64:6807/1132198485","heartbeat_front_addr":"172.21.15.64:6805/1132198485","state":["exists","up"]},{"osd":4,"uuid":"2a1288d7-618d-4f44-8346-930cf56d4e80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":3500786459},{"type":"v1","addr":"172.21.15.64:6809","nonce":3500786459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":3500786459},{"type":"v1","addr":"172.21.15.64:6811","nonce":3500786459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":3500786459},{"type":"v1","addr":"172.21.15.64:6815","nonce":3500786459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":3500786459},{"type":"v1","addr":"172.21.15.64:6813","nonce":3500786459}]},"public_addr":"172.21.15.64:6809/3500786459","cluster_addr":"172.21.15.64:6811/3500786459","heartbeat_back_addr":"172.21.15.64:6815/3500786459","heartbeat_front_addr":"172.21.15.64:6813/3500786459","state":["exists","up"]},{"osd":5,"uuid":"3ce719a8-34e9-4890-bd40-d22201e7bad8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2604542492},{"type":"v1","addr":"172.21.15.64:6817","nonce":2604542492}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":2604542492},{"type":"v1","addr":"172.21.15.64:6819","nonce":2604542492}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":2604542492},{"type":"v1","addr":"172.21.15.64:6823","nonce":2604542492}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":2604542492},{"type":"v1","addr":"172.21.15.64:6821","nonce":2604542492}]},"public_addr":"172.21.15.64:6817/2604542492","cluster_addr":"172.21.15.64:6819/2604542492","heartbeat_back_addr":"172.21.15.64:6823/2604542492","heartbeat_front_addr":"172.21.15.64:6821/2604542492","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T22:05:54.224881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T22:05:54.668978+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T22:05:54.315876+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T22:05:54.314795+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T22:05:54.322403+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T22:05:54.750633+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T22:06:03.081 DEBUG:teuthology.misc:6 of 6 OSDs are up 2024-09-21T22:06:03.081 INFO:tasks.ceph:Creating RBD pool 2024-09-21T22:06:03.081 DEBUG:teuthology.orchestra.run.smithi039:> sudo ceph --cluster ceph osd pool create rbd 8 2024-09-21T22:06:04.084 INFO:teuthology.orchestra.run.smithi039.stderr:pool 'rbd' created 2024-09-21T22:06:04.108 DEBUG:teuthology.orchestra.run.smithi039:> rbd --cluster ceph pool init rbd 2024-09-21T22:06:07.140 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-21T22:06:07.140 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-21T22:06:07.140 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-21T22:06:07.141 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-21T22:06:07.141 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-21T22:06:07.494 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T22:06:07.512 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":5,"flags":0,"active_gid":4105,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":3757327736},{"type":"v1","addr":"172.21.15.64:6825","nonce":3757327736}]},"active_addr":"172.21.15.64:6825/3757327736","active_change":"2024-09-21T22:05:50.767466+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.64:0","nonce":3080746788}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.64:0","nonce":728381110}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.64:0","nonce":480466644}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.64:0","nonce":1055871292}]}]} 2024-09-21T22:06:07.513 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-21T22:06:07.514 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-21T22:06:07.514 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-21T22:06:07.818 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T22:06:07.818 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":19,"fsid":"00bb09a3-e410-4e18-a2b2-08eaf61978e6","created":"2024-09-21T22:05:45.303341+0000","modified":"2024-09-21T22:06:07.105773+0000","last_up_change":"2024-09-21T22:05:58.958630+0000","last_in_change":"2024-09-21T22:05:48.083837+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T22:06:00.868591+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-21T22:06:03.535857+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":19,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8336c397-2018-4c75-ab31-2e5f6ad564b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":3054061468},{"type":"v1","addr":"172.21.15.39:6801","nonce":3054061468}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":3054061468},{"type":"v1","addr":"172.21.15.39:6803","nonce":3054061468}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":3054061468},{"type":"v1","addr":"172.21.15.39:6810","nonce":3054061468}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":3054061468},{"type":"v1","addr":"172.21.15.39:6806","nonce":3054061468}]},"public_addr":"172.21.15.39:6801/3054061468","cluster_addr":"172.21.15.39:6803/3054061468","heartbeat_back_addr":"172.21.15.39:6810/3054061468","heartbeat_front_addr":"172.21.15.39:6806/3054061468","state":["exists","up"]},{"osd":1,"uuid":"bca64af6-7427-4c5d-97a4-a183b1f3a7d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":2876495139},{"type":"v1","addr":"172.21.15.39:6817","nonce":2876495139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":2876495139},{"type":"v1","addr":"172.21.15.39:6819","nonce":2876495139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":2876495139},{"type":"v1","addr":"172.21.15.39:6823","nonce":2876495139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":2876495139},{"type":"v1","addr":"172.21.15.39:6821","nonce":2876495139}]},"public_addr":"172.21.15.39:6817/2876495139","cluster_addr":"172.21.15.39:6819/2876495139","heartbeat_back_addr":"172.21.15.39:6823/2876495139","heartbeat_front_addr":"172.21.15.39:6821/2876495139","state":["exists","up"]},{"osd":2,"uuid":"5f2678ed-6a23-4a67-96a4-c0ca91254bd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6805","nonce":1262038957},{"type":"v1","addr":"172.21.15.39:6807","nonce":1262038957}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6809","nonce":1262038957},{"type":"v1","addr":"172.21.15.39:6811","nonce":1262038957}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":1262038957},{"type":"v1","addr":"172.21.15.39:6815","nonce":1262038957}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":1262038957},{"type":"v1","addr":"172.21.15.39:6813","nonce":1262038957}]},"public_addr":"172.21.15.39:6807/1262038957","cluster_addr":"172.21.15.39:6811/1262038957","heartbeat_back_addr":"172.21.15.39:6815/1262038957","heartbeat_front_addr":"172.21.15.39:6813/1262038957","state":["exists","up"]},{"osd":3,"uuid":"aba30555-ef9e-47c9-a38b-ee1a4b8092f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":1132198485},{"type":"v1","addr":"172.21.15.64:6801","nonce":1132198485}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":1132198485},{"type":"v1","addr":"172.21.15.64:6803","nonce":1132198485}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":1132198485},{"type":"v1","addr":"172.21.15.64:6807","nonce":1132198485}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":1132198485},{"type":"v1","addr":"172.21.15.64:6805","nonce":1132198485}]},"public_addr":"172.21.15.64:6801/1132198485","cluster_addr":"172.21.15.64:6803/1132198485","heartbeat_back_addr":"172.21.15.64:6807/1132198485","heartbeat_front_addr":"172.21.15.64:6805/1132198485","state":["exists","up"]},{"osd":4,"uuid":"2a1288d7-618d-4f44-8346-930cf56d4e80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":3500786459},{"type":"v1","addr":"172.21.15.64:6809","nonce":3500786459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":3500786459},{"type":"v1","addr":"172.21.15.64:6811","nonce":3500786459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":3500786459},{"type":"v1","addr":"172.21.15.64:6815","nonce":3500786459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":3500786459},{"type":"v1","addr":"172.21.15.64:6813","nonce":3500786459}]},"public_addr":"172.21.15.64:6809/3500786459","cluster_addr":"172.21.15.64:6811/3500786459","heartbeat_back_addr":"172.21.15.64:6815/3500786459","heartbeat_front_addr":"172.21.15.64:6813/3500786459","state":["exists","up"]},{"osd":5,"uuid":"3ce719a8-34e9-4890-bd40-d22201e7bad8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2604542492},{"type":"v1","addr":"172.21.15.64:6817","nonce":2604542492}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":2604542492},{"type":"v1","addr":"172.21.15.64:6819","nonce":2604542492}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":2604542492},{"type":"v1","addr":"172.21.15.64:6823","nonce":2604542492}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":2604542492},{"type":"v1","addr":"172.21.15.64:6821","nonce":2604542492}]},"public_addr":"172.21.15.64:6817/2604542492","cluster_addr":"172.21.15.64:6819/2604542492","heartbeat_back_addr":"172.21.15.64:6823/2604542492","heartbeat_front_addr":"172.21.15.64:6821/2604542492","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T22:05:54.224881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T22:05:54.668978+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T22:05:54.315876+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T22:05:54.314795+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T22:05:54.322403+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T22:05:54.750633+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T22:06:07.836 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-21T22:06:07.836 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-21T22:06:08.128 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T22:06:08.128 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":19,"fsid":"00bb09a3-e410-4e18-a2b2-08eaf61978e6","created":"2024-09-21T22:05:45.303341+0000","modified":"2024-09-21T22:06:07.105773+0000","last_up_change":"2024-09-21T22:05:58.958630+0000","last_in_change":"2024-09-21T22:05:48.083837+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T22:06:00.868591+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-21T22:06:03.535857+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":19,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8336c397-2018-4c75-ab31-2e5f6ad564b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":3054061468},{"type":"v1","addr":"172.21.15.39:6801","nonce":3054061468}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":3054061468},{"type":"v1","addr":"172.21.15.39:6803","nonce":3054061468}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":3054061468},{"type":"v1","addr":"172.21.15.39:6810","nonce":3054061468}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":3054061468},{"type":"v1","addr":"172.21.15.39:6806","nonce":3054061468}]},"public_addr":"172.21.15.39:6801/3054061468","cluster_addr":"172.21.15.39:6803/3054061468","heartbeat_back_addr":"172.21.15.39:6810/3054061468","heartbeat_front_addr":"172.21.15.39:6806/3054061468","state":["exists","up"]},{"osd":1,"uuid":"bca64af6-7427-4c5d-97a4-a183b1f3a7d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":2876495139},{"type":"v1","addr":"172.21.15.39:6817","nonce":2876495139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":2876495139},{"type":"v1","addr":"172.21.15.39:6819","nonce":2876495139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":2876495139},{"type":"v1","addr":"172.21.15.39:6823","nonce":2876495139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":2876495139},{"type":"v1","addr":"172.21.15.39:6821","nonce":2876495139}]},"public_addr":"172.21.15.39:6817/2876495139","cluster_addr":"172.21.15.39:6819/2876495139","heartbeat_back_addr":"172.21.15.39:6823/2876495139","heartbeat_front_addr":"172.21.15.39:6821/2876495139","state":["exists","up"]},{"osd":2,"uuid":"5f2678ed-6a23-4a67-96a4-c0ca91254bd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6805","nonce":1262038957},{"type":"v1","addr":"172.21.15.39:6807","nonce":1262038957}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6809","nonce":1262038957},{"type":"v1","addr":"172.21.15.39:6811","nonce":1262038957}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":1262038957},{"type":"v1","addr":"172.21.15.39:6815","nonce":1262038957}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":1262038957},{"type":"v1","addr":"172.21.15.39:6813","nonce":1262038957}]},"public_addr":"172.21.15.39:6807/1262038957","cluster_addr":"172.21.15.39:6811/1262038957","heartbeat_back_addr":"172.21.15.39:6815/1262038957","heartbeat_front_addr":"172.21.15.39:6813/1262038957","state":["exists","up"]},{"osd":3,"uuid":"aba30555-ef9e-47c9-a38b-ee1a4b8092f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":1132198485},{"type":"v1","addr":"172.21.15.64:6801","nonce":1132198485}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":1132198485},{"type":"v1","addr":"172.21.15.64:6803","nonce":1132198485}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":1132198485},{"type":"v1","addr":"172.21.15.64:6807","nonce":1132198485}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":1132198485},{"type":"v1","addr":"172.21.15.64:6805","nonce":1132198485}]},"public_addr":"172.21.15.64:6801/1132198485","cluster_addr":"172.21.15.64:6803/1132198485","heartbeat_back_addr":"172.21.15.64:6807/1132198485","heartbeat_front_addr":"172.21.15.64:6805/1132198485","state":["exists","up"]},{"osd":4,"uuid":"2a1288d7-618d-4f44-8346-930cf56d4e80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":3500786459},{"type":"v1","addr":"172.21.15.64:6809","nonce":3500786459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":3500786459},{"type":"v1","addr":"172.21.15.64:6811","nonce":3500786459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":3500786459},{"type":"v1","addr":"172.21.15.64:6815","nonce":3500786459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":3500786459},{"type":"v1","addr":"172.21.15.64:6813","nonce":3500786459}]},"public_addr":"172.21.15.64:6809/3500786459","cluster_addr":"172.21.15.64:6811/3500786459","heartbeat_back_addr":"172.21.15.64:6815/3500786459","heartbeat_front_addr":"172.21.15.64:6813/3500786459","state":["exists","up"]},{"osd":5,"uuid":"3ce719a8-34e9-4890-bd40-d22201e7bad8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":2604542492},{"type":"v1","addr":"172.21.15.64:6817","nonce":2604542492}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":2604542492},{"type":"v1","addr":"172.21.15.64:6819","nonce":2604542492}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":2604542492},{"type":"v1","addr":"172.21.15.64:6823","nonce":2604542492}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":2604542492},{"type":"v1","addr":"172.21.15.64:6821","nonce":2604542492}]},"public_addr":"172.21.15.64:6817/2604542492","cluster_addr":"172.21.15.64:6819/2604542492","heartbeat_back_addr":"172.21.15.64:6823/2604542492","heartbeat_front_addr":"172.21.15.64:6821/2604542492","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T22:05:54.224881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T22:05:54.668978+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T22:05:54.315876+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T22:05:54.314795+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T22:05:54.322403+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T22:05:54.750633+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T22:06:08.146 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-21T22:06:08.146 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-21T22:06:08.147 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-21T22:06:08.147 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-21T22:06:08.148 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-21T22:06:08.148 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-21T22:06:08.315 INFO:teuthology.orchestra.run.smithi039.stdout:47244640264 2024-09-21T22:06:08.315 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-21T22:06:08.336 INFO:teuthology.orchestra.run.smithi039.stdout:47244640264 2024-09-21T22:06:08.336 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-21T22:06:08.383 INFO:teuthology.orchestra.run.smithi039.stdout:47244640261 2024-09-21T22:06:08.383 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-21T22:06:08.389 INFO:teuthology.orchestra.run.smithi039.stdout:47244640263 2024-09-21T22:06:08.389 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-21T22:06:08.396 INFO:teuthology.orchestra.run.smithi039.stdout:47244640263 2024-09-21T22:06:08.396 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-21T22:06:08.594 INFO:teuthology.orchestra.run.smithi039.stdout:47244640262 2024-09-21T22:06:08.595 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-21T22:06:08.712 INFO:teuthology.orchestra.run.smithi039.stdout:47244640260 2024-09-21T22:06:08.731 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640261 got 47244640260 for osd.4 2024-09-21T22:06:08.763 INFO:teuthology.orchestra.run.smithi039.stdout:47244640263 2024-09-21T22:06:08.787 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640264 got 47244640263 for osd.3 2024-09-21T22:06:08.810 INFO:teuthology.orchestra.run.smithi039.stdout:47244640263 2024-09-21T22:06:08.833 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640264 got 47244640263 for osd.0 2024-09-21T22:06:08.844 INFO:teuthology.orchestra.run.smithi039.stdout:47244640262 2024-09-21T22:06:08.858 INFO:teuthology.orchestra.run.smithi039.stdout:47244640262 2024-09-21T22:06:08.861 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640263 got 47244640262 for osd.1 2024-09-21T22:06:08.875 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640263 got 47244640262 for osd.2 2024-09-21T22:06:08.888 INFO:teuthology.orchestra.run.smithi039.stdout:47244640261 2024-09-21T22:06:08.905 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640262 got 47244640261 for osd.5 2024-09-21T22:06:09.731 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-21T22:06:09.788 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-21T22:06:09.835 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-21T22:06:09.862 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-21T22:06:09.876 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-21T22:06:09.906 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-21T22:06:10.108 INFO:teuthology.orchestra.run.smithi039.stdout:47244640261 2024-09-21T22:06:10.120 INFO:teuthology.orchestra.run.smithi039.stdout:47244640264 2024-09-21T22:06:10.136 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640261 got 47244640261 for osd.4 2024-09-21T22:06:10.136 DEBUG:teuthology.parallel:result is None 2024-09-21T22:06:10.139 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640264 got 47244640264 for osd.3 2024-09-21T22:06:10.140 DEBUG:teuthology.parallel:result is None 2024-09-21T22:06:10.202 INFO:teuthology.orchestra.run.smithi039.stdout:47244640264 2024-09-21T22:06:10.220 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640264 got 47244640264 for osd.0 2024-09-21T22:06:10.220 DEBUG:teuthology.parallel:result is None 2024-09-21T22:06:10.262 INFO:teuthology.orchestra.run.smithi039.stdout:47244640263 2024-09-21T22:06:10.279 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640263 got 47244640263 for osd.1 2024-09-21T22:06:10.279 DEBUG:teuthology.parallel:result is None 2024-09-21T22:06:10.285 INFO:teuthology.orchestra.run.smithi039.stdout:47244640263 2024-09-21T22:06:10.302 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640263 got 47244640263 for osd.2 2024-09-21T22:06:10.302 DEBUG:teuthology.parallel:result is None 2024-09-21T22:06:10.327 INFO:teuthology.orchestra.run.smithi039.stdout:47244640262 2024-09-21T22:06:10.343 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640262 got 47244640262 for osd.5 2024-09-21T22:06:10.344 DEBUG:teuthology.parallel:result is None 2024-09-21T22:06:10.344 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-21T22:06:10.344 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T22:06:10.664 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T22:06:10.665 INFO:teuthology.orchestra.run.smithi039.stderr:dumped all 2024-09-21T22:06:10.681 INFO:teuthology.orchestra.run.smithi039.stdout:{"pg_ready":true,"pg_map":{"version":20,"stamp":"2024-09-21T22:06:10.015814+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":175872,"kb_used_data":1176,"kb_used_omap":9,"kb_used_meta":174518,"kb_avail":566055168,"statfs":{"total":579820584960,"available":579640492032,"internally_reserved":0,"allocated":1204224,"data_stored":1544350,"data_compressed":13122,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":9568,"internal_metadata":178707104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.922970"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-21T22:06:07.121777+0000","last_change":"2024-09-21T22:06:07.122328+0000","last_active":"2024-09-21T22:06:07.121777+0000","last_peered":"2024-09-21T22:06:07.121777+0000","last_clean":"2024-09-21T22:06:07.121777+0000","last_became_active":"2024-09-21T22:06:05.220996+0000","last_became_peered":"2024-09-21T22:06:05.220996+0000","last_unstale":"2024-09-21T22:06:07.121777+0000","last_undegraded":"2024-09-21T22:06:07.121777+0000","last_fullsized":"2024-09-21T22:06:07.121777+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T23:45:49.178013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001473755,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-21T22:06:07.133265+0000","last_change":"2024-09-21T22:06:07.133757+0000","last_active":"2024-09-21T22:06:07.133265+0000","last_peered":"2024-09-21T22:06:07.133265+0000","last_clean":"2024-09-21T22:06:07.133265+0000","last_became_active":"2024-09-21T22:06:05.109065+0000","last_became_peered":"2024-09-21T22:06:05.109065+0000","last_unstale":"2024-09-21T22:06:07.133265+0000","last_undegraded":"2024-09-21T22:06:07.133265+0000","last_fullsized":"2024-09-21T22:06:07.133265+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:31:12.654893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00088865700000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-21T22:06:07.121241+0000","last_change":"2024-09-21T22:06:07.121697+0000","last_active":"2024-09-21T22:06:07.121241+0000","last_peered":"2024-09-21T22:06:07.121241+0000","last_clean":"2024-09-21T22:06:07.121241+0000","last_became_active":"2024-09-21T22:06:05.412385+0000","last_became_peered":"2024-09-21T22:06:05.412385+0000","last_unstale":"2024-09-21T22:06:07.121241+0000","last_undegraded":"2024-09-21T22:06:07.121241+0000","last_fullsized":"2024-09-21T22:06:07.121241+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T08:39:59.544519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091949900000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-21T22:06:07.133415+0000","last_change":"2024-09-21T22:06:07.133814+0000","last_active":"2024-09-21T22:06:07.133415+0000","last_peered":"2024-09-21T22:06:07.133415+0000","last_clean":"2024-09-21T22:06:07.133415+0000","last_became_active":"2024-09-21T22:06:05.412985+0000","last_became_peered":"2024-09-21T22:06:05.412985+0000","last_unstale":"2024-09-21T22:06:07.133415+0000","last_undegraded":"2024-09-21T22:06:07.133415+0000","last_fullsized":"2024-09-21T22:06:07.133415+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:28:32.737961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00090834100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"19'2","reported_seq":21,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-21T22:06:07.130034+0000","last_change":"2024-09-21T22:06:07.130034+0000","last_active":"2024-09-21T22:06:07.130034+0000","last_peered":"2024-09-21T22:06:07.130034+0000","last_clean":"2024-09-21T22:06:07.130034+0000","last_became_active":"2024-09-21T22:06:05.102579+0000","last_became_peered":"2024-09-21T22:06:05.102579+0000","last_unstale":"2024-09-21T22:06:07.130034+0000","last_undegraded":"2024-09-21T22:06:07.130034+0000","last_fullsized":"2024-09-21T22:06:07.130034+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T05:18:35.468678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026694230000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-21T22:06:07.122955+0000","last_change":"2024-09-21T22:06:07.123496+0000","last_active":"2024-09-21T22:06:07.122955+0000","last_peered":"2024-09-21T22:06:07.122955+0000","last_clean":"2024-09-21T22:06:07.122955+0000","last_became_active":"2024-09-21T22:06:05.104941+0000","last_became_peered":"2024-09-21T22:06:05.104941+0000","last_unstale":"2024-09-21T22:06:07.122955+0000","last_undegraded":"2024-09-21T22:06:07.122955+0000","last_fullsized":"2024-09-21T22:06:07.122955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:46:29.901819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010052629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-21T22:06:07.121148+0000","last_change":"2024-09-21T22:06:07.121573+0000","last_active":"2024-09-21T22:06:07.121148+0000","last_peered":"2024-09-21T22:06:07.121148+0000","last_clean":"2024-09-21T22:06:07.121148+0000","last_became_active":"2024-09-21T22:06:05.108187+0000","last_became_peered":"2024-09-21T22:06:05.108187+0000","last_unstale":"2024-09-21T22:06:07.121148+0000","last_undegraded":"2024-09-21T22:06:07.121148+0000","last_fullsized":"2024-09-21T22:06:07.121148+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T05:05:08.108341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079478100000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"17'1","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-21T22:06:07.127794+0000","last_change":"2024-09-21T22:06:07.128525+0000","last_active":"2024-09-21T22:06:07.127794+0000","last_peered":"2024-09-21T22:06:07.127794+0000","last_clean":"2024-09-21T22:06:07.127794+0000","last_became_active":"2024-09-21T22:06:05.101507+0000","last_became_peered":"2024-09-21T22:06:05.101507+0000","last_unstale":"2024-09-21T22:06:07.127794+0000","last_undegraded":"2024-09-21T22:06:07.127794+0000","last_fullsized":"2024-09-21T22:06:07.127794+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:43:06.377355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001125968,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"15'117","reported_seq":170,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-21T22:06:07.120548+0000","last_change":"2024-09-21T22:06:02.096716+0000","last_active":"2024-09-21T22:06:07.120548+0000","last_peered":"2024-09-21T22:06:07.120548+0000","last_clean":"2024-09-21T22:06:07.120548+0000","last_became_active":"2024-09-21T22:06:02.094354+0000","last_became_peered":"2024-09-21T22:06:02.094354+0000","last_unstale":"2024-09-21T22:06:07.120548+0000","last_undegraded":"2024-09-21T22:06:07.120548+0000","last_fullsized":"2024-09-21T22:06:07.120548+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:00.990331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:00.990331+0000","last_clean_scrub_stamp":"2024-09-21T22:06:00.990331+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:19:52.359189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":13122,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640263,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27216,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344624,"statfs":{"total":96636764160,"available":96608894976,"internally_reserved":0,"allocated":184320,"data_stored":60615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640262,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27208,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":180224,"data_stored":60596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640265,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":237568,"data_stored":650964,"data_compressed":6561,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640264,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31368,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340472,"statfs":{"total":96636764160,"available":96604643328,"internally_reserved":0,"allocated":180224,"data_stored":60596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640264,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31376,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340464,"statfs":{"total":96636764160,"available":96604635136,"internally_reserved":0,"allocated":184320,"data_stored":60615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640265,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31420,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340420,"statfs":{"total":96636764160,"available":96604590080,"internally_reserved":0,"allocated":237568,"data_stored":650964,"data_compressed":6561,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6561,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6561,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T22:06:10.682 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T22:06:10.968 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T22:06:10.968 INFO:teuthology.orchestra.run.smithi039.stderr:dumped all 2024-09-21T22:06:10.985 INFO:teuthology.orchestra.run.smithi039.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-09-21T22:06:10.788816+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":175872,"kb_used_data":1176,"kb_used_omap":9,"kb_used_meta":174518,"kb_avail":566055168,"statfs":{"total":579820584960,"available":579640492032,"internally_reserved":0,"allocated":1204224,"data_stored":1544350,"data_compressed":13122,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":9568,"internal_metadata":178707104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.695972"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-21T22:06:07.121777+0000","last_change":"2024-09-21T22:06:07.122328+0000","last_active":"2024-09-21T22:06:07.121777+0000","last_peered":"2024-09-21T22:06:07.121777+0000","last_clean":"2024-09-21T22:06:07.121777+0000","last_became_active":"2024-09-21T22:06:05.220996+0000","last_became_peered":"2024-09-21T22:06:05.220996+0000","last_unstale":"2024-09-21T22:06:07.121777+0000","last_undegraded":"2024-09-21T22:06:07.121777+0000","last_fullsized":"2024-09-21T22:06:07.121777+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T23:45:49.178013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001473755,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-21T22:06:07.133265+0000","last_change":"2024-09-21T22:06:07.133757+0000","last_active":"2024-09-21T22:06:07.133265+0000","last_peered":"2024-09-21T22:06:07.133265+0000","last_clean":"2024-09-21T22:06:07.133265+0000","last_became_active":"2024-09-21T22:06:05.109065+0000","last_became_peered":"2024-09-21T22:06:05.109065+0000","last_unstale":"2024-09-21T22:06:07.133265+0000","last_undegraded":"2024-09-21T22:06:07.133265+0000","last_fullsized":"2024-09-21T22:06:07.133265+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:31:12.654893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00088865700000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-21T22:06:07.121241+0000","last_change":"2024-09-21T22:06:07.121697+0000","last_active":"2024-09-21T22:06:07.121241+0000","last_peered":"2024-09-21T22:06:07.121241+0000","last_clean":"2024-09-21T22:06:07.121241+0000","last_became_active":"2024-09-21T22:06:05.412385+0000","last_became_peered":"2024-09-21T22:06:05.412385+0000","last_unstale":"2024-09-21T22:06:07.121241+0000","last_undegraded":"2024-09-21T22:06:07.121241+0000","last_fullsized":"2024-09-21T22:06:07.121241+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T08:39:59.544519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091949900000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-21T22:06:07.133415+0000","last_change":"2024-09-21T22:06:07.133814+0000","last_active":"2024-09-21T22:06:07.133415+0000","last_peered":"2024-09-21T22:06:07.133415+0000","last_clean":"2024-09-21T22:06:07.133415+0000","last_became_active":"2024-09-21T22:06:05.412985+0000","last_became_peered":"2024-09-21T22:06:05.412985+0000","last_unstale":"2024-09-21T22:06:07.133415+0000","last_undegraded":"2024-09-21T22:06:07.133415+0000","last_fullsized":"2024-09-21T22:06:07.133415+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:28:32.737961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00090834100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"19'2","reported_seq":21,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-21T22:06:07.130034+0000","last_change":"2024-09-21T22:06:07.130034+0000","last_active":"2024-09-21T22:06:07.130034+0000","last_peered":"2024-09-21T22:06:07.130034+0000","last_clean":"2024-09-21T22:06:07.130034+0000","last_became_active":"2024-09-21T22:06:05.102579+0000","last_became_peered":"2024-09-21T22:06:05.102579+0000","last_unstale":"2024-09-21T22:06:07.130034+0000","last_undegraded":"2024-09-21T22:06:07.130034+0000","last_fullsized":"2024-09-21T22:06:07.130034+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T05:18:35.468678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026694230000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-21T22:06:07.122955+0000","last_change":"2024-09-21T22:06:07.123496+0000","last_active":"2024-09-21T22:06:07.122955+0000","last_peered":"2024-09-21T22:06:07.122955+0000","last_clean":"2024-09-21T22:06:07.122955+0000","last_became_active":"2024-09-21T22:06:05.104941+0000","last_became_peered":"2024-09-21T22:06:05.104941+0000","last_unstale":"2024-09-21T22:06:07.122955+0000","last_undegraded":"2024-09-21T22:06:07.122955+0000","last_fullsized":"2024-09-21T22:06:07.122955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:46:29.901819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010052629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-21T22:06:07.121148+0000","last_change":"2024-09-21T22:06:07.121573+0000","last_active":"2024-09-21T22:06:07.121148+0000","last_peered":"2024-09-21T22:06:07.121148+0000","last_clean":"2024-09-21T22:06:07.121148+0000","last_became_active":"2024-09-21T22:06:05.108187+0000","last_became_peered":"2024-09-21T22:06:05.108187+0000","last_unstale":"2024-09-21T22:06:07.121148+0000","last_undegraded":"2024-09-21T22:06:07.121148+0000","last_fullsized":"2024-09-21T22:06:07.121148+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T05:05:08.108341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079478100000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"17'1","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-21T22:06:07.127794+0000","last_change":"2024-09-21T22:06:07.128525+0000","last_active":"2024-09-21T22:06:07.127794+0000","last_peered":"2024-09-21T22:06:07.127794+0000","last_clean":"2024-09-21T22:06:07.127794+0000","last_became_active":"2024-09-21T22:06:05.101507+0000","last_became_peered":"2024-09-21T22:06:05.101507+0000","last_unstale":"2024-09-21T22:06:07.127794+0000","last_undegraded":"2024-09-21T22:06:07.127794+0000","last_fullsized":"2024-09-21T22:06:07.127794+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:43:06.377355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001125968,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"15'117","reported_seq":170,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-21T22:06:07.120548+0000","last_change":"2024-09-21T22:06:02.096716+0000","last_active":"2024-09-21T22:06:07.120548+0000","last_peered":"2024-09-21T22:06:07.120548+0000","last_clean":"2024-09-21T22:06:07.120548+0000","last_became_active":"2024-09-21T22:06:02.094354+0000","last_became_peered":"2024-09-21T22:06:02.094354+0000","last_unstale":"2024-09-21T22:06:07.120548+0000","last_undegraded":"2024-09-21T22:06:07.120548+0000","last_fullsized":"2024-09-21T22:06:07.120548+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:00.990331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:00.990331+0000","last_clean_scrub_stamp":"2024-09-21T22:06:00.990331+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:19:52.359189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":13122,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640263,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27216,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344624,"statfs":{"total":96636764160,"available":96608894976,"internally_reserved":0,"allocated":184320,"data_stored":60615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640262,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27208,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":180224,"data_stored":60596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640265,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":237568,"data_stored":650964,"data_compressed":6561,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640264,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31368,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340472,"statfs":{"total":96636764160,"available":96604643328,"internally_reserved":0,"allocated":180224,"data_stored":60596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640264,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31376,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340464,"statfs":{"total":96636764160,"available":96604635136,"internally_reserved":0,"allocated":184320,"data_stored":60615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640265,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31420,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340420,"statfs":{"total":96636764160,"available":96604590080,"internally_reserved":0,"allocated":237568,"data_stored":650964,"data_compressed":6561,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6561,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6561,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T22:06:10.986 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-21T22:06:10.986 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-21T22:06:10.986 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-21T22:06:10.986 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-21T22:06:11.299 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T22:06:11.300 INFO:teuthology.orchestra.run.smithi039.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-21T22:06:11.317 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-21T22:06:11.318 INFO:teuthology.run_tasks:Running task thrashosds... 2024-09-21T22:06:11.336 INFO:tasks.thrashosds:config is {'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-09-21T22:06:11.336 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'squid'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zlib', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub'], 'sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'extra_packages': ['rbd-nbd'], 'flavor': 'default', 'sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'squid', 'sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7'}} 2024-09-21T22:06:11.337 INFO:tasks.thrashosds:config is {'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0, 'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5} 2024-09-21T22:06:11.337 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-09-21T22:06:11.655 INFO:tasks.thrashosds:Beginning thrashosds... 2024-09-21T22:06:11.656 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-21T22:06:11.656 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T22:06:11.951 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T22:06:11.952 INFO:teuthology.orchestra.run.smithi039.stderr:dumped all 2024-09-21T22:06:11.970 INFO:teuthology.orchestra.run.smithi039.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-09-21T22:06:10.788816+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":175872,"kb_used_data":1176,"kb_used_omap":9,"kb_used_meta":174518,"kb_avail":566055168,"statfs":{"total":579820584960,"available":579640492032,"internally_reserved":0,"allocated":1204224,"data_stored":1544350,"data_compressed":13122,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":9568,"internal_metadata":178707104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.695972"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-21T22:06:07.121777+0000","last_change":"2024-09-21T22:06:07.122328+0000","last_active":"2024-09-21T22:06:07.121777+0000","last_peered":"2024-09-21T22:06:07.121777+0000","last_clean":"2024-09-21T22:06:07.121777+0000","last_became_active":"2024-09-21T22:06:05.220996+0000","last_became_peered":"2024-09-21T22:06:05.220996+0000","last_unstale":"2024-09-21T22:06:07.121777+0000","last_undegraded":"2024-09-21T22:06:07.121777+0000","last_fullsized":"2024-09-21T22:06:07.121777+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T23:45:49.178013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001473755,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-21T22:06:07.133265+0000","last_change":"2024-09-21T22:06:07.133757+0000","last_active":"2024-09-21T22:06:07.133265+0000","last_peered":"2024-09-21T22:06:07.133265+0000","last_clean":"2024-09-21T22:06:07.133265+0000","last_became_active":"2024-09-21T22:06:05.109065+0000","last_became_peered":"2024-09-21T22:06:05.109065+0000","last_unstale":"2024-09-21T22:06:07.133265+0000","last_undegraded":"2024-09-21T22:06:07.133265+0000","last_fullsized":"2024-09-21T22:06:07.133265+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:31:12.654893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00088865700000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-21T22:06:07.121241+0000","last_change":"2024-09-21T22:06:07.121697+0000","last_active":"2024-09-21T22:06:07.121241+0000","last_peered":"2024-09-21T22:06:07.121241+0000","last_clean":"2024-09-21T22:06:07.121241+0000","last_became_active":"2024-09-21T22:06:05.412385+0000","last_became_peered":"2024-09-21T22:06:05.412385+0000","last_unstale":"2024-09-21T22:06:07.121241+0000","last_undegraded":"2024-09-21T22:06:07.121241+0000","last_fullsized":"2024-09-21T22:06:07.121241+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T08:39:59.544519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091949900000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-21T22:06:07.133415+0000","last_change":"2024-09-21T22:06:07.133814+0000","last_active":"2024-09-21T22:06:07.133415+0000","last_peered":"2024-09-21T22:06:07.133415+0000","last_clean":"2024-09-21T22:06:07.133415+0000","last_became_active":"2024-09-21T22:06:05.412985+0000","last_became_peered":"2024-09-21T22:06:05.412985+0000","last_unstale":"2024-09-21T22:06:07.133415+0000","last_undegraded":"2024-09-21T22:06:07.133415+0000","last_fullsized":"2024-09-21T22:06:07.133415+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:28:32.737961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00090834100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"19'2","reported_seq":21,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-21T22:06:07.130034+0000","last_change":"2024-09-21T22:06:07.130034+0000","last_active":"2024-09-21T22:06:07.130034+0000","last_peered":"2024-09-21T22:06:07.130034+0000","last_clean":"2024-09-21T22:06:07.130034+0000","last_became_active":"2024-09-21T22:06:05.102579+0000","last_became_peered":"2024-09-21T22:06:05.102579+0000","last_unstale":"2024-09-21T22:06:07.130034+0000","last_undegraded":"2024-09-21T22:06:07.130034+0000","last_fullsized":"2024-09-21T22:06:07.130034+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T05:18:35.468678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026694230000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-21T22:06:07.122955+0000","last_change":"2024-09-21T22:06:07.123496+0000","last_active":"2024-09-21T22:06:07.122955+0000","last_peered":"2024-09-21T22:06:07.122955+0000","last_clean":"2024-09-21T22:06:07.122955+0000","last_became_active":"2024-09-21T22:06:05.104941+0000","last_became_peered":"2024-09-21T22:06:05.104941+0000","last_unstale":"2024-09-21T22:06:07.122955+0000","last_undegraded":"2024-09-21T22:06:07.122955+0000","last_fullsized":"2024-09-21T22:06:07.122955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:46:29.901819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010052629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-21T22:06:07.121148+0000","last_change":"2024-09-21T22:06:07.121573+0000","last_active":"2024-09-21T22:06:07.121148+0000","last_peered":"2024-09-21T22:06:07.121148+0000","last_clean":"2024-09-21T22:06:07.121148+0000","last_became_active":"2024-09-21T22:06:05.108187+0000","last_became_peered":"2024-09-21T22:06:05.108187+0000","last_unstale":"2024-09-21T22:06:07.121148+0000","last_undegraded":"2024-09-21T22:06:07.121148+0000","last_fullsized":"2024-09-21T22:06:07.121148+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T05:05:08.108341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079478100000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"17'1","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-21T22:06:07.127794+0000","last_change":"2024-09-21T22:06:07.128525+0000","last_active":"2024-09-21T22:06:07.127794+0000","last_peered":"2024-09-21T22:06:07.127794+0000","last_clean":"2024-09-21T22:06:07.127794+0000","last_became_active":"2024-09-21T22:06:05.101507+0000","last_became_peered":"2024-09-21T22:06:05.101507+0000","last_unstale":"2024-09-21T22:06:07.127794+0000","last_undegraded":"2024-09-21T22:06:07.127794+0000","last_fullsized":"2024-09-21T22:06:07.127794+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:43:06.377355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001125968,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"15'117","reported_seq":170,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-21T22:06:07.120548+0000","last_change":"2024-09-21T22:06:02.096716+0000","last_active":"2024-09-21T22:06:07.120548+0000","last_peered":"2024-09-21T22:06:07.120548+0000","last_clean":"2024-09-21T22:06:07.120548+0000","last_became_active":"2024-09-21T22:06:02.094354+0000","last_became_peered":"2024-09-21T22:06:02.094354+0000","last_unstale":"2024-09-21T22:06:07.120548+0000","last_undegraded":"2024-09-21T22:06:07.120548+0000","last_fullsized":"2024-09-21T22:06:07.120548+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:00.990331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:00.990331+0000","last_clean_scrub_stamp":"2024-09-21T22:06:00.990331+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:19:52.359189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":13122,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640263,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27216,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344624,"statfs":{"total":96636764160,"available":96608894976,"internally_reserved":0,"allocated":184320,"data_stored":60615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640262,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27208,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":180224,"data_stored":60596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640265,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":237568,"data_stored":650964,"data_compressed":6561,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640264,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31368,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340472,"statfs":{"total":96636764160,"available":96604643328,"internally_reserved":0,"allocated":180224,"data_stored":60596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640264,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31376,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340464,"statfs":{"total":96636764160,"available":96604635136,"internally_reserved":0,"allocated":184320,"data_stored":60615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640265,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31420,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340420,"statfs":{"total":96636764160,"available":96604590080,"internally_reserved":0,"allocated":237568,"data_stored":650964,"data_compressed":6561,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6561,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6561,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T22:06:11.970 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T22:06:12.273 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T22:06:12.274 INFO:teuthology.orchestra.run.smithi039.stderr:dumped all 2024-09-21T22:06:12.292 INFO:teuthology.orchestra.run.smithi039.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-09-21T22:06:10.788816+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":175872,"kb_used_data":1176,"kb_used_omap":9,"kb_used_meta":174518,"kb_avail":566055168,"statfs":{"total":579820584960,"available":579640492032,"internally_reserved":0,"allocated":1204224,"data_stored":1544350,"data_compressed":13122,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":9568,"internal_metadata":178707104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.695972"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-21T22:06:07.121777+0000","last_change":"2024-09-21T22:06:07.122328+0000","last_active":"2024-09-21T22:06:07.121777+0000","last_peered":"2024-09-21T22:06:07.121777+0000","last_clean":"2024-09-21T22:06:07.121777+0000","last_became_active":"2024-09-21T22:06:05.220996+0000","last_became_peered":"2024-09-21T22:06:05.220996+0000","last_unstale":"2024-09-21T22:06:07.121777+0000","last_undegraded":"2024-09-21T22:06:07.121777+0000","last_fullsized":"2024-09-21T22:06:07.121777+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T23:45:49.178013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001473755,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-21T22:06:07.133265+0000","last_change":"2024-09-21T22:06:07.133757+0000","last_active":"2024-09-21T22:06:07.133265+0000","last_peered":"2024-09-21T22:06:07.133265+0000","last_clean":"2024-09-21T22:06:07.133265+0000","last_became_active":"2024-09-21T22:06:05.109065+0000","last_became_peered":"2024-09-21T22:06:05.109065+0000","last_unstale":"2024-09-21T22:06:07.133265+0000","last_undegraded":"2024-09-21T22:06:07.133265+0000","last_fullsized":"2024-09-21T22:06:07.133265+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:31:12.654893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00088865700000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-21T22:06:07.121241+0000","last_change":"2024-09-21T22:06:07.121697+0000","last_active":"2024-09-21T22:06:07.121241+0000","last_peered":"2024-09-21T22:06:07.121241+0000","last_clean":"2024-09-21T22:06:07.121241+0000","last_became_active":"2024-09-21T22:06:05.412385+0000","last_became_peered":"2024-09-21T22:06:05.412385+0000","last_unstale":"2024-09-21T22:06:07.121241+0000","last_undegraded":"2024-09-21T22:06:07.121241+0000","last_fullsized":"2024-09-21T22:06:07.121241+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T08:39:59.544519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091949900000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-21T22:06:07.133415+0000","last_change":"2024-09-21T22:06:07.133814+0000","last_active":"2024-09-21T22:06:07.133415+0000","last_peered":"2024-09-21T22:06:07.133415+0000","last_clean":"2024-09-21T22:06:07.133415+0000","last_became_active":"2024-09-21T22:06:05.412985+0000","last_became_peered":"2024-09-21T22:06:05.412985+0000","last_unstale":"2024-09-21T22:06:07.133415+0000","last_undegraded":"2024-09-21T22:06:07.133415+0000","last_fullsized":"2024-09-21T22:06:07.133415+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:28:32.737961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00090834100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"19'2","reported_seq":21,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-21T22:06:07.130034+0000","last_change":"2024-09-21T22:06:07.130034+0000","last_active":"2024-09-21T22:06:07.130034+0000","last_peered":"2024-09-21T22:06:07.130034+0000","last_clean":"2024-09-21T22:06:07.130034+0000","last_became_active":"2024-09-21T22:06:05.102579+0000","last_became_peered":"2024-09-21T22:06:05.102579+0000","last_unstale":"2024-09-21T22:06:07.130034+0000","last_undegraded":"2024-09-21T22:06:07.130034+0000","last_fullsized":"2024-09-21T22:06:07.130034+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T05:18:35.468678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026694230000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-21T22:06:07.122955+0000","last_change":"2024-09-21T22:06:07.123496+0000","last_active":"2024-09-21T22:06:07.122955+0000","last_peered":"2024-09-21T22:06:07.122955+0000","last_clean":"2024-09-21T22:06:07.122955+0000","last_became_active":"2024-09-21T22:06:05.104941+0000","last_became_peered":"2024-09-21T22:06:05.104941+0000","last_unstale":"2024-09-21T22:06:07.122955+0000","last_undegraded":"2024-09-21T22:06:07.122955+0000","last_fullsized":"2024-09-21T22:06:07.122955+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:46:29.901819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010052629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-21T22:06:07.121148+0000","last_change":"2024-09-21T22:06:07.121573+0000","last_active":"2024-09-21T22:06:07.121148+0000","last_peered":"2024-09-21T22:06:07.121148+0000","last_clean":"2024-09-21T22:06:07.121148+0000","last_became_active":"2024-09-21T22:06:05.108187+0000","last_became_peered":"2024-09-21T22:06:05.108187+0000","last_unstale":"2024-09-21T22:06:07.121148+0000","last_undegraded":"2024-09-21T22:06:07.121148+0000","last_fullsized":"2024-09-21T22:06:07.121148+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T05:05:08.108341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079478100000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"17'1","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-21T22:06:07.127794+0000","last_change":"2024-09-21T22:06:07.128525+0000","last_active":"2024-09-21T22:06:07.127794+0000","last_peered":"2024-09-21T22:06:07.127794+0000","last_clean":"2024-09-21T22:06:07.127794+0000","last_became_active":"2024-09-21T22:06:05.101507+0000","last_became_peered":"2024-09-21T22:06:05.101507+0000","last_unstale":"2024-09-21T22:06:07.127794+0000","last_undegraded":"2024-09-21T22:06:07.127794+0000","last_fullsized":"2024-09-21T22:06:07.127794+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:43:06.377355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001125968,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"15'117","reported_seq":170,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-21T22:06:07.120548+0000","last_change":"2024-09-21T22:06:02.096716+0000","last_active":"2024-09-21T22:06:07.120548+0000","last_peered":"2024-09-21T22:06:07.120548+0000","last_clean":"2024-09-21T22:06:07.120548+0000","last_became_active":"2024-09-21T22:06:02.094354+0000","last_became_peered":"2024-09-21T22:06:02.094354+0000","last_unstale":"2024-09-21T22:06:07.120548+0000","last_undegraded":"2024-09-21T22:06:07.120548+0000","last_fullsized":"2024-09-21T22:06:07.120548+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:00.990331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:00.990331+0000","last_clean_scrub_stamp":"2024-09-21T22:06:00.990331+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:19:52.359189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":1180736,"data_compressed":13122,"data_compressed_allocated":73728,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640263,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27216,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344624,"statfs":{"total":96636764160,"available":96608894976,"internally_reserved":0,"allocated":184320,"data_stored":60615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640262,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27208,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":180224,"data_stored":60596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640265,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":237568,"data_stored":650964,"data_compressed":6561,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640264,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31368,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340472,"statfs":{"total":96636764160,"available":96604643328,"internally_reserved":0,"allocated":180224,"data_stored":60596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640264,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31376,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340464,"statfs":{"total":96636764160,"available":96604635136,"internally_reserved":0,"allocated":184320,"data_stored":60615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640265,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31420,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340420,"statfs":{"total":96636764160,"available":96604590080,"internally_reserved":0,"allocated":237568,"data_stored":650964,"data_compressed":6561,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6561,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6561,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T22:06:12.293 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-21T22:06:12.293 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump 2024-09-21T22:06:12.594 INFO:teuthology.orchestra.run.smithi039.stdout:epoch 20 2024-09-21T22:06:12.595 INFO:teuthology.orchestra.run.smithi039.stdout:fsid 00bb09a3-e410-4e18-a2b2-08eaf61978e6 2024-09-21T22:06:12.595 INFO:teuthology.orchestra.run.smithi039.stdout:created 2024-09-21T22:05:45.303341+0000 2024-09-21T22:06:12.595 INFO:teuthology.orchestra.run.smithi039.stdout:modified 2024-09-21T22:06:10.007899+0000 2024-09-21T22:06:12.595 INFO:teuthology.orchestra.run.smithi039.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-09-21T22:06:12.595 INFO:teuthology.orchestra.run.smithi039.stdout:crush_version 4 2024-09-21T22:06:12.595 INFO:teuthology.orchestra.run.smithi039.stdout:full_ratio 0.95 2024-09-21T22:06:12.595 INFO:teuthology.orchestra.run.smithi039.stdout:backfillfull_ratio 0.9 2024-09-21T22:06:12.595 INFO:teuthology.orchestra.run.smithi039.stdout:nearfull_ratio 0.85 2024-09-21T22:06:12.595 INFO:teuthology.orchestra.run.smithi039.stdout:require_min_compat_client luminous 2024-09-21T22:06:12.595 INFO:teuthology.orchestra.run.smithi039.stdout:min_compat_client jewel 2024-09-21T22:06:12.595 INFO:teuthology.orchestra.run.smithi039.stdout:require_osd_release squid 2024-09-21T22:06:12.595 INFO:teuthology.orchestra.run.smithi039.stdout:stretch_mode_enabled false 2024-09-21T22:06:12.596 INFO:teuthology.orchestra.run.smithi039.stdout:pool 1 '.mgr' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 15 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 6.06 2024-09-21T22:06:12.596 INFO:teuthology.orchestra.run.smithi039.stdout:pool 2 'rbd' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 19 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 2.25 2024-09-21T22:06:12.596 INFO:teuthology.orchestra.run.smithi039.stdout:max_osd 6 2024-09-21T22:06:12.596 INFO:teuthology.orchestra.run.smithi039.stdout:osd.0 up in weight 1 up_from 11 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.39:6800/3054061468,v1:172.21.15.39:6801/3054061468] [v2:172.21.15.39:6802/3054061468,v1:172.21.15.39:6803/3054061468] exists,up 8336c397-2018-4c75-ab31-2e5f6ad564b5 2024-09-21T22:06:12.596 INFO:teuthology.orchestra.run.smithi039.stdout:osd.1 up in weight 1 up_from 11 up_thru 16 down_at 0 last_clean_interval [0,0) [v2:172.21.15.39:6816/2876495139,v1:172.21.15.39:6817/2876495139] [v2:172.21.15.39:6818/2876495139,v1:172.21.15.39:6819/2876495139] exists,up bca64af6-7427-4c5d-97a4-a183b1f3a7d9 2024-09-21T22:06:12.596 INFO:teuthology.orchestra.run.smithi039.stdout:osd.2 up in weight 1 up_from 11 up_thru 16 down_at 0 last_clean_interval [0,0) [v2:172.21.15.39:6805/1262038957,v1:172.21.15.39:6807/1262038957] [v2:172.21.15.39:6809/1262038957,v1:172.21.15.39:6811/1262038957] exists,up 5f2678ed-6a23-4a67-96a4-c0ca91254bd7 2024-09-21T22:06:12.596 INFO:teuthology.orchestra.run.smithi039.stdout:osd.3 up in weight 1 up_from 11 up_thru 16 down_at 0 last_clean_interval [0,0) [v2:172.21.15.64:6800/1132198485,v1:172.21.15.64:6801/1132198485] [v2:172.21.15.64:6802/1132198485,v1:172.21.15.64:6803/1132198485] exists,up aba30555-ef9e-47c9-a38b-ee1a4b8092f5 2024-09-21T22:06:12.596 INFO:teuthology.orchestra.run.smithi039.stdout:osd.4 up in weight 1 up_from 11 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.64:6808/3500786459,v1:172.21.15.64:6809/3500786459] [v2:172.21.15.64:6810/3500786459,v1:172.21.15.64:6811/3500786459] exists,up 2a1288d7-618d-4f44-8346-930cf56d4e80 2024-09-21T22:06:12.596 INFO:teuthology.orchestra.run.smithi039.stdout:osd.5 up in weight 1 up_from 11 up_thru 16 down_at 0 last_clean_interval [0,0) [v2:172.21.15.64:6816/2604542492,v1:172.21.15.64:6817/2604542492] [v2:172.21.15.64:6818/2604542492,v1:172.21.15.64:6819/2604542492] exists,up 3ce719a8-34e9-4890-bd40-d22201e7bad8 2024-09-21T22:06:12.612 INFO:tasks.ceph.ceph_manager.ceph:['osd.0 up in weight 1 up_from 11 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.39:6800/3054061468,v1:172.21.15.39:6801/3054061468] [v2:172.21.15.39:6802/3054061468,v1:172.21.15.39:6803/3054061468] exists,up 8336c397-2018-4c75-ab31-2e5f6ad564b5', 'osd.1 up in weight 1 up_from 11 up_thru 16 down_at 0 last_clean_interval [0,0) [v2:172.21.15.39:6816/2876495139,v1:172.21.15.39:6817/2876495139] [v2:172.21.15.39:6818/2876495139,v1:172.21.15.39:6819/2876495139] exists,up bca64af6-7427-4c5d-97a4-a183b1f3a7d9', 'osd.2 up in weight 1 up_from 11 up_thru 16 down_at 0 last_clean_interval [0,0) [v2:172.21.15.39:6805/1262038957,v1:172.21.15.39:6807/1262038957] [v2:172.21.15.39:6809/1262038957,v1:172.21.15.39:6811/1262038957] exists,up 5f2678ed-6a23-4a67-96a4-c0ca91254bd7', 'osd.3 up in weight 1 up_from 11 up_thru 16 down_at 0 last_clean_interval [0,0) [v2:172.21.15.64:6800/1132198485,v1:172.21.15.64:6801/1132198485] [v2:172.21.15.64:6802/1132198485,v1:172.21.15.64:6803/1132198485] exists,up aba30555-ef9e-47c9-a38b-ee1a4b8092f5', 'osd.4 up in weight 1 up_from 11 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.64:6808/3500786459,v1:172.21.15.64:6809/3500786459] [v2:172.21.15.64:6810/3500786459,v1:172.21.15.64:6811/3500786459] exists,up 2a1288d7-618d-4f44-8346-930cf56d4e80', 'osd.5 up in weight 1 up_from 11 up_thru 16 down_at 0 last_clean_interval [0,0) [v2:172.21.15.64:6816/2604542492,v1:172.21.15.64:6817/2604542492] [v2:172.21.15.64:6818/2604542492,v1:172.21.15.64:6819/2604542492] exists,up 3ce719a8-34e9-4890-bd40-d22201e7bad8'] 2024-09-21T22:06:12.612 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok config show 2024-09-21T22:06:12.716 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:06:12.716 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "mon.a", 2024-09-21T22:06:12.716 INFO:teuthology.orchestra.run.smithi039.stdout: "cluster": "ceph", 2024-09-21T22:06:12.716 INFO:teuthology.orchestra.run.smithi039.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-09-21T22:06:12.716 INFO:teuthology.orchestra.run.smithi039.stdout: "admin_socket_mode": "", 2024-09-21T22:06:12.716 INFO:teuthology.orchestra.run.smithi039.stdout: "auth_allow_insecure_global_id_reclaim": "true", 2024-09-21T22:06:12.717 INFO:teuthology.orchestra.run.smithi039.stdout: "auth_client_required": "cephx, none", 2024-09-21T22:06:12.717 INFO:teuthology.orchestra.run.smithi039.stdout: "auth_cluster_required": "cephx", 2024-09-21T22:06:12.717 INFO:teuthology.orchestra.run.smithi039.stdout: "auth_debug": "true", 2024-09-21T22:06:12.717 INFO:teuthology.orchestra.run.smithi039.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-09-21T22:06:12.717 INFO:teuthology.orchestra.run.smithi039.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-09-21T22:06:12.717 INFO:teuthology.orchestra.run.smithi039.stdout: "auth_service_required": "cephx", 2024-09-21T22:06:12.717 INFO:teuthology.orchestra.run.smithi039.stdout: "auth_service_ticket_ttl": "240.000000", 2024-09-21T22:06:12.717 INFO:teuthology.orchestra.run.smithi039.stdout: "auth_supported": "cephx", 2024-09-21T22:06:12.717 INFO:teuthology.orchestra.run.smithi039.stdout: "bdev_aio": "true", 2024-09-21T22:06:12.717 INFO:teuthology.orchestra.run.smithi039.stdout: "bdev_aio_max_queue_depth": "1024", 2024-09-21T22:06:12.717 INFO:teuthology.orchestra.run.smithi039.stdout: "bdev_aio_poll_ms": "250", 2024-09-21T22:06:12.717 INFO:teuthology.orchestra.run.smithi039.stdout: "bdev_aio_reap_max": "16", 2024-09-21T22:06:12.717 INFO:teuthology.orchestra.run.smithi039.stdout: "bdev_async_discard": "false", 2024-09-21T22:06:12.717 INFO:teuthology.orchestra.run.smithi039.stdout: "bdev_block_size": "4096", 2024-09-21T22:06:12.717 INFO:teuthology.orchestra.run.smithi039.stdout: "bdev_debug_aio": "false", 2024-09-21T22:06:12.718 INFO:teuthology.orchestra.run.smithi039.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-09-21T22:06:12.718 INFO:teuthology.orchestra.run.smithi039.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-09-21T22:06:12.718 INFO:teuthology.orchestra.run.smithi039.stdout: "bdev_debug_inflight_ios": "false", 2024-09-21T22:06:12.718 INFO:teuthology.orchestra.run.smithi039.stdout: "bdev_enable_discard": "false", 2024-09-21T22:06:12.718 INFO:teuthology.orchestra.run.smithi039.stdout: "bdev_flock_retry": "3", 2024-09-21T22:06:12.718 INFO:teuthology.orchestra.run.smithi039.stdout: "bdev_flock_retry_interval": "0.100000", 2024-09-21T22:06:12.718 INFO:teuthology.orchestra.run.smithi039.stdout: "bdev_inject_crash": "0", 2024-09-21T22:06:12.718 INFO:teuthology.orchestra.run.smithi039.stdout: "bdev_inject_crash_flush_delay": "2", 2024-09-21T22:06:12.718 INFO:teuthology.orchestra.run.smithi039.stdout: "bdev_ioring": "false", 2024-09-21T22:06:12.718 INFO:teuthology.orchestra.run.smithi039.stdout: "bdev_ioring_hipri": "false", 2024-09-21T22:06:12.718 INFO:teuthology.orchestra.run.smithi039.stdout: "bdev_ioring_sqthread_poll": "false", 2024-09-21T22:06:12.718 INFO:teuthology.orchestra.run.smithi039.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-09-21T22:06:12.718 INFO:teuthology.orchestra.run.smithi039.stdout: "bdev_read_buffer_alignment": "4096", 2024-09-21T22:06:12.718 INFO:teuthology.orchestra.run.smithi039.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-09-21T22:06:12.718 INFO:teuthology.orchestra.run.smithi039.stdout: "bdev_stalled_read_warn_lifetime": "86400", 2024-09-21T22:06:12.718 INFO:teuthology.orchestra.run.smithi039.stdout: "bdev_stalled_read_warn_threshold": "1", 2024-09-21T22:06:12.719 INFO:teuthology.orchestra.run.smithi039.stdout: "bdev_type": "", 2024-09-21T22:06:12.719 INFO:teuthology.orchestra.run.smithi039.stdout: "bluefs_alloc_size": "1048576", 2024-09-21T22:06:12.719 INFO:teuthology.orchestra.run.smithi039.stdout: "bluefs_allocator": "hybrid", 2024-09-21T22:06:12.719 INFO:teuthology.orchestra.run.smithi039.stdout: "bluefs_buffered_io": "true", 2024-09-21T22:06:12.719 INFO:teuthology.orchestra.run.smithi039.stdout: "bluefs_check_for_zeros": "false", 2024-09-21T22:06:12.719 INFO:teuthology.orchestra.run.smithi039.stdout: "bluefs_check_volume_selector_often": "false", 2024-09-21T22:06:12.719 INFO:teuthology.orchestra.run.smithi039.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-09-21T22:06:12.719 INFO:teuthology.orchestra.run.smithi039.stdout: "bluefs_compact_log_sync": "false", 2024-09-21T22:06:12.719 INFO:teuthology.orchestra.run.smithi039.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-09-21T22:06:12.719 INFO:teuthology.orchestra.run.smithi039.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-09-21T22:06:12.719 INFO:teuthology.orchestra.run.smithi039.stdout: "bluefs_log_compact_min_size": "16777216", 2024-09-21T22:06:12.719 INFO:teuthology.orchestra.run.smithi039.stdout: "bluefs_log_replay_check_allocations": "true", 2024-09-21T22:06:12.719 INFO:teuthology.orchestra.run.smithi039.stdout: "bluefs_max_log_runway": "4194304", 2024-09-21T22:06:12.719 INFO:teuthology.orchestra.run.smithi039.stdout: "bluefs_max_prefetch": "1048576", 2024-09-21T22:06:12.719 INFO:teuthology.orchestra.run.smithi039.stdout: "bluefs_min_flush_size": "524288", 2024-09-21T22:06:12.719 INFO:teuthology.orchestra.run.smithi039.stdout: "bluefs_min_log_runway": "1048576", 2024-09-21T22:06:12.720 INFO:teuthology.orchestra.run.smithi039.stdout: "bluefs_replay_recovery": "false", 2024-09-21T22:06:12.720 INFO:teuthology.orchestra.run.smithi039.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-09-21T22:06:12.720 INFO:teuthology.orchestra.run.smithi039.stdout: "bluefs_shared_alloc_size": "65536", 2024-09-21T22:06:12.720 INFO:teuthology.orchestra.run.smithi039.stdout: "bluefs_sync_write": "false", 2024-09-21T22:06:12.720 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-09-21T22:06:12.720 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-09-21T22:06:12.720 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-09-21T22:06:12.720 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_allocation_from_file": "true", 2024-09-21T22:06:12.720 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_allocator": "hybrid", 2024-09-21T22:06:12.720 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-09-21T22:06:12.720 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-09-21T22:06:12.720 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-09-21T22:06:12.720 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-09-21T22:06:12.720 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_bdev_label_multi": "true", 2024-09-21T22:06:12.720 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_bdev_label_multi_upgrade": "false", 2024-09-21T22:06:12.720 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_bdev_label_require_all": "true", 2024-09-21T22:06:12.721 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-09-21T22:06:12.721 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-09-21T22:06:12.721 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_blobid_prealloc": "10240", 2024-09-21T22:06:12.721 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_block_create": "true", 2024-09-21T22:06:12.721 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_block_db_create": "false", 2024-09-21T22:06:12.721 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_block_db_path": "", 2024-09-21T22:06:12.721 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_block_db_size": "0", 2024-09-21T22:06:12.721 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_block_path": "", 2024-09-21T22:06:12.721 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_block_preallocate_file": "false", 2024-09-21T22:06:12.721 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_block_size": "107374182400", 2024-09-21T22:06:12.721 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_block_wal_create": "false", 2024-09-21T22:06:12.721 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_block_wal_path": "", 2024-09-21T22:06:12.721 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_block_wal_size": "100663296", 2024-09-21T22:06:12.721 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_bluefs": "true", 2024-09-21T22:06:12.721 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-09-21T22:06:12.722 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_bluefs_env_mirror": "false", 2024-09-21T22:06:12.722 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-09-21T22:06:12.722 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-09-21T22:06:12.722 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-09-21T22:06:12.722 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-09-21T22:06:12.722 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-09-21T22:06:12.722 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-09-21T22:06:12.722 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_autotune": "true", 2024-09-21T22:06:12.722 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-09-21T22:06:12.722 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-09-21T22:06:12.722 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-09-21T22:06:12.722 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-09-21T22:06:12.722 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_size": "0", 2024-09-21T22:06:12.722 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-09-21T22:06:12.722 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-09-21T22:06:12.722 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-09-21T22:06:12.723 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-09-21T22:06:12.723 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_type": "2q", 2024-09-21T22:06:12.723 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-09-21T22:06:12.723 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_clone_cow": "true", 2024-09-21T22:06:12.723 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_compression_algorithm": "snappy", 2024-09-21T22:06:12.723 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_compression_max_blob_size": "0", 2024-09-21T22:06:12.723 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-09-21T22:06:12.723 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-09-21T22:06:12.723 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_compression_min_blob_size": "0", 2024-09-21T22:06:12.723 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-09-21T22:06:12.723 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-09-21T22:06:12.723 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_compression_mode": "none", 2024-09-21T22:06:12.723 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-09-21T22:06:12.723 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_csum_type": "crc32c", 2024-09-21T22:06:12.723 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_debug_enforce_settings": "default", 2024-09-21T22:06:12.724 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_debug_freelist": "false", 2024-09-21T22:06:12.724 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_debug_fsck_abort": "false", 2024-09-21T22:06:12.724 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-09-21T22:06:12.724 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-09-21T22:06:12.724 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_debug_inject_read_err": "false", 2024-09-21T22:06:12.724 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_debug_legacy_omap": "false", 2024-09-21T22:06:12.724 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-09-21T22:06:12.724 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-09-21T22:06:12.724 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-09-21T22:06:12.724 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-09-21T22:06:12.724 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-09-21T22:06:12.724 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-09-21T22:06:12.724 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-09-21T22:06:12.725 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_debug_small_allocations": "0", 2024-09-21T22:06:12.725 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-09-21T22:06:12.725 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_default_buffered_read": "true", 2024-09-21T22:06:12.725 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_default_buffered_write": "false", 2024-09-21T22:06:12.725 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_deferred_batch_ops": "0", 2024-09-21T22:06:12.725 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-09-21T22:06:12.725 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-09-21T22:06:12.725 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_elastic_shared_blobs": "true", 2024-09-21T22:06:12.725 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-09-21T22:06:12.725 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-09-21T22:06:12.725 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-09-21T22:06:12.725 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-09-21T22:06:12.725 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-09-21T22:06:12.725 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_fail_eio": "false", 2024-09-21T22:06:12.725 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-09-21T22:06:12.725 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-09-21T22:06:12.726 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-09-21T22:06:12.726 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-09-21T22:06:12.726 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_fsck_on_mkfs": "true", 2024-09-21T22:06:12.726 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-09-21T22:06:12.726 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_fsck_on_mount": "false", 2024-09-21T22:06:12.726 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-09-21T22:06:12.726 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_fsck_on_umount": "false", 2024-09-21T22:06:12.726 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-09-21T22:06:12.726 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-09-21T22:06:12.726 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-09-21T22:06:12.726 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-09-21T22:06:12.726 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-09-21T22:06:12.726 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-09-21T22:06:12.726 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-09-21T22:06:12.727 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-09-21T22:06:12.727 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_ignore_data_csum": "false", 2024-09-21T22:06:12.727 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-09-21T22:06:12.727 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_kvbackend": "rocksdb", 2024-09-21T22:06:12.727 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-09-21T22:06:12.727 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-09-21T22:06:12.727 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_log_op_age": "5.000000", 2024-09-21T22:06:12.727 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_max_alloc_size": "0", 2024-09-21T22:06:12.727 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_max_blob_size": "0", 2024-09-21T22:06:12.727 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-09-21T22:06:12.727 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-09-21T22:06:12.727 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_max_defer_interval": "3.000000", 2024-09-21T22:06:12.727 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_max_deferred_txc": "32", 2024-09-21T22:06:12.727 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_min_alloc_size": "0", 2024-09-21T22:06:12.727 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-09-21T22:06:12.727 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-09-21T22:06:12.728 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_nid_prealloc": "1024", 2024-09-21T22:06:12.728 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_prefer_deferred_size": "0", 2024-09-21T22:06:12.728 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-09-21T22:06:12.728 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-09-21T22:06:12.728 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_qfsck_on_mount": "true", 2024-09-21T22:06:12.728 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_retry_disk_reads": "3", 2024-09-21T22:06:12.728 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_rocksdb_cf": "true", 2024-09-21T22:06:12.728 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-09-21T22:06:12.728 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-09-21T22:06:12.728 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_rocksdb_options_annex": "", 2024-09-21T22:06:12.728 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_slow_ops_warn_lifetime": "86400", 2024-09-21T22:06:12.728 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_slow_ops_warn_threshold": "1", 2024-09-21T22:06:12.728 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_spdk_coremask": "0x1", 2024-09-21T22:06:12.728 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_spdk_io_sleep": "5", 2024-09-21T22:06:12.728 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_spdk_max_io_completion": "0", 2024-09-21T22:06:12.728 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_spdk_mem": "512", 2024-09-21T22:06:12.729 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_sync_submit_transaction": "false", 2024-09-21T22:06:12.729 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_throttle_bytes": "67108864", 2024-09-21T22:06:12.729 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_throttle_cost_per_io": "0", 2024-09-21T22:06:12.729 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-09-21T22:06:12.729 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-09-21T22:06:12.729 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-09-21T22:06:12.729 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-09-21T22:06:12.729 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_tracing": "false", 2024-09-21T22:06:12.729 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-09-21T22:06:12.729 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-09-21T22:06:12.729 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_volume_selection_reserved": "0", 2024-09-21T22:06:12.729 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-09-21T22:06:12.729 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-09-21T22:06:12.729 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-09-21T22:06:12.729 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-09-21T22:06:12.729 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-09-21T22:06:12.730 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-09-21T22:06:12.730 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_zero_block_detection": "false", 2024-09-21T22:06:12.730 INFO:teuthology.orchestra.run.smithi039.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-09-21T22:06:12.730 INFO:teuthology.orchestra.run.smithi039.stdout: "cephfs_mirror_action_update_interval": "2", 2024-09-21T22:06:12.730 INFO:teuthology.orchestra.run.smithi039.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-09-21T22:06:12.730 INFO:teuthology.orchestra.run.smithi039.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-09-21T22:06:12.730 INFO:teuthology.orchestra.run.smithi039.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-09-21T22:06:12.730 INFO:teuthology.orchestra.run.smithi039.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-09-21T22:06:12.730 INFO:teuthology.orchestra.run.smithi039.stdout: "cephfs_mirror_mount_timeout": "10", 2024-09-21T22:06:12.730 INFO:teuthology.orchestra.run.smithi039.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-09-21T22:06:12.730 INFO:teuthology.orchestra.run.smithi039.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-09-21T22:06:12.730 INFO:teuthology.orchestra.run.smithi039.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-09-21T22:06:12.730 INFO:teuthology.orchestra.run.smithi039.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-09-21T22:06:12.730 INFO:teuthology.orchestra.run.smithi039.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-09-21T22:06:12.730 INFO:teuthology.orchestra.run.smithi039.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-09-21T22:06:12.731 INFO:teuthology.orchestra.run.smithi039.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-09-21T22:06:12.731 INFO:teuthology.orchestra.run.smithi039.stdout: "cephx_cluster_require_signatures": "false", 2024-09-21T22:06:12.731 INFO:teuthology.orchestra.run.smithi039.stdout: "cephx_cluster_require_version": "2", 2024-09-21T22:06:12.731 INFO:teuthology.orchestra.run.smithi039.stdout: "cephx_require_signatures": "false", 2024-09-21T22:06:12.731 INFO:teuthology.orchestra.run.smithi039.stdout: "cephx_require_version": "2", 2024-09-21T22:06:12.731 INFO:teuthology.orchestra.run.smithi039.stdout: "cephx_service_require_signatures": "false", 2024-09-21T22:06:12.731 INFO:teuthology.orchestra.run.smithi039.stdout: "cephx_service_require_version": "2", 2024-09-21T22:06:12.731 INFO:teuthology.orchestra.run.smithi039.stdout: "cephx_sign_messages": "true", 2024-09-21T22:06:12.731 INFO:teuthology.orchestra.run.smithi039.stdout: "chdir": "", 2024-09-21T22:06:12.731 INFO:teuthology.orchestra.run.smithi039.stdout: "client_acl_type": "", 2024-09-21T22:06:12.731 INFO:teuthology.orchestra.run.smithi039.stdout: "client_asio_thread_count": "2", 2024-09-21T22:06:12.731 INFO:teuthology.orchestra.run.smithi039.stdout: "client_cache_mid": "0.750000", 2024-09-21T22:06:12.731 INFO:teuthology.orchestra.run.smithi039.stdout: "client_cache_size": "16384", 2024-09-21T22:06:12.731 INFO:teuthology.orchestra.run.smithi039.stdout: "client_caps_release_delay": "5", 2024-09-21T22:06:12.731 INFO:teuthology.orchestra.run.smithi039.stdout: "client_check_pool_perm": "true", 2024-09-21T22:06:12.731 INFO:teuthology.orchestra.run.smithi039.stdout: "client_collect_and_send_global_metrics": "false", 2024-09-21T22:06:12.732 INFO:teuthology.orchestra.run.smithi039.stdout: "client_debug_force_sync_read": "false", 2024-09-21T22:06:12.732 INFO:teuthology.orchestra.run.smithi039.stdout: "client_debug_getattr_caps": "false", 2024-09-21T22:06:12.732 INFO:teuthology.orchestra.run.smithi039.stdout: "client_debug_inject_features": "", 2024-09-21T22:06:12.732 INFO:teuthology.orchestra.run.smithi039.stdout: "client_debug_inject_tick_delay": "0", 2024-09-21T22:06:12.732 INFO:teuthology.orchestra.run.smithi039.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-09-21T22:06:12.732 INFO:teuthology.orchestra.run.smithi039.stdout: "client_die_on_failed_remount": "false", 2024-09-21T22:06:12.732 INFO:teuthology.orchestra.run.smithi039.stdout: "client_dirsize_rbytes": "true", 2024-09-21T22:06:12.732 INFO:teuthology.orchestra.run.smithi039.stdout: "client_force_lazyio": "false", 2024-09-21T22:06:12.732 INFO:teuthology.orchestra.run.smithi039.stdout: "client_fs": "", 2024-09-21T22:06:12.732 INFO:teuthology.orchestra.run.smithi039.stdout: "client_inject_fixed_oldest_tid": "false", 2024-09-21T22:06:12.732 INFO:teuthology.orchestra.run.smithi039.stdout: "client_inject_release_failure": "false", 2024-09-21T22:06:12.732 INFO:teuthology.orchestra.run.smithi039.stdout: "client_max_inline_size": "4096", 2024-09-21T22:06:12.732 INFO:teuthology.orchestra.run.smithi039.stdout: "client_max_retries_on_remount_failure": "5", 2024-09-21T22:06:12.732 INFO:teuthology.orchestra.run.smithi039.stdout: "client_mds_namespace": "", 2024-09-21T22:06:12.732 INFO:teuthology.orchestra.run.smithi039.stdout: "client_metadata": "", 2024-09-21T22:06:12.732 INFO:teuthology.orchestra.run.smithi039.stdout: "client_mount_gid": "-1", 2024-09-21T22:06:12.733 INFO:teuthology.orchestra.run.smithi039.stdout: "client_mount_timeout": "300", 2024-09-21T22:06:12.733 INFO:teuthology.orchestra.run.smithi039.stdout: "client_mount_uid": "-1", 2024-09-21T22:06:12.733 INFO:teuthology.orchestra.run.smithi039.stdout: "client_mountpoint": "/", 2024-09-21T22:06:12.733 INFO:teuthology.orchestra.run.smithi039.stdout: "client_notify_timeout": "10", 2024-09-21T22:06:12.733 INFO:teuthology.orchestra.run.smithi039.stdout: "client_oc": "true", 2024-09-21T22:06:12.733 INFO:teuthology.orchestra.run.smithi039.stdout: "client_oc_max_dirty": "104857600", 2024-09-21T22:06:12.733 INFO:teuthology.orchestra.run.smithi039.stdout: "client_oc_max_dirty_age": "5.000000", 2024-09-21T22:06:12.733 INFO:teuthology.orchestra.run.smithi039.stdout: "client_oc_max_objects": "1000", 2024-09-21T22:06:12.733 INFO:teuthology.orchestra.run.smithi039.stdout: "client_oc_size": "209715200", 2024-09-21T22:06:12.733 INFO:teuthology.orchestra.run.smithi039.stdout: "client_oc_target_dirty": "8388608", 2024-09-21T22:06:12.733 INFO:teuthology.orchestra.run.smithi039.stdout: "client_permissions": "true", 2024-09-21T22:06:12.733 INFO:teuthology.orchestra.run.smithi039.stdout: "client_quota": "true", 2024-09-21T22:06:12.733 INFO:teuthology.orchestra.run.smithi039.stdout: "client_quota_df": "true", 2024-09-21T22:06:12.733 INFO:teuthology.orchestra.run.smithi039.stdout: "client_readahead_max_bytes": "0", 2024-09-21T22:06:12.733 INFO:teuthology.orchestra.run.smithi039.stdout: "client_readahead_max_periods": "4", 2024-09-21T22:06:12.733 INFO:teuthology.orchestra.run.smithi039.stdout: "client_readahead_min": "131072", 2024-09-21T22:06:12.733 INFO:teuthology.orchestra.run.smithi039.stdout: "client_reconnect_stale": "false", 2024-09-21T22:06:12.734 INFO:teuthology.orchestra.run.smithi039.stdout: "client_shutdown_timeout": "30", 2024-09-21T22:06:12.734 INFO:teuthology.orchestra.run.smithi039.stdout: "client_snapdir": ".snap", 2024-09-21T22:06:12.734 INFO:teuthology.orchestra.run.smithi039.stdout: "client_tick_interval": "1", 2024-09-21T22:06:12.734 INFO:teuthology.orchestra.run.smithi039.stdout: "client_trace": "", 2024-09-21T22:06:12.734 INFO:teuthology.orchestra.run.smithi039.stdout: "client_try_dentry_invalidate": "false", 2024-09-21T22:06:12.734 INFO:teuthology.orchestra.run.smithi039.stdout: "client_use_faked_inos": "false", 2024-09-21T22:06:12.734 INFO:teuthology.orchestra.run.smithi039.stdout: "client_use_random_mds": "false", 2024-09-21T22:06:12.734 INFO:teuthology.orchestra.run.smithi039.stdout: "clog_to_graylog": "false", 2024-09-21T22:06:12.734 INFO:teuthology.orchestra.run.smithi039.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-09-21T22:06:12.734 INFO:teuthology.orchestra.run.smithi039.stdout: "clog_to_graylog_port": "12201", 2024-09-21T22:06:12.734 INFO:teuthology.orchestra.run.smithi039.stdout: "clog_to_monitors": "default=true", 2024-09-21T22:06:12.734 INFO:teuthology.orchestra.run.smithi039.stdout: "clog_to_syslog": "false", 2024-09-21T22:06:12.734 INFO:teuthology.orchestra.run.smithi039.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-09-21T22:06:12.734 INFO:teuthology.orchestra.run.smithi039.stdout: "clog_to_syslog_level": "info", 2024-09-21T22:06:12.734 INFO:teuthology.orchestra.run.smithi039.stdout: "cluster_addr": "-", 2024-09-21T22:06:12.734 INFO:teuthology.orchestra.run.smithi039.stdout: "cluster_network": "", 2024-09-21T22:06:12.735 INFO:teuthology.orchestra.run.smithi039.stdout: "cluster_network_interface": "", 2024-09-21T22:06:12.735 INFO:teuthology.orchestra.run.smithi039.stdout: "compressor_zlib_isal": "false", 2024-09-21T22:06:12.735 INFO:teuthology.orchestra.run.smithi039.stdout: "compressor_zlib_level": "5", 2024-09-21T22:06:12.735 INFO:teuthology.orchestra.run.smithi039.stdout: "compressor_zlib_winsize": "-15", 2024-09-21T22:06:12.735 INFO:teuthology.orchestra.run.smithi039.stdout: "compressor_zstd_level": "1", 2024-09-21T22:06:12.735 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image": "docker.io/ceph/daemon-base:latest-master-devel", 2024-09-21T22:06:12.735 INFO:teuthology.orchestra.run.smithi039.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-09-21T22:06:12.735 INFO:teuthology.orchestra.run.smithi039.stdout: "crimson_alien_op_num_threads": "6", 2024-09-21T22:06:12.735 INFO:teuthology.orchestra.run.smithi039.stdout: "crimson_alien_thread_cpu_cores": "", 2024-09-21T22:06:12.735 INFO:teuthology.orchestra.run.smithi039.stdout: "crimson_osd_obc_lru_size": "512", 2024-09-21T22:06:12.735 INFO:teuthology.orchestra.run.smithi039.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-09-21T22:06:12.735 INFO:teuthology.orchestra.run.smithi039.stdout: "crimson_osd_stat_interval": "0", 2024-09-21T22:06:12.735 INFO:teuthology.orchestra.run.smithi039.stdout: "crimson_seastar_cpu_cores": "", 2024-09-21T22:06:12.735 INFO:teuthology.orchestra.run.smithi039.stdout: "crimson_seastar_num_threads": "0", 2024-09-21T22:06:12.735 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location": "", 2024-09-21T22:06:12.735 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location_hook": "", 2024-09-21T22:06:12.736 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_location_hook_timeout": "10", 2024-09-21T22:06:12.736 INFO:teuthology.orchestra.run.smithi039.stdout: "daemonize": "false", 2024-09-21T22:06:12.736 INFO:teuthology.orchestra.run.smithi039.stdout: "daos_pool": "tank", 2024-09-21T22:06:12.736 INFO:teuthology.orchestra.run.smithi039.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-09-21T22:06:12.736 INFO:teuthology.orchestra.run.smithi039.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-09-21T22:06:12.736 INFO:teuthology.orchestra.run.smithi039.stdout: "dbstore_db_name_prefix": "dbstore", 2024-09-21T22:06:12.736 INFO:teuthology.orchestra.run.smithi039.stdout: "debug_alienstore": "0/5", 2024-09-21T22:06:12.736 INFO:teuthology.orchestra.run.smithi039.stdout: "debug_allow_any_pool_priority": "false", 2024-09-21T22:06:12.736 INFO:teuthology.orchestra.run.smithi039.stdout: "debug_asok": "1/5", 2024-09-21T22:06:12.736 INFO:teuthology.orchestra.run.smithi039.stdout: "debug_asok_assert_abort": "false", 2024-09-21T22:06:12.736 INFO:teuthology.orchestra.run.smithi039.stdout: "debug_asserts_on_shutdown": "true", 2024-09-21T22:06:12.736 INFO:teuthology.orchestra.run.smithi039.stdout: "debug_auth": "20/20", 2024-09-21T22:06:12.736 INFO:teuthology.orchestra.run.smithi039.stdout: "debug_bdev": "1/3", 2024-09-21T22:06:12.736 INFO:teuthology.orchestra.run.smithi039.stdout: "debug_bluefs": "1/5", 2024-09-21T22:06:12.736 INFO:teuthology.orchestra.run.smithi039.stdout: "debug_bluestore": "1/5", 2024-09-21T22:06:12.736 INFO:teuthology.orchestra.run.smithi039.stdout: "debug_buffer": "0/1", 2024-09-21T22:06:12.737 INFO:teuthology.orchestra.run.smithi039.stdout: "debug_ceph_exporter": "1/5", 2024-09-21T22:06:12.737 INFO:teuthology.orchestra.run.smithi039.stdout: "debug_cephfs_mirror": "0/5", 2024-09-21T22:06:12.737 INFO:teuthology.orchestra.run.smithi039.stdout: "debug_cephsqlite": "0/5", 2024-09-21T22:06:12.737 INFO:teuthology.orchestra.run.smithi039.stdout: "debug_client": "0/5", 2024-09-21T22:06:12.737 INFO:teuthology.orchestra.run.smithi039.stdout: "debug_compressor": "1/5", 2024-09-21T22:06:12.737 INFO:teuthology.orchestra.run.smithi039.stdout: "debug_context": "0/1", 2024-09-21T22:06:12.737 INFO:teuthology.orchestra.run.smithi039.stdout: "debug_crimson_interrupt": "0/5", 2024-09-21T22:06:12.737 INFO:teuthology.orchestra.run.smithi039.stdout: "debug_crush": "1/1", 2024-09-21T22:06:12.737 INFO:teuthology.orchestra.run.smithi039.stdout: "debug_crypto": "1/5", 2024-09-21T22:06:12.737 INFO:teuthology.orchestra.run.smithi039.stdout: "debug_cyanstore": "0/5", 2024-09-21T22:06:12.737 INFO:teuthology.orchestra.run.smithi039.stdout: "debug_deliberately_leak_memory": "false", 2024-09-21T22:06:12.737 INFO:teuthology.orchestra.run.smithi039.stdout: "debug_disable_randomized_ping": "false", 2024-09-21T22:06:12.737 INFO:teuthology.orchestra.run.smithi039.stdout: "debug_dpdk": "1/5", 2024-09-21T22:06:12.737 INFO:teuthology.orchestra.run.smithi039.stdout: "debug_eventtrace": "1/5", 2024-09-21T22:06:12.737 INFO:teuthology.orchestra.run.smithi039.stdout: "debug_filer": "0/1", 2024-09-21T22:06:12.737 INFO:teuthology.orchestra.run.smithi039.stdout: "debug_filestore": "1/3", 2024-09-21T22:06:12.738 INFO:teuthology.orchestra.run.smithi039.stdout: "debug_finisher": "1/1", 2024-09-21T22:06:12.738 INFO:teuthology.orchestra.run.smithi039.stdout: "debug_fuse": "1/5", 2024-09-21T22:06:12.738 INFO:teuthology.orchestra.run.smithi039.stdout: "debug_heartbeat_testing_span": "0", 2024-09-21T22:06:12.738 INFO:teuthology.orchestra.run.smithi039.stdout: "debug_heartbeatmap": "1/5", 2024-09-21T22:06:12.738 INFO:teuthology.orchestra.run.smithi039.stdout: "debug_immutable_obj_cache": "0/5", 2024-09-21T22:06:12.738 INFO:teuthology.orchestra.run.smithi039.stdout: "debug_javaclient": "1/5", 2024-09-21T22:06:12.738 INFO:teuthology.orchestra.run.smithi039.stdout: "debug_journal": "1/3", 2024-09-21T22:06:12.738 INFO:teuthology.orchestra.run.smithi039.stdout: "debug_journaler": "0/5", 2024-09-21T22:06:12.738 INFO:teuthology.orchestra.run.smithi039.stdout: "debug_kstore": "1/5", 2024-09-21T22:06:12.738 INFO:teuthology.orchestra.run.smithi039.stdout: "debug_lockdep": "0/1", 2024-09-21T22:06:12.738 INFO:teuthology.orchestra.run.smithi039.stdout: "debug_mclock": "1/5", 2024-09-21T22:06:12.738 INFO:teuthology.orchestra.run.smithi039.stdout: "debug_mds": "1/5", 2024-09-21T22:06:12.738 INFO:teuthology.orchestra.run.smithi039.stdout: "debug_mds_balancer": "1/5", 2024-09-21T22:06:12.738 INFO:teuthology.orchestra.run.smithi039.stdout: "debug_mds_locker": "1/5", 2024-09-21T22:06:12.739 INFO:teuthology.orchestra.run.smithi039.stdout: "debug_mds_log": "1/5", 2024-09-21T22:06:12.739 INFO:teuthology.orchestra.run.smithi039.stdout: "debug_mds_log_expire": "1/5", 2024-09-21T22:06:12.739 INFO:teuthology.orchestra.run.smithi039.stdout: "debug_mds_migrator": "1/5", 2024-09-21T22:06:12.739 INFO:teuthology.orchestra.run.smithi039.stdout: "debug_mds_quiesce": "3/5", 2024-09-21T22:06:12.739 INFO:teuthology.orchestra.run.smithi039.stdout: "debug_memstore": "1/5", 2024-09-21T22:06:12.739 INFO:teuthology.orchestra.run.smithi039.stdout: "debug_mgr": "2/5", 2024-09-21T22:06:12.739 INFO:teuthology.orchestra.run.smithi039.stdout: "debug_mgrc": "1/5", 2024-09-21T22:06:12.739 INFO:teuthology.orchestra.run.smithi039.stdout: "debug_mon": "20/20", 2024-09-21T22:06:12.739 INFO:teuthology.orchestra.run.smithi039.stdout: "debug_monc": "0/10", 2024-09-21T22:06:12.739 INFO:teuthology.orchestra.run.smithi039.stdout: "debug_ms": "1/1", 2024-09-21T22:06:12.739 INFO:teuthology.orchestra.run.smithi039.stdout: "debug_none": "0/5", 2024-09-21T22:06:12.739 INFO:teuthology.orchestra.run.smithi039.stdout: "debug_objclass": "0/5", 2024-09-21T22:06:12.739 INFO:teuthology.orchestra.run.smithi039.stdout: "debug_objectcacher": "0/5", 2024-09-21T22:06:12.739 INFO:teuthology.orchestra.run.smithi039.stdout: "debug_objecter": "0/1", 2024-09-21T22:06:12.739 INFO:teuthology.orchestra.run.smithi039.stdout: "debug_optracker": "0/5", 2024-09-21T22:06:12.739 INFO:teuthology.orchestra.run.smithi039.stdout: "debug_osd": "1/5", 2024-09-21T22:06:12.740 INFO:teuthology.orchestra.run.smithi039.stdout: "debug_paxos": "20/20", 2024-09-21T22:06:12.740 INFO:teuthology.orchestra.run.smithi039.stdout: "debug_perfcounter": "1/5", 2024-09-21T22:06:12.740 INFO:teuthology.orchestra.run.smithi039.stdout: "debug_prioritycache": "1/5", 2024-09-21T22:06:12.740 INFO:teuthology.orchestra.run.smithi039.stdout: "debug_rados": "0/5", 2024-09-21T22:06:12.740 INFO:teuthology.orchestra.run.smithi039.stdout: "debug_rbd": "0/5", 2024-09-21T22:06:12.740 INFO:teuthology.orchestra.run.smithi039.stdout: "debug_rbd_mirror": "0/5", 2024-09-21T22:06:12.740 INFO:teuthology.orchestra.run.smithi039.stdout: "debug_rbd_pwl": "0/5", 2024-09-21T22:06:12.740 INFO:teuthology.orchestra.run.smithi039.stdout: "debug_rbd_replay": "0/5", 2024-09-21T22:06:12.740 INFO:teuthology.orchestra.run.smithi039.stdout: "debug_refs": "0/0", 2024-09-21T22:06:12.740 INFO:teuthology.orchestra.run.smithi039.stdout: "debug_reserver": "1/1", 2024-09-21T22:06:12.740 INFO:teuthology.orchestra.run.smithi039.stdout: "debug_rgw": "1/5", 2024-09-21T22:06:12.740 INFO:teuthology.orchestra.run.smithi039.stdout: "debug_rgw_access": "1/5", 2024-09-21T22:06:12.740 INFO:teuthology.orchestra.run.smithi039.stdout: "debug_rgw_datacache": "1/5", 2024-09-21T22:06:12.740 INFO:teuthology.orchestra.run.smithi039.stdout: "debug_rgw_dbstore": "1/5", 2024-09-21T22:06:12.740 INFO:teuthology.orchestra.run.smithi039.stdout: "debug_rgw_flight": "1/5", 2024-09-21T22:06:12.741 INFO:teuthology.orchestra.run.smithi039.stdout: "debug_rgw_lifecycle": "1/5", 2024-09-21T22:06:12.741 INFO:teuthology.orchestra.run.smithi039.stdout: "debug_rgw_sync": "1/5", 2024-09-21T22:06:12.741 INFO:teuthology.orchestra.run.smithi039.stdout: "debug_rocksdb": "4/5", 2024-09-21T22:06:12.741 INFO:teuthology.orchestra.run.smithi039.stdout: "debug_seastore": "0/5", 2024-09-21T22:06:12.741 INFO:teuthology.orchestra.run.smithi039.stdout: "debug_seastore_backref": "0/5", 2024-09-21T22:06:12.741 INFO:teuthology.orchestra.run.smithi039.stdout: "debug_seastore_cache": "0/5", 2024-09-21T22:06:12.741 INFO:teuthology.orchestra.run.smithi039.stdout: "debug_seastore_cleaner": "0/5", 2024-09-21T22:06:12.741 INFO:teuthology.orchestra.run.smithi039.stdout: "debug_seastore_device": "0/5", 2024-09-21T22:06:12.741 INFO:teuthology.orchestra.run.smithi039.stdout: "debug_seastore_epm": "0/5", 2024-09-21T22:06:12.741 INFO:teuthology.orchestra.run.smithi039.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-09-21T22:06:12.741 INFO:teuthology.orchestra.run.smithi039.stdout: "debug_seastore_journal": "0/5", 2024-09-21T22:06:12.741 INFO:teuthology.orchestra.run.smithi039.stdout: "debug_seastore_lba": "0/5", 2024-09-21T22:06:12.741 INFO:teuthology.orchestra.run.smithi039.stdout: "debug_seastore_odata": "0/5", 2024-09-21T22:06:12.741 INFO:teuthology.orchestra.run.smithi039.stdout: "debug_seastore_omap": "0/5", 2024-09-21T22:06:12.741 INFO:teuthology.orchestra.run.smithi039.stdout: "debug_seastore_onode": "0/5", 2024-09-21T22:06:12.742 INFO:teuthology.orchestra.run.smithi039.stdout: "debug_seastore_t": "0/5", 2024-09-21T22:06:12.742 INFO:teuthology.orchestra.run.smithi039.stdout: "debug_seastore_tm": "0/5", 2024-09-21T22:06:12.742 INFO:teuthology.orchestra.run.smithi039.stdout: "debug_striper": "0/1", 2024-09-21T22:06:12.742 INFO:teuthology.orchestra.run.smithi039.stdout: "debug_test": "0/5", 2024-09-21T22:06:12.742 INFO:teuthology.orchestra.run.smithi039.stdout: "debug_throttle": "1/1", 2024-09-21T22:06:12.742 INFO:teuthology.orchestra.run.smithi039.stdout: "debug_timer": "0/1", 2024-09-21T22:06:12.742 INFO:teuthology.orchestra.run.smithi039.stdout: "debug_tp": "0/5", 2024-09-21T22:06:12.742 INFO:teuthology.orchestra.run.smithi039.stdout: "debug_trace": "1/5", 2024-09-21T22:06:12.742 INFO:teuthology.orchestra.run.smithi039.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-09-21T22:06:12.742 INFO:teuthology.orchestra.run.smithi039.stdout: "device_failure_prediction_mode": "none", 2024-09-21T22:06:12.742 INFO:teuthology.orchestra.run.smithi039.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-09-21T22:06:12.742 INFO:teuthology.orchestra.run.smithi039.stdout: "erasure_code_dir": "/usr/lib/ceph/erasure-code", 2024-09-21T22:06:12.742 INFO:teuthology.orchestra.run.smithi039.stdout: "err_to_graylog": "false", 2024-09-21T22:06:12.742 INFO:teuthology.orchestra.run.smithi039.stdout: "err_to_journald": "false", 2024-09-21T22:06:12.743 INFO:teuthology.orchestra.run.smithi039.stdout: "err_to_stderr": "true", 2024-09-21T22:06:12.743 INFO:teuthology.orchestra.run.smithi039.stdout: "err_to_syslog": "false", 2024-09-21T22:06:12.743 INFO:teuthology.orchestra.run.smithi039.stdout: "event_tracing": "false", 2024-09-21T22:06:12.743 INFO:teuthology.orchestra.run.smithi039.stdout: "exporter_addr": "0.0.0.0", 2024-09-21T22:06:12.743 INFO:teuthology.orchestra.run.smithi039.stdout: "exporter_http_port": "9926", 2024-09-21T22:06:12.743 INFO:teuthology.orchestra.run.smithi039.stdout: "exporter_prio_limit": "5", 2024-09-21T22:06:12.743 INFO:teuthology.orchestra.run.smithi039.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-09-21T22:06:12.743 INFO:teuthology.orchestra.run.smithi039.stdout: "exporter_sort_metrics": "true", 2024-09-21T22:06:12.743 INFO:teuthology.orchestra.run.smithi039.stdout: "exporter_stats_period": "5", 2024-09-21T22:06:12.743 INFO:teuthology.orchestra.run.smithi039.stdout: "fake_statfs_for_testing": "0", 2024-09-21T22:06:12.743 INFO:teuthology.orchestra.run.smithi039.stdout: "fatal_signal_handlers": "true", 2024-09-21T22:06:12.743 INFO:teuthology.orchestra.run.smithi039.stdout: "filer_max_purge_ops": "10", 2024-09-21T22:06:12.743 INFO:teuthology.orchestra.run.smithi039.stdout: "filer_max_truncate_ops": "128", 2024-09-21T22:06:12.743 INFO:teuthology.orchestra.run.smithi039.stdout: "filestore_apply_finisher_threads": "1", 2024-09-21T22:06:12.743 INFO:teuthology.orchestra.run.smithi039.stdout: "filestore_blackhole": "false", 2024-09-21T22:06:12.744 INFO:teuthology.orchestra.run.smithi039.stdout: "filestore_btrfs_clone_range": "true", 2024-09-21T22:06:12.744 INFO:teuthology.orchestra.run.smithi039.stdout: "filestore_btrfs_snap": "true", 2024-09-21T22:06:12.744 INFO:teuthology.orchestra.run.smithi039.stdout: "filestore_caller_concurrency": "10", 2024-09-21T22:06:12.744 INFO:teuthology.orchestra.run.smithi039.stdout: "filestore_collect_device_partition_information": "true", 2024-09-21T22:06:12.744 INFO:teuthology.orchestra.run.smithi039.stdout: "filestore_commit_timeout": "600.000000", 2024-09-21T22:06:12.744 INFO:teuthology.orchestra.run.smithi039.stdout: "filestore_debug_inject_read_err": "false", 2024-09-21T22:06:12.744 INFO:teuthology.orchestra.run.smithi039.stdout: "filestore_debug_omap_check": "false", 2024-09-21T22:06:12.744 INFO:teuthology.orchestra.run.smithi039.stdout: "filestore_debug_random_read_err": "0.000000", 2024-09-21T22:06:12.744 INFO:teuthology.orchestra.run.smithi039.stdout: "filestore_debug_verify_split": "false", 2024-09-21T22:06:12.744 INFO:teuthology.orchestra.run.smithi039.stdout: "filestore_dump_file": "", 2024-09-21T22:06:12.744 INFO:teuthology.orchestra.run.smithi039.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-09-21T22:06:12.744 INFO:teuthology.orchestra.run.smithi039.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-09-21T22:06:12.744 INFO:teuthology.orchestra.run.smithi039.stdout: "filestore_fadvise": "true", 2024-09-21T22:06:12.744 INFO:teuthology.orchestra.run.smithi039.stdout: "filestore_fail_eio": "true", 2024-09-21T22:06:12.744 INFO:teuthology.orchestra.run.smithi039.stdout: "filestore_fd_cache_shards": "16", 2024-09-21T22:06:12.744 INFO:teuthology.orchestra.run.smithi039.stdout: "filestore_fd_cache_size": "128", 2024-09-21T22:06:12.745 INFO:teuthology.orchestra.run.smithi039.stdout: "filestore_fiemap": "false", 2024-09-21T22:06:12.745 INFO:teuthology.orchestra.run.smithi039.stdout: "filestore_fiemap_threshold": "4096", 2024-09-21T22:06:12.745 INFO:teuthology.orchestra.run.smithi039.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-09-21T22:06:12.745 INFO:teuthology.orchestra.run.smithi039.stdout: "filestore_index_retry_probability": "0.000000", 2024-09-21T22:06:12.745 INFO:teuthology.orchestra.run.smithi039.stdout: "filestore_inject_stall": "0", 2024-09-21T22:06:12.745 INFO:teuthology.orchestra.run.smithi039.stdout: "filestore_journal_parallel": "false", 2024-09-21T22:06:12.745 INFO:teuthology.orchestra.run.smithi039.stdout: "filestore_journal_trailing": "false", 2024-09-21T22:06:12.745 INFO:teuthology.orchestra.run.smithi039.stdout: "filestore_journal_writeahead": "false", 2024-09-21T22:06:12.745 INFO:teuthology.orchestra.run.smithi039.stdout: "filestore_kill_at": "0", 2024-09-21T22:06:12.745 INFO:teuthology.orchestra.run.smithi039.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-09-21T22:06:12.745 INFO:teuthology.orchestra.run.smithi039.stdout: "filestore_max_inline_xattr_size": "0", 2024-09-21T22:06:12.745 INFO:teuthology.orchestra.run.smithi039.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-09-21T22:06:12.745 INFO:teuthology.orchestra.run.smithi039.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-09-21T22:06:12.745 INFO:teuthology.orchestra.run.smithi039.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-09-21T22:06:12.746 INFO:teuthology.orchestra.run.smithi039.stdout: "filestore_max_inline_xattrs": "0", 2024-09-21T22:06:12.746 INFO:teuthology.orchestra.run.smithi039.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-09-21T22:06:12.746 INFO:teuthology.orchestra.run.smithi039.stdout: "filestore_max_inline_xattrs_other": "2", 2024-09-21T22:06:12.746 INFO:teuthology.orchestra.run.smithi039.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-09-21T22:06:12.746 INFO:teuthology.orchestra.run.smithi039.stdout: "filestore_max_sync_interval": "5.000000", 2024-09-21T22:06:12.746 INFO:teuthology.orchestra.run.smithi039.stdout: "filestore_max_xattr_value_size": "0", 2024-09-21T22:06:12.746 INFO:teuthology.orchestra.run.smithi039.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-09-21T22:06:12.746 INFO:teuthology.orchestra.run.smithi039.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-09-21T22:06:12.746 INFO:teuthology.orchestra.run.smithi039.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-09-21T22:06:12.746 INFO:teuthology.orchestra.run.smithi039.stdout: "filestore_merge_threshold": "-10", 2024-09-21T22:06:12.746 INFO:teuthology.orchestra.run.smithi039.stdout: "filestore_min_sync_interval": "0.010000", 2024-09-21T22:06:12.746 INFO:teuthology.orchestra.run.smithi039.stdout: "filestore_odsync_write": "false", 2024-09-21T22:06:12.746 INFO:teuthology.orchestra.run.smithi039.stdout: "filestore_omap_backend": "rocksdb", 2024-09-21T22:06:12.746 INFO:teuthology.orchestra.run.smithi039.stdout: "filestore_omap_backend_path": "", 2024-09-21T22:06:12.747 INFO:teuthology.orchestra.run.smithi039.stdout: "filestore_omap_header_cache_size": "1024", 2024-09-21T22:06:12.747 INFO:teuthology.orchestra.run.smithi039.stdout: "filestore_ondisk_finisher_threads": "1", 2024-09-21T22:06:12.747 INFO:teuthology.orchestra.run.smithi039.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-09-21T22:06:12.747 INFO:teuthology.orchestra.run.smithi039.stdout: "filestore_op_thread_timeout": "60", 2024-09-21T22:06:12.747 INFO:teuthology.orchestra.run.smithi039.stdout: "filestore_op_threads": "2", 2024-09-21T22:06:12.747 INFO:teuthology.orchestra.run.smithi039.stdout: "filestore_punch_hole": "false", 2024-09-21T22:06:12.747 INFO:teuthology.orchestra.run.smithi039.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-09-21T22:06:12.747 INFO:teuthology.orchestra.run.smithi039.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-09-21T22:06:12.747 INFO:teuthology.orchestra.run.smithi039.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-09-21T22:06:12.747 INFO:teuthology.orchestra.run.smithi039.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-09-21T22:06:12.747 INFO:teuthology.orchestra.run.smithi039.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-09-21T22:06:12.747 INFO:teuthology.orchestra.run.smithi039.stdout: "filestore_queue_max_bytes": "104857600", 2024-09-21T22:06:12.747 INFO:teuthology.orchestra.run.smithi039.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-09-21T22:06:12.747 INFO:teuthology.orchestra.run.smithi039.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-09-21T22:06:12.747 INFO:teuthology.orchestra.run.smithi039.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-09-21T22:06:12.748 INFO:teuthology.orchestra.run.smithi039.stdout: "filestore_queue_max_ops": "50", 2024-09-21T22:06:12.748 INFO:teuthology.orchestra.run.smithi039.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-09-21T22:06:12.748 INFO:teuthology.orchestra.run.smithi039.stdout: "filestore_seek_data_hole": "false", 2024-09-21T22:06:12.748 INFO:teuthology.orchestra.run.smithi039.stdout: "filestore_sloppy_crc": "false", 2024-09-21T22:06:12.748 INFO:teuthology.orchestra.run.smithi039.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-09-21T22:06:12.748 INFO:teuthology.orchestra.run.smithi039.stdout: "filestore_splice": "false", 2024-09-21T22:06:12.748 INFO:teuthology.orchestra.run.smithi039.stdout: "filestore_split_multiple": "2", 2024-09-21T22:06:12.748 INFO:teuthology.orchestra.run.smithi039.stdout: "filestore_split_rand_factor": "20", 2024-09-21T22:06:12.748 INFO:teuthology.orchestra.run.smithi039.stdout: "filestore_update_to": "1000", 2024-09-21T22:06:12.748 INFO:teuthology.orchestra.run.smithi039.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-09-21T22:06:12.748 INFO:teuthology.orchestra.run.smithi039.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-09-21T22:06:12.748 INFO:teuthology.orchestra.run.smithi039.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-09-21T22:06:12.748 INFO:teuthology.orchestra.run.smithi039.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-09-21T22:06:12.748 INFO:teuthology.orchestra.run.smithi039.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-09-21T22:06:12.749 INFO:teuthology.orchestra.run.smithi039.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-09-21T22:06:12.749 INFO:teuthology.orchestra.run.smithi039.stdout: "filestore_wbthrottle_enable": "true", 2024-09-21T22:06:12.749 INFO:teuthology.orchestra.run.smithi039.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-09-21T22:06:12.749 INFO:teuthology.orchestra.run.smithi039.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-09-21T22:06:12.749 INFO:teuthology.orchestra.run.smithi039.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-09-21T22:06:12.749 INFO:teuthology.orchestra.run.smithi039.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-09-21T22:06:12.749 INFO:teuthology.orchestra.run.smithi039.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-09-21T22:06:12.749 INFO:teuthology.orchestra.run.smithi039.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-09-21T22:06:12.749 INFO:teuthology.orchestra.run.smithi039.stdout: "filestore_xfs_extsize": "false", 2024-09-21T22:06:12.749 INFO:teuthology.orchestra.run.smithi039.stdout: "filestore_zfs_snap": "false", 2024-09-21T22:06:12.749 INFO:teuthology.orchestra.run.smithi039.stdout: "fio_dir": "/tmp/fio", 2024-09-21T22:06:12.749 INFO:teuthology.orchestra.run.smithi039.stdout: "fsid": "00bb09a3-e410-4e18-a2b2-08eaf61978e6", 2024-09-21T22:06:12.749 INFO:teuthology.orchestra.run.smithi039.stdout: "fuse_allow_other": "true", 2024-09-21T22:06:12.749 INFO:teuthology.orchestra.run.smithi039.stdout: "fuse_atomic_o_trunc": "true", 2024-09-21T22:06:12.749 INFO:teuthology.orchestra.run.smithi039.stdout: "fuse_big_writes": "true", 2024-09-21T22:06:12.750 INFO:teuthology.orchestra.run.smithi039.stdout: "fuse_debug": "false", 2024-09-21T22:06:12.750 INFO:teuthology.orchestra.run.smithi039.stdout: "fuse_default_permissions": "false", 2024-09-21T22:06:12.750 INFO:teuthology.orchestra.run.smithi039.stdout: "fuse_disable_pagecache": "false", 2024-09-21T22:06:12.750 INFO:teuthology.orchestra.run.smithi039.stdout: "fuse_max_write": "0", 2024-09-21T22:06:12.750 INFO:teuthology.orchestra.run.smithi039.stdout: "fuse_multithreaded": "true", 2024-09-21T22:06:12.750 INFO:teuthology.orchestra.run.smithi039.stdout: "fuse_require_active_mds": "true", 2024-09-21T22:06:12.750 INFO:teuthology.orchestra.run.smithi039.stdout: "fuse_set_user_groups": "true", 2024-09-21T22:06:12.750 INFO:teuthology.orchestra.run.smithi039.stdout: "fuse_splice_move": "true", 2024-09-21T22:06:12.750 INFO:teuthology.orchestra.run.smithi039.stdout: "fuse_splice_read": "true", 2024-09-21T22:06:12.750 INFO:teuthology.orchestra.run.smithi039.stdout: "fuse_splice_write": "true", 2024-09-21T22:06:12.750 INFO:teuthology.orchestra.run.smithi039.stdout: "fuse_syncfs_on_mksnap": "true", 2024-09-21T22:06:12.750 INFO:teuthology.orchestra.run.smithi039.stdout: "fuse_use_invalidate_cb": "true", 2024-09-21T22:06:12.750 INFO:teuthology.orchestra.run.smithi039.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-09-21T22:06:12.751 INFO:teuthology.orchestra.run.smithi039.stdout: "gss_target_name": "ceph", 2024-09-21T22:06:12.751 INFO:teuthology.orchestra.run.smithi039.stdout: "heartbeat_file": "", 2024-09-21T22:06:12.751 INFO:teuthology.orchestra.run.smithi039.stdout: "heartbeat_inject_failure": "0", 2024-09-21T22:06:12.751 INFO:teuthology.orchestra.run.smithi039.stdout: "heartbeat_interval": "5", 2024-09-21T22:06:12.751 INFO:teuthology.orchestra.run.smithi039.stdout: "host": "smithi039", 2024-09-21T22:06:12.751 INFO:teuthology.orchestra.run.smithi039.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-09-21T22:06:12.751 INFO:teuthology.orchestra.run.smithi039.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-09-21T22:06:12.751 INFO:teuthology.orchestra.run.smithi039.stdout: "immutable_object_cache_max_size": "1073741824", 2024-09-21T22:06:12.751 INFO:teuthology.orchestra.run.smithi039.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-09-21T22:06:12.751 INFO:teuthology.orchestra.run.smithi039.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-09-21T22:06:12.751 INFO:teuthology.orchestra.run.smithi039.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-09-21T22:06:12.751 INFO:teuthology.orchestra.run.smithi039.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-09-21T22:06:12.751 INFO:teuthology.orchestra.run.smithi039.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-09-21T22:06:12.751 INFO:teuthology.orchestra.run.smithi039.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-09-21T22:06:12.752 INFO:teuthology.orchestra.run.smithi039.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-09-21T22:06:12.752 INFO:teuthology.orchestra.run.smithi039.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-09-21T22:06:12.752 INFO:teuthology.orchestra.run.smithi039.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-09-21T22:06:12.752 INFO:teuthology.orchestra.run.smithi039.stdout: "immutable_object_cache_watermark": "0.900000", 2024-09-21T22:06:12.752 INFO:teuthology.orchestra.run.smithi039.stdout: "inject_early_sigterm": "false", 2024-09-21T22:06:12.752 INFO:teuthology.orchestra.run.smithi039.stdout: "jaeger_agent_port": "6799", 2024-09-21T22:06:12.752 INFO:teuthology.orchestra.run.smithi039.stdout: "jaeger_tracing_enable": "false", 2024-09-21T22:06:12.752 INFO:teuthology.orchestra.run.smithi039.stdout: "journal_aio": "true", 2024-09-21T22:06:12.752 INFO:teuthology.orchestra.run.smithi039.stdout: "journal_align_min_size": "65536", 2024-09-21T22:06:12.752 INFO:teuthology.orchestra.run.smithi039.stdout: "journal_block_align": "true", 2024-09-21T22:06:12.752 INFO:teuthology.orchestra.run.smithi039.stdout: "journal_block_size": "4096", 2024-09-21T22:06:12.752 INFO:teuthology.orchestra.run.smithi039.stdout: "journal_dio": "true", 2024-09-21T22:06:12.752 INFO:teuthology.orchestra.run.smithi039.stdout: "journal_discard": "false", 2024-09-21T22:06:12.752 INFO:teuthology.orchestra.run.smithi039.stdout: "journal_force_aio": "false", 2024-09-21T22:06:12.753 INFO:teuthology.orchestra.run.smithi039.stdout: "journal_ignore_corruption": "false", 2024-09-21T22:06:12.753 INFO:teuthology.orchestra.run.smithi039.stdout: "journal_max_write_bytes": "10485760", 2024-09-21T22:06:12.753 INFO:teuthology.orchestra.run.smithi039.stdout: "journal_max_write_entries": "100", 2024-09-21T22:06:12.753 INFO:teuthology.orchestra.run.smithi039.stdout: "journal_replay_from": "0", 2024-09-21T22:06:12.753 INFO:teuthology.orchestra.run.smithi039.stdout: "journal_throttle_high_multiple": "0.000000", 2024-09-21T22:06:12.753 INFO:teuthology.orchestra.run.smithi039.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-09-21T22:06:12.753 INFO:teuthology.orchestra.run.smithi039.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-09-21T22:06:12.753 INFO:teuthology.orchestra.run.smithi039.stdout: "journal_throttle_max_multiple": "0.000000", 2024-09-21T22:06:12.753 INFO:teuthology.orchestra.run.smithi039.stdout: "journal_write_header_frequency": "0", 2024-09-21T22:06:12.753 INFO:teuthology.orchestra.run.smithi039.stdout: "journal_zero_on_create": "false", 2024-09-21T22:06:12.753 INFO:teuthology.orchestra.run.smithi039.stdout: "journaler_prefetch_periods": "10", 2024-09-21T22:06:12.753 INFO:teuthology.orchestra.run.smithi039.stdout: "journaler_prezero_periods": "5", 2024-09-21T22:06:12.753 INFO:teuthology.orchestra.run.smithi039.stdout: "journaler_write_head_interval": "15", 2024-09-21T22:06:12.753 INFO:teuthology.orchestra.run.smithi039.stdout: "key": "", 2024-09-21T22:06:12.754 INFO:teuthology.orchestra.run.smithi039.stdout: "keyfile": "", 2024-09-21T22:06:12.754 INFO:teuthology.orchestra.run.smithi039.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-09-21T22:06:12.754 INFO:teuthology.orchestra.run.smithi039.stdout: "kstore_backend": "rocksdb", 2024-09-21T22:06:12.754 INFO:teuthology.orchestra.run.smithi039.stdout: "kstore_default_stripe_size": "65536", 2024-09-21T22:06:12.754 INFO:teuthology.orchestra.run.smithi039.stdout: "kstore_fsck_on_mount": "false", 2024-09-21T22:06:12.754 INFO:teuthology.orchestra.run.smithi039.stdout: "kstore_fsck_on_mount_deep": "true", 2024-09-21T22:06:12.754 INFO:teuthology.orchestra.run.smithi039.stdout: "kstore_max_bytes": "67108864", 2024-09-21T22:06:12.754 INFO:teuthology.orchestra.run.smithi039.stdout: "kstore_max_ops": "512", 2024-09-21T22:06:12.754 INFO:teuthology.orchestra.run.smithi039.stdout: "kstore_nid_prealloc": "1024", 2024-09-21T22:06:12.754 INFO:teuthology.orchestra.run.smithi039.stdout: "kstore_onode_map_size": "1024", 2024-09-21T22:06:12.754 INFO:teuthology.orchestra.run.smithi039.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-09-21T22:06:12.754 INFO:teuthology.orchestra.run.smithi039.stdout: "kstore_sync_submit_transaction": "false", 2024-09-21T22:06:12.754 INFO:teuthology.orchestra.run.smithi039.stdout: "kstore_sync_transaction": "false", 2024-09-21T22:06:12.754 INFO:teuthology.orchestra.run.smithi039.stdout: "librados_thread_count": "2", 2024-09-21T22:06:12.755 INFO:teuthology.orchestra.run.smithi039.stdout: "lockdep": "false", 2024-09-21T22:06:12.755 INFO:teuthology.orchestra.run.smithi039.stdout: "lockdep_force_backtrace": "false", 2024-09-21T22:06:12.755 INFO:teuthology.orchestra.run.smithi039.stdout: "log_coarse_timestamps": "true", 2024-09-21T22:06:12.755 INFO:teuthology.orchestra.run.smithi039.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-09-21T22:06:12.755 INFO:teuthology.orchestra.run.smithi039.stdout: "log_flush_on_exit": "false", 2024-09-21T22:06:12.755 INFO:teuthology.orchestra.run.smithi039.stdout: "log_graylog_host": "127.0.0.1", 2024-09-21T22:06:12.755 INFO:teuthology.orchestra.run.smithi039.stdout: "log_graylog_port": "12201", 2024-09-21T22:06:12.755 INFO:teuthology.orchestra.run.smithi039.stdout: "log_max_new": "1000", 2024-09-21T22:06:12.755 INFO:teuthology.orchestra.run.smithi039.stdout: "log_max_recent": "10000", 2024-09-21T22:06:12.755 INFO:teuthology.orchestra.run.smithi039.stdout: "log_stderr_prefix": "", 2024-09-21T22:06:12.755 INFO:teuthology.orchestra.run.smithi039.stdout: "log_stop_at_utilization": "0.970000", 2024-09-21T22:06:12.755 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": "true", 2024-09-21T22:06:12.755 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_graylog": "false", 2024-09-21T22:06:12.756 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_journald": "false", 2024-09-21T22:06:12.756 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_stderr": "false", 2024-09-21T22:06:12.756 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_syslog": "false", 2024-09-21T22:06:12.756 INFO:teuthology.orchestra.run.smithi039.stdout: "max_rotating_auth_attempts": "10", 2024-09-21T22:06:12.756 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-09-21T22:06:12.756 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_action_on_write_error": "1", 2024-09-21T22:06:12.756 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_alternate_name_max": "8192", 2024-09-21T22:06:12.756 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_asio_thread_count": "2", 2024-09-21T22:06:12.756 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_bal_export_pin": "true", 2024-09-21T22:06:12.756 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_bal_fragment_dirs": "true", 2024-09-21T22:06:12.756 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-09-21T22:06:12.756 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_bal_fragment_interval": "5", 2024-09-21T22:06:12.756 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_bal_fragment_size_max": "100000", 2024-09-21T22:06:12.756 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_bal_idle_threshold": "0.000000", 2024-09-21T22:06:12.757 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_bal_interval": "10", 2024-09-21T22:06:12.757 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_bal_max": "-1", 2024-09-21T22:06:12.757 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_bal_max_until": "-1", 2024-09-21T22:06:12.757 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_bal_merge_size": "50", 2024-09-21T22:06:12.757 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_bal_midchunk": "0.300000", 2024-09-21T22:06:12.757 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_bal_min_rebalance": "0.100000", 2024-09-21T22:06:12.757 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_bal_min_start": "0.200000", 2024-09-21T22:06:12.757 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_bal_minchunk": "0.001000", 2024-09-21T22:06:12.757 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_bal_mode": "0", 2024-09-21T22:06:12.757 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_bal_need_max": "1.200000", 2024-09-21T22:06:12.757 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_bal_need_min": "0.800000", 2024-09-21T22:06:12.757 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_bal_overload_epochs": "2", 2024-09-21T22:06:12.757 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-09-21T22:06:12.757 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_bal_sample_interval": "3.000000", 2024-09-21T22:06:12.757 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_bal_split_bits": "3", 2024-09-21T22:06:12.758 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_bal_split_rd": "25000.000000", 2024-09-21T22:06:12.758 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_bal_split_size": "10000", 2024-09-21T22:06:12.758 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_bal_split_wr": "10000.000000", 2024-09-21T22:06:12.758 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_bal_target_decay": "10.000000", 2024-09-21T22:06:12.758 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-09-21T22:06:12.758 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_beacon_grace": "15.000000", 2024-09-21T22:06:12.758 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_beacon_interval": "4.000000", 2024-09-21T22:06:12.758 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_beacon_mon_down_grace": "60", 2024-09-21T22:06:12.758 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_cache_memory_limit": "4294967296", 2024-09-21T22:06:12.758 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_cache_mid": "0.700000", 2024-09-21T22:06:12.758 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_cache_quiesce_decay_rate": "1.000000", 2024-09-21T22:06:12.758 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_cache_quiesce_delay": "0", 2024-09-21T22:06:12.759 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_cache_quiesce_sleep": "200", 2024-09-21T22:06:12.759 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_cache_quiesce_splitauth": "true", 2024-09-21T22:06:12.759 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_cache_quiesce_threshold": "524288", 2024-09-21T22:06:12.759 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_cache_release_free_interval": "10", 2024-09-21T22:06:12.759 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_cache_reservation": "0.050000", 2024-09-21T22:06:12.759 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-09-21T22:06:12.759 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_cache_trim_interval": "1", 2024-09-21T22:06:12.759 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_cache_trim_threshold": "262144", 2024-09-21T22:06:12.759 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-09-21T22:06:12.759 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-09-21T22:06:12.759 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_client_delegate_inos_pct": "50", 2024-09-21T22:06:12.759 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_client_prealloc_inos": "1000", 2024-09-21T22:06:12.759 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-09-21T22:06:12.759 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_connect_bootstrapping": "false", 2024-09-21T22:06:12.759 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_damage_table_max_entries": "10000", 2024-09-21T22:06:12.760 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-09-21T22:06:12.760 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_debug_auth_pins": "false", 2024-09-21T22:06:12.760 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_debug_frag": "false", 2024-09-21T22:06:12.760 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_debug_scatterstat": "false", 2024-09-21T22:06:12.760 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_debug_subtrees": "false", 2024-09-21T22:06:12.760 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_decay_halflife": "5.000000", 2024-09-21T22:06:12.760 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_default_dir_hash": "2", 2024-09-21T22:06:12.760 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_defer_session_stale": "true", 2024-09-21T22:06:12.760 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_deny_all_reconnect": "false", 2024-09-21T22:06:12.760 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_dir_keys_per_op": "16384", 2024-09-21T22:06:12.760 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_dir_max_commit_size": "10", 2024-09-21T22:06:12.760 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_dir_max_entries": "0", 2024-09-21T22:06:12.760 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_dir_prefetch": "true", 2024-09-21T22:06:12.760 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_dirstat_min_interval": "1.000000", 2024-09-21T22:06:12.760 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_dump_cache_after_rejoin": "false", 2024-09-21T22:06:12.761 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_dump_cache_on_map": "false", 2024-09-21T22:06:12.761 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_dump_cache_threshold_file": "0", 2024-09-21T22:06:12.761 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-09-21T22:06:12.761 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_early_reply": "true", 2024-09-21T22:06:12.761 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_enable_op_tracker": "true", 2024-09-21T22:06:12.761 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_enforce_unique_name": "true", 2024-09-21T22:06:12.761 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_export_ephemeral_distributed": "true", 2024-09-21T22:06:12.761 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-09-21T22:06:12.761 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_export_ephemeral_random": "true", 2024-09-21T22:06:12.761 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-09-21T22:06:12.761 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-09-21T22:06:12.761 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_forward_all_requests_to_auth": "false", 2024-09-21T22:06:12.761 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-09-21T22:06:12.761 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-09-21T22:06:12.761 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-09-21T22:06:12.762 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-09-21T22:06:12.762 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_health_cache_threshold": "1.500000", 2024-09-21T22:06:12.762 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_health_summarize_threshold": "10", 2024-09-21T22:06:12.762 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_heartbeat_grace": "15.000000", 2024-09-21T22:06:12.762 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_heartbeat_reset_grace": "1000", 2024-09-21T22:06:12.762 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_inject_health_dummy": "false", 2024-09-21T22:06:12.762 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-09-21T22:06:12.762 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_inject_migrator_session_race": "false", 2024-09-21T22:06:12.762 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-09-21T22:06:12.762 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-09-21T22:06:12.762 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-09-21T22:06:12.762 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_join_fs": "", 2024-09-21T22:06:12.762 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_journal_format": "1", 2024-09-21T22:06:12.762 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_kill_after_journal_logs_flushed": "false", 2024-09-21T22:06:12.762 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_kill_create_at": "0", 2024-09-21T22:06:12.763 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_kill_export_at": "0", 2024-09-21T22:06:12.763 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_kill_import_at": "0", 2024-09-21T22:06:12.763 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_kill_journal_at": "0", 2024-09-21T22:06:12.763 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_kill_journal_expire_at": "0", 2024-09-21T22:06:12.763 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_kill_journal_replay_at": "0", 2024-09-21T22:06:12.763 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_kill_link_at": "0", 2024-09-21T22:06:12.763 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_kill_mdstable_at": "0", 2024-09-21T22:06:12.763 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_kill_openc_at": "0", 2024-09-21T22:06:12.763 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_kill_rename_at": "0", 2024-09-21T22:06:12.763 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_kill_shutdown_at": "0", 2024-09-21T22:06:12.763 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_log_event_large_threshold": "524288", 2024-09-21T22:06:12.763 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_log_events_per_segment": "1024", 2024-09-21T22:06:12.763 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_log_major_segment_event_ratio": "12", 2024-09-21T22:06:12.763 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_log_max_events": "-1", 2024-09-21T22:06:12.763 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_log_max_segments": "128", 2024-09-21T22:06:12.763 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_log_pause": "false", 2024-09-21T22:06:12.764 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_log_segment_size": "0", 2024-09-21T22:06:12.764 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_log_skip_corrupt_events": "false", 2024-09-21T22:06:12.764 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_log_skip_unbounded_events": "false", 2024-09-21T22:06:12.764 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-09-21T22:06:12.764 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_log_trim_threshold": "128", 2024-09-21T22:06:12.764 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-09-21T22:06:12.764 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_log_warn_factor": "2.000000", 2024-09-21T22:06:12.764 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_max_caps_per_client": "1048576", 2024-09-21T22:06:12.764 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_max_completed_flushes": "100000", 2024-09-21T22:06:12.764 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_max_completed_requests": "100000", 2024-09-21T22:06:12.764 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_max_export_size": "20971520", 2024-09-21T22:06:12.764 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_max_file_recover": "32", 2024-09-21T22:06:12.764 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_max_purge_files": "64", 2024-09-21T22:06:12.764 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_max_purge_ops": "8192", 2024-09-21T22:06:12.764 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-09-21T22:06:12.764 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-09-21T22:06:12.765 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_max_snaps_per_dir": "100", 2024-09-21T22:06:12.765 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_metrics_update_interval": "2", 2024-09-21T22:06:12.765 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_min_caps_per_client": "100", 2024-09-21T22:06:12.765 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_min_caps_working_set": "10000", 2024-09-21T22:06:12.765 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-09-21T22:06:12.765 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_numa_node": "-1", 2024-09-21T22:06:12.765 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-09-21T22:06:12.765 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_op_complaint_time": "30.000000", 2024-09-21T22:06:12.765 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_op_history_duration": "600", 2024-09-21T22:06:12.765 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_op_history_size": "20", 2024-09-21T22:06:12.765 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_op_history_slow_op_size": "20", 2024-09-21T22:06:12.765 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_op_history_slow_op_threshold": "10", 2024-09-21T22:06:12.765 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_op_log_threshold": "5", 2024-09-21T22:06:12.765 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_ping_grace": "15", 2024-09-21T22:06:12.765 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_ping_interval": "5", 2024-09-21T22:06:12.765 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-09-21T22:06:12.766 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-09-21T22:06:12.766 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_recall_max_caps": "30000", 2024-09-21T22:06:12.766 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-09-21T22:06:12.766 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_recall_max_decay_threshold": "131072", 2024-09-21T22:06:12.766 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-09-21T22:06:12.766 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_recall_warning_threshold": "262144", 2024-09-21T22:06:12.766 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_reconnect_timeout": "45.000000", 2024-09-21T22:06:12.766 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_replay_interval": "1.000000", 2024-09-21T22:06:12.766 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-09-21T22:06:12.766 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-09-21T22:06:12.766 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_root_ino_gid": "0", 2024-09-21T22:06:12.766 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_root_ino_uid": "0", 2024-09-21T22:06:12.766 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-09-21T22:06:12.766 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_session_blocklist_on_evict": "true", 2024-09-21T22:06:12.767 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_session_blocklist_on_timeout": "true", 2024-09-21T22:06:12.767 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-09-21T22:06:12.767 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-09-21T22:06:12.767 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-09-21T22:06:12.767 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-09-21T22:06:12.767 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-09-21T22:06:12.767 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_session_metadata_threshold": "16777216", 2024-09-21T22:06:12.767 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-09-21T22:06:12.767 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_shutdown_check": "0", 2024-09-21T22:06:12.767 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_skip_ino": "0", 2024-09-21T22:06:12.767 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_sleep_rank_change": "0.000000", 2024-09-21T22:06:12.767 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_snap_max_uid": "4294967294", 2024-09-21T22:06:12.767 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_snap_min_uid": "0", 2024-09-21T22:06:12.767 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_snap_rstat": "false", 2024-09-21T22:06:12.767 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_standby_replay_damaged": "false", 2024-09-21T22:06:12.767 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_symlink_recovery": "true", 2024-09-21T22:06:12.768 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_task_status_update_interval": "2.000000", 2024-09-21T22:06:12.768 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_thrash_exports": "0", 2024-09-21T22:06:12.768 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_thrash_fragments": "0", 2024-09-21T22:06:12.768 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_tick_interval": "5.000000", 2024-09-21T22:06:12.768 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_valgrind_exit": "false", 2024-09-21T22:06:12.768 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_verify_backtrace": "1", 2024-09-21T22:06:12.768 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_verify_scatter": "false", 2024-09-21T22:06:12.768 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_wipe_ino_prealloc": "false", 2024-09-21T22:06:12.768 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_wipe_sessions": "false", 2024-09-21T22:06:12.768 INFO:teuthology.orchestra.run.smithi039.stdout: "mempool_debug": "false", 2024-09-21T22:06:12.768 INFO:teuthology.orchestra.run.smithi039.stdout: "memstore_debug_omit_block_device_write": "false", 2024-09-21T22:06:12.768 INFO:teuthology.orchestra.run.smithi039.stdout: "memstore_device_bytes": "1073741824", 2024-09-21T22:06:12.768 INFO:teuthology.orchestra.run.smithi039.stdout: "memstore_page_set": "false", 2024-09-21T22:06:12.768 INFO:teuthology.orchestra.run.smithi039.stdout: "memstore_page_size": "65536", 2024-09-21T22:06:12.768 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr_client_bytes": "134217728", 2024-09-21T22:06:12.768 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr_client_messages": "512", 2024-09-21T22:06:12.769 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-09-21T22:06:12.769 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr_connect_retry_interval": "1.000000", 2024-09-21T22:06:12.769 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-09-21T22:06:12.769 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-09-21T22:06:12.769 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr_disabled_modules": "", 2024-09-21T22:06:12.769 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr_enable_op_tracker": "true", 2024-09-21T22:06:12.769 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-09-21T22:06:12.769 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr_max_pg_num_change": "128", 2024-09-21T22:06:12.769 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr_mds_bytes": "134217728", 2024-09-21T22:06:12.769 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr_mds_messages": "128", 2024-09-21T22:06:12.769 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-09-21T22:06:12.769 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr_mon_bytes": "134217728", 2024-09-21T22:06:12.769 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr_mon_messages": "128", 2024-09-21T22:06:12.769 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr_num_op_tracker_shard": "32", 2024-09-21T22:06:12.769 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr_op_complaint_time": "30.000000", 2024-09-21T22:06:12.769 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr_op_history_duration": "600", 2024-09-21T22:06:12.770 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr_op_history_size": "20", 2024-09-21T22:06:12.770 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr_op_history_slow_op_size": "20", 2024-09-21T22:06:12.770 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr_op_history_slow_op_threshold": "10.000000", 2024-09-21T22:06:12.770 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr_op_log_threshold": "5", 2024-09-21T22:06:12.770 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr_osd_bytes": "536870912", 2024-09-21T22:06:12.770 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr_osd_messages": "8192", 2024-09-21T22:06:12.770 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr_pool": "true", 2024-09-21T22:06:12.770 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr_service_beacon_grace": "60.000000", 2024-09-21T22:06:12.770 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr_standby_modules": "true", 2024-09-21T22:06:12.770 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr_stats_period": "5", 2024-09-21T22:06:12.770 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr_stats_threshold": "5", 2024-09-21T22:06:12.770 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr_tick_period": "2", 2024-09-21T22:06:12.770 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-09-21T22:06:12.770 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_accept_timeout_factor": "2.000000", 2024-09-21T22:06:12.770 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_allow_pool_delete": "true", 2024-09-21T22:06:12.771 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_allow_pool_size_one": "true", 2024-09-21T22:06:12.771 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_auth_validate_all_caps": "true", 2024-09-21T22:06:12.771 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-09-21T22:06:12.771 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-09-21T22:06:12.771 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_client_bytes": "104857600", 2024-09-21T22:06:12.771 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_client_directed_command_retry": "5", 2024-09-21T22:06:12.771 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_client_hunt_interval": "3.000000", 2024-09-21T22:06:12.771 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-09-21T22:06:12.771 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-09-21T22:06:12.771 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-09-21T22:06:12.771 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_client_hunt_parallel": "3", 2024-09-21T22:06:12.771 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_client_log_interval": "1.000000", 2024-09-21T22:06:12.771 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-09-21T22:06:12.771 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_client_ping_interval": "10.000000", 2024-09-21T22:06:12.772 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_client_ping_timeout": "30.000000", 2024-09-21T22:06:12.772 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_client_target_rank": "-1", 2024-09-21T22:06:12.772 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_clock_drift_allowed": "1.000000", 2024-09-21T22:06:12.772 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-09-21T22:06:12.772 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-09-21T22:06:12.772 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_cluster_log_file_level": "debug", 2024-09-21T22:06:12.772 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_cluster_log_to_file": "true", 2024-09-21T22:06:12.772 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_cluster_log_to_graylog": "false", 2024-09-21T22:06:12.772 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-09-21T22:06:12.772 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-09-21T22:06:12.772 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_cluster_log_to_journald": "false", 2024-09-21T22:06:12.772 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_cluster_log_to_stderr": "false", 2024-09-21T22:06:12.772 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-09-21T22:06:12.772 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-09-21T22:06:12.772 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_cluster_log_to_syslog_level": "info", 2024-09-21T22:06:12.772 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_compact_on_bootstrap": "false", 2024-09-21T22:06:12.773 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_compact_on_start": "false", 2024-09-21T22:06:12.773 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_compact_on_trim": "true", 2024-09-21T22:06:12.773 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_con_tracker_persist_interval": "10", 2024-09-21T22:06:12.773 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_con_tracker_score_halflife": "43200", 2024-09-21T22:06:12.773 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_config_key_max_entry_size": "65536", 2024-09-21T22:06:12.773 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_cpu_threads": "4", 2024-09-21T22:06:12.773 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_crush_min_required_version": "hammer", 2024-09-21T22:06:12.773 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_daemon_bytes": "419430400", 2024-09-21T22:06:12.773 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-09-21T22:06:12.773 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_data_avail_crit": "5", 2024-09-21T22:06:12.773 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_data_avail_warn": "5", 2024-09-21T22:06:12.773 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_data_size_warn": "16106127360", 2024-09-21T22:06:12.773 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_debug_block_osdmap_trim": "false", 2024-09-21T22:06:12.773 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-09-21T22:06:12.773 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_debug_dump_json": "false", 2024-09-21T22:06:12.774 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-09-21T22:06:12.774 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_debug_dump_transactions": "false", 2024-09-21T22:06:12.774 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_debug_extra_checks": "false", 2024-09-21T22:06:12.774 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-09-21T22:06:12.774 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-09-21T22:06:12.774 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_debug_no_require_reef": "false", 2024-09-21T22:06:12.774 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_debug_no_require_squid": "false", 2024-09-21T22:06:12.774 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-09-21T22:06:12.774 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_delta_reset_interval": "10.000000", 2024-09-21T22:06:12.774 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_dns_srv_name": "ceph-mon", 2024-09-21T22:06:12.774 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_down_mkfs_grace": "120", 2024-09-21T22:06:12.774 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_down_uptime_grace": "60", 2024-09-21T22:06:12.774 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_election_default_strategy": "1", 2024-09-21T22:06:12.774 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_election_timeout": "5.000000", 2024-09-21T22:06:12.775 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-09-21T22:06:12.775 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_elector_ping_divisor": "2", 2024-09-21T22:06:12.775 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_elector_ping_timeout": "2.000000", 2024-09-21T22:06:12.775 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_enable_op_tracker": "true", 2024-09-21T22:06:12.775 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_fake_pool_delete": "false", 2024-09-21T22:06:12.775 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_force_quorum_join": "false", 2024-09-21T22:06:12.775 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_fsmap_prune_threshold": "300", 2024-09-21T22:06:12.775 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_globalid_prealloc": "10000", 2024-09-21T22:06:12.775 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_health_detail_to_clog": "false", 2024-09-21T22:06:12.775 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_health_log_update_period": "5", 2024-09-21T22:06:12.775 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_health_max_detail": "50", 2024-09-21T22:06:12.775 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_health_to_clog": "true", 2024-09-21T22:06:12.775 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_health_to_clog_interval": "600", 2024-09-21T22:06:12.775 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-09-21T22:06:12.775 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_host": "172.21.15.39,[v2:172.21.15.39:3301,v1:172.21.15.39:6790],172.21.15.64", 2024-09-21T22:06:12.776 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_host_override": "", 2024-09-21T22:06:12.776 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_initial_members": "", 2024-09-21T22:06:12.776 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-09-21T22:06:12.776 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-09-21T22:06:12.776 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-09-21T22:06:12.776 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-09-21T22:06:12.776 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_keyvaluedb": "rocksdb", 2024-09-21T22:06:12.776 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_lease": "5.000000", 2024-09-21T22:06:12.776 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-09-21T22:06:12.776 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-09-21T22:06:12.776 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_log_full_interval": "50", 2024-09-21T22:06:12.776 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_log_max": "10000", 2024-09-21T22:06:12.776 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_log_max_summary": "50", 2024-09-21T22:06:12.776 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_max_log_entries_per_event": "4096", 2024-09-21T22:06:12.776 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_max_log_epochs": "500", 2024-09-21T22:06:12.776 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_max_mdsmap_epochs": "500", 2024-09-21T22:06:12.777 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_max_mgrmap_epochs": "500", 2024-09-21T22:06:12.777 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_max_osd": "10000", 2024-09-21T22:06:12.777 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_max_pg_per_osd": "10000", 2024-09-21T22:06:12.777 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_max_pool_pg_num": "65536", 2024-09-21T22:06:12.777 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-09-21T22:06:12.777 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-09-21T22:06:12.777 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_mds_force_trim_to": "0", 2024-09-21T22:06:12.777 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_mds_skip_sanity": "false", 2024-09-21T22:06:12.777 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_memory_autotune": "true", 2024-09-21T22:06:12.777 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_memory_target": "2147483648", 2024-09-21T22:06:12.777 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_mgr_beacon_grace": "30", 2024-09-21T22:06:12.777 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-09-21T22:06:12.777 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_mgr_digest_period": "5", 2024-09-21T22:06:12.777 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_mgr_inactive_grace": "60", 2024-09-21T22:06:12.777 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_mgr_mkfs_grace": "240", 2024-09-21T22:06:12.778 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-09-21T22:06:12.778 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_min_osdmap_epochs": "500", 2024-09-21T22:06:12.778 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_op_complaint_time": "30", 2024-09-21T22:06:12.778 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_op_history_duration": "600", 2024-09-21T22:06:12.778 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_op_history_size": "20", 2024-09-21T22:06:12.778 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_op_history_slow_op_size": "20", 2024-09-21T22:06:12.778 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_op_history_slow_op_threshold": "10", 2024-09-21T22:06:12.778 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_op_log_threshold": "5", 2024-09-21T22:06:12.778 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-09-21T22:06:12.778 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-09-21T22:06:12.778 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-09-21T22:06:12.778 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_osd_auto_mark_in": "false", 2024-09-21T22:06:12.778 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_osd_auto_mark_new_in": "true", 2024-09-21T22:06:12.779 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-09-21T22:06:12.779 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-09-21T22:06:12.779 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_osd_cache_size": "500", 2024-09-21T22:06:12.779 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_osd_cache_size_min": "134217728", 2024-09-21T22:06:12.779 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_osd_crush_smoke_test": "true", 2024-09-21T22:06:12.779 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_osd_destroyed_out_interval": "600", 2024-09-21T22:06:12.779 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_osd_down_out_interval": "600", 2024-09-21T22:06:12.779 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-09-21T22:06:12.779 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-09-21T22:06:12.779 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_osd_force_trim_to": "0", 2024-09-21T22:06:12.779 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_osd_full_ratio": "0.950000", 2024-09-21T22:06:12.779 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-09-21T22:06:12.779 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_osd_laggy_halflife": "3600", 2024-09-21T22:06:12.779 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_osd_laggy_max_interval": "300", 2024-09-21T22:06:12.780 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_osd_laggy_weight": "0.300000", 2024-09-21T22:06:12.780 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-09-21T22:06:12.780 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_osd_max_creating_pgs": "1024", 2024-09-21T22:06:12.780 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_osd_max_initial_pgs": "1024", 2024-09-21T22:06:12.780 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_osd_min_down_reporters": "2", 2024-09-21T22:06:12.780 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-09-21T22:06:12.780 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-09-21T22:06:12.780 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-09-21T22:06:12.780 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_osd_prime_pg_temp": "true", 2024-09-21T22:06:12.780 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-09-21T22:06:12.780 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-09-21T22:06:12.780 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_osd_report_timeout": "900", 2024-09-21T22:06:12.780 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-09-21T22:06:12.780 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-09-21T22:06:12.780 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_osd_warn_num_repaired": "10", 2024-09-21T22:06:12.781 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_osd_warn_op_age": "32.000000", 2024-09-21T22:06:12.781 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-09-21T22:06:12.781 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_osdmap_full_prune_interval": "10", 2024-09-21T22:06:12.781 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_osdmap_full_prune_min": "10000", 2024-09-21T22:06:12.781 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_osdmap_full_prune_txsize": "100", 2024-09-21T22:06:12.781 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-09-21T22:06:12.781 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_pg_stuck_threshold": "60", 2024-09-21T22:06:12.781 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-09-21T22:06:12.781 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_pg_warn_min_objects": "10000", 2024-09-21T22:06:12.781 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_pg_warn_min_per_osd": "0", 2024-09-21T22:06:12.781 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-09-21T22:06:12.781 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_pool_quota_crit_threshold": "0", 2024-09-21T22:06:12.781 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_pool_quota_warn_threshold": "0", 2024-09-21T22:06:12.781 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_probe_timeout": "2.000000", 2024-09-21T22:06:12.782 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_reweight_max_change": "0.050000", 2024-09-21T22:06:12.782 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_reweight_max_osds": "4", 2024-09-21T22:06:12.782 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-09-21T22:06:12.782 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-09-21T22:06:12.782 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-09-21T22:06:12.782 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-09-21T22:06:12.782 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-09-21T22:06:12.782 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_scrub_interval": "86400", 2024-09-21T22:06:12.782 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_scrub_max_keys": "100", 2024-09-21T22:06:12.782 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_scrub_timeout": "300", 2024-09-21T22:06:12.782 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_session_timeout": "300", 2024-09-21T22:06:12.782 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_smart_report_timeout": "5", 2024-09-21T22:06:12.782 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_stat_smooth_intervals": "6", 2024-09-21T22:06:12.782 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-09-21T22:06:12.783 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_stretch_pool_min_size": "2", 2024-09-21T22:06:12.783 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_stretch_pool_size": "4", 2024-09-21T22:06:12.783 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-09-21T22:06:12.783 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_subscribe_interval": "86400.000000", 2024-09-21T22:06:12.783 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_sync_debug": "false", 2024-09-21T22:06:12.783 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_sync_max_payload_keys": "2000", 2024-09-21T22:06:12.783 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_sync_max_payload_size": "1048576", 2024-09-21T22:06:12.783 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_sync_provider_kill_at": "0", 2024-09-21T22:06:12.783 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_sync_requester_kill_at": "0", 2024-09-21T22:06:12.783 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_sync_timeout": "60.000000", 2024-09-21T22:06:12.783 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_target_pg_per_osd": "100", 2024-09-21T22:06:12.783 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_tick_interval": "5", 2024-09-21T22:06:12.783 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_timecheck_interval": "300.000000", 2024-09-21T22:06:12.784 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-09-21T22:06:12.784 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_use_min_delay_socket": "false", 2024-09-21T22:06:12.784 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_warn_older_version_delay": "604800", 2024-09-21T22:06:12.784 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-09-21T22:06:12.784 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-09-21T22:06:12.784 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-09-21T22:06:12.784 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_warn_on_filestore_osds": "false", 2024-09-21T22:06:12.784 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-09-21T22:06:12.784 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-09-21T22:06:12.784 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-09-21T22:06:12.784 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_warn_on_misplaced": "false", 2024-09-21T22:06:12.784 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-09-21T22:06:12.784 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_warn_on_older_version": "true", 2024-09-21T22:06:12.784 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-09-21T22:06:12.785 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_warn_on_pool_no_app": "false", 2024-09-21T22:06:12.785 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_warn_on_pool_no_redundancy": "false", 2024-09-21T22:06:12.785 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-09-21T22:06:12.785 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-09-21T22:06:12.785 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-09-21T22:06:12.785 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_warn_on_too_few_osds": "false", 2024-09-21T22:06:12.785 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-09-21T22:06:12.785 INFO:teuthology.orchestra.run.smithi039.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-09-21T22:06:12.785 INFO:teuthology.orchestra.run.smithi039.stdout: "monmap": "", 2024-09-21T22:06:12.785 INFO:teuthology.orchestra.run.smithi039.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-09-21T22:06:12.785 INFO:teuthology.orchestra.run.smithi039.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-09-21T22:06:12.785 INFO:teuthology.orchestra.run.smithi039.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-09-21T22:06:12.785 INFO:teuthology.orchestra.run.smithi039.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-09-21T22:06:12.785 INFO:teuthology.orchestra.run.smithi039.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-09-21T22:06:12.785 INFO:teuthology.orchestra.run.smithi039.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-09-21T22:06:12.786 INFO:teuthology.orchestra.run.smithi039.stdout: "motr_tracing_enabled": "false", 2024-09-21T22:06:12.786 INFO:teuthology.orchestra.run.smithi039.stdout: "ms_async_op_threads": "3", 2024-09-21T22:06:12.786 INFO:teuthology.orchestra.run.smithi039.stdout: "ms_async_rdma_buffer_size": "131072", 2024-09-21T22:06:12.786 INFO:teuthology.orchestra.run.smithi039.stdout: "ms_async_rdma_cm": "false", 2024-09-21T22:06:12.786 INFO:teuthology.orchestra.run.smithi039.stdout: "ms_async_rdma_device_name": "", 2024-09-21T22:06:12.786 INFO:teuthology.orchestra.run.smithi039.stdout: "ms_async_rdma_dscp": "96", 2024-09-21T22:06:12.786 INFO:teuthology.orchestra.run.smithi039.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-09-21T22:06:12.786 INFO:teuthology.orchestra.run.smithi039.stdout: "ms_async_rdma_gid_idx": "0", 2024-09-21T22:06:12.786 INFO:teuthology.orchestra.run.smithi039.stdout: "ms_async_rdma_local_gid": "", 2024-09-21T22:06:12.786 INFO:teuthology.orchestra.run.smithi039.stdout: "ms_async_rdma_polling_us": "1000", 2024-09-21T22:06:12.786 INFO:teuthology.orchestra.run.smithi039.stdout: "ms_async_rdma_port_num": "1", 2024-09-21T22:06:12.786 INFO:teuthology.orchestra.run.smithi039.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-09-21T22:06:12.786 INFO:teuthology.orchestra.run.smithi039.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-09-21T22:06:12.787 INFO:teuthology.orchestra.run.smithi039.stdout: "ms_async_rdma_roce_ver": "1", 2024-09-21T22:06:12.787 INFO:teuthology.orchestra.run.smithi039.stdout: "ms_async_rdma_send_buffers": "1024", 2024-09-21T22:06:12.787 INFO:teuthology.orchestra.run.smithi039.stdout: "ms_async_rdma_sl": "3", 2024-09-21T22:06:12.787 INFO:teuthology.orchestra.run.smithi039.stdout: "ms_async_rdma_support_srq": "true", 2024-09-21T22:06:12.787 INFO:teuthology.orchestra.run.smithi039.stdout: "ms_async_rdma_type": "ib", 2024-09-21T22:06:12.787 INFO:teuthology.orchestra.run.smithi039.stdout: "ms_async_reap_threshold": "5", 2024-09-21T22:06:12.787 INFO:teuthology.orchestra.run.smithi039.stdout: "ms_bind_before_connect": "false", 2024-09-21T22:06:12.787 INFO:teuthology.orchestra.run.smithi039.stdout: "ms_bind_ipv4": "true", 2024-09-21T22:06:12.787 INFO:teuthology.orchestra.run.smithi039.stdout: "ms_bind_ipv6": "false", 2024-09-21T22:06:12.787 INFO:teuthology.orchestra.run.smithi039.stdout: "ms_bind_msgr1": "true", 2024-09-21T22:06:12.787 INFO:teuthology.orchestra.run.smithi039.stdout: "ms_bind_msgr2": "true", 2024-09-21T22:06:12.787 INFO:teuthology.orchestra.run.smithi039.stdout: "ms_bind_port_max": "7568", 2024-09-21T22:06:12.787 INFO:teuthology.orchestra.run.smithi039.stdout: "ms_bind_port_min": "6800", 2024-09-21T22:06:12.787 INFO:teuthology.orchestra.run.smithi039.stdout: "ms_bind_prefer_ipv4": "false", 2024-09-21T22:06:12.788 INFO:teuthology.orchestra.run.smithi039.stdout: "ms_bind_retry_count": "3", 2024-09-21T22:06:12.788 INFO:teuthology.orchestra.run.smithi039.stdout: "ms_bind_retry_delay": "5", 2024-09-21T22:06:12.788 INFO:teuthology.orchestra.run.smithi039.stdout: "ms_blackhole_client": "false", 2024-09-21T22:06:12.788 INFO:teuthology.orchestra.run.smithi039.stdout: "ms_blackhole_mds": "false", 2024-09-21T22:06:12.788 INFO:teuthology.orchestra.run.smithi039.stdout: "ms_blackhole_mgr": "false", 2024-09-21T22:06:12.788 INFO:teuthology.orchestra.run.smithi039.stdout: "ms_blackhole_mon": "false", 2024-09-21T22:06:12.788 INFO:teuthology.orchestra.run.smithi039.stdout: "ms_blackhole_osd": "false", 2024-09-21T22:06:12.788 INFO:teuthology.orchestra.run.smithi039.stdout: "ms_client_mode": "crc secure", 2024-09-21T22:06:12.788 INFO:teuthology.orchestra.run.smithi039.stdout: "ms_cluster_mode": "crc secure", 2024-09-21T22:06:12.788 INFO:teuthology.orchestra.run.smithi039.stdout: "ms_cluster_type": "", 2024-09-21T22:06:12.788 INFO:teuthology.orchestra.run.smithi039.stdout: "ms_compress_secure": "false", 2024-09-21T22:06:12.788 INFO:teuthology.orchestra.run.smithi039.stdout: "ms_connection_idle_timeout": "900", 2024-09-21T22:06:12.788 INFO:teuthology.orchestra.run.smithi039.stdout: "ms_connection_ready_timeout": "10", 2024-09-21T22:06:12.788 INFO:teuthology.orchestra.run.smithi039.stdout: "ms_crc_data": "true", 2024-09-21T22:06:12.789 INFO:teuthology.orchestra.run.smithi039.stdout: "ms_crc_header": "true", 2024-09-21T22:06:12.789 INFO:teuthology.orchestra.run.smithi039.stdout: "ms_die_on_bad_msg": "false", 2024-09-21T22:06:12.789 INFO:teuthology.orchestra.run.smithi039.stdout: "ms_die_on_bug": "true", 2024-09-21T22:06:12.789 INFO:teuthology.orchestra.run.smithi039.stdout: "ms_die_on_old_message": "true", 2024-09-21T22:06:12.789 INFO:teuthology.orchestra.run.smithi039.stdout: "ms_die_on_skipped_message": "false", 2024-09-21T22:06:12.789 INFO:teuthology.orchestra.run.smithi039.stdout: "ms_die_on_unhandled_msg": "false", 2024-09-21T22:06:12.789 INFO:teuthology.orchestra.run.smithi039.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-09-21T22:06:12.789 INFO:teuthology.orchestra.run.smithi039.stdout: "ms_dpdk_coremask": "0xF", 2024-09-21T22:06:12.789 INFO:teuthology.orchestra.run.smithi039.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-09-21T22:06:12.789 INFO:teuthology.orchestra.run.smithi039.stdout: "ms_dpdk_devs_allowlist": "", 2024-09-21T22:06:12.789 INFO:teuthology.orchestra.run.smithi039.stdout: "ms_dpdk_enable_tso": "true", 2024-09-21T22:06:12.789 INFO:teuthology.orchestra.run.smithi039.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-09-21T22:06:12.789 INFO:teuthology.orchestra.run.smithi039.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-09-21T22:06:12.789 INFO:teuthology.orchestra.run.smithi039.stdout: "ms_dpdk_hugepages": "", 2024-09-21T22:06:12.789 INFO:teuthology.orchestra.run.smithi039.stdout: "ms_dpdk_hw_flow_control": "true", 2024-09-21T22:06:12.790 INFO:teuthology.orchestra.run.smithi039.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-09-21T22:06:12.790 INFO:teuthology.orchestra.run.smithi039.stdout: "ms_dpdk_lro": "true", 2024-09-21T22:06:12.790 INFO:teuthology.orchestra.run.smithi039.stdout: "ms_dpdk_memory_channel": "4", 2024-09-21T22:06:12.790 INFO:teuthology.orchestra.run.smithi039.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-09-21T22:06:12.790 INFO:teuthology.orchestra.run.smithi039.stdout: "ms_dpdk_pmd": "", 2024-09-21T22:06:12.790 INFO:teuthology.orchestra.run.smithi039.stdout: "ms_dpdk_port_id": "0", 2024-09-21T22:06:12.790 INFO:teuthology.orchestra.run.smithi039.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-09-21T22:06:12.790 INFO:teuthology.orchestra.run.smithi039.stdout: "ms_dump_corrupt_message_level": "1", 2024-09-21T22:06:12.790 INFO:teuthology.orchestra.run.smithi039.stdout: "ms_dump_on_send": "false", 2024-09-21T22:06:12.790 INFO:teuthology.orchestra.run.smithi039.stdout: "ms_initial_backoff": "0.200000", 2024-09-21T22:06:12.790 INFO:teuthology.orchestra.run.smithi039.stdout: "ms_inject_delay_max": "1.000000", 2024-09-21T22:06:12.790 INFO:teuthology.orchestra.run.smithi039.stdout: "ms_inject_delay_probability": "0.000000", 2024-09-21T22:06:12.790 INFO:teuthology.orchestra.run.smithi039.stdout: "ms_inject_delay_type": "", 2024-09-21T22:06:12.791 INFO:teuthology.orchestra.run.smithi039.stdout: "ms_inject_internal_delays": "0.000000", 2024-09-21T22:06:12.791 INFO:teuthology.orchestra.run.smithi039.stdout: "ms_inject_network_congestion": "0", 2024-09-21T22:06:12.791 INFO:teuthology.orchestra.run.smithi039.stdout: "ms_inject_socket_failures": "5000", 2024-09-21T22:06:12.791 INFO:teuthology.orchestra.run.smithi039.stdout: "ms_learn_addr_from_peer": "true", 2024-09-21T22:06:12.791 INFO:teuthology.orchestra.run.smithi039.stdout: "ms_max_accept_failures": "4", 2024-09-21T22:06:12.791 INFO:teuthology.orchestra.run.smithi039.stdout: "ms_max_backoff": "15.000000", 2024-09-21T22:06:12.791 INFO:teuthology.orchestra.run.smithi039.stdout: "ms_mon_client_mode": "secure crc", 2024-09-21T22:06:12.791 INFO:teuthology.orchestra.run.smithi039.stdout: "ms_mon_cluster_mode": "secure crc", 2024-09-21T22:06:12.791 INFO:teuthology.orchestra.run.smithi039.stdout: "ms_mon_service_mode": "secure crc", 2024-09-21T22:06:12.791 INFO:teuthology.orchestra.run.smithi039.stdout: "ms_osd_compress_min_size": "1024", 2024-09-21T22:06:12.791 INFO:teuthology.orchestra.run.smithi039.stdout: "ms_osd_compress_mode": "none", 2024-09-21T22:06:12.791 INFO:teuthology.orchestra.run.smithi039.stdout: "ms_osd_compression_algorithm": "snappy", 2024-09-21T22:06:12.791 INFO:teuthology.orchestra.run.smithi039.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-09-21T22:06:12.792 INFO:teuthology.orchestra.run.smithi039.stdout: "ms_pq_min_cost": "65536", 2024-09-21T22:06:12.792 INFO:teuthology.orchestra.run.smithi039.stdout: "ms_public_type": "", 2024-09-21T22:06:12.792 INFO:teuthology.orchestra.run.smithi039.stdout: "ms_service_mode": "crc secure", 2024-09-21T22:06:12.792 INFO:teuthology.orchestra.run.smithi039.stdout: "ms_tcp_listen_backlog": "512", 2024-09-21T22:06:12.792 INFO:teuthology.orchestra.run.smithi039.stdout: "ms_tcp_nodelay": "true", 2024-09-21T22:06:12.792 INFO:teuthology.orchestra.run.smithi039.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-09-21T22:06:12.792 INFO:teuthology.orchestra.run.smithi039.stdout: "ms_tcp_rcvbuf": "0", 2024-09-21T22:06:12.792 INFO:teuthology.orchestra.run.smithi039.stdout: "ms_type": "async+posix", 2024-09-21T22:06:12.792 INFO:teuthology.orchestra.run.smithi039.stdout: "no_config_file": "false", 2024-09-21T22:06:12.792 INFO:teuthology.orchestra.run.smithi039.stdout: "objecter_completion_locks_per_session": "32", 2024-09-21T22:06:12.792 INFO:teuthology.orchestra.run.smithi039.stdout: "objecter_debug_inject_relock_delay": "false", 2024-09-21T22:06:12.792 INFO:teuthology.orchestra.run.smithi039.stdout: "objecter_inflight_op_bytes": "104857600", 2024-09-21T22:06:12.792 INFO:teuthology.orchestra.run.smithi039.stdout: "objecter_inflight_ops": "1024", 2024-09-21T22:06:12.792 INFO:teuthology.orchestra.run.smithi039.stdout: "objecter_inject_no_watch_ping": "false", 2024-09-21T22:06:12.792 INFO:teuthology.orchestra.run.smithi039.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-09-21T22:06:12.793 INFO:teuthology.orchestra.run.smithi039.stdout: "objecter_tick_interval": "5.000000", 2024-09-21T22:06:12.793 INFO:teuthology.orchestra.run.smithi039.stdout: "objecter_timeout": "10.000000", 2024-09-21T22:06:12.793 INFO:teuthology.orchestra.run.smithi039.stdout: "objectstore_blackhole": "false", 2024-09-21T22:06:12.793 INFO:teuthology.orchestra.run.smithi039.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-09-21T22:06:12.793 INFO:teuthology.orchestra.run.smithi039.stdout: "openssl_engine_opts": "", 2024-09-21T22:06:12.793 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_agent_delay_time": "5.000000", 2024-09-21T22:06:12.793 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_agent_hist_halflife": "1000", 2024-09-21T22:06:12.793 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_agent_max_low_ops": "2", 2024-09-21T22:06:12.793 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_agent_max_ops": "4", 2024-09-21T22:06:12.793 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-09-21T22:06:12.793 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_agent_quantize_effort": "0.100000", 2024-09-21T22:06:12.793 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_agent_slop": "0.020000", 2024-09-21T22:06:12.793 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-09-21T22:06:12.793 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_allow_recovery_below_min_size": "true", 2024-09-21T22:06:12.794 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_asio_thread_count": "2", 2024-09-21T22:06:12.794 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_async_recovery_min_cost": "100", 2024-09-21T22:06:12.794 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_auto_mark_unfound_lost": "false", 2024-09-21T22:06:12.794 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_backfill_retry_interval": "30.000000", 2024-09-21T22:06:12.794 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_backfill_scan_max": "512", 2024-09-21T22:06:12.794 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_backfill_scan_min": "64", 2024-09-21T22:06:12.794 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_backoff_on_degraded": "false", 2024-09-21T22:06:12.794 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_backoff_on_peering": "false", 2024-09-21T22:06:12.794 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_backoff_on_unfound": "true", 2024-09-21T22:06:12.794 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_beacon_report_interval": "300", 2024-09-21T22:06:12.794 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_bench_duration": "30", 2024-09-21T22:06:12.794 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-09-21T22:06:12.794 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_bench_max_block_size": "67108864", 2024-09-21T22:06:12.795 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_bench_small_size_max_iops": "100", 2024-09-21T22:06:12.795 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_blkin_trace_all": "false", 2024-09-21T22:06:12.795 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_blocked_scrub_grace_period": "120", 2024-09-21T22:06:12.795 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-09-21T22:06:12.795 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-09-21T22:06:12.795 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-09-21T22:06:12.795 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_check_for_log_corruption": "false", 2024-09-21T22:06:12.795 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-09-21T22:06:12.795 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-21T22:06:12.795 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_class_dir": "/usr/lib/rados-classes", 2024-09-21T22:06:12.795 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-21T22:06:12.795 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_class_update_on_start": "true", 2024-09-21T22:06:12.795 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_client_message_cap": "256", 2024-09-21T22:06:12.795 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_client_message_size_cap": "524288000", 2024-09-21T22:06:12.795 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_client_op_priority": "63", 2024-09-21T22:06:12.795 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_client_watch_timeout": "30", 2024-09-21T22:06:12.796 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_command_max_records": "256", 2024-09-21T22:06:12.796 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_compact_on_start": "false", 2024-09-21T22:06:12.796 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-09-21T22:06:12.796 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_crush_chooseleaf_type": "0", 2024-09-21T22:06:12.796 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_crush_initial_weight": "-1.000000", 2024-09-21T22:06:12.796 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_crush_update_on_start": "true", 2024-09-21T22:06:12.796 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_crush_update_weight_set": "true", 2024-09-21T22:06:12.796 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-09-21T22:06:12.796 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-09-21T22:06:12.796 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-09-21T22:06:12.796 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_debug_drop_ping_duration": "0", 2024-09-21T22:06:12.796 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-09-21T22:06:12.796 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_debug_feed_pullee": "-1", 2024-09-21T22:06:12.796 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-09-21T22:06:12.796 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-09-21T22:06:12.796 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-09-21T22:06:12.797 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_debug_misdirected_ops": "false", 2024-09-21T22:06:12.797 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_debug_no_acting_change": "false", 2024-09-21T22:06:12.797 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_debug_no_purge_strays": "false", 2024-09-21T22:06:12.797 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_debug_op_order": "false", 2024-09-21T22:06:12.797 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_debug_pg_log_writeout": "false", 2024-09-21T22:06:12.797 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_debug_pretend_recovery_active": "false", 2024-09-21T22:06:12.797 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-09-21T22:06:12.797 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-09-21T22:06:12.797 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_debug_shutdown": "false", 2024-09-21T22:06:12.797 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-09-21T22:06:12.797 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-09-21T22:06:12.797 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_debug_trim_objects": "false", 2024-09-21T22:06:12.797 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_debug_verify_cached_snaps": "false", 2024-09-21T22:06:12.797 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_debug_verify_missing_on_start": "false", 2024-09-21T22:06:12.797 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_debug_verify_snaps": "false", 2024-09-21T22:06:12.798 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-09-21T22:06:12.798 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-09-21T22:06:12.798 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_deep_scrub_keys": "1024", 2024-09-21T22:06:12.798 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-09-21T22:06:12.798 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-09-21T22:06:12.798 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-09-21T22:06:12.798 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_deep_scrub_stride": "524288", 2024-09-21T22:06:12.798 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-09-21T22:06:12.798 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_default_data_pool_replay_window": "5", 2024-09-21T22:06:12.798 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_default_notify_timeout": "30", 2024-09-21T22:06:12.798 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_delete_sleep": "0.000000", 2024-09-21T22:06:12.798 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-09-21T22:06:12.798 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-09-21T22:06:12.798 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-09-21T22:06:12.799 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_discard_disconnected_ops": "true", 2024-09-21T22:06:12.799 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_enable_op_tracker": "true", 2024-09-21T22:06:12.799 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-09-21T22:06:12.799 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_extblkdev_plugins": "vdo", 2024-09-21T22:06:12.799 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-09-21T22:06:12.799 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-09-21T22:06:12.799 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_fast_info": "true", 2024-09-21T22:06:12.799 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_fast_shutdown": "true", 2024-09-21T22:06:12.799 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-09-21T22:06:12.799 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_fast_shutdown_timeout": "15", 2024-09-21T22:06:12.799 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-09-21T22:06:12.799 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-09-21T22:06:12.799 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-09-21T22:06:12.799 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_function_tracing": "false", 2024-09-21T22:06:12.800 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_heartbeat_grace": "20", 2024-09-21T22:06:12.800 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_heartbeat_interval": "6", 2024-09-21T22:06:12.800 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-09-21T22:06:12.800 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_heartbeat_min_peers": "10", 2024-09-21T22:06:12.800 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_heartbeat_min_size": "2000", 2024-09-21T22:06:12.800 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_heartbeat_stale": "600", 2024-09-21T22:06:12.800 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-09-21T22:06:12.800 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_hit_set_max_size": "100000", 2024-09-21T22:06:12.800 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_hit_set_min_size": "1000", 2024-09-21T22:06:12.800 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-09-21T22:06:12.800 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-09-21T22:06:12.800 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-09-21T22:06:12.800 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-09-21T22:06:12.800 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-09-21T22:06:12.800 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_journal_flush_on_shutdown": "true", 2024-09-21T22:06:12.800 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_journal_size": "5120", 2024-09-21T22:06:12.801 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_kill_backfill_at": "0", 2024-09-21T22:06:12.801 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_loop_before_reset_tphandle": "64", 2024-09-21T22:06:12.801 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_map_cache_size": "50", 2024-09-21T22:06:12.801 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_map_dedup": "true", 2024-09-21T22:06:12.801 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_map_message_max": "40", 2024-09-21T22:06:12.801 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_map_message_max_bytes": "10485760", 2024-09-21T22:06:12.801 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_map_share_max_epochs": "40", 2024-09-21T22:06:12.801 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_max_attr_name_len": "100", 2024-09-21T22:06:12.801 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_max_attr_size": "0", 2024-09-21T22:06:12.801 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_max_backfills": "1", 2024-09-21T22:06:12.801 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_max_markdown_count": "5", 2024-09-21T22:06:12.801 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_max_markdown_period": "600", 2024-09-21T22:06:12.801 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_max_object_name_len": "2048", 2024-09-21T22:06:12.801 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_max_object_namespace_len": "256", 2024-09-21T22:06:12.801 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_max_object_size": "134217728", 2024-09-21T22:06:12.801 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-09-21T22:06:12.802 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_max_omap_entries_per_request": "1024", 2024-09-21T22:06:12.802 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_max_pg_blocked_by": "16", 2024-09-21T22:06:12.802 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_max_pg_log_entries": "10000", 2024-09-21T22:06:12.802 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-09-21T22:06:12.802 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_max_pgls": "1024", 2024-09-21T22:06:12.802 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_max_push_cost": "8388608", 2024-09-21T22:06:12.802 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_max_push_objects": "10", 2024-09-21T22:06:12.802 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_max_scrubs": "3", 2024-09-21T22:06:12.802 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-09-21T22:06:12.802 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_max_trimming_pgs": "2", 2024-09-21T22:06:12.802 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_max_write_op_reply_len": "64", 2024-09-21T22:06:12.802 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_max_write_size": "90", 2024-09-21T22:06:12.802 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-09-21T22:06:12.802 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-09-21T22:06:12.803 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-09-21T22:06:12.803 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-09-21T22:06:12.803 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-09-21T22:06:12.803 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-09-21T22:06:12.803 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-09-21T22:06:12.803 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_mclock_override_recovery_settings": "false", 2024-09-21T22:06:12.803 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_mclock_profile": "balanced", 2024-09-21T22:06:12.803 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-09-21T22:06:12.803 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-09-21T22:06:12.803 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-09-21T22:06:12.803 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-09-21T22:06:12.803 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-09-21T22:06:12.803 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-09-21T22:06:12.803 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-09-21T22:06:12.803 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-09-21T22:06:12.804 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-09-21T22:06:12.804 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-09-21T22:06:12.804 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_mclock_skip_benchmark": "false", 2024-09-21T22:06:12.804 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_memory_base": "805306368", 2024-09-21T22:06:12.804 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_memory_cache_min": "134217728", 2024-09-21T22:06:12.804 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-09-21T22:06:12.804 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-09-21T22:06:12.804 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_memory_target": "4294967296", 2024-09-21T22:06:12.804 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_memory_target_autotune": "false", 2024-09-21T22:06:12.804 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-09-21T22:06:12.804 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_min_pg_log_entries": "250", 2024-09-21T22:06:12.804 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_min_recovery_priority": "0", 2024-09-21T22:06:12.804 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_mon_heartbeat_interval": "30", 2024-09-21T22:06:12.804 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-09-21T22:06:12.804 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_mon_report_interval": "5", 2024-09-21T22:06:12.805 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_mon_report_max_in_flight": "2", 2024-09-21T22:06:12.805 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-09-21T22:06:12.805 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_num_cache_shards": "32", 2024-09-21T22:06:12.805 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_num_op_tracker_shard": "32", 2024-09-21T22:06:12.805 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_numa_auto_affinity": "true", 2024-09-21T22:06:12.805 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_numa_node": "-1", 2024-09-21T22:06:12.805 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_numa_prefer_iface": "true", 2024-09-21T22:06:12.805 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-09-21T22:06:12.805 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_objecter_finishers": "1", 2024-09-21T22:06:12.805 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_objectstore": "bluestore", 2024-09-21T22:06:12.805 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_objectstore_fuse": "false", 2024-09-21T22:06:12.805 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_objectstore_tracing": "false", 2024-09-21T22:06:12.805 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_op_complaint_time": "30.000000", 2024-09-21T22:06:12.805 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_op_history_duration": "600", 2024-09-21T22:06:12.805 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_op_history_size": "20", 2024-09-21T22:06:12.805 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_op_history_slow_op_size": "20", 2024-09-21T22:06:12.806 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-09-21T22:06:12.806 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_op_log_threshold": "5", 2024-09-21T22:06:12.806 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_op_num_shards": "0", 2024-09-21T22:06:12.806 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_op_num_shards_hdd": "5", 2024-09-21T22:06:12.806 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_op_num_shards_ssd": "8", 2024-09-21T22:06:12.806 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_op_num_threads_per_shard": "0", 2024-09-21T22:06:12.806 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-09-21T22:06:12.806 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-09-21T22:06:12.806 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-09-21T22:06:12.806 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_op_pq_min_cost": "65536", 2024-09-21T22:06:12.806 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_op_queue": "mclock_scheduler", 2024-09-21T22:06:12.806 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_op_queue_cut_off": "high", 2024-09-21T22:06:12.806 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_op_thread_suicide_timeout": "150", 2024-09-21T22:06:12.806 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_op_thread_timeout": "15", 2024-09-21T22:06:12.806 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_open_classes_on_start": "true", 2024-09-21T22:06:12.807 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_os_flags": "0", 2024-09-21T22:06:12.807 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_peering_op_priority": "255", 2024-09-21T22:06:12.807 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_pg_delete_cost": "1048576", 2024-09-21T22:06:12.807 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_pg_delete_priority": "5", 2024-09-21T22:06:12.807 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-09-21T22:06:12.807 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-09-21T22:06:12.807 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_pg_log_dups_tracked": "3000", 2024-09-21T22:06:12.807 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_pg_log_trim_max": "10000", 2024-09-21T22:06:12.807 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_pg_log_trim_min": "100", 2024-09-21T22:06:12.807 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-09-21T22:06:12.807 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_pg_object_context_cache_count": "64", 2024-09-21T22:06:12.807 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-09-21T22:06:12.807 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-09-21T22:06:12.807 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-09-21T22:06:12.807 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-09-21T22:06:12.807 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-09-21T22:06:12.808 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-09-21T22:06:12.808 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-09-21T22:06:12.808 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-09-21T22:06:12.808 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_pool_default_crimson": "false", 2024-09-21T22:06:12.808 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_pool_default_crush_rule": "-1", 2024-09-21T22:06:12.808 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_pool_default_ec_fast_read": "false", 2024-09-21T22:06:12.808 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-09-21T22:06:12.808 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_pool_default_flag_bulk": "false", 2024-09-21T22:06:12.808 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-09-21T22:06:12.808 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_pool_default_flag_nodelete": "false", 2024-09-21T22:06:12.808 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-09-21T22:06:12.808 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-09-21T22:06:12.808 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_pool_default_flags": "0", 2024-09-21T22:06:12.808 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-09-21T22:06:12.809 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_pool_default_min_size": "0", 2024-09-21T22:06:12.809 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-09-21T22:06:12.809 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_pool_default_pg_num": "32", 2024-09-21T22:06:12.809 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_pool_default_pgp_num": "0", 2024-09-21T22:06:12.809 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-09-21T22:06:12.809 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_pool_default_read_ratio": "70", 2024-09-21T22:06:12.809 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_pool_default_size": "2", 2024-09-21T22:06:12.809 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_pool_default_type": "replicated", 2024-09-21T22:06:12.809 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-09-21T22:06:12.809 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_pool_use_gmt_hitset": "true", 2024-09-21T22:06:12.809 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_push_per_object_cost": "1000", 2024-09-21T22:06:12.809 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_read_ec_check_for_errors": "false", 2024-09-21T22:06:12.809 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_recover_clone_overlap": "true", 2024-09-21T22:06:12.809 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_recover_clone_overlap_limit": "10", 2024-09-21T22:06:12.809 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_recovery_cost": "20971520", 2024-09-21T22:06:12.809 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_recovery_delay_start": "0.000000", 2024-09-21T22:06:12.810 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_recovery_max_active": "0", 2024-09-21T22:06:12.810 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_recovery_max_active_hdd": "3", 2024-09-21T22:06:12.810 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_recovery_max_active_ssd": "10", 2024-09-21T22:06:12.810 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_recovery_max_chunk": "8388608", 2024-09-21T22:06:12.810 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-09-21T22:06:12.810 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_recovery_max_single_start": "1", 2024-09-21T22:06:12.810 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_recovery_op_priority": "3", 2024-09-21T22:06:12.810 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_recovery_op_warn_multiple": "16", 2024-09-21T22:06:12.810 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_recovery_priority": "5", 2024-09-21T22:06:12.810 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_recovery_retry_interval": "30.000000", 2024-09-21T22:06:12.810 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_recovery_sleep": "0.000000", 2024-09-21T22:06:12.810 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-09-21T22:06:12.810 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-09-21T22:06:12.810 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-09-21T22:06:12.811 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_repair_during_recovery": "false", 2024-09-21T22:06:12.811 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_requested_scrub_priority": "120", 2024-09-21T22:06:12.811 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-09-21T22:06:12.811 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_rollback_to_cluster_snap": "", 2024-09-21T22:06:12.811 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_scrub_auto_repair": "false", 2024-09-21T22:06:12.811 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-09-21T22:06:12.811 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-09-21T22:06:12.811 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_scrub_begin_hour": "0", 2024-09-21T22:06:12.811 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_scrub_begin_week_day": "0", 2024-09-21T22:06:12.811 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_scrub_chunk_max": "25", 2024-09-21T22:06:12.811 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_scrub_chunk_min": "5", 2024-09-21T22:06:12.811 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_scrub_cost": "52428800", 2024-09-21T22:06:12.811 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-09-21T22:06:12.811 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_scrub_during_recovery": "false", 2024-09-21T22:06:12.811 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_scrub_end_hour": "0", 2024-09-21T22:06:12.811 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_scrub_end_week_day": "0", 2024-09-21T22:06:12.812 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_scrub_event_cost": "4096", 2024-09-21T22:06:12.812 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-09-21T22:06:12.812 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-09-21T22:06:12.812 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_scrub_invalid_stats": "true", 2024-09-21T22:06:12.812 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_scrub_load_threshold": "0.500000", 2024-09-21T22:06:12.812 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_scrub_max_interval": "604800.000000", 2024-09-21T22:06:12.812 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_scrub_max_preemptions": "5", 2024-09-21T22:06:12.812 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_scrub_min_interval": "86400.000000", 2024-09-21T22:06:12.812 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_scrub_priority": "5", 2024-09-21T22:06:12.812 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_scrub_sleep": "0.000000", 2024-09-21T22:06:12.812 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-09-21T22:06:12.812 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-09-21T22:06:12.812 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_shutdown_pgref_assert": "false", 2024-09-21T22:06:12.812 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_skip_data_digest": "false", 2024-09-21T22:06:12.812 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_smart_report_timeout": "5", 2024-09-21T22:06:12.812 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_snap_trim_cost": "1048576", 2024-09-21T22:06:12.813 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_snap_trim_priority": "5", 2024-09-21T22:06:12.813 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_snap_trim_sleep": "0.000000", 2024-09-21T22:06:12.813 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-09-21T22:06:12.813 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-09-21T22:06:12.813 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-09-21T22:06:12.813 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-09-21T22:06:12.813 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_stats_update_period_scrubbing": "15", 2024-09-21T22:06:12.813 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-09-21T22:06:12.813 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_target_transaction_size": "30", 2024-09-21T22:06:12.813 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-09-21T22:06:12.813 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-09-21T22:06:12.813 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-09-21T22:06:12.813 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-09-21T22:06:12.813 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-09-21T22:06:12.813 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-09-21T22:06:12.814 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-09-21T22:06:12.814 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_tier_default_cache_mode": "writeback", 2024-09-21T22:06:12.814 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-09-21T22:06:12.814 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-09-21T22:06:12.814 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_tracing": "false", 2024-09-21T22:06:12.814 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_use_stale_snap": "false", 2024-09-21T22:06:12.814 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-09-21T22:06:12.814 INFO:teuthology.orchestra.run.smithi039.stdout: "osdc_blkin_trace_all": "false", 2024-09-21T22:06:12.814 INFO:teuthology.orchestra.run.smithi039.stdout: "paxos_kill_at": "0", 2024-09-21T22:06:12.814 INFO:teuthology.orchestra.run.smithi039.stdout: "paxos_max_join_drift": "10", 2024-09-21T22:06:12.814 INFO:teuthology.orchestra.run.smithi039.stdout: "paxos_min": "500", 2024-09-21T22:06:12.814 INFO:teuthology.orchestra.run.smithi039.stdout: "paxos_min_wait": "0.050000", 2024-09-21T22:06:12.814 INFO:teuthology.orchestra.run.smithi039.stdout: "paxos_propose_interval": "1.000000", 2024-09-21T22:06:12.814 INFO:teuthology.orchestra.run.smithi039.stdout: "paxos_service_trim_max": "500", 2024-09-21T22:06:12.814 INFO:teuthology.orchestra.run.smithi039.stdout: "paxos_service_trim_max_multiplier": "20", 2024-09-21T22:06:12.815 INFO:teuthology.orchestra.run.smithi039.stdout: "paxos_service_trim_min": "250", 2024-09-21T22:06:12.815 INFO:teuthology.orchestra.run.smithi039.stdout: "paxos_stash_full_interval": "25", 2024-09-21T22:06:12.815 INFO:teuthology.orchestra.run.smithi039.stdout: "paxos_trim_max": "500", 2024-09-21T22:06:12.815 INFO:teuthology.orchestra.run.smithi039.stdout: "paxos_trim_min": "250", 2024-09-21T22:06:12.815 INFO:teuthology.orchestra.run.smithi039.stdout: "perf": "true", 2024-09-21T22:06:12.815 INFO:teuthology.orchestra.run.smithi039.stdout: "pid_file": "/var/run/ceph/ceph-mon.a.pid", 2024-09-21T22:06:12.815 INFO:teuthology.orchestra.run.smithi039.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-09-21T22:06:12.815 INFO:teuthology.orchestra.run.smithi039.stdout: "plugin_dir": "/usr/lib/ceph", 2024-09-21T22:06:12.815 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "-", 2024-09-21T22:06:12.815 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrv": "", 2024-09-21T22:06:12.815 INFO:teuthology.orchestra.run.smithi039.stdout: "public_bind_addr": "-", 2024-09-21T22:06:12.815 INFO:teuthology.orchestra.run.smithi039.stdout: "public_network": "", 2024-09-21T22:06:12.815 INFO:teuthology.orchestra.run.smithi039.stdout: "public_network_interface": "", 2024-09-21T22:06:12.815 INFO:teuthology.orchestra.run.smithi039.stdout: "qat_compressor_busy_polling": "false", 2024-09-21T22:06:12.816 INFO:teuthology.orchestra.run.smithi039.stdout: "qat_compressor_enabled": "false", 2024-09-21T22:06:12.816 INFO:teuthology.orchestra.run.smithi039.stdout: "qat_compressor_session_max_number": "256", 2024-09-21T22:06:12.816 INFO:teuthology.orchestra.run.smithi039.stdout: "rados_mon_op_timeout": "0", 2024-09-21T22:06:12.816 INFO:teuthology.orchestra.run.smithi039.stdout: "rados_osd_op_timeout": "0", 2024-09-21T22:06:12.816 INFO:teuthology.orchestra.run.smithi039.stdout: "rados_tracing": "false", 2024-09-21T22:06:12.816 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_atime_update_interval": "60", 2024-09-21T22:06:12.816 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-09-21T22:06:12.816 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_balance_parent_reads": "false", 2024-09-21T22:06:12.816 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_balance_snap_reads": "false", 2024-09-21T22:06:12.816 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_blkin_trace_all": "false", 2024-09-21T22:06:12.816 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_blocklist_expire_seconds": "0", 2024-09-21T22:06:12.816 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_blocklist_on_break_lock": "true", 2024-09-21T22:06:12.816 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_cache": "true", 2024-09-21T22:06:12.816 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_cache_block_writes_upfront": "false", 2024-09-21T22:06:12.816 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_cache_max_dirty": "25165824", 2024-09-21T22:06:12.817 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-09-21T22:06:12.817 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_cache_max_dirty_object": "0", 2024-09-21T22:06:12.817 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_cache_policy": "writearound", 2024-09-21T22:06:12.817 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_cache_size": "33554432", 2024-09-21T22:06:12.817 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_cache_target_dirty": "16777216", 2024-09-21T22:06:12.817 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-09-21T22:06:12.817 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_clone_copy_on_read": "false", 2024-09-21T22:06:12.817 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_compression_hint": "none", 2024-09-21T22:06:12.817 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_concurrent_management_ops": "10", 2024-09-21T22:06:12.817 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-09-21T22:06:12.817 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_default_clone_format": "auto", 2024-09-21T22:06:12.817 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_default_data_pool": "", 2024-09-21T22:06:12.817 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_default_features": "61", 2024-09-21T22:06:12.817 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_default_format": "2", 2024-09-21T22:06:12.817 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_default_map_options": "", 2024-09-21T22:06:12.817 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_default_order": "22", 2024-09-21T22:06:12.818 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_default_pool": "rbd", 2024-09-21T22:06:12.818 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-09-21T22:06:12.818 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_default_stripe_count": "0", 2024-09-21T22:06:12.818 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_default_stripe_unit": "0", 2024-09-21T22:06:12.818 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_disable_zero_copy_writes": "true", 2024-09-21T22:06:12.818 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_discard_granularity_bytes": "65536", 2024-09-21T22:06:12.818 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-09-21T22:06:12.818 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_enable_alloc_hint": "true", 2024-09-21T22:06:12.818 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-09-21T22:06:12.818 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_io_scheduler": "simple", 2024-09-21T22:06:12.818 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-09-21T22:06:12.818 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_journal_commit_age": "5.000000", 2024-09-21T22:06:12.818 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-09-21T22:06:12.818 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-09-21T22:06:12.818 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-09-21T22:06:12.819 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-09-21T22:06:12.819 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_journal_object_flush_interval": "0", 2024-09-21T22:06:12.819 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-09-21T22:06:12.819 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-09-21T22:06:12.819 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_journal_order": "24", 2024-09-21T22:06:12.819 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_journal_pool": "", 2024-09-21T22:06:12.819 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_journal_splay_width": "4", 2024-09-21T22:06:12.819 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_localize_parent_reads": "false", 2024-09-21T22:06:12.819 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_localize_snap_reads": "false", 2024-09-21T22:06:12.819 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-09-21T22:06:12.819 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-09-21T22:06:12.819 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-09-21T22:06:12.819 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-09-21T22:06:12.819 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-09-21T22:06:12.819 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-09-21T22:06:12.819 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_mirror_image_policy_type": "simple", 2024-09-21T22:06:12.819 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-09-21T22:06:12.820 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-09-21T22:06:12.820 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-09-21T22:06:12.820 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-09-21T22:06:12.820 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-09-21T22:06:12.820 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-09-21T22:06:12.820 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-09-21T22:06:12.820 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_mirror_memory_autotune": "true", 2024-09-21T22:06:12.820 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_mirror_memory_base": "805306368", 2024-09-21T22:06:12.820 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-09-21T22:06:12.820 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-09-21T22:06:12.820 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-09-21T22:06:12.820 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-09-21T22:06:12.820 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_mirror_memory_target": "4294967296", 2024-09-21T22:06:12.820 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-09-21T22:06:12.820 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-09-21T22:06:12.820 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-09-21T22:06:12.821 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_mirroring_delete_delay": "0", 2024-09-21T22:06:12.821 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-09-21T22:06:12.821 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_mirroring_replay_delay": "0", 2024-09-21T22:06:12.821 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-09-21T22:06:12.821 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-09-21T22:06:12.821 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_move_to_trash_on_remove": "false", 2024-09-21T22:06:12.821 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-09-21T22:06:12.821 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_mtime_update_interval": "60", 2024-09-21T22:06:12.821 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_non_blocking_aio": "true", 2024-09-21T22:06:12.821 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_op_thread_timeout": "60", 2024-09-21T22:06:12.821 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_op_threads": "1", 2024-09-21T22:06:12.821 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_parent_cache_enabled": "false", 2024-09-21T22:06:12.821 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_persistent_cache_mode": "disabled", 2024-09-21T22:06:12.821 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_persistent_cache_path": "/tmp", 2024-09-21T22:06:12.821 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_persistent_cache_size": "1073741824", 2024-09-21T22:06:12.821 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_plugins": "", 2024-09-21T22:06:12.822 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_qos_bps_burst": "0", 2024-09-21T22:06:12.822 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-09-21T22:06:12.822 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_qos_bps_limit": "0", 2024-09-21T22:06:12.822 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_qos_exclude_ops": "0", 2024-09-21T22:06:12.822 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_qos_iops_burst": "0", 2024-09-21T22:06:12.822 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-09-21T22:06:12.822 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_qos_iops_limit": "0", 2024-09-21T22:06:12.822 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_qos_read_bps_burst": "0", 2024-09-21T22:06:12.822 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-09-21T22:06:12.822 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_qos_read_bps_limit": "0", 2024-09-21T22:06:12.822 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_qos_read_iops_burst": "0", 2024-09-21T22:06:12.822 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-09-21T22:06:12.822 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_qos_read_iops_limit": "0", 2024-09-21T22:06:12.822 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_qos_schedule_tick_min": "50", 2024-09-21T22:06:12.822 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_qos_write_bps_burst": "0", 2024-09-21T22:06:12.823 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-09-21T22:06:12.823 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_qos_write_bps_limit": "0", 2024-09-21T22:06:12.823 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_qos_write_iops_burst": "0", 2024-09-21T22:06:12.823 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-09-21T22:06:12.823 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_qos_write_iops_limit": "0", 2024-09-21T22:06:12.823 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_quiesce_notification_attempts": "10", 2024-09-21T22:06:12.823 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_read_from_replica_policy": "default", 2024-09-21T22:06:12.823 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-09-21T22:06:12.823 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_readahead_max_bytes": "524288", 2024-09-21T22:06:12.823 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_readahead_trigger_requests": "10", 2024-09-21T22:06:12.823 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_request_timed_out_seconds": "30", 2024-09-21T22:06:12.823 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_skip_partial_discard": "true", 2024-09-21T22:06:12.823 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-09-21T22:06:12.823 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_tracing": "false", 2024-09-21T22:06:12.823 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_validate_names": "true", 2024-09-21T22:06:12.823 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_validate_pool": "true", 2024-09-21T22:06:12.824 INFO:teuthology.orchestra.run.smithi039.stdout: "restapi_base_url": "", 2024-09-21T22:06:12.824 INFO:teuthology.orchestra.run.smithi039.stdout: "restapi_log_level": "", 2024-09-21T22:06:12.824 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_account_default_quota_max_objects": "-1", 2024-09-21T22:06:12.824 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_account_default_quota_max_size": "-1", 2024-09-21T22:06:12.824 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_acl_grants_max_num": "100", 2024-09-21T22:06:12.824 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_admin_entry": "admin", 2024-09-21T22:06:12.824 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-09-21T22:06:12.824 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_backend_store": "rados", 2024-09-21T22:06:12.824 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_barbican_url": "", 2024-09-21T22:06:12.824 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_beast_enable_async": "true", 2024-09-21T22:06:12.824 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_bucket_counters_cache": "false", 2024-09-21T22:06:12.824 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-09-21T22:06:12.824 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-09-21T22:06:12.824 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-09-21T22:06:12.824 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_bucket_index_max_aio": "128", 2024-09-21T22:06:12.824 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-09-21T22:06:12.825 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-09-21T22:06:12.825 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_bucket_quota_ttl": "600", 2024-09-21T22:06:12.825 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-09-21T22:06:12.825 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_cache_enabled": "true", 2024-09-21T22:06:12.825 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_cache_expiry_interval": "900", 2024-09-21T22:06:12.825 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_cache_lru_size": "25000", 2024-09-21T22:06:12.825 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_config_store": "rados", 2024-09-21T22:06:12.825 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_content_length_compat": "false", 2024-09-21T22:06:12.825 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_copy_obj_progress": "true", 2024-09-21T22:06:12.825 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-09-21T22:06:12.825 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_cors_rules_max_num": "100", 2024-09-21T22:06:12.825 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_cross_domain_policy": "", 2024-09-21T22:06:12.825 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_crypt_default_encryption_key": "", 2024-09-21T22:06:12.825 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_crypt_kmip_addr": "", 2024-09-21T22:06:12.825 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_crypt_kmip_ca_path": "", 2024-09-21T22:06:12.825 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_crypt_kmip_client_cert": "", 2024-09-21T22:06:12.826 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_crypt_kmip_client_key": "", 2024-09-21T22:06:12.826 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-09-21T22:06:12.826 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_crypt_kmip_password": "", 2024-09-21T22:06:12.826 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-09-21T22:06:12.826 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_crypt_kmip_username": "", 2024-09-21T22:06:12.826 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_crypt_require_ssl": "true", 2024-09-21T22:06:12.826 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-09-21T22:06:12.826 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-09-21T22:06:12.826 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-09-21T22:06:12.826 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-09-21T22:06:12.826 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-09-21T22:06:12.826 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-09-21T22:06:12.826 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-09-21T22:06:12.827 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-09-21T22:06:12.827 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-09-21T22:06:12.827 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-09-21T22:06:12.827 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-09-21T22:06:12.827 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-09-21T22:06:12.827 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-09-21T22:06:12.827 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-09-21T22:06:12.827 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_crypt_suppress_logs": "true", 2024-09-21T22:06:12.827 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_crypt_vault_addr": "", 2024-09-21T22:06:12.827 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_crypt_vault_auth": "token", 2024-09-21T22:06:12.827 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_crypt_vault_namespace": "", 2024-09-21T22:06:12.827 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_crypt_vault_prefix": "", 2024-09-21T22:06:12.827 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-09-21T22:06:12.827 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-09-21T22:06:12.827 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-09-21T22:06:12.828 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-09-21T22:06:12.828 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_crypt_vault_token_file": "", 2024-09-21T22:06:12.828 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-09-21T22:06:12.828 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_curl_buffersize": "524288", 2024-09-21T22:06:12.828 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_curl_low_speed_limit": "1024", 2024-09-21T22:06:12.828 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_curl_low_speed_time": "300", 2024-09-21T22:06:12.828 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_curl_tcp_keepalive": "0", 2024-09-21T22:06:12.828 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-09-21T22:06:12.828 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-09-21T22:06:12.828 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-09-21T22:06:12.828 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-09-21T22:06:12.828 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-09-21T22:06:12.828 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_d3n_l1_fadvise": "4", 2024-09-21T22:06:12.828 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-09-21T22:06:12.828 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-09-21T22:06:12.828 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-09-21T22:06:12.829 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_d4n_host": "127.0.0.1", 2024-09-21T22:06:12.829 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_d4n_port": "6379", 2024-09-21T22:06:12.829 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-09-21T22:06:12.829 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_data_log_changes_size": "1000", 2024-09-21T22:06:12.829 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_data_log_num_shards": "128", 2024-09-21T22:06:12.829 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-09-21T22:06:12.829 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_data_log_window": "30", 2024-09-21T22:06:12.829 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_data_notify_interval_msec": "0", 2024-09-21T22:06:12.829 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_data_sync_poll_interval": "20", 2024-09-21T22:06:12.829 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_data_sync_spawn_window": "20", 2024-09-21T22:06:12.829 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-09-21T22:06:12.829 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-09-21T22:06:12.829 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-09-21T22:06:12.829 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_default_data_log_backing": "fifo", 2024-09-21T22:06:12.829 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-09-21T22:06:12.829 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_default_region_info_oid": "default.region", 2024-09-21T22:06:12.830 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-09-21T22:06:12.830 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-09-21T22:06:12.830 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_defer_to_bucket_acls": "", 2024-09-21T22:06:12.830 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-09-21T22:06:12.830 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-09-21T22:06:12.830 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-09-21T22:06:12.830 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-09-21T22:06:12.830 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-09-21T22:06:12.830 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-09-21T22:06:12.830 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-09-21T22:06:12.830 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-09-21T22:06:12.830 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_dmclock_data_res": "500.000000", 2024-09-21T22:06:12.830 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-09-21T22:06:12.830 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-09-21T22:06:12.830 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-09-21T22:06:12.830 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-09-21T22:06:12.831 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_dns_name": "", 2024-09-21T22:06:12.831 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_dns_s3website_name": "", 2024-09-21T22:06:12.831 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_dynamic_resharding": "true", 2024-09-21T22:06:12.831 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-09-21T22:06:12.831 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_enable_gc_threads": "true", 2024-09-21T22:06:12.831 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_enable_lc_threads": "true", 2024-09-21T22:06:12.831 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_enable_ops_log": "false", 2024-09-21T22:06:12.831 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_enable_quota_threads": "true", 2024-09-21T22:06:12.831 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_enable_static_website": "false", 2024-09-21T22:06:12.831 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_enable_usage_log": "false", 2024-09-21T22:06:12.831 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_enforce_swift_acls": "true", 2024-09-21T22:06:12.831 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_exit_timeout_secs": "120", 2024-09-21T22:06:12.831 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_expose_bucket": "false", 2024-09-21T22:06:12.831 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_extended_http_attrs": "", 2024-09-21T22:06:12.831 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_filter": "none", 2024-09-21T22:06:12.832 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-09-21T22:06:12.832 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_frontends": "beast port=7480", 2024-09-21T22:06:12.832 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_gc_max_concurrent_io": "10", 2024-09-21T22:06:12.832 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_gc_max_deferred": "50", 2024-09-21T22:06:12.832 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-09-21T22:06:12.832 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_gc_max_objs": "32", 2024-09-21T22:06:12.832 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_gc_max_queue_size": "134213632", 2024-09-21T22:06:12.832 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_gc_max_trim_chunk": "16", 2024-09-21T22:06:12.832 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_gc_obj_min_wait": "7200", 2024-09-21T22:06:12.832 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_gc_processor_max_time": "3600", 2024-09-21T22:06:12.832 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_gc_processor_period": "3600", 2024-09-21T22:06:12.832 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-09-21T22:06:12.832 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_get_obj_window_size": "16777216", 2024-09-21T22:06:12.832 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_healthcheck_disabling_path": "", 2024-09-21T22:06:12.832 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_ignore_get_invalid_range": "false", 2024-09-21T22:06:12.833 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_init_timeout": "300", 2024-09-21T22:06:12.833 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-09-21T22:06:12.833 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-09-21T22:06:12.833 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_kafka_connection_idle": "300", 2024-09-21T22:06:12.833 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_kafka_message_timeout": "5000", 2024-09-21T22:06:12.833 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_kafka_sleep_timeout": "10", 2024-09-21T22:06:12.833 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-09-21T22:06:12.833 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-09-21T22:06:12.833 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-09-21T22:06:12.833 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_keystone_admin_domain": "", 2024-09-21T22:06:12.833 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_keystone_admin_password": "", 2024-09-21T22:06:12.833 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_keystone_admin_password_path": "", 2024-09-21T22:06:12.833 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_keystone_admin_project": "", 2024-09-21T22:06:12.833 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_keystone_admin_tenant": "", 2024-09-21T22:06:12.834 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_keystone_admin_token": "", 2024-09-21T22:06:12.834 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_keystone_admin_token_path": "", 2024-09-21T22:06:12.834 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_keystone_admin_user": "", 2024-09-21T22:06:12.834 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_keystone_api_version": "2", 2024-09-21T22:06:12.834 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_keystone_barbican_domain": "", 2024-09-21T22:06:12.834 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_keystone_barbican_password": "", 2024-09-21T22:06:12.834 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_keystone_barbican_project": "", 2024-09-21T22:06:12.834 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_keystone_barbican_tenant": "", 2024-09-21T22:06:12.834 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_keystone_barbican_user": "", 2024-09-21T22:06:12.834 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-09-21T22:06:12.834 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_keystone_implicit_tenants": "false", 2024-09-21T22:06:12.834 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-09-21T22:06:12.834 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_keystone_service_token_enabled": "false", 2024-09-21T22:06:12.834 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_keystone_token_cache_size": "10000", 2024-09-21T22:06:12.834 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_keystone_url": "", 2024-09-21T22:06:12.835 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_keystone_verify_ssl": "true", 2024-09-21T22:06:12.835 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_lc_debug_interval": "-1", 2024-09-21T22:06:12.835 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_lc_lock_max_time": "90", 2024-09-21T22:06:12.835 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_lc_max_objs": "32", 2024-09-21T22:06:12.835 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_lc_max_rules": "1000", 2024-09-21T22:06:12.835 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_lc_max_worker": "3", 2024-09-21T22:06:12.835 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_lc_max_wp_worker": "3", 2024-09-21T22:06:12.835 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_lc_thread_delay": "0", 2024-09-21T22:06:12.835 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-09-21T22:06:12.835 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_ldap_dnattr": "uid", 2024-09-21T22:06:12.835 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-09-21T22:06:12.835 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_ldap_searchfilter": "", 2024-09-21T22:06:12.835 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-09-21T22:06:12.835 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_ldap_uri": "", 2024-09-21T22:06:12.835 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-09-21T22:06:12.835 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-09-21T22:06:12.836 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-09-21T22:06:12.836 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_log_http_headers": "", 2024-09-21T22:06:12.836 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_log_nonexistent_bucket": "false", 2024-09-21T22:06:12.836 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-09-21T22:06:12.836 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_log_object_name_utc": "false", 2024-09-21T22:06:12.836 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-09-21T22:06:12.836 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-09-21T22:06:12.836 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_max_attr_name_len": "0", 2024-09-21T22:06:12.836 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_max_attr_size": "0", 2024-09-21T22:06:12.836 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_max_attrs_num_in_req": "0", 2024-09-21T22:06:12.836 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_max_chunk_size": "4194304", 2024-09-21T22:06:12.836 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_max_concurrent_requests": "1024", 2024-09-21T22:06:12.836 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-09-21T22:06:12.836 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_max_dynamic_shards": "1999", 2024-09-21T22:06:12.836 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_max_listing_results": "1000", 2024-09-21T22:06:12.836 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_max_notify_retries": "10", 2024-09-21T22:06:12.837 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_max_objs_per_shard": "100000", 2024-09-21T22:06:12.837 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_max_put_param_size": "1048576", 2024-09-21T22:06:12.837 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_max_put_size": "5368709120", 2024-09-21T22:06:12.837 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_max_slo_entries": "1000", 2024-09-21T22:06:12.837 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_md_log_max_shards": "64", 2024-09-21T22:06:12.837 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_md_notify_interval_msec": "200", 2024-09-21T22:06:12.837 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_meta_sync_poll_interval": "20", 2024-09-21T22:06:12.837 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_meta_sync_spawn_window": "20", 2024-09-21T22:06:12.837 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-09-21T22:06:12.837 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_mp_lock_max_time": "600", 2024-09-21T22:06:12.837 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-09-21T22:06:12.837 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_multipart_min_part_size": "5242880", 2024-09-21T22:06:12.837 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-09-21T22:06:12.837 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-09-21T22:06:12.837 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_nfs_fhcache_size": "2017", 2024-09-21T22:06:12.838 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-09-21T22:06:12.838 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-09-21T22:06:12.838 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_nfs_lru_lanes": "5", 2024-09-21T22:06:12.838 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_nfs_max_gc": "300", 2024-09-21T22:06:12.838 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-09-21T22:06:12.838 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_nfs_run_gc_threads": "false", 2024-09-21T22:06:12.838 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_nfs_run_lc_threads": "false", 2024-09-21T22:06:12.838 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_nfs_run_quota_threads": "false", 2024-09-21T22:06:12.838 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_nfs_run_sync_thread": "false", 2024-09-21T22:06:12.838 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-09-21T22:06:12.838 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-09-21T22:06:12.838 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_num_async_rados_threads": "32", 2024-09-21T22:06:12.838 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_num_control_oids": "8", 2024-09-21T22:06:12.838 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_numa_node": "-1", 2024-09-21T22:06:12.838 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_obj_stripe_size": "4194304", 2024-09-21T22:06:12.839 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-09-21T22:06:12.839 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_objexp_chunk_size": "100", 2024-09-21T22:06:12.839 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_objexp_gc_interval": "600", 2024-09-21T22:06:12.839 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_objexp_hints_num_shards": "127", 2024-09-21T22:06:12.839 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-09-21T22:06:12.839 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-09-21T22:06:12.839 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_op_thread_timeout": "600", 2024-09-21T22:06:12.839 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_op_tracing": "false", 2024-09-21T22:06:12.839 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_opa_token": "", 2024-09-21T22:06:12.839 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_opa_url": "", 2024-09-21T22:06:12.839 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_opa_verify_ssl": "true", 2024-09-21T22:06:12.839 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-09-21T22:06:12.839 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-09-21T22:06:12.839 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_ops_log_rados": "false", 2024-09-21T22:06:12.839 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_ops_log_socket_path": "", 2024-09-21T22:06:12.840 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-09-21T22:06:12.840 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-09-21T22:06:12.840 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-09-21T22:06:12.840 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_period_push_interval": "2.000000", 2024-09-21T22:06:12.840 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_period_push_interval_max": "30.000000", 2024-09-21T22:06:12.840 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_period_root_pool": ".rgw.root", 2024-09-21T22:06:12.840 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-09-21T22:06:12.840 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-09-21T22:06:12.840 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_posix_cache_lanes": "3", 2024-09-21T22:06:12.840 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-09-21T22:06:12.840 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_posix_cache_max_buckets": "100", 2024-09-21T22:06:12.840 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_posix_cache_partitions": "3", 2024-09-21T22:06:12.840 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-09-21T22:06:12.840 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_print_continue": "true", 2024-09-21T22:06:12.840 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_print_prohibited_content_length": "false", 2024-09-21T22:06:12.841 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-09-21T22:06:12.841 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-09-21T22:06:12.841 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-09-21T22:06:12.841 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-09-21T22:06:12.841 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_rados_tracing": "false", 2024-09-21T22:06:12.841 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_realm": "", 2024-09-21T22:06:12.841 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_realm_id": "", 2024-09-21T22:06:12.841 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-09-21T22:06:12.841 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_region": "", 2024-09-21T22:06:12.841 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_region_root_pool": ".rgw.root", 2024-09-21T22:06:12.841 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_relaxed_region_enforcement": "false", 2024-09-21T22:06:12.841 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-09-21T22:06:12.841 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-09-21T22:06:12.841 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_request_uri": "", 2024-09-21T22:06:12.841 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_reshard_batch_size": "64", 2024-09-21T22:06:12.841 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-09-21T22:06:12.842 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_reshard_max_aio": "128", 2024-09-21T22:06:12.842 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_reshard_num_logs": "16", 2024-09-21T22:06:12.842 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_reshard_thread_interval": "600", 2024-09-21T22:06:12.842 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_resolve_cname": "false", 2024-09-21T22:06:12.842 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_rest_getusage_op_compat": "false", 2024-09-21T22:06:12.842 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_run_sync_thread": "true", 2024-09-21T22:06:12.842 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_s3_auth_disable_signature_url": "false", 2024-09-21T22:06:12.842 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-09-21T22:06:12.842 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_s3_auth_use_keystone": "false", 2024-09-21T22:06:12.842 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_s3_auth_use_ldap": "false", 2024-09-21T22:06:12.842 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_s3_auth_use_rados": "true", 2024-09-21T22:06:12.842 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_s3_auth_use_sts": "false", 2024-09-21T22:06:12.842 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-09-21T22:06:12.842 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_s3_success_create_obj_status": "0", 2024-09-21T22:06:12.842 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-09-21T22:06:12.842 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_scheduler_type": "throttler", 2024-09-21T22:06:12.843 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_script_uri": "", 2024-09-21T22:06:12.843 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_service_provider_name": "", 2024-09-21T22:06:12.843 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-09-21T22:06:12.843 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_sts_client_id": "", 2024-09-21T22:06:12.843 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_sts_client_secret": "", 2024-09-21T22:06:12.843 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_sts_entry": "sts", 2024-09-21T22:06:12.843 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_sts_key": "sts", 2024-09-21T22:06:12.843 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_sts_max_session_duration": "43200", 2024-09-21T22:06:12.843 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_sts_min_session_duration": "900", 2024-09-21T22:06:12.843 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_sts_token_introspection_url": "", 2024-09-21T22:06:12.843 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_swift_account_in_url": "false", 2024-09-21T22:06:12.843 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_swift_auth_entry": "auth", 2024-09-21T22:06:12.843 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_swift_auth_url": "", 2024-09-21T22:06:12.843 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_swift_custom_header": "", 2024-09-21T22:06:12.843 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_swift_enforce_content_length": "false", 2024-09-21T22:06:12.843 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_swift_need_stats": "true", 2024-09-21T22:06:12.844 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_swift_tenant_name": "", 2024-09-21T22:06:12.844 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_swift_token_expiration": "86400", 2024-09-21T22:06:12.844 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_swift_url": "", 2024-09-21T22:06:12.844 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_swift_url_prefix": "swift", 2024-09-21T22:06:12.844 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_swift_versioning_enabled": "false", 2024-09-21T22:06:12.844 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-09-21T22:06:12.844 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-09-21T22:06:12.844 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_sync_lease_period": "120", 2024-09-21T22:06:12.844 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-09-21T22:06:12.844 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_sync_log_trim_interval": "1200", 2024-09-21T22:06:12.844 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-09-21T22:06:12.844 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-09-21T22:06:12.844 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-09-21T22:06:12.844 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_sync_obj_etag_verify": "false", 2024-09-21T22:06:12.844 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_sync_trace_history_size": "4096", 2024-09-21T22:06:12.844 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-09-21T22:06:12.845 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-09-21T22:06:12.845 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_thread_pool_size": "512", 2024-09-21T22:06:12.845 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_topic_persistency_max_retries": "0", 2024-09-21T22:06:12.845 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-09-21T22:06:12.845 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-09-21T22:06:12.845 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_topic_require_publish_policy": "false", 2024-09-21T22:06:12.845 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_torrent_comment": "", 2024-09-21T22:06:12.845 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_torrent_createby": "", 2024-09-21T22:06:12.845 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_torrent_encoding": "", 2024-09-21T22:06:12.845 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_torrent_flag": "false", 2024-09-21T22:06:12.845 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_torrent_max_size": "5368709120", 2024-09-21T22:06:12.845 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_torrent_origin": "", 2024-09-21T22:06:12.845 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_torrent_sha_unit": "524288", 2024-09-21T22:06:12.845 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_torrent_tracker": "", 2024-09-21T22:06:12.845 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_trust_forwarded_https": "false", 2024-09-21T22:06:12.845 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-09-21T22:06:12.846 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_usage_log_tick_interval": "30", 2024-09-21T22:06:12.846 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_usage_max_shards": "32", 2024-09-21T22:06:12.846 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_usage_max_user_shards": "1", 2024-09-21T22:06:12.846 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_use_opa_authz": "false", 2024-09-21T22:06:12.846 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_user_counters_cache": "false", 2024-09-21T22:06:12.846 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_user_counters_cache_size": "10000", 2024-09-21T22:06:12.846 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-09-21T22:06:12.846 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_user_default_quota_max_size": "-1", 2024-09-21T22:06:12.846 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_user_max_buckets": "1000", 2024-09-21T22:06:12.846 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_user_policies_max_num": "100", 2024-09-21T22:06:12.846 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-09-21T22:06:12.846 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-09-21T22:06:12.846 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_user_quota_sync_interval": "86400", 2024-09-21T22:06:12.846 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-09-21T22:06:12.847 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_user_unique_email": "true", 2024-09-21T22:06:12.847 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_verify_ssl": "true", 2024-09-21T22:06:12.847 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_website_routing_rules_max_num": "50", 2024-09-21T22:06:12.847 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_zone": "", 2024-09-21T22:06:12.847 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_zone_id": "", 2024-09-21T22:06:12.847 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-09-21T22:06:12.847 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_zonegroup": "", 2024-09-21T22:06:12.847 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_zonegroup_id": "", 2024-09-21T22:06:12.847 INFO:teuthology.orchestra.run.smithi039.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-09-21T22:06:12.847 INFO:teuthology.orchestra.run.smithi039.stdout: "rgwlc_auto_session_clear": "true", 2024-09-21T22:06:12.847 INFO:teuthology.orchestra.run.smithi039.stdout: "rgwlc_skip_bucket_step": "false", 2024-09-21T22:06:12.847 INFO:teuthology.orchestra.run.smithi039.stdout: "rocksdb_block_size": "4096", 2024-09-21T22:06:12.847 INFO:teuthology.orchestra.run.smithi039.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-09-21T22:06:12.847 INFO:teuthology.orchestra.run.smithi039.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-09-21T22:06:12.847 INFO:teuthology.orchestra.run.smithi039.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-09-21T22:06:12.847 INFO:teuthology.orchestra.run.smithi039.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-09-21T22:06:12.848 INFO:teuthology.orchestra.run.smithi039.stdout: "rocksdb_cache_shard_bits": "4", 2024-09-21T22:06:12.848 INFO:teuthology.orchestra.run.smithi039.stdout: "rocksdb_cache_size": "536870912", 2024-09-21T22:06:12.848 INFO:teuthology.orchestra.run.smithi039.stdout: "rocksdb_cache_type": "binned_lru", 2024-09-21T22:06:12.848 INFO:teuthology.orchestra.run.smithi039.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-09-21T22:06:12.848 INFO:teuthology.orchestra.run.smithi039.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-09-21T22:06:12.848 INFO:teuthology.orchestra.run.smithi039.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-09-21T22:06:12.848 INFO:teuthology.orchestra.run.smithi039.stdout: "rocksdb_collect_compaction_stats": "false", 2024-09-21T22:06:12.848 INFO:teuthology.orchestra.run.smithi039.stdout: "rocksdb_collect_extended_stats": "false", 2024-09-21T22:06:12.848 INFO:teuthology.orchestra.run.smithi039.stdout: "rocksdb_collect_memory_stats": "false", 2024-09-21T22:06:12.848 INFO:teuthology.orchestra.run.smithi039.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-09-21T22:06:12.848 INFO:teuthology.orchestra.run.smithi039.stdout: "rocksdb_index_type": "binary_search", 2024-09-21T22:06:12.848 INFO:teuthology.orchestra.run.smithi039.stdout: "rocksdb_log_to_ceph_log": "true", 2024-09-21T22:06:12.848 INFO:teuthology.orchestra.run.smithi039.stdout: "rocksdb_metadata_block_size": "4096", 2024-09-21T22:06:12.848 INFO:teuthology.orchestra.run.smithi039.stdout: "rocksdb_partition_filters": "false", 2024-09-21T22:06:12.848 INFO:teuthology.orchestra.run.smithi039.stdout: "rocksdb_perf": "false", 2024-09-21T22:06:12.849 INFO:teuthology.orchestra.run.smithi039.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-09-21T22:06:12.849 INFO:teuthology.orchestra.run.smithi039.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-09-21T22:06:12.849 INFO:teuthology.orchestra.run.smithi039.stdout: "rotating_keys_renewal_timeout": "10", 2024-09-21T22:06:12.849 INFO:teuthology.orchestra.run.smithi039.stdout: "run_dir": "/var/run/ceph", 2024-09-21T22:06:12.849 INFO:teuthology.orchestra.run.smithi039.stdout: "seastore_block_create": "true", 2024-09-21T22:06:12.849 INFO:teuthology.orchestra.run.smithi039.stdout: "seastore_cache_lru_size": "67108864", 2024-09-21T22:06:12.849 INFO:teuthology.orchestra.run.smithi039.stdout: "seastore_cbjournal_size": "5368709120", 2024-09-21T22:06:12.849 INFO:teuthology.orchestra.run.smithi039.stdout: "seastore_data_delta_based_overwrite": "0", 2024-09-21T22:06:12.849 INFO:teuthology.orchestra.run.smithi039.stdout: "seastore_default_max_object_size": "16777216", 2024-09-21T22:06:12.849 INFO:teuthology.orchestra.run.smithi039.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-09-21T22:06:12.849 INFO:teuthology.orchestra.run.smithi039.stdout: "seastore_device_size": "53687091200", 2024-09-21T22:06:12.849 INFO:teuthology.orchestra.run.smithi039.stdout: "seastore_disable_end_to_end_data_protection": "true", 2024-09-21T22:06:12.849 INFO:teuthology.orchestra.run.smithi039.stdout: "seastore_full_integrity_check": "false", 2024-09-21T22:06:12.849 INFO:teuthology.orchestra.run.smithi039.stdout: "seastore_journal_batch_capacity": "16", 2024-09-21T22:06:12.849 INFO:teuthology.orchestra.run.smithi039.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-09-21T22:06:12.850 INFO:teuthology.orchestra.run.smithi039.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-09-21T22:06:12.850 INFO:teuthology.orchestra.run.smithi039.stdout: "seastore_journal_iodepth_limit": "5", 2024-09-21T22:06:12.850 INFO:teuthology.orchestra.run.smithi039.stdout: "seastore_main_device_type": "SSD", 2024-09-21T22:06:12.850 INFO:teuthology.orchestra.run.smithi039.stdout: "seastore_max_concurrent_transactions": "8", 2024-09-21T22:06:12.850 INFO:teuthology.orchestra.run.smithi039.stdout: "seastore_max_data_allocation_size": "32768", 2024-09-21T22:06:12.850 INFO:teuthology.orchestra.run.smithi039.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-09-21T22:06:12.850 INFO:teuthology.orchestra.run.smithi039.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-09-21T22:06:12.850 INFO:teuthology.orchestra.run.smithi039.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-09-21T22:06:12.850 INFO:teuthology.orchestra.run.smithi039.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-09-21T22:06:12.850 INFO:teuthology.orchestra.run.smithi039.stdout: "seastore_segment_size": "67108864", 2024-09-21T22:06:12.850 INFO:teuthology.orchestra.run.smithi039.stdout: "set_keepcaps": "false", 2024-09-21T22:06:12.850 INFO:teuthology.orchestra.run.smithi039.stdout: "setgroup": "", 2024-09-21T22:06:12.850 INFO:teuthology.orchestra.run.smithi039.stdout: "setuser": "", 2024-09-21T22:06:12.850 INFO:teuthology.orchestra.run.smithi039.stdout: "setuser_match_path": "", 2024-09-21T22:06:12.851 INFO:teuthology.orchestra.run.smithi039.stdout: "target_max_misplaced_ratio": "0.050000", 2024-09-21T22:06:12.851 INFO:teuthology.orchestra.run.smithi039.stdout: "thp": "false", 2024-09-21T22:06:12.851 INFO:teuthology.orchestra.run.smithi039.stdout: "threadpool_default_timeout": "60", 2024-09-21T22:06:12.851 INFO:teuthology.orchestra.run.smithi039.stdout: "threadpool_empty_queue_max_wait": "2", 2024-09-21T22:06:12.851 INFO:teuthology.orchestra.run.smithi039.stdout: "throttler_perf_counter": "true" 2024-09-21T22:06:12.851 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:06:12.853 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-09-21T22:06:12.988 INFO:teuthology.orchestra.run.smithi039.stdout:mon.a: {} 2024-09-21T22:06:12.989 INFO:teuthology.orchestra.run.smithi039.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-21T22:06:13.001 INFO:teuthology.orchestra.run.smithi039.stdout:mon.b: {} 2024-09-21T22:06:13.002 INFO:teuthology.orchestra.run.smithi039.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-21T22:06:13.011 INFO:teuthology.orchestra.run.smithi039.stdout:mon.c: {} 2024-09-21T22:06:13.011 INFO:teuthology.orchestra.run.smithi039.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-21T22:06:13.026 DEBUG:teuthology.orchestra.run.smithi039:> type ceph-objectstore-tool 2024-09-21T22:06:13.030 INFO:teuthology.orchestra.run.smithi039.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-21T22:06:13.030 DEBUG:teuthology.orchestra.run.smithi064:> type ceph-objectstore-tool 2024-09-21T22:06:13.035 INFO:teuthology.orchestra.run.smithi064.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-21T22:06:13.037 INFO:teuthology.run_tasks:Running task workunit... 2024-09-21T22:06:13.049 INFO:tasks.workunit:Pulling workunits from ref 4e45e6b490e6cbe082cdf155fea780438ec53fc7 2024-09-21T22:06:13.050 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-21T22:06:13.050 DEBUG:teuthology.orchestra.run.smithi157:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-21T22:06:13.050 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-09-21T22:06:13.050 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-09-21T22:06:13.050 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T22:06:13.050 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-09-21T22:06:13.050 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight-by-utilization 110 0.05 2024-09-21T22:06:13.051 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-09-21T22:06:13.051 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-09-21T22:06:13.051 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T22:06:13.051 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-09-21T22:06:13.052 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T22:06:13.052 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-09-21T22:06:13.052 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-21T22:06:13.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T22:06:13.057 INFO:teuthology.orchestra.run.smithi157.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-21T22:06:13.058 DEBUG:teuthology.orchestra.run.smithi157:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-09-21T22:06:13.075 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:06:13.073+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:06:13.106 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-09-21T22:06:13.106 DEBUG:teuthology.orchestra.run.smithi157:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-09-21T22:06:13.152 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:06:13.148+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:06:13.154 INFO:tasks.workunit:timeout=3h 2024-09-21T22:06:13.154 INFO:tasks.workunit:cleanup=True 2024-09-21T22:06:13.155 DEBUG:teuthology.orchestra.run.smithi157:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 4e45e6b490e6cbe082cdf155fea780438ec53fc7 2024-09-21T22:06:13.192 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:06:13.192 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:06:13.192 INFO:teuthology.orchestra.run.smithi039.stdout: "num_ops": 0 2024-09-21T22:06:13.192 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:06:13.205 INFO:tasks.workunit.client.0.smithi157.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-21T22:06:13.207 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T22:06:13.208 INFO:teuthology.orchestra.run.smithi039.stdout:osd.0: {} 2024-09-21T22:06:13.208 INFO:teuthology.orchestra.run.smithi039.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T22:06:13.226 INFO:teuthology.orchestra.run.smithi039.stdout:osd.1: {} 2024-09-21T22:06:13.226 INFO:teuthology.orchestra.run.smithi039.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T22:06:13.244 INFO:teuthology.orchestra.run.smithi039.stdout:osd.2: {} 2024-09-21T22:06:13.244 INFO:teuthology.orchestra.run.smithi039.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T22:06:13.252 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:06:13.249+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:06:13.265 INFO:teuthology.orchestra.run.smithi039.stdout:osd.3: {} 2024-09-21T22:06:13.265 INFO:teuthology.orchestra.run.smithi039.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T22:06:13.286 INFO:teuthology.orchestra.run.smithi039.stdout:osd.4: {} 2024-09-21T22:06:13.286 INFO:teuthology.orchestra.run.smithi039.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T22:06:13.307 INFO:teuthology.orchestra.run.smithi039.stdout:osd.5: {} 2024-09-21T22:06:13.307 INFO:teuthology.orchestra.run.smithi039.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T22:06:13.317 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:13.318 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:13.331 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T22:06:13.372 INFO:teuthology.orchestra.run.smithi039.stdout:moved 0 / 18 (0%) 2024-09-21T22:06:13.372 INFO:teuthology.orchestra.run.smithi039.stdout:avg 3 2024-09-21T22:06:13.372 INFO:teuthology.orchestra.run.smithi039.stdout:stddev 1.52753 -> 1.52753 (expected baseline 1.58114) 2024-09-21T22:06:13.372 INFO:teuthology.orchestra.run.smithi039.stdout:min osd.4 with 1 -> 1 pgs (0.333333 -> 0.333333 * mean) 2024-09-21T22:06:13.372 INFO:teuthology.orchestra.run.smithi039.stdout:max osd.1 with 5 -> 5 pgs (1.66667 -> 1.66667 * mean) 2024-09-21T22:06:13.372 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T22:06:13.372 INFO:teuthology.orchestra.run.smithi039.stdout:oload 110 2024-09-21T22:06:13.372 INFO:teuthology.orchestra.run.smithi039.stdout:max_change 0.05 2024-09-21T22:06:13.372 INFO:teuthology.orchestra.run.smithi039.stdout:max_change_osds 4 2024-09-21T22:06:13.372 INFO:teuthology.orchestra.run.smithi039.stdout:average_utilization 0.0003 2024-09-21T22:06:13.372 INFO:teuthology.orchestra.run.smithi039.stdout:overload_utilization 0.0003 2024-09-21T22:06:13.373 INFO:teuthology.orchestra.run.smithi039.stderr:no change 2024-09-21T22:06:13.373 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:06:13.369+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:06:13.389 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-09-21T22:06:13.390 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd test-reweight-by-utilization 1000 1.0 2024-09-21T22:06:13.455 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:06:13.448+0000 7fc8d39f7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25294) UID: 0 2024-09-21T22:06:13.480 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:13.480 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:13.494 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T22:06:13.555 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:06:13.548+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:06:13.614 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:13.615 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:13.627 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T22:06:13.655 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:06:13.652+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:06:13.708 INFO:teuthology.orchestra.run.smithi039.stdout:moved 0 / 18 (0%) 2024-09-21T22:06:13.708 INFO:teuthology.orchestra.run.smithi039.stdout:avg 3 2024-09-21T22:06:13.708 INFO:teuthology.orchestra.run.smithi039.stdout:stddev 1.52753 -> 1.52753 (expected baseline 1.58114) 2024-09-21T22:06:13.708 INFO:teuthology.orchestra.run.smithi039.stdout:min osd.4 with 1 -> 1 pgs (0.333333 -> 0.333333 * mean) 2024-09-21T22:06:13.708 INFO:teuthology.orchestra.run.smithi039.stdout:max osd.1 with 5 -> 5 pgs (1.66667 -> 1.66667 * mean) 2024-09-21T22:06:13.709 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T22:06:13.709 INFO:teuthology.orchestra.run.smithi039.stdout:oload 1000 2024-09-21T22:06:13.709 INFO:teuthology.orchestra.run.smithi039.stdout:max_change 1 2024-09-21T22:06:13.709 INFO:teuthology.orchestra.run.smithi039.stdout:max_change_osds 4 2024-09-21T22:06:13.709 INFO:teuthology.orchestra.run.smithi039.stdout:average_utilization 0.0003 2024-09-21T22:06:13.709 INFO:teuthology.orchestra.run.smithi039.stdout:overload_utilization 0.0031 2024-09-21T22:06:13.709 INFO:teuthology.orchestra.run.smithi039.stderr:no change 2024-09-21T22:06:13.734 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-09-21T22:06:13.734 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd test-reweight-by-utilization 1000 0.05 2024-09-21T22:06:13.753 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:13.753 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:13.756 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:06:13.752+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:06:13.767 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T22:06:13.857 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:06:13.852+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:06:13.930 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:13.930 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:13.943 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T22:06:13.958 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:06:13.952+0000 7fc8d39f7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25294) UID: 0 2024-09-21T22:06:14.017 INFO:teuthology.orchestra.run.smithi039.stdout:moved 0 / 18 (0%) 2024-09-21T22:06:14.017 INFO:teuthology.orchestra.run.smithi039.stdout:avg 3 2024-09-21T22:06:14.017 INFO:teuthology.orchestra.run.smithi039.stdout:stddev 1.52753 -> 1.52753 (expected baseline 1.58114) 2024-09-21T22:06:14.017 INFO:teuthology.orchestra.run.smithi039.stdout:min osd.4 with 1 -> 1 pgs (0.333333 -> 0.333333 * mean) 2024-09-21T22:06:14.017 INFO:teuthology.orchestra.run.smithi039.stdout:max osd.1 with 5 -> 5 pgs (1.66667 -> 1.66667 * mean) 2024-09-21T22:06:14.017 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T22:06:14.017 INFO:teuthology.orchestra.run.smithi039.stdout:oload 1000 2024-09-21T22:06:14.017 INFO:teuthology.orchestra.run.smithi039.stdout:max_change 0.05 2024-09-21T22:06:14.017 INFO:teuthology.orchestra.run.smithi039.stdout:max_change_osds 4 2024-09-21T22:06:14.017 INFO:teuthology.orchestra.run.smithi039.stdout:average_utilization 0.0003 2024-09-21T22:06:14.017 INFO:teuthology.orchestra.run.smithi039.stdout:overload_utilization 0.0031 2024-09-21T22:06:14.018 INFO:teuthology.orchestra.run.smithi039.stderr:no change 2024-09-21T22:06:14.034 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-09-21T22:06:14.035 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight-by-utilization 1000 3.0 2024-09-21T22:06:14.054 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:14.054 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:14.059 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:06:14.057+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:06:14.060 INFO:teuthology.orchestra.run.smithi039.stderr:noscrub is set 2024-09-21T22:06:14.070 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T22:06:14.182 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:06:14.181+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:06:14.246 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:14.246 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:14.262 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T22:06:14.263 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:06:14.260+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:06:14.329 INFO:teuthology.orchestra.run.smithi039.stdout:moved 0 / 18 (0%) 2024-09-21T22:06:14.329 INFO:teuthology.orchestra.run.smithi039.stdout:avg 3 2024-09-21T22:06:14.329 INFO:teuthology.orchestra.run.smithi039.stdout:stddev 1.52753 -> 1.52753 (expected baseline 1.58114) 2024-09-21T22:06:14.329 INFO:teuthology.orchestra.run.smithi039.stdout:min osd.4 with 1 -> 1 pgs (0.333333 -> 0.333333 * mean) 2024-09-21T22:06:14.329 INFO:teuthology.orchestra.run.smithi039.stdout:max osd.1 with 5 -> 5 pgs (1.66667 -> 1.66667 * mean) 2024-09-21T22:06:14.329 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T22:06:14.329 INFO:teuthology.orchestra.run.smithi039.stdout:oload 1000 2024-09-21T22:06:14.329 INFO:teuthology.orchestra.run.smithi039.stdout:max_change 3 2024-09-21T22:06:14.329 INFO:teuthology.orchestra.run.smithi039.stdout:max_change_osds 4 2024-09-21T22:06:14.329 INFO:teuthology.orchestra.run.smithi039.stdout:average_utilization 0.0003 2024-09-21T22:06:14.330 INFO:teuthology.orchestra.run.smithi039.stdout:overload_utilization 0.0031 2024-09-21T22:06:14.330 INFO:teuthology.orchestra.run.smithi039.stderr:no change 2024-09-21T22:06:14.346 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-09-21T22:06:14.346 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight-by-utilization 1000 1.0 2024-09-21T22:06:14.390 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:06:14.389+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:06:14.425 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:14.425 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:14.438 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T22:06:14.486 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:06:14.485+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:06:14.564 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:06:14.560+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:06:14.572 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:14.572 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:14.586 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T22:06:14.671 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:06:14.668+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:06:14.680 INFO:teuthology.orchestra.run.smithi039.stdout:moved 0 / 18 (0%) 2024-09-21T22:06:14.681 INFO:teuthology.orchestra.run.smithi039.stdout:avg 3 2024-09-21T22:06:14.681 INFO:teuthology.orchestra.run.smithi039.stdout:stddev 1.52753 -> 1.52753 (expected baseline 1.58114) 2024-09-21T22:06:14.681 INFO:teuthology.orchestra.run.smithi039.stdout:min osd.4 with 1 -> 1 pgs (0.333333 -> 0.333333 * mean) 2024-09-21T22:06:14.681 INFO:teuthology.orchestra.run.smithi039.stdout:max osd.1 with 5 -> 5 pgs (1.66667 -> 1.66667 * mean) 2024-09-21T22:06:14.681 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T22:06:14.681 INFO:teuthology.orchestra.run.smithi039.stdout:oload 1000 2024-09-21T22:06:14.681 INFO:teuthology.orchestra.run.smithi039.stdout:max_change 1 2024-09-21T22:06:14.681 INFO:teuthology.orchestra.run.smithi039.stdout:max_change_osds 4 2024-09-21T22:06:14.681 INFO:teuthology.orchestra.run.smithi039.stdout:average_utilization 0.0003 2024-09-21T22:06:14.681 INFO:teuthology.orchestra.run.smithi039.stdout:overload_utilization 0.0031 2024-09-21T22:06:14.681 INFO:teuthology.orchestra.run.smithi039.stderr:no change 2024-09-21T22:06:14.750 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:14.750 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:14.764 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T22:06:14.807 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:06:14.804+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:06:14.866 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:06:14.860+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:06:14.937 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:14.937 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:14.951 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T22:06:14.967 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:06:14.965+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:06:15.068 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:06:15.064+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:06:15.125 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:15.125 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:15.139 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T22:06:15.169 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:06:15.165+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:06:15.269 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:06:15.264+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:06:15.304 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:15.304 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:15.318 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T22:06:15.324 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T22:06:15.407 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:06:15.404+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:06:15.470 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:06:15.469+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:06:15.481 INFO:teuthology.orchestra.run.smithi039.stdout:osd.0: {} 2024-09-21T22:06:15.482 INFO:teuthology.orchestra.run.smithi039.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T22:06:15.490 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:15.490 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:15.497 INFO:teuthology.orchestra.run.smithi039.stdout:osd.1: {} 2024-09-21T22:06:15.497 INFO:teuthology.orchestra.run.smithi039.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T22:06:15.504 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T22:06:15.513 INFO:teuthology.orchestra.run.smithi039.stdout:osd.2: {} 2024-09-21T22:06:15.513 INFO:teuthology.orchestra.run.smithi039.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T22:06:15.531 INFO:teuthology.orchestra.run.smithi039.stdout:osd.3: {} 2024-09-21T22:06:15.531 INFO:teuthology.orchestra.run.smithi039.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T22:06:15.549 INFO:teuthology.orchestra.run.smithi039.stdout:osd.4: {} 2024-09-21T22:06:15.549 INFO:teuthology.orchestra.run.smithi039.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T22:06:15.568 INFO:teuthology.orchestra.run.smithi039.stdout:osd.5: {} 2024-09-21T22:06:15.568 INFO:teuthology.orchestra.run.smithi039.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T22:06:15.571 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:06:15.569+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:06:15.672 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:06:15.672 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:06:15.673 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-09-21T22:06:15.673 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:06:15.673 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:06:15.669+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:06:15.686 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T22:06:15.774 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:06:15.769+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:06:15.811 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:06:15.811 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:06:15.812 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-21T22:06:15.812 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-21T22:06:15.812 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:06:15.825 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T22:06:15.874 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:06:15.868+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:06:15.956 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:06:15.956 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-21T22:06:15.956 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-21T22:06:15.956 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-21T22:06:15.956 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:15.956 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4224.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-21T22:06:15.956 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:05.105895+0000", 2024-09-21T22:06:15.957 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.848868448999999, 2024-09-21T22:06:15.957 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0034226500000000002, 2024-09-21T22:06:15.957 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:15.957 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:15.957 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:15.957 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:15.957 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4224", 2024-09-21T22:06:15.957 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.39:0/795178185", 2024-09-21T22:06:15.957 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1 2024-09-21T22:06:15.957 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:15.957 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:15.957 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:15.957 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:15.957 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.105895+0000", 2024-09-21T22:06:15.958 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:15.958 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:15.958 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:15.958 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:15.958 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.105895+0000", 2024-09-21T22:06:15.958 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:15.958 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:15.959 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:15.959 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:15.959 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.105905+0000", 2024-09-21T22:06:15.959 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8069999999999994e-06 2024-09-21T22:06:15.959 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:15.959 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:15.959 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:15.959 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.105919+0000", 2024-09-21T22:06:15.959 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3976e-05 2024-09-21T22:06:15.959 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:15.959 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:15.959 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:15.959 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.105955+0000", 2024-09-21T22:06:15.959 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6776999999999997e-05 2024-09-21T22:06:15.960 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:15.960 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:15.960 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:15.960 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.105983+0000", 2024-09-21T22:06:15.960 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7535999999999999e-05 2024-09-21T22:06:15.960 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:15.960 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:15.960 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:15.960 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.106162+0000", 2024-09-21T22:06:15.960 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000178839 2024-09-21T22:06:15.960 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:15.960 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:15.960 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:15.960 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.106437+0000", 2024-09-21T22:06:15.961 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027542099999999998 2024-09-21T22:06:15.961 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:15.961 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:15.961 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-09-21T22:06:15.961 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.106726+0000", 2024-09-21T22:06:15.961 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00028856800000000001 2024-09-21T22:06:15.961 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:15.961 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:15.961 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:15.961 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.108194+0000", 2024-09-21T22:06:15.961 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001468095 2024-09-21T22:06:15.961 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:15.961 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:15.961 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:15.961 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.109130+0000", 2024-09-21T22:06:15.962 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00093653599999999997 2024-09-21T22:06:15.962 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:15.962 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:15.962 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:15.962 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.109216+0000", 2024-09-21T22:06:15.962 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5191999999999993e-05 2024-09-21T22:06:15.962 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:15.962 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:15.962 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:15.962 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.109317+0000", 2024-09-21T22:06:15.962 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000101903 2024-09-21T22:06:15.962 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:15.962 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:15.962 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:15.963 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:15.963 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:15.963 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4224.0:1 2.3 e17/16 ondisk, result = 0)", 2024-09-21T22:06:15.963 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:05.108920+0000", 2024-09-21T22:06:15.963 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.845842942000001, 2024-09-21T22:06:15.963 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00041366599999999998, 2024-09-21T22:06:15.963 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:15.963 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:15.963 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:06:15.963 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:15.963 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:15.963 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:15.963 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.108920+0000", 2024-09-21T22:06:15.963 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:15.964 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:15.964 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:15.964 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:15.964 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.108920+0000", 2024-09-21T22:06:15.964 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:15.964 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:15.964 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:15.964 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:15.964 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.108922+0000", 2024-09-21T22:06:15.964 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.703e-06 2024-09-21T22:06:15.964 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:15.964 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:15.964 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:15.964 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.108938+0000", 2024-09-21T22:06:15.964 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6350000000000001e-05 2024-09-21T22:06:15.965 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:15.965 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:15.965 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:15.965 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.108974+0000", 2024-09-21T22:06:15.965 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5612999999999999e-05 2024-09-21T22:06:15.965 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:15.965 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:15.965 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:15.965 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.108989+0000", 2024-09-21T22:06:15.965 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4576000000000001e-05 2024-09-21T22:06:15.965 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:15.965 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:15.965 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:15.965 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.109108+0000", 2024-09-21T22:06:15.966 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011901899999999999 2024-09-21T22:06:15.966 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:15.966 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:15.966 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:15.966 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.109120+0000", 2024-09-21T22:06:15.966 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.293e-05 2024-09-21T22:06:15.966 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:15.966 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:15.966 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:15.966 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.109334+0000", 2024-09-21T22:06:15.966 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000213475 2024-09-21T22:06:15.966 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:15.966 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:15.966 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:15.967 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:15.967 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:15.967 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4224.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e17)", 2024-09-21T22:06:15.967 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:05.115572+0000", 2024-09-21T22:06:15.967 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.839191034000001, 2024-09-21T22:06:15.967 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00041313, 2024-09-21T22:06:15.967 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:15.967 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:15.967 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-21T22:06:15.968 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:15.968 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4224", 2024-09-21T22:06:15.968 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.39:0/795178185", 2024-09-21T22:06:15.968 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2 2024-09-21T22:06:15.968 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:15.968 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:15.968 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:15.968 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:15.968 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.115572+0000", 2024-09-21T22:06:15.968 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:15.968 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:15.968 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:15.968 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:15.968 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.115572+0000", 2024-09-21T22:06:15.968 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:15.969 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:15.969 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:15.969 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:15.969 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.115574+0000", 2024-09-21T22:06:15.969 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0530000000000001e-06 2024-09-21T22:06:15.969 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:15.969 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:15.969 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:15.969 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.115585+0000", 2024-09-21T22:06:15.969 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0947e-05 2024-09-21T22:06:15.969 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:15.969 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:15.969 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:15.969 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.115615+0000", 2024-09-21T22:06:15.970 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9408999999999999e-05 2024-09-21T22:06:15.970 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:15.970 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:15.970 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:15.970 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.115637+0000", 2024-09-21T22:06:15.970 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2073e-05 2024-09-21T22:06:15.970 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:15.970 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:15.970 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:15.970 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.115768+0000", 2024-09-21T22:06:15.970 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013153899999999999 2024-09-21T22:06:15.970 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:15.970 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:15.970 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:15.971 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.115985+0000", 2024-09-21T22:06:15.971 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000217109 2024-09-21T22:06:15.971 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:15.971 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:15.971 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:15.971 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:15.971 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:15.971 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4224.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e17)", 2024-09-21T22:06:15.971 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:05.116456+0000", 2024-09-21T22:06:15.971 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.838307488, 2024-09-21T22:06:15.971 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00030938500000000002, 2024-09-21T22:06:15.971 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:15.971 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:15.971 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-21T22:06:15.972 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:15.972 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4224", 2024-09-21T22:06:15.972 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.39:0/795178185", 2024-09-21T22:06:15.972 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3 2024-09-21T22:06:15.972 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:15.972 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:15.972 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:15.972 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:15.972 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.116456+0000", 2024-09-21T22:06:15.972 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:15.972 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:15.972 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:15.972 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:15.972 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.116456+0000", 2024-09-21T22:06:15.972 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:15.973 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:15.973 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:15.973 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:15.973 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.116458+0000", 2024-09-21T22:06:15.973 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.77e-06 2024-09-21T22:06:15.973 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:15.973 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:15.973 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:15.973 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.116466+0000", 2024-09-21T22:06:15.973 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1799999999999996e-06 2024-09-21T22:06:15.973 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:15.973 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:15.973 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:15.973 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.116492+0000", 2024-09-21T22:06:15.973 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6523999999999999e-05 2024-09-21T22:06:15.974 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:15.974 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:15.974 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:15.974 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.116513+0000", 2024-09-21T22:06:15.974 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0400000000000001e-05 2024-09-21T22:06:15.974 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:15.974 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:15.974 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:15.974 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.116616+0000", 2024-09-21T22:06:15.974 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000103862 2024-09-21T22:06:15.974 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:15.974 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:15.974 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:15.974 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.116765+0000", 2024-09-21T22:06:15.975 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014864899999999999 2024-09-21T22:06:15.975 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:15.975 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:15.975 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:15.975 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:15.975 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:15.975 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4224.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-21T22:06:15.975 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:06.109464+0000", 2024-09-21T22:06:15.975 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.8452988099999992, 2024-09-21T22:06:15.975 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0070431790000000001, 2024-09-21T22:06:15.975 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:15.975 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:15.975 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:15.975 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:15.976 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4224", 2024-09-21T22:06:15.976 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.39:0/795178185", 2024-09-21T22:06:15.976 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 5 2024-09-21T22:06:15.976 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:15.976 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:15.976 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:15.976 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:15.976 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.109464+0000", 2024-09-21T22:06:15.976 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:15.976 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:15.976 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:15.976 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:15.976 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.109464+0000", 2024-09-21T22:06:15.976 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:15.977 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:15.977 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:15.977 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:15.977 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.109467+0000", 2024-09-21T22:06:15.977 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.066e-06 2024-09-21T22:06:15.977 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:15.977 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:15.977 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:15.977 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.109485+0000", 2024-09-21T22:06:15.977 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7464000000000001e-05 2024-09-21T22:06:15.977 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:15.977 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:15.977 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:15.977 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.109518+0000", 2024-09-21T22:06:15.978 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3343000000000002e-05 2024-09-21T22:06:15.978 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:15.978 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:15.978 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:15.978 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.109537+0000", 2024-09-21T22:06:15.978 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8743e-05 2024-09-21T22:06:15.978 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:15.978 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:15.978 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:15.978 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.109708+0000", 2024-09-21T22:06:15.978 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017073899999999999 2024-09-21T22:06:15.978 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:15.978 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:15.978 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op must wait for map", 2024-09-21T22:06:15.979 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.109725+0000", 2024-09-21T22:06:15.979 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7013000000000001e-05 2024-09-21T22:06:15.979 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:15.979 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:15.979 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:15.979 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.113449+0000", 2024-09-21T22:06:15.979 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0037244579999999999 2024-09-21T22:06:15.979 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:15.979 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:15.979 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:15.979 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.113659+0000", 2024-09-21T22:06:15.979 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020968199999999999 2024-09-21T22:06:15.979 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:15.979 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:15.979 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-09-21T22:06:15.980 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.113872+0000", 2024-09-21T22:06:15.980 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000213062 2024-09-21T22:06:15.980 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:15.980 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:15.980 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:15.980 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.115375+0000", 2024-09-21T22:06:15.980 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015028979999999999 2024-09-21T22:06:15.980 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:15.980 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:15.980 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:15.980 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.116356+0000", 2024-09-21T22:06:15.980 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00098098200000000003 2024-09-21T22:06:15.980 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:15.980 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:15.981 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:15.981 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.116442+0000", 2024-09-21T22:06:15.981 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6533000000000007e-05 2024-09-21T22:06:15.981 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:15.981 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:15.981 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:15.981 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.116508+0000", 2024-09-21T22:06:15.981 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5196000000000004e-05 2024-09-21T22:06:15.981 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:15.981 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:15.981 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:15.981 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:15.981 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:15.981 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4224.0:5 2.2 e17/16 ondisk, result = 0)", 2024-09-21T22:06:15.981 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:06.116133+0000", 2024-09-21T22:06:15.982 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.8386303250000005, 2024-09-21T22:06:15.982 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00038649700000000002, 2024-09-21T22:06:15.982 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:15.982 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:15.982 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:06:15.982 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:15.982 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:15.982 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:15.983 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.116133+0000", 2024-09-21T22:06:15.983 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:15.983 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:15.983 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:15.983 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:15.983 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.116133+0000", 2024-09-21T22:06:15.983 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:15.983 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:15.983 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:15.983 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:15.983 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.116135+0000", 2024-09-21T22:06:15.983 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8870000000000001e-06 2024-09-21T22:06:15.983 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:15.983 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:15.983 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:15.984 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.116146+0000", 2024-09-21T22:06:15.984 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0953e-05 2024-09-21T22:06:15.984 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:15.984 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:15.984 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:15.984 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.116175+0000", 2024-09-21T22:06:15.984 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9383e-05 2024-09-21T22:06:15.984 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:15.984 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:15.984 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:15.984 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.116190+0000", 2024-09-21T22:06:15.984 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4613000000000001e-05 2024-09-21T22:06:15.984 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:15.984 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:15.985 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:15.985 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.116328+0000", 2024-09-21T22:06:15.985 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013848300000000001 2024-09-21T22:06:15.985 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:15.985 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:15.985 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:15.985 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.116345+0000", 2024-09-21T22:06:15.985 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6883000000000001e-05 2024-09-21T22:06:15.985 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:15.985 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:15.985 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:15.985 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.116519+0000", 2024-09-21T22:06:15.985 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017429500000000001 2024-09-21T22:06:15.985 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:15.986 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:15.986 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:15.986 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:15.986 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:15.986 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4224.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-21T22:06:15.986 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:07.124322+0000", 2024-09-21T22:06:15.986 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.8304410200000003, 2024-09-21T22:06:15.986 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0067709010000000002, 2024-09-21T22:06:15.986 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:15.986 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:15.986 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:15.986 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:15.986 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4224", 2024-09-21T22:06:15.986 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.39:0/795178185", 2024-09-21T22:06:15.987 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 7 2024-09-21T22:06:15.987 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:15.987 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:15.987 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:15.987 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:15.987 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.124322+0000", 2024-09-21T22:06:15.987 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:15.987 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:15.987 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:15.987 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:15.987 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.124322+0000", 2024-09-21T22:06:15.987 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:15.987 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:15.987 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:15.988 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:15.988 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.124325+0000", 2024-09-21T22:06:15.988 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1489999999999998e-06 2024-09-21T22:06:15.988 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:15.988 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:15.988 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:15.988 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.124342+0000", 2024-09-21T22:06:15.988 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.666e-05 2024-09-21T22:06:15.988 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:15.988 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:15.988 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:15.988 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.124379+0000", 2024-09-21T22:06:15.988 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6539999999999999e-05 2024-09-21T22:06:15.988 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:15.988 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:15.989 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:15.989 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.124406+0000", 2024-09-21T22:06:15.989 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7886e-05 2024-09-21T22:06:15.989 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:15.989 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:15.989 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:15.989 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.124535+0000", 2024-09-21T22:06:15.989 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000128749 2024-09-21T22:06:15.989 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:15.989 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:15.989 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op must wait for map", 2024-09-21T22:06:15.990 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.124547+0000", 2024-09-21T22:06:15.990 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.136e-05 2024-09-21T22:06:15.990 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:15.990 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:15.990 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:15.990 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.127774+0000", 2024-09-21T22:06:15.990 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0032275519999999999 2024-09-21T22:06:15.990 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:15.990 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:15.990 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:15.990 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.128014+0000", 2024-09-21T22:06:15.990 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023956800000000001 2024-09-21T22:06:15.990 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:15.990 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:15.991 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-09-21T22:06:15.991 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.128462+0000", 2024-09-21T22:06:15.991 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00044797700000000001 2024-09-21T22:06:15.991 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:15.991 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:15.991 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:15.991 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.130185+0000", 2024-09-21T22:06:15.991 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00172313 2024-09-21T22:06:15.991 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:15.991 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:15.991 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:15.991 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.130937+0000", 2024-09-21T22:06:15.991 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00075244799999999996 2024-09-21T22:06:15.991 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:15.992 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:15.992 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:15.992 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.131025+0000", 2024-09-21T22:06:15.992 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7998999999999996e-05 2024-09-21T22:06:15.992 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:15.992 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:15.992 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:15.992 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.131093+0000", 2024-09-21T22:06:15.992 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7883e-05 2024-09-21T22:06:15.992 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:15.992 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:15.992 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:15.992 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:15.992 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:15.992 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4224.0:7 2.2 e18/16 ondisk, result = 0)", 2024-09-21T22:06:15.993 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:07.130777+0000", 2024-09-21T22:06:15.993 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.8239859739999993, 2024-09-21T22:06:15.993 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00032786799999999999, 2024-09-21T22:06:15.993 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:15.993 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:15.993 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:06:15.993 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:15.993 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:15.993 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:15.993 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.130777+0000", 2024-09-21T22:06:15.993 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:15.993 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:15.993 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:15.993 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:15.994 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.130777+0000", 2024-09-21T22:06:15.994 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:15.994 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:15.994 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:15.994 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:15.994 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.130779+0000", 2024-09-21T22:06:15.994 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7e-06 2024-09-21T22:06:15.994 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:15.994 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:15.994 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:15.994 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.130788+0000", 2024-09-21T22:06:15.994 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6670000000000002e-06 2024-09-21T22:06:15.994 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:15.994 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:15.994 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:15.995 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.130814+0000", 2024-09-21T22:06:15.995 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6590000000000001e-05 2024-09-21T22:06:15.995 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:15.995 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:15.995 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:15.995 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.130828+0000", 2024-09-21T22:06:15.995 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.415e-05 2024-09-21T22:06:15.995 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:15.995 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:15.995 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:15.995 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.130915+0000", 2024-09-21T22:06:15.995 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6759000000000001e-05 2024-09-21T22:06:15.995 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:15.995 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:15.995 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:15.996 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.130928+0000", 2024-09-21T22:06:15.996 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.328e-05 2024-09-21T22:06:15.996 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:15.996 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:15.996 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:15.996 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.131105+0000", 2024-09-21T22:06:15.996 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017672200000000001 2024-09-21T22:06:15.996 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:15.996 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:15.996 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:15.996 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:15.996 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:15.996 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:06:15.998 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T22:06:15.998 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:06:15.993+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:06:16.088 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-21T22:06:16.134 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:06:16.133+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:06:16.136 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:06:16.136 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:06:16.136 INFO:teuthology.orchestra.run.smithi039.stdout: "num_ops": 0 2024-09-21T22:06:16.136 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:06:16.151 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T22:06:16.200 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:06:16.197+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:06:16.301 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:06:16.296+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:06:16.317 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:06:16.317 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:06:16.317 INFO:teuthology.orchestra.run.smithi039.stdout: "complaint_time": 30, 2024-09-21T22:06:16.317 INFO:teuthology.orchestra.run.smithi039.stdout: "num_blocked_ops": 0 2024-09-21T22:06:16.317 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:06:16.331 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T22:06:16.401 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:06:16.400+0000 7fc8d39f7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25294) UID: 0 2024-09-21T22:06:16.459 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:06:16.459 INFO:teuthology.orchestra.run.smithi039.stdout: "size": 20, 2024-09-21T22:06:16.459 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 600, 2024-09-21T22:06:16.459 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [ 2024-09-21T22:06:16.459 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.460 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:20 1.0 e14/13 1:30306672:devicehealth::main.db.0000000000000000:head v 14'10, mlcod=14'8)", 2024-09-21T22:06:16.460 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.137639+0000", 2024-09-21T22:06:16.460 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 14.319170186000001, 2024-09-21T22:06:16.460 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0039902100000000001, 2024-09-21T22:06:16.460 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:16.460 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:16.460 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:16.460 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:16.460 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.460 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:16.460 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.137639+0000", 2024-09-21T22:06:16.460 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:16.460 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.460 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.460 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:16.461 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.137639+0000", 2024-09-21T22:06:16.461 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:16.461 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.461 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.461 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:16.461 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.137641+0000", 2024-09-21T22:06:16.461 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.686e-06 2024-09-21T22:06:16.461 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.461 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.461 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:16.461 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.137934+0000", 2024-09-21T22:06:16.461 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00029319099999999999 2024-09-21T22:06:16.461 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.461 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.461 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:16.462 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.137963+0000", 2024-09-21T22:06:16.462 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9502999999999999e-05 2024-09-21T22:06:16.462 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.462 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.462 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:16.462 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.137989+0000", 2024-09-21T22:06:16.462 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5573000000000001e-05 2024-09-21T22:06:16.462 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.462 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.462 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:16.462 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.138086+0000", 2024-09-21T22:06:16.462 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.6968999999999999e-05 2024-09-21T22:06:16.462 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.463 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.463 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:16.463 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.138120+0000", 2024-09-21T22:06:16.463 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.4560000000000001e-05 2024-09-21T22:06:16.463 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.463 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.463 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:16.463 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.141539+0000", 2024-09-21T22:06:16.463 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0034183289999999999 2024-09-21T22:06:16.463 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.463 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.463 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:16.463 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.141629+0000", 2024-09-21T22:06:16.463 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.0399e-05 2024-09-21T22:06:16.463 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:16.464 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:16.464 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:16.464 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.464 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.464 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:23 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'13, mlcod=14'11)", 2024-09-21T22:06:16.464 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.144914+0000", 2024-09-21T22:06:16.464 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 14.311894868, 2024-09-21T22:06:16.464 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0040223230000000004, 2024-09-21T22:06:16.464 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:16.464 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:16.464 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:16.464 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:16.464 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.464 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:16.464 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.144914+0000", 2024-09-21T22:06:16.465 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:16.465 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.465 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.465 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:16.465 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.144914+0000", 2024-09-21T22:06:16.465 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:16.465 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.465 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.465 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:16.465 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.144916+0000", 2024-09-21T22:06:16.465 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6470000000000001e-06 2024-09-21T22:06:16.465 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.465 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.465 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:16.465 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.144929+0000", 2024-09-21T22:06:16.465 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2903e-05 2024-09-21T22:06:16.466 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.466 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.466 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:16.466 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.144958+0000", 2024-09-21T22:06:16.466 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9280000000000001e-05 2024-09-21T22:06:16.466 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.466 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.466 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:16.466 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.144971+0000", 2024-09-21T22:06:16.466 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3407000000000001e-05 2024-09-21T22:06:16.466 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.466 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.466 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:16.466 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.145106+0000", 2024-09-21T22:06:16.466 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013440500000000001 2024-09-21T22:06:16.467 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.467 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.467 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:16.467 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.145170+0000", 2024-09-21T22:06:16.467 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.4597000000000005e-05 2024-09-21T22:06:16.467 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.467 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.467 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:16.467 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.148854+0000", 2024-09-21T22:06:16.467 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0036833249999999999 2024-09-21T22:06:16.467 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.467 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.467 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:16.467 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.148937+0000", 2024-09-21T22:06:16.468 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.2758999999999998e-05 2024-09-21T22:06:16.468 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:16.468 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:16.468 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:16.468 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.468 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.468 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:31 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'17, mlcod=14'14)", 2024-09-21T22:06:16.468 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.163958+0000", 2024-09-21T22:06:16.468 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 14.292850601, 2024-09-21T22:06:16.468 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0033891149999999998, 2024-09-21T22:06:16.468 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:16.468 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:16.468 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:16.468 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:16.469 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.469 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:16.469 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.163958+0000", 2024-09-21T22:06:16.469 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:16.469 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.469 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.469 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:16.469 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.163958+0000", 2024-09-21T22:06:16.469 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:16.469 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.469 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.469 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:16.469 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.163960+0000", 2024-09-21T22:06:16.469 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.64e-06 2024-09-21T22:06:16.469 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.470 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.470 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:16.470 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.163971+0000", 2024-09-21T22:06:16.470 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0797e-05 2024-09-21T22:06:16.470 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.470 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.470 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:16.470 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.163995+0000", 2024-09-21T22:06:16.470 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3876e-05 2024-09-21T22:06:16.470 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.470 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.470 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:16.470 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.164008+0000", 2024-09-21T22:06:16.470 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3674e-05 2024-09-21T22:06:16.470 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.471 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.471 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:16.471 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.164203+0000", 2024-09-21T22:06:16.471 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00019457799999999999 2024-09-21T22:06:16.471 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.471 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.471 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:16.471 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.164265+0000", 2024-09-21T22:06:16.471 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.2415999999999993e-05 2024-09-21T22:06:16.471 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.471 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.471 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:16.471 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.167209+0000", 2024-09-21T22:06:16.471 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0029436380000000002 2024-09-21T22:06:16.471 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.472 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.472 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:16.472 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.167348+0000", 2024-09-21T22:06:16.472 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000138496 2024-09-21T22:06:16.472 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:16.472 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:16.472 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:16.472 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.472 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.472 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:32 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'18, mlcod=14'14)", 2024-09-21T22:06:16.472 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.165319+0000", 2024-09-21T22:06:16.472 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 14.291489751, 2024-09-21T22:06:16.472 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0033722280000000001, 2024-09-21T22:06:16.472 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:16.473 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:16.473 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:16.473 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:16.473 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.473 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:16.473 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.165319+0000", 2024-09-21T22:06:16.473 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:16.473 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.473 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.473 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:16.473 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.165319+0000", 2024-09-21T22:06:16.473 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:16.473 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.473 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.474 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:16.474 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.165321+0000", 2024-09-21T22:06:16.474 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5999999999999999e-06 2024-09-21T22:06:16.474 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.474 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.474 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:16.474 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.165377+0000", 2024-09-21T22:06:16.474 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.5877000000000001e-05 2024-09-21T22:06:16.474 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.474 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.474 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:16.474 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.165407+0000", 2024-09-21T22:06:16.474 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.0263000000000001e-05 2024-09-21T22:06:16.474 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.475 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.475 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:16.475 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.165441+0000", 2024-09-21T22:06:16.475 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.3966e-05 2024-09-21T22:06:16.475 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.475 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.475 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:16.475 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.165574+0000", 2024-09-21T22:06:16.475 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013327599999999999 2024-09-21T22:06:16.475 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.475 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.475 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:16.475 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.165635+0000", 2024-09-21T22:06:16.475 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.0949000000000002e-05 2024-09-21T22:06:16.476 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.476 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.476 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:16.476 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.168619+0000", 2024-09-21T22:06:16.476 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0029836979999999999 2024-09-21T22:06:16.476 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.476 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.476 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:16.476 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.168692+0000", 2024-09-21T22:06:16.476 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.2599000000000001e-05 2024-09-21T22:06:16.476 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:16.476 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:16.476 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:16.476 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.476 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.477 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:38 1.0 e14/13 1:30306672:devicehealth::main.db.0000000000000000:head v 14'24, mlcod=14'22)", 2024-09-21T22:06:16.477 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.180244+0000", 2024-09-21T22:06:16.477 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 14.276565106, 2024-09-21T22:06:16.477 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0041712219999999996, 2024-09-21T22:06:16.477 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:16.477 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:16.477 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:16.477 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:16.477 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.477 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:16.477 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.180244+0000", 2024-09-21T22:06:16.477 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:16.477 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.477 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.478 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:16.478 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.180244+0000", 2024-09-21T22:06:16.478 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:16.478 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.478 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.478 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:16.478 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.180246+0000", 2024-09-21T22:06:16.478 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6959999999999999e-06 2024-09-21T22:06:16.478 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.478 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.478 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:16.478 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.180508+0000", 2024-09-21T22:06:16.478 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00026275199999999999 2024-09-21T22:06:16.478 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.478 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.479 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:16.479 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.180538+0000", 2024-09-21T22:06:16.479 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9142999999999999e-05 2024-09-21T22:06:16.479 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.479 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.479 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:16.479 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.180552+0000", 2024-09-21T22:06:16.479 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4466e-05 2024-09-21T22:06:16.479 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.479 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.479 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:16.479 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.180677+0000", 2024-09-21T22:06:16.479 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000125413 2024-09-21T22:06:16.479 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.480 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.480 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:16.480 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.180739+0000", 2024-09-21T22:06:16.480 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.1122000000000002e-05 2024-09-21T22:06:16.480 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.480 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.480 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:16.480 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.184291+0000", 2024-09-21T22:06:16.480 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0035525510000000001 2024-09-21T22:06:16.480 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.480 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.480 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:16.480 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.184415+0000", 2024-09-21T22:06:16.480 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000124079 2024-09-21T22:06:16.480 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:16.481 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:16.481 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:16.481 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.481 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.481 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:54 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'32, mlcod=14'29)", 2024-09-21T22:06:16.481 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.214326+0000", 2024-09-21T22:06:16.481 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 14.242482869, 2024-09-21T22:06:16.481 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0035226569999999998, 2024-09-21T22:06:16.481 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:16.481 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:16.481 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:16.481 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:16.481 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.481 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:16.481 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.214326+0000", 2024-09-21T22:06:16.481 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:16.482 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.482 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.482 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:16.482 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.214326+0000", 2024-09-21T22:06:16.482 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:16.482 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.482 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.482 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:16.482 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.214328+0000", 2024-09-21T22:06:16.482 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6270000000000001e-06 2024-09-21T22:06:16.482 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.482 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.482 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:16.483 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.214340+0000", 2024-09-21T22:06:16.483 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2286000000000001e-05 2024-09-21T22:06:16.483 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.483 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.483 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:16.483 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.214369+0000", 2024-09-21T22:06:16.483 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9297000000000001e-05 2024-09-21T22:06:16.483 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.483 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.483 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:16.483 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.214383+0000", 2024-09-21T22:06:16.483 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3773e-05 2024-09-21T22:06:16.483 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.483 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.483 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:16.484 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.214513+0000", 2024-09-21T22:06:16.484 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000130056 2024-09-21T22:06:16.484 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.484 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.484 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:16.484 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.214568+0000", 2024-09-21T22:06:16.484 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.4873000000000001e-05 2024-09-21T22:06:16.484 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.484 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.484 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:16.484 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.217739+0000", 2024-09-21T22:06:16.484 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0031712720000000002 2024-09-21T22:06:16.484 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.484 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.484 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:16.484 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.217849+0000", 2024-09-21T22:06:16.485 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000109473 2024-09-21T22:06:16.485 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:16.485 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:16.485 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:16.485 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.485 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.485 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:57 1.0 e14/13 1:30306672:devicehealth::main.db.0000000000000000:head v 14'35, mlcod=14'33)", 2024-09-21T22:06:16.485 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.221549+0000", 2024-09-21T22:06:16.485 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 14.235260370000001, 2024-09-21T22:06:16.485 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0041720860000000002, 2024-09-21T22:06:16.485 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:16.485 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:16.485 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:16.485 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:16.485 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.486 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:16.486 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.221549+0000", 2024-09-21T22:06:16.486 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:16.486 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.486 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.486 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:16.486 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.221549+0000", 2024-09-21T22:06:16.486 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:16.486 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.486 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.486 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:16.486 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.221550+0000", 2024-09-21T22:06:16.486 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.717e-06 2024-09-21T22:06:16.486 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.487 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.487 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:16.487 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.221673+0000", 2024-09-21T22:06:16.487 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000122612 2024-09-21T22:06:16.487 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.487 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.487 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:16.487 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.221703+0000", 2024-09-21T22:06:16.487 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9539999999999998e-05 2024-09-21T22:06:16.487 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.487 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.487 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:16.487 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.221716+0000", 2024-09-21T22:06:16.487 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3149999999999999e-05 2024-09-21T22:06:16.487 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.488 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.488 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:16.488 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.221848+0000", 2024-09-21T22:06:16.488 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013201900000000001 2024-09-21T22:06:16.488 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.488 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.488 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:16.488 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.221908+0000", 2024-09-21T22:06:16.488 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.0452999999999999e-05 2024-09-21T22:06:16.488 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.488 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.488 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:16.488 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.225573+0000", 2024-09-21T22:06:16.488 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.003664676 2024-09-21T22:06:16.488 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.489 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.489 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:16.489 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.225721+0000", 2024-09-21T22:06:16.489 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00014791899999999999 2024-09-21T22:06:16.489 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:16.489 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:16.489 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:16.489 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.489 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.489 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:68 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'42, mlcod=14'39)", 2024-09-21T22:06:16.489 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.247413+0000", 2024-09-21T22:06:16.489 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 14.209396021, 2024-09-21T22:06:16.489 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0039572339999999996, 2024-09-21T22:06:16.489 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:16.490 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:16.490 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:16.490 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:16.490 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.490 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:16.490 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.247413+0000", 2024-09-21T22:06:16.490 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:16.490 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.490 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.490 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:16.490 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.247413+0000", 2024-09-21T22:06:16.490 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:16.490 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.490 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.490 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:16.490 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.247415+0000", 2024-09-21T22:06:16.491 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.883e-06 2024-09-21T22:06:16.491 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.491 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.491 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:16.491 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.247427+0000", 2024-09-21T22:06:16.491 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2557e-05 2024-09-21T22:06:16.491 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.491 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.491 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:16.491 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.247457+0000", 2024-09-21T22:06:16.491 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9966000000000001e-05 2024-09-21T22:06:16.491 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.491 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.491 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:16.491 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.247471+0000", 2024-09-21T22:06:16.492 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3747000000000001e-05 2024-09-21T22:06:16.492 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.492 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.492 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:16.492 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.247603+0000", 2024-09-21T22:06:16.492 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013171600000000001 2024-09-21T22:06:16.492 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.492 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.492 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:16.492 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.247656+0000", 2024-09-21T22:06:16.492 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.3529999999999997e-05 2024-09-21T22:06:16.492 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.492 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.492 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:16.492 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.251225+0000", 2024-09-21T22:06:16.492 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00356865 2024-09-21T22:06:16.493 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.493 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.493 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:16.493 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.251370+0000", 2024-09-21T22:06:16.493 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00014518500000000001 2024-09-21T22:06:16.493 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:16.493 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:16.493 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:16.493 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.493 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.493 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:69 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'43, mlcod=14'39)", 2024-09-21T22:06:16.493 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.248839+0000", 2024-09-21T22:06:16.493 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 14.207970091, 2024-09-21T22:06:16.493 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0034954470000000001, 2024-09-21T22:06:16.494 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:16.494 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:16.494 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:16.494 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:16.494 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.494 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:16.494 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.248839+0000", 2024-09-21T22:06:16.494 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:16.494 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.494 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.494 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:16.494 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.248839+0000", 2024-09-21T22:06:16.494 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:16.494 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.495 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.495 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:16.495 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.248841+0000", 2024-09-21T22:06:16.495 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6759999999999999e-06 2024-09-21T22:06:16.495 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.495 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.495 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:16.495 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.249058+0000", 2024-09-21T22:06:16.495 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000217455 2024-09-21T22:06:16.495 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.495 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.495 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:16.495 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.249085+0000", 2024-09-21T22:06:16.495 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.7002999999999999e-05 2024-09-21T22:06:16.495 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.496 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.496 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:16.496 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.249099+0000", 2024-09-21T22:06:16.496 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.402e-05 2024-09-21T22:06:16.496 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.496 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.496 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:16.496 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.249241+0000", 2024-09-21T22:06:16.496 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00014194899999999999 2024-09-21T22:06:16.496 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.496 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.496 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:16.496 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.249298+0000", 2024-09-21T22:06:16.496 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.6687000000000001e-05 2024-09-21T22:06:16.496 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.497 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.497 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:16.497 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.252266+0000", 2024-09-21T22:06:16.497 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.002968348 2024-09-21T22:06:16.497 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.497 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.497 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:16.497 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.252334+0000", 2024-09-21T22:06:16.497 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.8308999999999999e-05 2024-09-21T22:06:16.497 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:16.497 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:16.497 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:16.497 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.497 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.497 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:71 1.0 e14/13 1:30306672:devicehealth::main.db.0000000000000000:head v 14'45, mlcod=14'43)", 2024-09-21T22:06:16.497 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.256068+0000", 2024-09-21T22:06:16.498 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 14.200740618999999, 2024-09-21T22:06:16.498 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0041712579999999997, 2024-09-21T22:06:16.498 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:16.498 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:16.498 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:16.498 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:16.498 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.498 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:16.498 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.256068+0000", 2024-09-21T22:06:16.498 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:16.498 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.498 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.498 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:16.499 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.256068+0000", 2024-09-21T22:06:16.499 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:16.499 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.499 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.499 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:16.499 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.256070+0000", 2024-09-21T22:06:16.499 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.61e-06 2024-09-21T22:06:16.499 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.499 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.499 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:16.499 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.256129+0000", 2024-09-21T22:06:16.499 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.8702999999999997e-05 2024-09-21T22:06:16.499 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.499 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.499 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:16.500 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.256158+0000", 2024-09-21T22:06:16.500 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9533e-05 2024-09-21T22:06:16.500 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.500 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.500 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:16.500 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.256173+0000", 2024-09-21T22:06:16.500 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4267000000000001e-05 2024-09-21T22:06:16.500 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.500 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.500 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:16.500 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.256304+0000", 2024-09-21T22:06:16.500 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000131352 2024-09-21T22:06:16.500 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.500 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.500 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:16.501 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.256357+0000", 2024-09-21T22:06:16.501 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.3492999999999998e-05 2024-09-21T22:06:16.501 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.501 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.501 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:16.501 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.260147+0000", 2024-09-21T22:06:16.501 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0037900410000000001 2024-09-21T22:06:16.501 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.501 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.501 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:16.501 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.260240+0000", 2024-09-21T22:06:16.501 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.2258999999999999e-05 2024-09-21T22:06:16.501 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:16.501 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:16.501 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:16.502 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.502 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.502 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:72 1.0 e14/13 1:30306672:devicehealth::main.db.0000000000000000:head v 14'46, mlcod=14'43)", 2024-09-21T22:06:16.502 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.258358+0000", 2024-09-21T22:06:16.502 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 14.198450626, 2024-09-21T22:06:16.502 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0041492919999999997, 2024-09-21T22:06:16.502 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:16.502 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:16.502 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:16.502 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:16.502 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.502 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:16.502 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.258358+0000", 2024-09-21T22:06:16.502 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:16.502 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.503 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.503 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:16.503 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.258358+0000", 2024-09-21T22:06:16.503 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:16.503 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.503 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.503 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:16.503 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.258360+0000", 2024-09-21T22:06:16.503 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.623e-06 2024-09-21T22:06:16.503 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.503 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.503 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:16.503 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.258415+0000", 2024-09-21T22:06:16.503 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.4756000000000002e-05 2024-09-21T22:06:16.504 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.504 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.504 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:16.504 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.258444+0000", 2024-09-21T22:06:16.504 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9164000000000001e-05 2024-09-21T22:06:16.504 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.504 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.504 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:16.504 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.258459+0000", 2024-09-21T22:06:16.504 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4996e-05 2024-09-21T22:06:16.504 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.504 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.504 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:16.504 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.258637+0000", 2024-09-21T22:06:16.504 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00017796500000000001 2024-09-21T22:06:16.505 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.505 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.505 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:16.505 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.258695+0000", 2024-09-21T22:06:16.505 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.8207e-05 2024-09-21T22:06:16.505 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.505 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.505 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:16.505 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.262421+0000", 2024-09-21T22:06:16.505 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0037261550000000001 2024-09-21T22:06:16.505 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.505 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.505 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:16.505 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.262508+0000", 2024-09-21T22:06:16.505 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.6426000000000002e-05 2024-09-21T22:06:16.505 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:16.506 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:16.506 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:16.506 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.506 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.506 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:87 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'57, mlcod=14'52)", 2024-09-21T22:06:16.506 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.290115+0000", 2024-09-21T22:06:16.506 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 14.166694181, 2024-09-21T22:06:16.506 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0030478200000000001, 2024-09-21T22:06:16.506 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:16.506 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:16.506 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:16.506 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:16.506 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.506 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:16.507 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.290115+0000", 2024-09-21T22:06:16.507 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:16.507 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.507 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.507 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:16.507 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.290115+0000", 2024-09-21T22:06:16.507 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:16.507 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.507 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.507 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:16.507 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.290116+0000", 2024-09-21T22:06:16.507 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3829999999999999e-06 2024-09-21T22:06:16.507 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.507 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.507 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:16.507 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.290191+0000", 2024-09-21T22:06:16.508 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.4562999999999997e-05 2024-09-21T22:06:16.508 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.508 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.508 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:16.508 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.290216+0000", 2024-09-21T22:06:16.508 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.565e-05 2024-09-21T22:06:16.508 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.508 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.508 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:16.508 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.290236+0000", 2024-09-21T22:06:16.508 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9326000000000001e-05 2024-09-21T22:06:16.508 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.508 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.508 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:16.508 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.290340+0000", 2024-09-21T22:06:16.509 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0001047 2024-09-21T22:06:16.509 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.509 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.509 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:16.509 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.290377+0000", 2024-09-21T22:06:16.509 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.6356000000000002e-05 2024-09-21T22:06:16.509 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.509 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.509 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:16.509 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.293096+0000", 2024-09-21T22:06:16.509 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.002718996 2024-09-21T22:06:16.509 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.509 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.509 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:16.509 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.293163+0000", 2024-09-21T22:06:16.510 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.6846000000000003e-05 2024-09-21T22:06:16.510 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:16.510 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:16.510 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:16.510 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.510 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.510 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:114 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'76, mlcod=14'73)", 2024-09-21T22:06:16.510 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.490030+0000", 2024-09-21T22:06:16.510 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 13.966778742000001, 2024-09-21T22:06:16.510 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.004227873, 2024-09-21T22:06:16.510 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:16.510 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:16.510 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:16.510 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:16.510 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.511 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:16.511 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.490030+0000", 2024-09-21T22:06:16.511 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:16.511 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.511 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.511 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:16.511 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.490030+0000", 2024-09-21T22:06:16.511 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:16.511 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.511 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.511 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:16.511 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.490031+0000", 2024-09-21T22:06:16.511 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.3300000000000001e-07 2024-09-21T22:06:16.511 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.511 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.512 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:16.512 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.490042+0000", 2024-09-21T22:06:16.512 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.1095e-05 2024-09-21T22:06:16.512 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.512 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.512 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:16.512 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.490053+0000", 2024-09-21T22:06:16.512 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0501e-05 2024-09-21T22:06:16.512 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.512 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.512 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:16.512 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.490060+0000", 2024-09-21T22:06:16.512 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.9829999999999999e-06 2024-09-21T22:06:16.512 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.512 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.512 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:16.512 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.490107+0000", 2024-09-21T22:06:16.513 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.7278999999999999e-05 2024-09-21T22:06:16.513 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.513 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.513 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:16.513 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.490124+0000", 2024-09-21T22:06:16.513 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7190999999999999e-05 2024-09-21T22:06:16.513 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.513 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.513 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:16.513 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.494229+0000", 2024-09-21T22:06:16.513 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0041050440000000004 2024-09-21T22:06:16.513 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.513 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.513 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:16.513 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.494258+0000", 2024-09-21T22:06:16.514 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9047000000000002e-05 2024-09-21T22:06:16.514 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:16.514 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:16.514 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:16.514 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.514 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.514 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:115 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'77, mlcod=14'73)", 2024-09-21T22:06:16.514 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.491380+0000", 2024-09-21T22:06:16.514 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 13.96542912, 2024-09-21T22:06:16.514 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0031550889999999998, 2024-09-21T22:06:16.514 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:16.514 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:16.514 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:16.514 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:16.514 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.515 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:16.515 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.491380+0000", 2024-09-21T22:06:16.515 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:16.515 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.515 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.515 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:16.515 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.491380+0000", 2024-09-21T22:06:16.515 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:16.515 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.517 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.517 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:16.517 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.491381+0000", 2024-09-21T22:06:16.518 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.1399999999999996e-07 2024-09-21T22:06:16.518 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.518 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.518 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:16.518 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.491435+0000", 2024-09-21T22:06:16.518 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.3797e-05 2024-09-21T22:06:16.518 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.518 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.518 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:16.518 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.491447+0000", 2024-09-21T22:06:16.518 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2642e-05 2024-09-21T22:06:16.518 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.518 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.518 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:16.519 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.491453+0000", 2024-09-21T22:06:16.519 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.4820000000000002e-06 2024-09-21T22:06:16.519 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.519 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.519 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:16.519 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.491502+0000", 2024-09-21T22:06:16.519 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.9038e-05 2024-09-21T22:06:16.519 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.519 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.519 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:16.519 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.491514+0000", 2024-09-21T22:06:16.519 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2483999999999999e-05 2024-09-21T22:06:16.519 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.519 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.519 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:16.520 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.494486+0000", 2024-09-21T22:06:16.520 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.002971338 2024-09-21T22:06:16.520 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.520 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.520 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:16.520 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.494535+0000", 2024-09-21T22:06:16.520 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.9493999999999997e-05 2024-09-21T22:06:16.520 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:16.520 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:16.520 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:16.520 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.520 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.520 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:118 1.0 e14/13 1:30306672:devicehealth::main.db.0000000000000000:head v 14'80, mlcod=14'78)", 2024-09-21T22:06:16.520 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.499045+0000", 2024-09-21T22:06:16.520 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 13.957764118, 2024-09-21T22:06:16.520 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0029889589999999998, 2024-09-21T22:06:16.521 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:16.521 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:16.521 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:16.521 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:16.521 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.521 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:16.521 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.499045+0000", 2024-09-21T22:06:16.521 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:16.521 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.521 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.521 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:16.522 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.499045+0000", 2024-09-21T22:06:16.522 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:16.522 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.522 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.522 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:16.522 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.499046+0000", 2024-09-21T22:06:16.522 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.0299999999999998e-07 2024-09-21T22:06:16.522 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.522 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.522 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:16.522 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.499110+0000", 2024-09-21T22:06:16.522 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.4688000000000001e-05 2024-09-21T22:06:16.522 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.522 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.523 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:16.523 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.499125+0000", 2024-09-21T22:06:16.523 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4387e-05 2024-09-21T22:06:16.523 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.523 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.523 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:16.523 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.499131+0000", 2024-09-21T22:06:16.523 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.2129999999999996e-06 2024-09-21T22:06:16.523 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.523 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.523 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:16.523 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.499255+0000", 2024-09-21T22:06:16.523 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00012424199999999999 2024-09-21T22:06:16.523 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.523 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.524 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:16.524 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.499312+0000", 2024-09-21T22:06:16.524 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.6709999999999997e-05 2024-09-21T22:06:16.524 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.524 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.524 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:16.524 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.502005+0000", 2024-09-21T22:06:16.524 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0026927169999999999 2024-09-21T22:06:16.524 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.524 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.524 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:16.524 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.502034+0000", 2024-09-21T22:06:16.524 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9198999999999999e-05 2024-09-21T22:06:16.524 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:16.524 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:16.525 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:16.525 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.525 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.525 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:136 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'94, mlcod=14'89)", 2024-09-21T22:06:16.525 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.903810+0000", 2024-09-21T22:06:16.525 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 13.552999147, 2024-09-21T22:06:16.525 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0062746829999999996, 2024-09-21T22:06:16.525 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:16.525 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:16.525 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:16.525 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:16.525 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.525 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:16.525 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.903810+0000", 2024-09-21T22:06:16.525 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:16.526 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.526 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.526 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:16.526 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.903810+0000", 2024-09-21T22:06:16.526 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:16.526 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.526 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.526 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:16.526 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.903810+0000", 2024-09-21T22:06:16.526 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.03e-07 2024-09-21T22:06:16.526 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.526 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.526 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:16.526 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.903814+0000", 2024-09-21T22:06:16.527 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.676e-06 2024-09-21T22:06:16.527 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.527 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.527 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:16.527 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.903822+0000", 2024-09-21T22:06:16.527 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.2879999999999993e-06 2024-09-21T22:06:16.527 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.527 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.527 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:16.527 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.903829+0000", 2024-09-21T22:06:16.527 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.2920000000000001e-06 2024-09-21T22:06:16.527 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.527 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.527 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:16.528 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.903879+0000", 2024-09-21T22:06:16.528 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.0253999999999997e-05 2024-09-21T22:06:16.528 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.528 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.528 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:16.528 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.903897+0000", 2024-09-21T22:06:16.528 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8468999999999999e-05 2024-09-21T22:06:16.528 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.528 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.528 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:16.528 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.910044+0000", 2024-09-21T22:06:16.528 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0061468950000000003 2024-09-21T22:06:16.528 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.529 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.529 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:16.529 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.910085+0000", 2024-09-21T22:06:16.529 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.0405999999999999e-05 2024-09-21T22:06:16.529 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:16.529 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:16.529 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:16.529 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.529 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.529 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:137 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'95, mlcod=14'90)", 2024-09-21T22:06:16.529 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.904439+0000", 2024-09-21T22:06:16.529 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 13.552369751000001, 2024-09-21T22:06:16.529 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0072625019999999997, 2024-09-21T22:06:16.530 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:16.530 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:16.530 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:16.530 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:16.530 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.530 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:16.530 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.904439+0000", 2024-09-21T22:06:16.530 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:16.530 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.530 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.530 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:16.530 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.904439+0000", 2024-09-21T22:06:16.530 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:16.530 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.531 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.531 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:16.531 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.904440+0000", 2024-09-21T22:06:16.531 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8400000000000005e-07 2024-09-21T22:06:16.531 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.531 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.531 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:16.531 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.904498+0000", 2024-09-21T22:06:16.531 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.8320999999999999e-05 2024-09-21T22:06:16.531 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.531 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.531 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:16.531 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.904507+0000", 2024-09-21T22:06:16.531 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.7790000000000001e-06 2024-09-21T22:06:16.531 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.532 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.532 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:16.532 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.904511+0000", 2024-09-21T22:06:16.532 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.9700000000000001e-06 2024-09-21T22:06:16.532 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.532 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.532 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:16.532 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.904551+0000", 2024-09-21T22:06:16.532 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.9910000000000002e-05 2024-09-21T22:06:16.532 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.532 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.532 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:16.532 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.904567+0000", 2024-09-21T22:06:16.532 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5903999999999999e-05 2024-09-21T22:06:16.533 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.533 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.533 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:16.533 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.911666+0000", 2024-09-21T22:06:16.533 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0070996929999999998 2024-09-21T22:06:16.533 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.533 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.533 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:16.533 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.911702+0000", 2024-09-21T22:06:16.533 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.5441000000000002e-05 2024-09-21T22:06:16.533 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:16.533 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:16.533 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:16.534 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.534 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.534 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:138 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'96, mlcod=14'90)", 2024-09-21T22:06:16.534 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.905402+0000", 2024-09-21T22:06:16.534 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 13.551406774, 2024-09-21T22:06:16.534 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0063366890000000004, 2024-09-21T22:06:16.534 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:16.534 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:16.534 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:16.534 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:16.534 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.534 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:16.534 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905402+0000", 2024-09-21T22:06:16.535 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:16.535 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.535 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.535 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:16.535 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905402+0000", 2024-09-21T22:06:16.535 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:16.535 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.535 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.535 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:16.535 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905403+0000", 2024-09-21T22:06:16.535 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.9200000000000002e-07 2024-09-21T22:06:16.535 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.535 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.535 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:16.535 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905406+0000", 2024-09-21T22:06:16.536 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.326e-06 2024-09-21T22:06:16.536 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.536 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.536 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:16.536 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905413+0000", 2024-09-21T22:06:16.536 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.3320000000000003e-06 2024-09-21T22:06:16.536 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.536 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.536 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:16.536 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905417+0000", 2024-09-21T22:06:16.536 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.7139999999999999e-06 2024-09-21T22:06:16.536 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.536 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.536 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:16.536 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.910115+0000", 2024-09-21T22:06:16.537 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0046983939999999998 2024-09-21T22:06:16.537 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.537 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.537 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:16.537 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.910140+0000", 2024-09-21T22:06:16.537 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4689999999999999e-05 2024-09-21T22:06:16.537 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.537 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.537 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:16.537 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.911706+0000", 2024-09-21T22:06:16.537 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0015658479999999999 2024-09-21T22:06:16.537 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.537 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.537 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:16.538 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.911739+0000", 2024-09-21T22:06:16.538 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.2993000000000001e-05 2024-09-21T22:06:16.538 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:16.538 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:16.538 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:16.538 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.538 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.538 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:139 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'97, mlcod=14'90)", 2024-09-21T22:06:16.538 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.905983+0000", 2024-09-21T22:06:16.538 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 13.550825855999999, 2024-09-21T22:06:16.538 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0057906030000000001, 2024-09-21T22:06:16.538 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:16.538 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:16.539 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:16.539 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:16.539 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.539 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:16.539 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905983+0000", 2024-09-21T22:06:16.539 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:16.539 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.539 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.539 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:16.539 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905983+0000", 2024-09-21T22:06:16.539 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:16.539 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.539 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.539 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:16.540 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905984+0000", 2024-09-21T22:06:16.540 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.15e-07 2024-09-21T22:06:16.540 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.540 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.540 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:16.540 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905986+0000", 2024-09-21T22:06:16.540 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.852e-06 2024-09-21T22:06:16.540 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.540 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.540 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:16.540 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905993+0000", 2024-09-21T22:06:16.540 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.652e-06 2024-09-21T22:06:16.540 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.540 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.540 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:16.541 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905996+0000", 2024-09-21T22:06:16.541 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.3519999999999998e-06 2024-09-21T22:06:16.541 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.541 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.541 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:16.541 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.910519+0000", 2024-09-21T22:06:16.541 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0045230820000000003 2024-09-21T22:06:16.541 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.541 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.541 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:16.541 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.910548+0000", 2024-09-21T22:06:16.541 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8399999999999999e-05 2024-09-21T22:06:16.541 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.541 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.541 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:16.542 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.911742+0000", 2024-09-21T22:06:16.542 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011939909999999999 2024-09-21T22:06:16.542 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.542 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.542 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:16.542 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.911774+0000", 2024-09-21T22:06:16.542 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.1958999999999997e-05 2024-09-21T22:06:16.542 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:16.542 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:16.542 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:16.542 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.542 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.542 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:140 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'98, mlcod=14'90)", 2024-09-21T22:06:16.543 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.906441+0000", 2024-09-21T22:06:16.543 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 13.550367726999999, 2024-09-21T22:06:16.543 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0057052989999999996, 2024-09-21T22:06:16.543 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:16.543 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:16.543 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:16.543 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:16.543 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.543 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:16.543 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.906441+0000", 2024-09-21T22:06:16.543 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:16.543 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.543 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.543 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:16.543 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.906441+0000", 2024-09-21T22:06:16.544 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:16.544 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.544 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.544 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:16.544 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.906442+0000", 2024-09-21T22:06:16.544 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.7e-07 2024-09-21T22:06:16.544 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.544 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.544 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:16.544 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.906511+0000", 2024-09-21T22:06:16.544 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.8857999999999997e-05 2024-09-21T22:06:16.544 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.544 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.545 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:16.545 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.906518+0000", 2024-09-21T22:06:16.545 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.1690000000000003e-06 2024-09-21T22:06:16.545 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.545 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.545 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:16.545 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.906521+0000", 2024-09-21T22:06:16.545 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.4829999999999999e-06 2024-09-21T22:06:16.545 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.545 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.545 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:16.545 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.910939+0000", 2024-09-21T22:06:16.545 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0044176809999999997 2024-09-21T22:06:16.545 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.545 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.546 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:16.546 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.910959+0000", 2024-09-21T22:06:16.546 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9898000000000002e-05 2024-09-21T22:06:16.546 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.546 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.546 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:16.546 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.912113+0000", 2024-09-21T22:06:16.546 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001153811 2024-09-21T22:06:16.546 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:16.546 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:16.546 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:16.546 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.912147+0000", 2024-09-21T22:06:16.546 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.4029000000000002e-05 2024-09-21T22:06:16.547 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:16.547 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:16.547 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:16.547 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:16.547 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:16.547 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:06:16.548 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T22:06:16.549 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:06:16.545+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:06:16.649 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:06:16.644+0000 7fc8d39f7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25294) UID: 0 2024-09-21T22:06:16.722 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:06:16.722 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:06:16.722 INFO:teuthology.orchestra.run.smithi039.stdout: "num_ops": 0 2024-09-21T22:06:16.722 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:06:16.736 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T22:06:16.750 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:06:16.748+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:06:16.851 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:06:16.849+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:06:16.861 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:06:16.861 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:06:16.861 INFO:teuthology.orchestra.run.smithi039.stdout: "complaint_time": 30, 2024-09-21T22:06:16.862 INFO:teuthology.orchestra.run.smithi039.stdout: "num_blocked_ops": 0 2024-09-21T22:06:16.862 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:06:16.875 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T22:06:16.951 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:06:16.948+0000 7fc8d39f7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25294) UID: 0 2024-09-21T22:06:17.046 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:06:17.046 INFO:teuthology.orchestra.run.smithi039.stdout: "size": 20, 2024-09-21T22:06:17.046 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 600, 2024-09-21T22:06:17.046 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [ 2024-09-21T22:06:17.047 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:17.047 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4224.0:5 2.2 e18/16 2:5cea7035:::rbd_info:head v 18'1, mlcod=0'0)", 2024-09-21T22:06:17.047 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:06.114955+0000", 2024-09-21T22:06:17.047 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 10.932139111, 2024-09-21T22:06:17.047 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0017716170000000001, 2024-09-21T22:06:17.047 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:17.047 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:17.047 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:17.047 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:17.047 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:17.048 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:17.048 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:06.114955+0000", 2024-09-21T22:06:17.048 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:17.048 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:17.048 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:17.048 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:17.048 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:06.114955+0000", 2024-09-21T22:06:17.048 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:17.048 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:17.048 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:17.048 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:17.048 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:06.114956+0000", 2024-09-21T22:06:17.048 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.9299999999999997e-07 2024-09-21T22:06:17.048 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:17.049 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:17.049 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:17.049 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:06.114964+0000", 2024-09-21T22:06:17.049 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.9840000000000001e-06 2024-09-21T22:06:17.049 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:17.049 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:17.049 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:17.049 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:06.115001+0000", 2024-09-21T22:06:17.049 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.7719999999999998e-05 2024-09-21T22:06:17.050 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:17.050 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:17.050 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:17.050 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:06.115019+0000", 2024-09-21T22:06:17.050 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7943000000000001e-05 2024-09-21T22:06:17.050 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:17.050 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:17.050 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:17.050 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:06.115168+0000", 2024-09-21T22:06:17.050 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000148589 2024-09-21T22:06:17.050 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:17.051 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:17.051 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:17.051 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:06.115225+0000", 2024-09-21T22:06:17.051 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.6975999999999998e-05 2024-09-21T22:06:17.051 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:17.051 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:17.051 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:17.051 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:06.116585+0000", 2024-09-21T22:06:17.051 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00136038 2024-09-21T22:06:17.051 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:17.051 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:17.051 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:17.052 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:06.116726+0000", 2024-09-21T22:06:17.052 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00014133200000000001 2024-09-21T22:06:17.052 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:17.052 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:17.052 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:17.052 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:17.052 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:17.052 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4224.0:7 2.2 e19/16 2:5cea7035:::rbd_info:head v 19'2, mlcod=0'0)", 2024-09-21T22:06:17.052 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:07.129567+0000", 2024-09-21T22:06:17.052 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 9.9175271019999993, 2024-09-21T22:06:17.052 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0018460239999999999, 2024-09-21T22:06:17.052 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:17.052 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:17.052 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:17.053 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:17.053 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:17.053 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:17.053 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:07.129567+0000", 2024-09-21T22:06:17.053 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:17.053 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:17.053 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:17.053 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:17.053 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:07.129567+0000", 2024-09-21T22:06:17.053 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:17.053 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:17.053 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:17.053 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:17.053 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:07.129567+0000", 2024-09-21T22:06:17.053 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.9499999999999998e-07 2024-09-21T22:06:17.053 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:17.054 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:17.054 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:17.054 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:07.129580+0000", 2024-09-21T22:06:17.054 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2335e-05 2024-09-21T22:06:17.054 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:17.054 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:17.054 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:17.054 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:07.129606+0000", 2024-09-21T22:06:17.054 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.6528000000000001e-05 2024-09-21T22:06:17.054 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:17.054 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:17.054 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:17.054 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:07.129628+0000", 2024-09-21T22:06:17.054 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1549999999999999e-05 2024-09-21T22:06:17.055 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:17.055 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:17.055 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:17.055 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:07.129744+0000", 2024-09-21T22:06:17.055 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000116551 2024-09-21T22:06:17.055 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:17.055 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:17.055 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:17.055 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:07.129780+0000", 2024-09-21T22:06:17.055 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.5447000000000001e-05 2024-09-21T22:06:17.055 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:17.055 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:17.055 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:17.055 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:07.131348+0000", 2024-09-21T22:06:17.056 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001568631 2024-09-21T22:06:17.056 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:17.056 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:17.056 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:17.056 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:07.131413+0000", 2024-09-21T22:06:17.056 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.4586999999999997e-05 2024-09-21T22:06:17.056 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:17.056 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:17.056 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:17.056 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:17.056 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:17.056 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:06:17.058 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:06:17.052+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:06:17.061 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T22:06:17.128 INFO:teuthology.orchestra.run.smithi039.stderr:nodeep-scrub is set 2024-09-21T22:06:17.159 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:06:17.152+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:06:17.192 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:06:17.193 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:06:17.193 INFO:teuthology.orchestra.run.smithi039.stdout: "num_ops": 0 2024-09-21T22:06:17.193 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:06:17.206 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T22:06:17.259 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:06:17.257+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:06:17.335 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:06:17.335 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:06:17.335 INFO:teuthology.orchestra.run.smithi039.stdout: "complaint_time": 30, 2024-09-21T22:06:17.335 INFO:teuthology.orchestra.run.smithi039.stdout: "num_blocked_ops": 0 2024-09-21T22:06:17.335 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:06:17.349 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T22:06:17.360 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:06:17.356+0000 7fc8d39f7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25294) UID: 0 2024-09-21T22:06:17.461 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:06:17.457+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:06:17.509 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:06:17.509 INFO:teuthology.orchestra.run.smithi039.stdout: "size": 20, 2024-09-21T22:06:17.509 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 600, 2024-09-21T22:06:17.509 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [ 2024-09-21T22:06:17.509 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:17.509 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4224.0:1 2.3 e17/16 2:c4c92e5a:::rbd_trash:head v 17'1, mlcod=0'0)", 2024-09-21T22:06:17.509 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:05.107810+0000", 2024-09-21T22:06:17.509 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 12.401919693, 2024-09-21T22:06:17.510 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0017679200000000001, 2024-09-21T22:06:17.510 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:17.510 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:17.510 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:17.510 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:17.510 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:17.510 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:17.510 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.107810+0000", 2024-09-21T22:06:17.510 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:17.510 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:17.510 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:17.510 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:17.510 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.107810+0000", 2024-09-21T22:06:17.510 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:17.511 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:17.511 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:17.511 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:17.511 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.107811+0000", 2024-09-21T22:06:17.511 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.6700000000000003e-07 2024-09-21T22:06:17.511 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:17.511 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:17.511 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:17.511 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.107822+0000", 2024-09-21T22:06:17.511 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0443e-05 2024-09-21T22:06:17.511 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:17.511 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:17.511 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:17.511 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.107853+0000", 2024-09-21T22:06:17.511 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.1046000000000001e-05 2024-09-21T22:06:17.511 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:17.512 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:17.512 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:17.512 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.107876+0000", 2024-09-21T22:06:17.512 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3547e-05 2024-09-21T22:06:17.512 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:17.512 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:17.512 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:17.512 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.108002+0000", 2024-09-21T22:06:17.512 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000125599 2024-09-21T22:06:17.512 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:17.512 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:17.512 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:17.512 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.108069+0000", 2024-09-21T22:06:17.512 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.7162999999999994e-05 2024-09-21T22:06:17.512 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:17.512 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:17.513 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:17.513 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.109385+0000", 2024-09-21T22:06:17.513 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0013162899999999999 2024-09-21T22:06:17.513 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:17.513 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:17.513 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:17.513 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.109578+0000", 2024-09-21T22:06:17.513 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000193065 2024-09-21T22:06:17.513 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:17.513 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:17.513 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:17.513 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:17.513 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:17.513 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:06:17.523 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T22:06:17.566 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:06:17.565+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:06:17.585 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T22:06:17.659 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:06:17.660 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:06:17.660 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-09-21T22:06:17.660 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:06:17.662 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:06:17.661+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:06:17.674 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T22:06:17.726 INFO:teuthology.orchestra.run.smithi039.stdout:osd.0: {} 2024-09-21T22:06:17.726 INFO:teuthology.orchestra.run.smithi039.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T22:06:17.742 INFO:teuthology.orchestra.run.smithi039.stdout:osd.1: {} 2024-09-21T22:06:17.742 INFO:teuthology.orchestra.run.smithi039.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T22:06:17.760 INFO:teuthology.orchestra.run.smithi039.stdout:osd.2: {} 2024-09-21T22:06:17.761 INFO:teuthology.orchestra.run.smithi039.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T22:06:17.763 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:06:17.760+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:06:17.779 INFO:teuthology.orchestra.run.smithi039.stdout:osd.3: {} 2024-09-21T22:06:17.780 INFO:teuthology.orchestra.run.smithi039.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T22:06:17.800 INFO:teuthology.orchestra.run.smithi039.stdout:osd.4: {} 2024-09-21T22:06:17.800 INFO:teuthology.orchestra.run.smithi039.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T22:06:17.801 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:17.801 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:17.816 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T22:06:17.821 INFO:teuthology.orchestra.run.smithi039.stdout:osd.5: {} 2024-09-21T22:06:17.821 INFO:teuthology.orchestra.run.smithi039.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T22:06:17.903 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:06:17.900+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:06:17.965 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:06:17.960+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:06:17.984 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:17.985 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:17.999 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T22:06:18.066 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:06:18.065+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:06:18.161 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:18.161 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:18.167 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:06:18.165+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:06:18.176 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T22:06:18.268 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:06:18.264+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:06:18.303 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:18.303 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:18.317 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T22:06:18.403 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:06:18.400+0000 7fc8d39f7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25294) UID: 0 2024-09-21T22:06:18.469 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:06:18.465+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:06:18.486 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:18.486 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:18.500 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T22:06:18.570 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:06:18.569+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:06:18.662 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:18.663 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:18.671 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:06:18.669+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:06:18.677 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T22:06:18.772 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:06:18.768+0000 7fc8d39f7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25294) UID: 0 2024-09-21T22:06:18.799 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:18.799 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:18.814 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T22:06:18.873 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:06:18.869+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:06:18.974 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:06:18.973+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:06:18.979 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:18.979 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:18.994 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T22:06:19.074 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:06:19.073+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:06:19.131 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:19.131 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:19.147 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T22:06:19.149 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-21T22:06:19.190 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:06:19.189+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:06:19.276 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:06:19.273+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:06:19.309 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:19.309 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:19.324 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T22:06:19.410 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:06:19.409+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:06:19.478 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:06:19.476+0000 7fc8d39f7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25294) UID: 0 2024-09-21T22:06:19.484 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:19.484 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:19.499 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T22:06:19.586 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:06:19.585+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:06:19.668 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:19.669 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:19.680 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:06:19.677+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:06:19.681 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T22:06:19.699 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-09-21T22:06:19.699 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T22:06:19.699 INFO:tasks.thrashosds.thrasher:Removing osd 0, in_osds are: [0, 1, 2, 3, 4, 5] 2024-09-21T22:06:19.699 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 0 2024-09-21T22:06:19.780 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:06:19.777+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:06:19.840 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T22:06:19.847 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:19.847 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:19.863 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T22:06:19.882 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:06:19.876+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:06:19.983 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:06:19.981+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:06:19.991 INFO:teuthology.orchestra.run.smithi039.stdout:osd.0: {} 2024-09-21T22:06:19.991 INFO:teuthology.orchestra.run.smithi039.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T22:06:20.007 INFO:teuthology.orchestra.run.smithi039.stdout:osd.1: {} 2024-09-21T22:06:20.008 INFO:teuthology.orchestra.run.smithi039.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T22:06:20.033 INFO:teuthology.orchestra.run.smithi039.stdout:osd.2: {} 2024-09-21T22:06:20.033 INFO:teuthology.orchestra.run.smithi039.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T22:06:20.034 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:06:20.034 INFO:teuthology.orchestra.run.smithi039.stdout: "size": 20, 2024-09-21T22:06:20.034 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 600, 2024-09-21T22:06:20.034 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [ 2024-09-21T22:06:20.034 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:20.035 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4224.0:5 2.2 e18/16 2:5cea7035:::rbd_info:head v 18'1, mlcod=0'0)", 2024-09-21T22:06:20.035 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:06.114955+0000", 2024-09-21T22:06:20.035 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 13.920355112999999, 2024-09-21T22:06:20.035 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0017716170000000001, 2024-09-21T22:06:20.035 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:20.035 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:20.035 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:20.035 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:20.035 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:20.035 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:20.035 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:06.114955+0000", 2024-09-21T22:06:20.036 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:20.036 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:20.036 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:20.036 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:20.036 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:06.114955+0000", 2024-09-21T22:06:20.036 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:20.036 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:20.036 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:20.036 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:20.036 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:06.114956+0000", 2024-09-21T22:06:20.036 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.9299999999999997e-07 2024-09-21T22:06:20.036 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:20.036 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:20.036 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:20.037 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:06.114964+0000", 2024-09-21T22:06:20.037 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.9840000000000001e-06 2024-09-21T22:06:20.037 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:20.037 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:20.037 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:20.037 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:06.115001+0000", 2024-09-21T22:06:20.037 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.7719999999999998e-05 2024-09-21T22:06:20.037 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:20.037 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:20.037 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:20.037 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:06.115019+0000", 2024-09-21T22:06:20.037 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7943000000000001e-05 2024-09-21T22:06:20.037 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:20.037 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:20.037 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:20.038 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:06.115168+0000", 2024-09-21T22:06:20.038 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000148589 2024-09-21T22:06:20.038 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:20.038 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:20.038 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:20.038 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:06.115225+0000", 2024-09-21T22:06:20.038 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.6975999999999998e-05 2024-09-21T22:06:20.038 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:20.038 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:20.038 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:20.038 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:06.116585+0000", 2024-09-21T22:06:20.038 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00136038 2024-09-21T22:06:20.039 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:20.039 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:20.039 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:20.039 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:06.116726+0000", 2024-09-21T22:06:20.039 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00014133200000000001 2024-09-21T22:06:20.039 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:20.039 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:20.039 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:20.039 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:20.039 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:20.039 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4224.0:7 2.2 e19/16 2:5cea7035:::rbd_info:head v 19'2, mlcod=0'0)", 2024-09-21T22:06:20.039 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:07.129567+0000", 2024-09-21T22:06:20.039 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 12.905743104000001, 2024-09-21T22:06:20.040 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0018460239999999999, 2024-09-21T22:06:20.040 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:20.040 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:20.040 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:20.040 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:20.040 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:20.040 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:20.040 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:07.129567+0000", 2024-09-21T22:06:20.040 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:20.040 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:20.040 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:20.040 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:20.040 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:07.129567+0000", 2024-09-21T22:06:20.041 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:20.041 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:20.041 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:20.041 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:20.041 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:07.129567+0000", 2024-09-21T22:06:20.041 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.9499999999999998e-07 2024-09-21T22:06:20.041 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:20.041 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:20.041 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:20.041 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:07.129580+0000", 2024-09-21T22:06:20.041 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2335e-05 2024-09-21T22:06:20.041 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:20.041 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:20.042 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:20.042 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:07.129606+0000", 2024-09-21T22:06:20.042 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.6528000000000001e-05 2024-09-21T22:06:20.042 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:20.042 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:20.042 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:20.042 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:07.129628+0000", 2024-09-21T22:06:20.042 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1549999999999999e-05 2024-09-21T22:06:20.042 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:20.042 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:20.042 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:20.042 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:07.129744+0000", 2024-09-21T22:06:20.042 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000116551 2024-09-21T22:06:20.043 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:20.043 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:20.043 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:20.043 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:07.129780+0000", 2024-09-21T22:06:20.043 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.5447000000000001e-05 2024-09-21T22:06:20.043 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:20.043 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:20.043 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:20.043 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:07.131348+0000", 2024-09-21T22:06:20.043 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001568631 2024-09-21T22:06:20.043 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:20.043 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:20.043 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:20.043 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:07.131413+0000", 2024-09-21T22:06:20.044 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.4586999999999997e-05 2024-09-21T22:06:20.044 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:20.044 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:20.044 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:20.044 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:20.044 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:20.044 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:06:20.048 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T22:06:20.054 INFO:tasks.ceph.mon.a.smithi039.stderr:2024-09-21T22:06:20.049+0000 7f9967196640 -1 mon.a@0(leader).osd e22 definitely_dead 0 2024-09-21T22:06:20.054 INFO:teuthology.orchestra.run.smithi039.stdout:osd.3: {} 2024-09-21T22:06:20.054 INFO:teuthology.orchestra.run.smithi039.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T22:06:20.074 INFO:teuthology.orchestra.run.smithi039.stdout:osd.4: {} 2024-09-21T22:06:20.075 INFO:teuthology.orchestra.run.smithi039.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T22:06:20.083 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:06:20.080+0000 7fc8d39f7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25294) UID: 0 2024-09-21T22:06:20.094 INFO:teuthology.orchestra.run.smithi039.stdout:osd.5: {} 2024-09-21T22:06:20.095 INFO:teuthology.orchestra.run.smithi039.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T22:06:20.148 INFO:teuthology.orchestra.run.smithi039.stderr:noscrub is unset 2024-09-21T22:06:20.148 INFO:teuthology.orchestra.run.smithi039.stderr:marked out osd.0. 2024-09-21T22:06:20.171 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:06:20.171 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:06:20.171 INFO:teuthology.orchestra.run.smithi039.stdout: "num_ops": 0 2024-09-21T22:06:20.171 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:06:20.185 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T22:06:20.185 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:06:20.181+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:06:20.284 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:06:20.280+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:06:20.337 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:06:20.337 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:06:20.337 INFO:teuthology.orchestra.run.smithi039.stdout: "complaint_time": 30, 2024-09-21T22:06:20.337 INFO:teuthology.orchestra.run.smithi039.stdout: "num_blocked_ops": 0 2024-09-21T22:06:20.337 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:06:20.350 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T22:06:20.385 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:06:20.380+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:06:20.470 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:06:20.470 INFO:teuthology.orchestra.run.smithi039.stdout: "size": 20, 2024-09-21T22:06:20.471 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 600, 2024-09-21T22:06:20.471 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [ 2024-09-21T22:06:20.471 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:20.471 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4224.0:1 2.3 e17/16 2:c4c92e5a:::rbd_trash:head v 17'1, mlcod=0'0)", 2024-09-21T22:06:20.471 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:05.107810+0000", 2024-09-21T22:06:20.471 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 15.363414838000001, 2024-09-21T22:06:20.471 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0017679200000000001, 2024-09-21T22:06:20.471 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:20.471 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:20.471 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:20.471 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:20.471 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:20.471 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:20.471 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.107810+0000", 2024-09-21T22:06:20.471 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:20.472 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:20.472 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:20.472 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:20.472 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.107810+0000", 2024-09-21T22:06:20.472 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:20.472 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:20.472 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:20.472 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:20.472 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.107811+0000", 2024-09-21T22:06:20.472 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.6700000000000003e-07 2024-09-21T22:06:20.472 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:20.472 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:20.472 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:20.472 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.107822+0000", 2024-09-21T22:06:20.472 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0443e-05 2024-09-21T22:06:20.472 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:20.473 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:20.473 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:20.473 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.107853+0000", 2024-09-21T22:06:20.473 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.1046000000000001e-05 2024-09-21T22:06:20.473 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:20.473 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:20.473 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:20.473 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.107876+0000", 2024-09-21T22:06:20.473 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3547e-05 2024-09-21T22:06:20.473 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:20.473 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:20.473 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:20.473 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.108002+0000", 2024-09-21T22:06:20.473 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000125599 2024-09-21T22:06:20.473 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:20.474 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:20.474 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:20.474 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.108069+0000", 2024-09-21T22:06:20.474 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.7162999999999994e-05 2024-09-21T22:06:20.474 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:20.474 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:20.474 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:20.474 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.109385+0000", 2024-09-21T22:06:20.474 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0013162899999999999 2024-09-21T22:06:20.474 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:20.474 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:20.474 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:20.474 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.109578+0000", 2024-09-21T22:06:20.474 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000193065 2024-09-21T22:06:20.474 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:20.474 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:20.475 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:20.475 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:20.475 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:20.475 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:06:20.484 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T22:06:20.485 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:06:20.481+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:06:20.586 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:06:20.585+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:06:20.619 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:06:20.619 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:06:20.620 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-09-21T22:06:20.620 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:06:20.635 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T22:06:20.687 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:06:20.680+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:06:20.754 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:06:20.754 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:06:20.754 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-21T22:06:20.754 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-21T22:06:20.754 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:06:20.768 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T22:06:20.811 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:06:20.808+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:06:20.889 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:06:20.884+0000 7fc8d39f7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25294) UID: 0 2024-09-21T22:06:20.944 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:06:20.944 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-21T22:06:20.944 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-21T22:06:20.944 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-21T22:06:20.944 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.944 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:06:20.945 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.036355+0000", 2024-09-21T22:06:20.945 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 18.903279225999999, 2024-09-21T22:06:20.945 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.062441934999999997, 2024-09-21T22:06:20.945 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:20.945 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:20.945 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:20.945 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:20.945 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:06:20.945 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:06:20.945 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1 2024-09-21T22:06:20.945 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.945 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:20.945 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.945 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:20.946 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036355+0000", 2024-09-21T22:06:20.946 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:20.946 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.946 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.946 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:20.946 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036355+0000", 2024-09-21T22:06:20.946 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:20.946 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.946 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.946 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:20.946 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036357+0000", 2024-09-21T22:06:20.946 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.446e-06 2024-09-21T22:06:20.946 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.946 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.947 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:20.947 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036373+0000", 2024-09-21T22:06:20.947 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5447e-05 2024-09-21T22:06:20.947 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.947 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.947 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:20.947 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036399+0000", 2024-09-21T22:06:20.947 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6423000000000001e-05 2024-09-21T22:06:20.947 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.947 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.947 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:20.947 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036420+0000", 2024-09-21T22:06:20.947 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1129999999999999e-05 2024-09-21T22:06:20.947 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.948 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.948 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:20.948 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036557+0000", 2024-09-21T22:06:20.948 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013686899999999999 2024-09-21T22:06:20.948 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.948 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.948 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-21T22:06:20.948 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036585+0000", 2024-09-21T22:06:20.948 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8362999999999999e-05 2024-09-21T22:06:20.948 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.948 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.948 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:20.948 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.095068+0000", 2024-09-21T22:06:20.948 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058482972000000001 2024-09-21T22:06:20.949 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.949 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.949 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:20.949 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.095516+0000", 2024-09-21T22:06:20.949 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00044802999999999998 2024-09-21T22:06:20.949 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.949 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.949 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:06:20.949 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.095869+0000", 2024-09-21T22:06:20.949 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000352104 2024-09-21T22:06:20.949 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.949 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.949 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:20.950 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.097055+0000", 2024-09-21T22:06:20.950 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011865599999999999 2024-09-21T22:06:20.950 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.950 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.950 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:20.950 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.098637+0000", 2024-09-21T22:06:20.950 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001581458 2024-09-21T22:06:20.950 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.950 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.950 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:20.950 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.098728+0000", 2024-09-21T22:06:20.950 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1043000000000002e-05 2024-09-21T22:06:20.950 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.950 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.950 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:20.951 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.098797+0000", 2024-09-21T22:06:20.951 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9090000000000004e-05 2024-09-21T22:06:20.951 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:20.951 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:20.951 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:20.951 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.951 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.951 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:06:20.951 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.161242+0000", 2024-09-21T22:06:20.951 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 18.778392314000001, 2024-09-21T22:06:20.951 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014477078000000001, 2024-09-21T22:06:20.951 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:20.951 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:20.951 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:20.951 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:20.952 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:06:20.952 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:06:20.952 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 35 2024-09-21T22:06:20.952 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.952 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:20.952 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.952 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:20.952 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161242+0000", 2024-09-21T22:06:20.952 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:20.952 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.952 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.952 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:20.952 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161242+0000", 2024-09-21T22:06:20.952 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:20.953 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.953 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.953 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:20.953 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161243+0000", 2024-09-21T22:06:20.953 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4729999999999999e-06 2024-09-21T22:06:20.953 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.953 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.953 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:20.953 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161385+0000", 2024-09-21T22:06:20.953 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014226899999999999 2024-09-21T22:06:20.953 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.953 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.953 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:20.953 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161417+0000", 2024-09-21T22:06:20.953 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1406999999999999e-05 2024-09-21T22:06:20.954 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.954 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.954 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:20.954 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161434+0000", 2024-09-21T22:06:20.954 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7623000000000001e-05 2024-09-21T22:06:20.954 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.954 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.954 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:20.954 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.169728+0000", 2024-09-21T22:06:20.954 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0082933090000000004 2024-09-21T22:06:20.954 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.954 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.954 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:20.954 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.169861+0000", 2024-09-21T22:06:20.955 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013367300000000001 2024-09-21T22:06:20.955 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.955 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.955 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:06:20.955 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.170186+0000", 2024-09-21T22:06:20.955 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00032466400000000001 2024-09-21T22:06:20.955 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.955 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.955 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:20.955 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.174759+0000", 2024-09-21T22:06:20.955 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0045727880000000004 2024-09-21T22:06:20.955 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.955 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.955 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:20.956 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.175497+0000", 2024-09-21T22:06:20.956 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00073803400000000004 2024-09-21T22:06:20.956 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.956 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.956 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:20.956 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.175607+0000", 2024-09-21T22:06:20.956 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010987199999999999 2024-09-21T22:06:20.956 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.956 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.956 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:20.956 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.175719+0000", 2024-09-21T22:06:20.956 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011196600000000001 2024-09-21T22:06:20.956 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:20.956 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:20.956 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:20.956 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.957 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.957 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:06:20.957 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.161483+0000", 2024-09-21T22:06:20.957 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 18.778150802999999, 2024-09-21T22:06:20.957 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016155732999999999, 2024-09-21T22:06:20.957 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:20.957 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:20.957 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:20.957 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:20.957 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:06:20.957 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:06:20.957 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 36 2024-09-21T22:06:20.957 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.958 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:20.958 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.958 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:20.958 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161483+0000", 2024-09-21T22:06:20.958 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:20.958 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.958 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.958 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:20.958 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161483+0000", 2024-09-21T22:06:20.958 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:20.958 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.958 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.958 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:20.958 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161485+0000", 2024-09-21T22:06:20.958 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5999999999999999e-06 2024-09-21T22:06:20.959 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.959 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.959 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:20.959 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161498+0000", 2024-09-21T22:06:20.959 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3166999999999999e-05 2024-09-21T22:06:20.959 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.959 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.959 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:20.959 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161526+0000", 2024-09-21T22:06:20.959 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7786000000000001e-05 2024-09-21T22:06:20.959 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.959 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.959 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:20.959 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161545+0000", 2024-09-21T22:06:20.960 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8899999999999999e-05 2024-09-21T22:06:20.960 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.960 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.960 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:20.960 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.172615+0000", 2024-09-21T22:06:20.960 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011070465 2024-09-21T22:06:20.960 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.960 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.960 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:20.960 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.172759+0000", 2024-09-21T22:06:20.960 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014416900000000001 2024-09-21T22:06:20.960 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.960 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.960 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:06:20.960 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.172967+0000", 2024-09-21T22:06:20.961 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020758500000000001 2024-09-21T22:06:20.961 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.961 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.961 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:20.961 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.174775+0000", 2024-09-21T22:06:20.961 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0018080990000000001 2024-09-21T22:06:20.961 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.961 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.961 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:20.961 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.177447+0000", 2024-09-21T22:06:20.961 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026717429999999999 2024-09-21T22:06:20.961 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.961 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.961 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:20.961 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.177525+0000", 2024-09-21T22:06:20.962 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7846e-05 2024-09-21T22:06:20.962 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.962 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.962 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:20.962 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.177639+0000", 2024-09-21T22:06:20.962 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000114373 2024-09-21T22:06:20.962 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:20.962 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:20.962 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:20.962 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.962 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.962 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:06:20.962 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.161577+0000", 2024-09-21T22:06:20.962 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 18.778057270000001, 2024-09-21T22:06:20.963 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016652991999999998, 2024-09-21T22:06:20.963 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:20.963 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:20.963 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:20.963 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:20.963 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:06:20.963 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:06:20.963 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 37 2024-09-21T22:06:20.963 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.963 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:20.963 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.963 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:20.963 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161577+0000", 2024-09-21T22:06:20.963 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:20.964 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.964 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.964 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:20.964 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161577+0000", 2024-09-21T22:06:20.964 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:20.964 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.964 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.964 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:20.964 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161578+0000", 2024-09-21T22:06:20.964 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5269999999999999e-06 2024-09-21T22:06:20.964 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.964 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.965 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:20.965 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161591+0000", 2024-09-21T22:06:20.965 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.296e-05 2024-09-21T22:06:20.965 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.965 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.965 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:20.965 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161629+0000", 2024-09-21T22:06:20.965 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7799000000000002e-05 2024-09-21T22:06:20.965 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.965 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.965 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:20.965 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161644+0000", 2024-09-21T22:06:20.965 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5333e-05 2024-09-21T22:06:20.965 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.966 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.966 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:20.966 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.173636+0000", 2024-09-21T22:06:20.966 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011992078 2024-09-21T22:06:20.966 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.966 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.966 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:20.966 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.173862+0000", 2024-09-21T22:06:20.966 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022556799999999999 2024-09-21T22:06:20.966 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.966 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.966 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:06:20.966 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.174193+0000", 2024-09-21T22:06:20.966 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000331088 2024-09-21T22:06:20.967 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.967 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.967 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:20.967 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.175740+0000", 2024-09-21T22:06:20.967 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015467910000000001 2024-09-21T22:06:20.967 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.967 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.967 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:20.967 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.178073+0000", 2024-09-21T22:06:20.967 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002333356 2024-09-21T22:06:20.967 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.967 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.967 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:20.967 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.178162+0000", 2024-09-21T22:06:20.968 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8306000000000005e-05 2024-09-21T22:06:20.968 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.968 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.968 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:20.968 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.178230+0000", 2024-09-21T22:06:20.968 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8186000000000001e-05 2024-09-21T22:06:20.968 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:20.968 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:20.968 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:20.968 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.968 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.968 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:06:20.968 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.280540+0000", 2024-09-21T22:06:20.968 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 18.659093698, 2024-09-21T22:06:20.968 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013246212, 2024-09-21T22:06:20.969 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:20.969 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:20.969 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:20.969 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:20.969 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:06:20.969 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:06:20.969 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 88 2024-09-21T22:06:20.969 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.969 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:20.969 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.969 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:20.969 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.280540+0000", 2024-09-21T22:06:20.969 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:20.969 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.969 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.970 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:20.970 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.280540+0000", 2024-09-21T22:06:20.970 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:20.970 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.970 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.970 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:20.970 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.280541+0000", 2024-09-21T22:06:20.970 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.54e-07 2024-09-21T22:06:20.970 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.970 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.970 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:20.970 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.280549+0000", 2024-09-21T22:06:20.970 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3329999999999997e-06 2024-09-21T22:06:20.970 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.971 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.971 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:20.971 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.280560+0000", 2024-09-21T22:06:20.971 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1003e-05 2024-09-21T22:06:20.971 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.971 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.971 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:20.971 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.280569+0000", 2024-09-21T22:06:20.971 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2430000000000005e-06 2024-09-21T22:06:20.971 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.971 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.971 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:20.971 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.291095+0000", 2024-09-21T22:06:20.972 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010526259 2024-09-21T22:06:20.972 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.972 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.972 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:20.972 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.291231+0000", 2024-09-21T22:06:20.972 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013561899999999999 2024-09-21T22:06:20.972 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.972 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.972 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:06:20.972 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.291457+0000", 2024-09-21T22:06:20.972 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022686200000000001 2024-09-21T22:06:20.972 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.972 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.972 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:20.973 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.293117+0000", 2024-09-21T22:06:20.973 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001659474 2024-09-21T22:06:20.973 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.973 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.973 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:20.973 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.293524+0000", 2024-09-21T22:06:20.973 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00040701300000000001 2024-09-21T22:06:20.973 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.973 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.973 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:20.973 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.293673+0000", 2024-09-21T22:06:20.973 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000148782 2024-09-21T22:06:20.973 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.973 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.973 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:20.974 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.293786+0000", 2024-09-21T22:06:20.974 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011377 2024-09-21T22:06:20.974 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:20.974 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:20.974 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:20.974 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.974 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.974 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:06:20.974 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.466485+0000", 2024-09-21T22:06:20.974 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 18.473148967, 2024-09-21T22:06:20.974 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014443981, 2024-09-21T22:06:20.974 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:20.974 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:20.974 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:20.975 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:20.975 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:06:20.975 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:06:20.975 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 107 2024-09-21T22:06:20.975 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.975 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:20.975 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.975 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:20.975 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.466485+0000", 2024-09-21T22:06:20.975 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:20.975 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.975 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.975 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:20.975 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.466485+0000", 2024-09-21T22:06:20.976 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:20.976 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.976 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.976 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:20.976 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.466486+0000", 2024-09-21T22:06:20.976 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6299999999999993e-07 2024-09-21T22:06:20.976 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.976 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.976 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:20.976 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.466493+0000", 2024-09-21T22:06:20.976 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9639999999999997e-06 2024-09-21T22:06:20.976 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.976 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.976 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:20.976 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.466507+0000", 2024-09-21T22:06:20.977 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4216000000000001e-05 2024-09-21T22:06:20.977 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.977 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.977 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:20.977 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.466515+0000", 2024-09-21T22:06:20.977 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.157e-06 2024-09-21T22:06:20.977 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.977 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.977 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:20.977 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.472033+0000", 2024-09-21T22:06:20.977 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0055175839999999999 2024-09-21T22:06:20.977 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.977 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.977 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:20.978 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.472174+0000", 2024-09-21T22:06:20.978 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000141612 2024-09-21T22:06:20.978 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.978 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.978 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:06:20.978 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.472381+0000", 2024-09-21T22:06:20.978 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000206961 2024-09-21T22:06:20.978 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.978 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.978 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:20.978 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.473924+0000", 2024-09-21T22:06:20.978 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015427119999999999 2024-09-21T22:06:20.978 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.978 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.979 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:20.979 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.480666+0000", 2024-09-21T22:06:20.979 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0067415410000000002 2024-09-21T22:06:20.979 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.979 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.979 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:20.979 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.480811+0000", 2024-09-21T22:06:20.979 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000145422 2024-09-21T22:06:20.979 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.979 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.979 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:20.979 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.480929+0000", 2024-09-21T22:06:20.979 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000117849 2024-09-21T22:06:20.979 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:20.980 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:20.980 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:20.980 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.980 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.980 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:06:20.980 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.467057+0000", 2024-09-21T22:06:20.980 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 18.472576707999998, 2024-09-21T22:06:20.980 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018350195, 2024-09-21T22:06:20.980 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:20.980 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:20.980 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:20.980 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:20.980 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:06:20.980 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:06:20.980 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 108 2024-09-21T22:06:20.981 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.981 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:20.981 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.981 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:20.981 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467057+0000", 2024-09-21T22:06:20.981 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:20.981 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.981 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.981 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:20.981 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467057+0000", 2024-09-21T22:06:20.981 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:20.981 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.981 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.981 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:20.981 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467059+0000", 2024-09-21T22:06:20.982 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.826e-06 2024-09-21T22:06:20.982 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.982 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.982 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:20.982 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467071+0000", 2024-09-21T22:06:20.982 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2224e-05 2024-09-21T22:06:20.982 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.982 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.982 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:20.982 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467098+0000", 2024-09-21T22:06:20.982 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6953000000000001e-05 2024-09-21T22:06:20.982 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.982 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.982 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:20.982 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467113+0000", 2024-09-21T22:06:20.983 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5119999999999999e-05 2024-09-21T22:06:20.983 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.983 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.983 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:20.983 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.474060+0000", 2024-09-21T22:06:20.983 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0069462919999999997 2024-09-21T22:06:20.983 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.983 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.983 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:20.983 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.474283+0000", 2024-09-21T22:06:20.983 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000223315 2024-09-21T22:06:20.983 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.983 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.983 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:06:20.984 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.474660+0000", 2024-09-21T22:06:20.984 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00037729400000000002 2024-09-21T22:06:20.984 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.984 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.984 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:20.984 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.480622+0000", 2024-09-21T22:06:20.984 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0059613369999999997 2024-09-21T22:06:20.984 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.984 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.984 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:20.984 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.485145+0000", 2024-09-21T22:06:20.984 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0045231960000000002 2024-09-21T22:06:20.984 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.984 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.985 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:20.985 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.485288+0000", 2024-09-21T22:06:20.985 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000143232 2024-09-21T22:06:20.985 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.985 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.985 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:20.985 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.485407+0000", 2024-09-21T22:06:20.985 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000119406 2024-09-21T22:06:20.985 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:20.985 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:20.985 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:20.985 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.985 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.985 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:06:20.985 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.467144+0000", 2024-09-21T22:06:20.986 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 18.472489489000001, 2024-09-21T22:06:20.986 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018907146999999999, 2024-09-21T22:06:20.986 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:20.986 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:20.986 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:20.986 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:20.986 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:06:20.986 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:06:20.986 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 109 2024-09-21T22:06:20.986 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.986 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:20.986 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.986 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:20.987 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467144+0000", 2024-09-21T22:06:20.987 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:20.987 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.987 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.987 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:20.987 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467144+0000", 2024-09-21T22:06:20.987 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:20.987 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.987 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.987 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:20.987 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467145+0000", 2024-09-21T22:06:20.987 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9999999999999996e-07 2024-09-21T22:06:20.987 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.987 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.988 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:20.988 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467331+0000", 2024-09-21T22:06:20.988 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018518899999999999 2024-09-21T22:06:20.988 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.988 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.988 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:20.988 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467350+0000", 2024-09-21T22:06:20.988 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8966000000000001e-05 2024-09-21T22:06:20.988 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.988 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.988 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:20.988 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467361+0000", 2024-09-21T22:06:20.988 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1233000000000001e-05 2024-09-21T22:06:20.988 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.988 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.989 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:20.989 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.475613+0000", 2024-09-21T22:06:20.989 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0082518629999999999 2024-09-21T22:06:20.989 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.989 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.989 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:20.989 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.475859+0000", 2024-09-21T22:06:20.989 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024633800000000003 2024-09-21T22:06:20.989 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.989 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.989 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:06:20.989 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.476223+0000", 2024-09-21T22:06:20.989 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00036408700000000003 2024-09-21T22:06:20.989 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.990 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.990 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:20.990 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.481046+0000", 2024-09-21T22:06:20.990 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.004823356 2024-09-21T22:06:20.990 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.990 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.990 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:20.990 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.485722+0000", 2024-09-21T22:06:20.990 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0046753640000000004 2024-09-21T22:06:20.990 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.990 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.990 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:20.990 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.485908+0000", 2024-09-21T22:06:20.990 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000185859 2024-09-21T22:06:20.991 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.991 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.991 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:20.991 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.486052+0000", 2024-09-21T22:06:20.991 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014399200000000001 2024-09-21T22:06:20.991 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:20.991 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:20.991 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:20.991 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.991 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.991 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:06:20.991 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.467384+0000", 2024-09-21T22:06:20.991 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 18.472249660999999, 2024-09-21T22:06:20.991 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.01894144, 2024-09-21T22:06:20.991 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:20.992 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:20.992 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:20.992 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:20.992 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:06:20.992 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:06:20.992 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 110 2024-09-21T22:06:20.992 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.992 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:20.992 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.992 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:20.992 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467384+0000", 2024-09-21T22:06:20.992 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:20.992 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.992 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.993 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:20.993 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467384+0000", 2024-09-21T22:06:20.993 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:20.993 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.993 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.993 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:20.993 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467385+0000", 2024-09-21T22:06:20.993 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0470000000000001e-06 2024-09-21T22:06:20.993 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.993 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.993 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:20.993 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467393+0000", 2024-09-21T22:06:20.994 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3869999999999997e-06 2024-09-21T22:06:20.994 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.994 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.994 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:20.994 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467413+0000", 2024-09-21T22:06:20.994 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9953e-05 2024-09-21T22:06:20.994 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.994 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.994 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:20.994 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467421+0000", 2024-09-21T22:06:20.994 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2400000000000007e-06 2024-09-21T22:06:20.994 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.994 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.994 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:20.995 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.479542+0000", 2024-09-21T22:06:20.995 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012120832 2024-09-21T22:06:20.995 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.995 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.995 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:20.995 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.479705+0000", 2024-09-21T22:06:20.995 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016362899999999999 2024-09-21T22:06:20.995 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.995 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.995 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:06:20.995 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.479914+0000", 2024-09-21T22:06:20.995 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020814499999999999 2024-09-21T22:06:20.995 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.995 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.995 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:20.996 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.485536+0000", 2024-09-21T22:06:20.996 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0056222800000000003 2024-09-21T22:06:20.996 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.996 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.996 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:20.996 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.486077+0000", 2024-09-21T22:06:20.996 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00054079900000000001 2024-09-21T22:06:20.996 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.996 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.996 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:20.996 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.486213+0000", 2024-09-21T22:06:20.996 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000136682 2024-09-21T22:06:20.996 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.996 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.997 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:20.997 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.486326+0000", 2024-09-21T22:06:20.997 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000112446 2024-09-21T22:06:20.997 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:20.997 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:20.997 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:20.997 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.997 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.997 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:06:20.997 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.467436+0000", 2024-09-21T22:06:20.997 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 18.472198161000001, 2024-09-21T22:06:20.997 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019152168000000001, 2024-09-21T22:06:20.997 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:20.997 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:20.997 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:20.998 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:20.998 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:06:20.998 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:06:20.998 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 111 2024-09-21T22:06:20.998 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.998 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:20.998 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.998 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:20.998 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467436+0000", 2024-09-21T22:06:20.998 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:20.998 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.998 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.998 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:20.999 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467436+0000", 2024-09-21T22:06:20.999 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:20.999 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.999 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.999 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:20.999 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467437+0000", 2024-09-21T22:06:20.999 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0699999999999996e-07 2024-09-21T22:06:20.999 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.999 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:20.999 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:20.999 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467445+0000", 2024-09-21T22:06:20.999 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.789e-06 2024-09-21T22:06:20.999 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:20.999 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.000 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:21.000 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467467+0000", 2024-09-21T22:06:21.000 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1253000000000001e-05 2024-09-21T22:06:21.000 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.000 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.000 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:21.000 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467475+0000", 2024-09-21T22:06:21.000 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0039999999999999e-06 2024-09-21T22:06:21.000 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.000 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.000 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:21.000 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.481164+0000", 2024-09-21T22:06:21.000 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013689114 2024-09-21T22:06:21.000 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.001 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.001 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:21.001 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.481394+0000", 2024-09-21T22:06:21.001 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000229931 2024-09-21T22:06:21.001 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.001 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.001 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:06:21.001 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.481779+0000", 2024-09-21T22:06:21.001 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00038545400000000003 2024-09-21T22:06:21.001 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.001 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.001 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:21.001 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.485675+0000", 2024-09-21T22:06:21.001 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0038958370000000001 2024-09-21T22:06:21.002 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.002 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.002 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:21.002 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.486346+0000", 2024-09-21T22:06:21.002 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00067119100000000002 2024-09-21T22:06:21.002 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.002 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.002 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:21.002 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.486478+0000", 2024-09-21T22:06:21.002 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013182600000000001 2024-09-21T22:06:21.002 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.002 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.002 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:21.003 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.486588+0000", 2024-09-21T22:06:21.003 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000110062 2024-09-21T22:06:21.003 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:21.003 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:21.003 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:21.003 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.003 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.003 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:06:21.003 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.467488+0000", 2024-09-21T22:06:21.003 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 18.472145534999999, 2024-09-21T22:06:21.003 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021038304000000001, 2024-09-21T22:06:21.003 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:21.003 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:21.003 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:21.004 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:21.004 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:06:21.004 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:06:21.004 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 112 2024-09-21T22:06:21.004 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.005 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:21.005 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.005 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:21.005 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467488+0000", 2024-09-21T22:06:21.006 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:21.006 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.006 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.006 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:21.006 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467488+0000", 2024-09-21T22:06:21.006 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:21.006 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.006 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.006 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:21.006 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467489+0000", 2024-09-21T22:06:21.006 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7999999999999995e-07 2024-09-21T22:06:21.006 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.006 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.007 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:21.007 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467687+0000", 2024-09-21T22:06:21.007 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019776200000000001 2024-09-21T22:06:21.007 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.007 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.007 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:21.007 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467704+0000", 2024-09-21T22:06:21.007 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7193e-05 2024-09-21T22:06:21.007 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.007 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.007 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:21.007 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467719+0000", 2024-09-21T22:06:21.007 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4703e-05 2024-09-21T22:06:21.007 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.008 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:21.008 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.482745+0000", 2024-09-21T22:06:21.008 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015025801 2024-09-21T22:06:21.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.008 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:21.008 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.483015+0000", 2024-09-21T22:06:21.008 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026999400000000002 2024-09-21T22:06:21.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.008 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:06:21.008 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.483440+0000", 2024-09-21T22:06:21.008 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000425697 2024-09-21T22:06:21.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.009 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.009 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:21.009 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.486701+0000", 2024-09-21T22:06:21.009 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0032611950000000002 2024-09-21T22:06:21.009 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.009 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.009 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:21.009 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.488262+0000", 2024-09-21T22:06:21.009 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015610439999999999 2024-09-21T22:06:21.009 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.009 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.009 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:21.009 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.488406+0000", 2024-09-21T22:06:21.009 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000143232 2024-09-21T22:06:21.009 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.010 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.010 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:21.010 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.488527+0000", 2024-09-21T22:06:21.010 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012100299999999999 2024-09-21T22:06:21.010 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:21.010 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:21.010 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:21.010 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.010 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.010 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:06:21.010 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.467786+0000", 2024-09-21T22:06:21.010 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 18.47184811, 2024-09-21T22:06:21.010 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026339741999999999, 2024-09-21T22:06:21.010 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:21.011 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:21.011 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:21.011 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:21.011 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:06:21.011 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:06:21.011 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 113 2024-09-21T22:06:21.011 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.011 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:21.011 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.011 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:21.011 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467786+0000", 2024-09-21T22:06:21.011 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:21.011 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.011 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.011 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:21.012 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467786+0000", 2024-09-21T22:06:21.012 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:21.012 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.012 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:21.012 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467787+0000", 2024-09-21T22:06:21.012 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.446e-06 2024-09-21T22:06:21.012 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.012 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:21.012 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467802+0000", 2024-09-21T22:06:21.012 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4769999999999999e-05 2024-09-21T22:06:21.012 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.012 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:21.013 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467826+0000", 2024-09-21T22:06:21.013 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3430000000000001e-05 2024-09-21T22:06:21.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.013 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.013 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:21.013 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467843+0000", 2024-09-21T22:06:21.013 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7065999999999999e-05 2024-09-21T22:06:21.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.013 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.013 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:21.013 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.486820+0000", 2024-09-21T22:06:21.013 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018977233999999999 2024-09-21T22:06:21.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.013 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.014 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:21.014 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.487065+0000", 2024-09-21T22:06:21.014 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024558799999999998 2024-09-21T22:06:21.014 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.014 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.014 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:06:21.014 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.487442+0000", 2024-09-21T22:06:21.014 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000376707 2024-09-21T22:06:21.014 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.014 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.014 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:21.014 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.492256+0000", 2024-09-21T22:06:21.014 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0048141620000000003 2024-09-21T22:06:21.014 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.015 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.015 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:21.015 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.493865+0000", 2024-09-21T22:06:21.015 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016090150000000001 2024-09-21T22:06:21.015 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.015 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.015 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:21.015 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.494007+0000", 2024-09-21T22:06:21.015 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000141359 2024-09-21T22:06:21.015 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.015 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.015 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:21.015 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.494126+0000", 2024-09-21T22:06:21.015 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000118965 2024-09-21T22:06:21.015 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:21.016 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:21.016 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:21.016 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.016 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.016 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:06:21.016 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.467878+0000", 2024-09-21T22:06:21.016 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 18.471755621, 2024-09-21T22:06:21.016 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.027783958000000001, 2024-09-21T22:06:21.016 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:21.016 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:21.016 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:21.016 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:21.016 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:06:21.016 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:06:21.017 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 114 2024-09-21T22:06:21.017 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.017 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:21.017 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.017 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:21.017 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467878+0000", 2024-09-21T22:06:21.017 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:21.017 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.017 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.017 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:21.017 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467878+0000", 2024-09-21T22:06:21.017 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:21.017 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.017 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.018 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:21.018 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467880+0000", 2024-09-21T22:06:21.018 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.223e-06 2024-09-21T22:06:21.018 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.018 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.018 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:21.018 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467891+0000", 2024-09-21T22:06:21.018 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1527e-05 2024-09-21T22:06:21.018 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.018 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.018 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:21.018 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467913+0000", 2024-09-21T22:06:21.018 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.143e-05 2024-09-21T22:06:21.018 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.019 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.019 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:21.019 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467930+0000", 2024-09-21T22:06:21.019 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7620000000000001e-05 2024-09-21T22:06:21.019 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.019 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.019 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:21.019 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.488690+0000", 2024-09-21T22:06:21.019 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020759588999999998 2024-09-21T22:06:21.019 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.019 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.019 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:21.019 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.488882+0000", 2024-09-21T22:06:21.019 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019236200000000001 2024-09-21T22:06:21.019 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.020 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.020 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:06:21.020 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.489093+0000", 2024-09-21T22:06:21.020 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000211318 2024-09-21T22:06:21.020 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.020 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.020 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:21.020 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.494152+0000", 2024-09-21T22:06:21.020 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0050582769999999999 2024-09-21T22:06:21.020 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.020 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.020 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:21.020 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.495373+0000", 2024-09-21T22:06:21.020 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001221747 2024-09-21T22:06:21.020 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.021 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.021 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:21.021 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.495543+0000", 2024-09-21T22:06:21.021 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000169259 2024-09-21T22:06:21.021 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.021 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.021 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:21.021 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.495662+0000", 2024-09-21T22:06:21.021 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011960600000000001 2024-09-21T22:06:21.021 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:21.021 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:21.021 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:21.021 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.021 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.021 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:06:21.022 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.468118+0000", 2024-09-21T22:06:21.022 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 18.471515879999998, 2024-09-21T22:06:21.022 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.027938877000000001, 2024-09-21T22:06:21.022 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:21.022 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:21.022 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:21.022 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:21.022 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:06:21.022 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:06:21.022 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 115 2024-09-21T22:06:21.022 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.022 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:21.022 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.023 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:21.023 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468118+0000", 2024-09-21T22:06:21.023 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:21.023 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.023 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.023 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:21.023 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468118+0000", 2024-09-21T22:06:21.023 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:21.023 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.023 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.023 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:21.023 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468120+0000", 2024-09-21T22:06:21.023 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7069999999999999e-06 2024-09-21T22:06:21.023 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.024 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.024 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:21.024 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468381+0000", 2024-09-21T22:06:21.024 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026135399999999999 2024-09-21T22:06:21.024 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.024 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.024 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:21.024 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468409+0000", 2024-09-21T22:06:21.024 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.781e-05 2024-09-21T22:06:21.024 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.024 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.024 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:21.024 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468433+0000", 2024-09-21T22:06:21.024 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4343999999999999e-05 2024-09-21T22:06:21.024 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.025 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.025 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:21.025 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.489785+0000", 2024-09-21T22:06:21.025 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021351221 2024-09-21T22:06:21.025 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.025 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.025 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:21.025 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.490018+0000", 2024-09-21T22:06:21.025 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023317800000000001 2024-09-21T22:06:21.025 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.025 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.025 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:06:21.025 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.490412+0000", 2024-09-21T22:06:21.025 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00039417400000000001 2024-09-21T22:06:21.025 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.026 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.026 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:21.026 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.495694+0000", 2024-09-21T22:06:21.026 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.005282335 2024-09-21T22:06:21.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.026 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.026 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:21.026 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.495852+0000", 2024-09-21T22:06:21.026 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015777200000000001 2024-09-21T22:06:21.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.026 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.026 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:21.026 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.495938+0000", 2024-09-21T22:06:21.027 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6493e-05 2024-09-21T22:06:21.027 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.027 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.027 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:21.027 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.496057+0000", 2024-09-21T22:06:21.027 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000118489 2024-09-21T22:06:21.027 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:21.027 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:21.027 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:21.027 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.027 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.027 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:06:21.027 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.468474+0000", 2024-09-21T22:06:21.027 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 18.471160119, 2024-09-21T22:06:21.028 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.028015895999999998, 2024-09-21T22:06:21.028 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:21.028 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:21.028 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:21.028 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:21.028 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:06:21.028 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:06:21.028 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 116 2024-09-21T22:06:21.028 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.028 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:21.028 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.028 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:21.028 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468474+0000", 2024-09-21T22:06:21.028 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:21.028 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.029 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.029 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:21.029 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468474+0000", 2024-09-21T22:06:21.029 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:21.029 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.029 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.029 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:21.029 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468476+0000", 2024-09-21T22:06:21.029 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7060000000000001e-06 2024-09-21T22:06:21.029 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.029 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.029 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:21.029 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468488+0000", 2024-09-21T22:06:21.029 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.234e-05 2024-09-21T22:06:21.030 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.030 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.030 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:21.030 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468517+0000", 2024-09-21T22:06:21.030 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.898e-05 2024-09-21T22:06:21.030 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.030 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.030 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:21.030 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468536+0000", 2024-09-21T22:06:21.030 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8899999999999999e-05 2024-09-21T22:06:21.030 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.030 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.030 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:21.030 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.492376+0000", 2024-09-21T22:06:21.031 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023840659 2024-09-21T22:06:21.031 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.031 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.031 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:21.031 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.492621+0000", 2024-09-21T22:06:21.031 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000244794 2024-09-21T22:06:21.031 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.031 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.031 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:06:21.031 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.493008+0000", 2024-09-21T22:06:21.031 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00038629099999999998 2024-09-21T22:06:21.031 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.031 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.031 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:21.031 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.496192+0000", 2024-09-21T22:06:21.032 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0031847220000000001 2024-09-21T22:06:21.032 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.032 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.032 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:21.032 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.496233+0000", 2024-09-21T22:06:21.032 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0410000000000001e-05 2024-09-21T22:06:21.032 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.032 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.032 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:21.032 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.496374+0000", 2024-09-21T22:06:21.032 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140872 2024-09-21T22:06:21.032 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.032 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.032 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:21.033 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.496490+0000", 2024-09-21T22:06:21.033 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000116222 2024-09-21T22:06:21.033 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:21.033 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:21.033 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:21.033 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.033 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.033 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:06:21.033 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.468568+0000", 2024-09-21T22:06:21.033 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 18.471066, 2024-09-21T22:06:21.033 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.028181578999999998, 2024-09-21T22:06:21.033 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:21.033 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:21.033 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:21.034 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:21.034 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:06:21.034 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:06:21.034 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 117 2024-09-21T22:06:21.034 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.034 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:21.034 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.034 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:21.034 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468568+0000", 2024-09-21T22:06:21.034 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:21.034 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.034 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.034 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:21.034 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468568+0000", 2024-09-21T22:06:21.035 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:21.035 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.035 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:21.035 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468569+0000", 2024-09-21T22:06:21.035 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4139999999999999e-06 2024-09-21T22:06:21.035 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.035 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:21.035 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468583+0000", 2024-09-21T22:06:21.035 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3409e-05 2024-09-21T22:06:21.035 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.035 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:21.036 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468608+0000", 2024-09-21T22:06:21.036 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5559999999999999e-05 2024-09-21T22:06:21.036 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.036 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.036 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:21.036 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468626+0000", 2024-09-21T22:06:21.036 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7863000000000002e-05 2024-09-21T22:06:21.036 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.036 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.036 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:21.036 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.494260+0000", 2024-09-21T22:06:21.036 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025633744999999999 2024-09-21T22:06:21.036 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.036 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.036 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:21.037 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.494519+0000", 2024-09-21T22:06:21.037 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025924800000000001 2024-09-21T22:06:21.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.037 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.037 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:06:21.037 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.494747+0000", 2024-09-21T22:06:21.037 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022792799999999999 2024-09-21T22:06:21.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.037 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.037 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:21.037 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.496519+0000", 2024-09-21T22:06:21.037 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017720030000000001 2024-09-21T22:06:21.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.037 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.037 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:21.038 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.496607+0000", 2024-09-21T22:06:21.038 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7873000000000004e-05 2024-09-21T22:06:21.038 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.038 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.038 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:21.038 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.496691+0000", 2024-09-21T22:06:21.038 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4009000000000002e-05 2024-09-21T22:06:21.038 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.038 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.038 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:21.038 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.496750+0000", 2024-09-21T22:06:21.038 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8526999999999997e-05 2024-09-21T22:06:21.038 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:21.039 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:21.039 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:21.039 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.039 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.039 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:06:21.039 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.898244+0000", 2024-09-21T22:06:21.039 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 18.041389697, 2024-09-21T22:06:21.039 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014749464, 2024-09-21T22:06:21.039 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:21.039 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:21.039 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:21.039 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:21.039 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:06:21.039 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:06:21.040 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 140 2024-09-21T22:06:21.040 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.040 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:21.040 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.040 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:21.040 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898244+0000", 2024-09-21T22:06:21.040 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:21.040 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.040 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.040 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:21.040 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898244+0000", 2024-09-21T22:06:21.040 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:21.040 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.040 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.040 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:21.041 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898245+0000", 2024-09-21T22:06:21.041 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3000000000000002e-07 2024-09-21T22:06:21.041 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.041 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.041 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:21.041 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898308+0000", 2024-09-21T22:06:21.041 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3468999999999998e-05 2024-09-21T22:06:21.041 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.041 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.041 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:21.041 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898315+0000", 2024-09-21T22:06:21.041 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.725e-06 2024-09-21T22:06:21.041 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.041 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.042 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:21.042 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898321+0000", 2024-09-21T22:06:21.042 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2090000000000002e-06 2024-09-21T22:06:21.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.042 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.042 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:21.042 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.905452+0000", 2024-09-21T22:06:21.042 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0071305989999999996 2024-09-21T22:06:21.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.042 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.042 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:21.042 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.905504+0000", 2024-09-21T22:06:21.042 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2638e-05 2024-09-21T22:06:21.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.043 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.043 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:06:21.043 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.905590+0000", 2024-09-21T22:06:21.043 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5333000000000004e-05 2024-09-21T22:06:21.043 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.043 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.043 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:21.043 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.911542+0000", 2024-09-21T22:06:21.043 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0059519819999999998 2024-09-21T22:06:21.043 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.043 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.043 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:21.043 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.912908+0000", 2024-09-21T22:06:21.044 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0013666640000000001 2024-09-21T22:06:21.044 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.044 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.044 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:21.044 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.912957+0000", 2024-09-21T22:06:21.044 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9203000000000003e-05 2024-09-21T22:06:21.044 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.044 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.044 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:21.044 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.912994+0000", 2024-09-21T22:06:21.044 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6312000000000001e-05 2024-09-21T22:06:21.044 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:21.044 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:21.044 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:21.044 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.045 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.045 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:06:21.045 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.898342+0000", 2024-09-21T22:06:21.045 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 18.041291559000001, 2024-09-21T22:06:21.045 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014787918000000001, 2024-09-21T22:06:21.045 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:21.045 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:21.045 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:21.045 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:21.045 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:06:21.045 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:06:21.045 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 141 2024-09-21T22:06:21.045 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.045 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:21.046 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.046 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:21.046 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898342+0000", 2024-09-21T22:06:21.046 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:21.046 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.046 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.046 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:21.046 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898342+0000", 2024-09-21T22:06:21.046 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:21.046 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.046 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.046 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:21.046 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898343+0000", 2024-09-21T22:06:21.046 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3000000000000001e-07 2024-09-21T22:06:21.047 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.047 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.047 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:21.047 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898346+0000", 2024-09-21T22:06:21.047 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6009999999999999e-06 2024-09-21T22:06:21.047 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.047 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.047 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:21.047 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898352+0000", 2024-09-21T22:06:21.047 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8089999999999998e-06 2024-09-21T22:06:21.047 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.047 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.047 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:21.048 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898358+0000", 2024-09-21T22:06:21.048 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4349999999999996e-06 2024-09-21T22:06:21.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.048 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:21.048 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.909387+0000", 2024-09-21T22:06:21.048 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011029105000000001 2024-09-21T22:06:21.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.048 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:21.048 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.909462+0000", 2024-09-21T22:06:21.048 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5192999999999996e-05 2024-09-21T22:06:21.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.048 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:06:21.049 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.909551+0000", 2024-09-21T22:06:21.049 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9046000000000001e-05 2024-09-21T22:06:21.049 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.049 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.049 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:21.049 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.911852+0000", 2024-09-21T22:06:21.049 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0023005579999999999 2024-09-21T22:06:21.049 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.049 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.049 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:21.049 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.913002+0000", 2024-09-21T22:06:21.049 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011502929999999999 2024-09-21T22:06:21.049 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.049 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.049 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:21.050 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.913060+0000", 2024-09-21T22:06:21.050 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8427000000000002e-05 2024-09-21T22:06:21.050 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.050 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.050 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:21.050 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.913130+0000", 2024-09-21T22:06:21.050 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0020999999999999e-05 2024-09-21T22:06:21.050 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:21.050 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:21.050 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:21.050 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.050 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.050 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:158 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T22:06:21.050 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:03.511201+0000", 2024-09-21T22:06:21.051 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 17.428433076000001, 2024-09-21T22:06:21.051 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013353538999999999, 2024-09-21T22:06:21.051 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:21.051 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:21.051 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:21.051 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:21.051 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:06:21.051 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:06:21.051 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 158 2024-09-21T22:06:21.051 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.051 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:21.051 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.051 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:21.051 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511201+0000", 2024-09-21T22:06:21.052 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:21.052 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.052 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.052 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:21.052 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511201+0000", 2024-09-21T22:06:21.052 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:21.052 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.052 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.052 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:21.052 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511202+0000", 2024-09-21T22:06:21.052 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9299999999999997e-07 2024-09-21T22:06:21.052 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.052 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.052 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:21.052 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511315+0000", 2024-09-21T22:06:21.053 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000113909 2024-09-21T22:06:21.053 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.053 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.053 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:21.053 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511333+0000", 2024-09-21T22:06:21.053 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7062999999999999e-05 2024-09-21T22:06:21.053 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.053 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.053 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:21.053 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511344+0000", 2024-09-21T22:06:21.053 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1177e-05 2024-09-21T22:06:21.053 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.053 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.053 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:21.054 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.517415+0000", 2024-09-21T22:06:21.054 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0060709830000000003 2024-09-21T22:06:21.054 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.054 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.054 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:21.054 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.517539+0000", 2024-09-21T22:06:21.054 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000124432 2024-09-21T22:06:21.054 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.054 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.054 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:06:21.054 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.517732+0000", 2024-09-21T22:06:21.054 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019262200000000001 2024-09-21T22:06:21.054 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.054 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.055 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:21.055 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.520709+0000", 2024-09-21T22:06:21.055 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0029767470000000001 2024-09-21T22:06:21.055 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.055 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.055 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:21.055 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.524389+0000", 2024-09-21T22:06:21.055 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003680975 2024-09-21T22:06:21.055 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.055 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.055 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:21.055 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.524486+0000", 2024-09-21T22:06:21.055 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6575000000000003e-05 2024-09-21T22:06:21.055 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.055 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.056 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:21.056 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.524554+0000", 2024-09-21T22:06:21.056 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8362999999999996e-05 2024-09-21T22:06:21.056 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:21.056 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:21.056 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:21.056 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.056 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.056 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T22:06:21.056 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:03.511368+0000", 2024-09-21T22:06:21.056 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 17.428266174000001, 2024-09-21T22:06:21.056 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013312133, 2024-09-21T22:06:21.056 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:21.056 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:21.057 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:21.057 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:21.057 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:06:21.057 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:06:21.057 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 159 2024-09-21T22:06:21.057 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.057 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:21.057 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.057 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:21.057 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511368+0000", 2024-09-21T22:06:21.057 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:21.057 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.057 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.057 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:21.058 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511368+0000", 2024-09-21T22:06:21.058 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:21.058 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.058 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.058 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:21.058 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511369+0000", 2024-09-21T22:06:21.058 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9599999999999998e-07 2024-09-21T22:06:21.058 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.058 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.058 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:21.058 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511382+0000", 2024-09-21T22:06:21.058 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3487e-05 2024-09-21T22:06:21.058 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.058 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.059 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:21.059 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511394+0000", 2024-09-21T22:06:21.059 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1337e-05 2024-09-21T22:06:21.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.059 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.059 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:21.059 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511402+0000", 2024-09-21T22:06:21.059 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0059999999999992e-06 2024-09-21T22:06:21.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.059 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.059 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:21.059 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.519376+0000", 2024-09-21T22:06:21.059 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0079746520000000005 2024-09-21T22:06:21.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.059 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.060 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:21.060 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.519498+0000", 2024-09-21T22:06:21.060 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012212900000000001 2024-09-21T22:06:21.060 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.060 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.060 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:06:21.060 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.519728+0000", 2024-09-21T22:06:21.060 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022985799999999999 2024-09-21T22:06:21.060 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.060 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.060 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:21.060 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.521344+0000", 2024-09-21T22:06:21.060 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016163169999999999 2024-09-21T22:06:21.060 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.060 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.061 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:21.061 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.524568+0000", 2024-09-21T22:06:21.061 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0032234220000000001 2024-09-21T22:06:21.061 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.061 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.061 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:21.061 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.524633+0000", 2024-09-21T22:06:21.061 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5126e-05 2024-09-21T22:06:21.061 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:21.061 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:21.061 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:21.061 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.524680+0000", 2024-09-21T22:06:21.061 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6903000000000001e-05 2024-09-21T22:06:21.061 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:21.061 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:21.062 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:21.062 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:21.062 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:21.062 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:06:21.063 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T22:06:21.064 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:06:21.060+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:06:21.165 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:06:21.160+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:06:21.232 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:06:21.232 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:06:21.232 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-09-21T22:06:21.232 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:06:21.245 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T22:06:21.265 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:06:21.261+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:06:21.366 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:06:21.360+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:06:21.417 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:06:21.418 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:06:21.418 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-21T22:06:21.418 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-21T22:06:21.418 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:06:21.430 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T22:06:21.467 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:06:21.465+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:06:21.567 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:06:21.565+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:06:21.598 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:06:21.598 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-21T22:06:21.599 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-21T22:06:21.599 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [] 2024-09-21T22:06:21.599 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:06:21.612 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T22:06:21.668 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:06:21.664+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:06:21.732 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:06:21.745 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:06:21.745 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-09-21T22:06:21.745 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:06:21.746 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T22:06:21.787 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:06:21.784+0000 7fc8d39f7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25294) UID: 0 2024-09-21T22:06:21.869 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:06:21.864+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:06:21.912 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:06:21.912 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:06:21.912 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-21T22:06:21.912 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-21T22:06:21.912 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:06:21.928 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T22:06:21.971 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:06:21.968+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:06:22.071 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:06:22.068+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:06:22.098 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:06:22.098 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-21T22:06:22.098 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-21T22:06:22.099 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-21T22:06:22.099 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:22.099 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4224.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-21T22:06:22.099 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:05.105895+0000", 2024-09-21T22:06:22.099 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 16.991425437, 2024-09-21T22:06:22.099 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0034226500000000002, 2024-09-21T22:06:22.099 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:22.099 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:22.099 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:22.099 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:22.099 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4224", 2024-09-21T22:06:22.099 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.39:0/795178185", 2024-09-21T22:06:22.099 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1 2024-09-21T22:06:22.099 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:22.099 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:22.100 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:22.100 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:22.100 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.105895+0000", 2024-09-21T22:06:22.100 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:22.100 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:22.100 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:22.100 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:22.100 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.105895+0000", 2024-09-21T22:06:22.100 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:22.100 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:22.100 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:22.100 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:22.100 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.105905+0000", 2024-09-21T22:06:22.100 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8069999999999994e-06 2024-09-21T22:06:22.100 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:22.101 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:22.101 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:22.101 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.105919+0000", 2024-09-21T22:06:22.101 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3976e-05 2024-09-21T22:06:22.101 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:22.101 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:22.101 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:22.101 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.105955+0000", 2024-09-21T22:06:22.101 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6776999999999997e-05 2024-09-21T22:06:22.101 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:22.101 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:22.101 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:22.102 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.105983+0000", 2024-09-21T22:06:22.102 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7535999999999999e-05 2024-09-21T22:06:22.102 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:22.102 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:22.102 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:22.102 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.106162+0000", 2024-09-21T22:06:22.102 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000178839 2024-09-21T22:06:22.102 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:22.102 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:22.102 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:22.102 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.106437+0000", 2024-09-21T22:06:22.102 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027542099999999998 2024-09-21T22:06:22.102 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:22.102 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:22.103 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-09-21T22:06:22.103 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.106726+0000", 2024-09-21T22:06:22.103 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00028856800000000001 2024-09-21T22:06:22.103 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:22.103 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:22.103 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:22.103 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.108194+0000", 2024-09-21T22:06:22.103 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001468095 2024-09-21T22:06:22.103 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:22.103 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:22.103 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:22.103 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.109130+0000", 2024-09-21T22:06:22.103 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00093653599999999997 2024-09-21T22:06:22.103 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:22.103 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:22.104 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:22.104 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.109216+0000", 2024-09-21T22:06:22.104 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5191999999999993e-05 2024-09-21T22:06:22.104 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:22.104 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:22.104 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:22.104 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.109317+0000", 2024-09-21T22:06:22.104 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000101903 2024-09-21T22:06:22.104 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:22.104 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:22.104 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:22.104 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:22.105 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:22.105 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4224.0:1 2.3 e17/16 ondisk, result = 0)", 2024-09-21T22:06:22.105 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:05.108920+0000", 2024-09-21T22:06:22.105 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 16.98839993, 2024-09-21T22:06:22.105 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00041366599999999998, 2024-09-21T22:06:22.105 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:22.105 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:22.105 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:06:22.105 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:22.105 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:22.105 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:22.105 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.108920+0000", 2024-09-21T22:06:22.105 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:22.105 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:22.105 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:22.106 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:22.106 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.108920+0000", 2024-09-21T22:06:22.106 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:22.106 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:22.106 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:22.106 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:22.106 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.108922+0000", 2024-09-21T22:06:22.106 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.703e-06 2024-09-21T22:06:22.106 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:22.106 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:22.106 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:22.106 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.108938+0000", 2024-09-21T22:06:22.106 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6350000000000001e-05 2024-09-21T22:06:22.107 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:22.107 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:22.107 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:22.107 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.108974+0000", 2024-09-21T22:06:22.107 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5612999999999999e-05 2024-09-21T22:06:22.107 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:22.107 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:22.107 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:22.107 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.108989+0000", 2024-09-21T22:06:22.107 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4576000000000001e-05 2024-09-21T22:06:22.107 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:22.107 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:22.107 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:22.107 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.109108+0000", 2024-09-21T22:06:22.108 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011901899999999999 2024-09-21T22:06:22.108 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:22.108 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:22.108 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:22.108 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.109120+0000", 2024-09-21T22:06:22.108 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.293e-05 2024-09-21T22:06:22.108 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:22.108 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:22.108 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:22.108 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.109334+0000", 2024-09-21T22:06:22.108 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000213475 2024-09-21T22:06:22.108 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:22.108 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:22.108 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:22.108 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:22.109 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:22.109 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4224.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e17)", 2024-09-21T22:06:22.109 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:05.115572+0000", 2024-09-21T22:06:22.109 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 16.981748022000001, 2024-09-21T22:06:22.109 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00041313, 2024-09-21T22:06:22.109 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:22.109 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:22.109 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-21T22:06:22.109 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:22.109 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4224", 2024-09-21T22:06:22.109 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.39:0/795178185", 2024-09-21T22:06:22.109 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2 2024-09-21T22:06:22.109 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:22.109 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:22.109 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:22.110 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:22.110 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.115572+0000", 2024-09-21T22:06:22.110 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:22.110 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:22.110 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:22.110 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:22.110 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.115572+0000", 2024-09-21T22:06:22.110 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:22.110 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:22.110 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:22.110 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:22.110 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.115574+0000", 2024-09-21T22:06:22.110 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0530000000000001e-06 2024-09-21T22:06:22.110 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:22.110 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:22.111 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:22.111 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.115585+0000", 2024-09-21T22:06:22.111 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0947e-05 2024-09-21T22:06:22.111 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:22.111 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:22.111 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:22.111 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.115615+0000", 2024-09-21T22:06:22.111 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9408999999999999e-05 2024-09-21T22:06:22.111 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:22.111 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:22.111 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:22.111 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.115637+0000", 2024-09-21T22:06:22.111 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2073e-05 2024-09-21T22:06:22.111 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:22.111 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:22.112 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:22.112 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.115768+0000", 2024-09-21T22:06:22.112 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013153899999999999 2024-09-21T22:06:22.112 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:22.112 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:22.112 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:22.112 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.115985+0000", 2024-09-21T22:06:22.112 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000217109 2024-09-21T22:06:22.112 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:22.112 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:22.112 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:22.112 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:22.112 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:22.112 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4224.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e17)", 2024-09-21T22:06:22.112 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:05.116456+0000", 2024-09-21T22:06:22.113 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 16.980864476000001, 2024-09-21T22:06:22.113 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00030938500000000002, 2024-09-21T22:06:22.113 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:22.113 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:22.113 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-21T22:06:22.113 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:22.113 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4224", 2024-09-21T22:06:22.113 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.39:0/795178185", 2024-09-21T22:06:22.113 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3 2024-09-21T22:06:22.113 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:22.113 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:22.113 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:22.113 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:22.113 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.116456+0000", 2024-09-21T22:06:22.114 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:22.114 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:22.114 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:22.114 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:22.114 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.116456+0000", 2024-09-21T22:06:22.114 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:22.114 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:22.114 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:22.114 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:22.114 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.116458+0000", 2024-09-21T22:06:22.114 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.77e-06 2024-09-21T22:06:22.114 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:22.114 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:22.114 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:22.114 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.116466+0000", 2024-09-21T22:06:22.115 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1799999999999996e-06 2024-09-21T22:06:22.115 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:22.115 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:22.115 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:22.115 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.116492+0000", 2024-09-21T22:06:22.115 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6523999999999999e-05 2024-09-21T22:06:22.115 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:22.115 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:22.115 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:22.115 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.116513+0000", 2024-09-21T22:06:22.115 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0400000000000001e-05 2024-09-21T22:06:22.115 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:22.115 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:22.115 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:22.116 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.116616+0000", 2024-09-21T22:06:22.116 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000103862 2024-09-21T22:06:22.116 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:22.116 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:22.116 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:22.116 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.116765+0000", 2024-09-21T22:06:22.116 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014864899999999999 2024-09-21T22:06:22.116 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:22.116 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:22.116 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:22.116 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:22.116 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:22.116 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4224.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-21T22:06:22.116 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:06.109464+0000", 2024-09-21T22:06:22.116 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 15.987855798, 2024-09-21T22:06:22.117 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0070431790000000001, 2024-09-21T22:06:22.117 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:22.117 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:22.117 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:22.117 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:22.117 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4224", 2024-09-21T22:06:22.117 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.39:0/795178185", 2024-09-21T22:06:22.117 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 5 2024-09-21T22:06:22.117 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:22.117 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:22.117 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:22.117 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:22.117 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.109464+0000", 2024-09-21T22:06:22.117 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:22.117 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:22.117 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:22.118 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:22.118 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.109464+0000", 2024-09-21T22:06:22.118 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:22.118 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:22.118 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:22.118 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:22.118 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.109467+0000", 2024-09-21T22:06:22.118 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.066e-06 2024-09-21T22:06:22.118 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:22.118 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:22.118 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:22.118 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.109485+0000", 2024-09-21T22:06:22.118 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7464000000000001e-05 2024-09-21T22:06:22.118 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:22.119 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:22.119 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:22.119 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.109518+0000", 2024-09-21T22:06:22.119 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3343000000000002e-05 2024-09-21T22:06:22.119 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:22.119 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:22.119 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:22.119 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.109537+0000", 2024-09-21T22:06:22.119 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8743e-05 2024-09-21T22:06:22.119 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:22.119 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:22.119 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:22.119 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.109708+0000", 2024-09-21T22:06:22.119 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017073899999999999 2024-09-21T22:06:22.120 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:22.120 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:22.120 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op must wait for map", 2024-09-21T22:06:22.120 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.109725+0000", 2024-09-21T22:06:22.120 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7013000000000001e-05 2024-09-21T22:06:22.120 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:22.120 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:22.120 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:22.120 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.113449+0000", 2024-09-21T22:06:22.120 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0037244579999999999 2024-09-21T22:06:22.120 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:22.120 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:22.120 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:22.120 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.113659+0000", 2024-09-21T22:06:22.121 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020968199999999999 2024-09-21T22:06:22.121 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:22.121 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:22.121 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-09-21T22:06:22.121 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.113872+0000", 2024-09-21T22:06:22.121 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000213062 2024-09-21T22:06:22.121 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:22.121 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:22.121 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:22.121 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.115375+0000", 2024-09-21T22:06:22.121 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015028979999999999 2024-09-21T22:06:22.121 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:22.121 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:22.121 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:22.122 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.116356+0000", 2024-09-21T22:06:22.122 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00098098200000000003 2024-09-21T22:06:22.122 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:22.122 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:22.122 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:22.122 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.116442+0000", 2024-09-21T22:06:22.122 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6533000000000007e-05 2024-09-21T22:06:22.122 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:22.122 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:22.122 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:22.122 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.116508+0000", 2024-09-21T22:06:22.122 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5196000000000004e-05 2024-09-21T22:06:22.122 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:22.122 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:22.122 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:22.122 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:22.123 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:22.123 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4224.0:5 2.2 e17/16 ondisk, result = 0)", 2024-09-21T22:06:22.123 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:06.116133+0000", 2024-09-21T22:06:22.123 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 15.981187313, 2024-09-21T22:06:22.123 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00038649700000000002, 2024-09-21T22:06:22.123 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:22.123 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:22.123 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:06:22.123 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:22.123 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:22.123 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:22.123 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.116133+0000", 2024-09-21T22:06:22.123 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:22.123 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:22.124 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:22.124 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:22.124 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.116133+0000", 2024-09-21T22:06:22.124 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:22.124 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:22.124 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:22.124 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:22.124 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.116135+0000", 2024-09-21T22:06:22.124 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8870000000000001e-06 2024-09-21T22:06:22.124 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:22.124 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:22.124 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:22.124 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.116146+0000", 2024-09-21T22:06:22.124 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0953e-05 2024-09-21T22:06:22.124 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:22.124 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:22.125 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:22.125 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.116175+0000", 2024-09-21T22:06:22.125 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9383e-05 2024-09-21T22:06:22.125 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:22.125 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:22.125 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:22.125 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.116190+0000", 2024-09-21T22:06:22.125 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4613000000000001e-05 2024-09-21T22:06:22.125 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:22.125 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:22.125 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:22.125 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.116328+0000", 2024-09-21T22:06:22.125 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013848300000000001 2024-09-21T22:06:22.125 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:22.125 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:22.125 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:22.126 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.116345+0000", 2024-09-21T22:06:22.126 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6883000000000001e-05 2024-09-21T22:06:22.126 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:22.126 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:22.126 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:22.126 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.116519+0000", 2024-09-21T22:06:22.126 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017429500000000001 2024-09-21T22:06:22.126 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:22.126 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:22.126 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:22.126 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:22.126 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:22.126 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4224.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-21T22:06:22.127 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:07.124322+0000", 2024-09-21T22:06:22.127 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 14.972998007999999, 2024-09-21T22:06:22.127 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0067709010000000002, 2024-09-21T22:06:22.127 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:22.127 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:22.127 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:22.127 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:22.127 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4224", 2024-09-21T22:06:22.127 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.39:0/795178185", 2024-09-21T22:06:22.127 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 7 2024-09-21T22:06:22.127 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:22.127 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:22.127 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:22.127 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:22.128 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.124322+0000", 2024-09-21T22:06:22.128 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:22.128 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:22.128 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:22.128 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:22.128 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.124322+0000", 2024-09-21T22:06:22.128 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:22.128 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:22.128 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:22.128 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:22.128 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.124325+0000", 2024-09-21T22:06:22.128 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1489999999999998e-06 2024-09-21T22:06:22.128 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:22.128 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:22.128 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:22.128 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.124342+0000", 2024-09-21T22:06:22.129 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.666e-05 2024-09-21T22:06:22.129 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:22.129 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:22.129 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:22.129 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.124379+0000", 2024-09-21T22:06:22.129 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6539999999999999e-05 2024-09-21T22:06:22.129 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:22.129 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:22.129 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:22.129 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.124406+0000", 2024-09-21T22:06:22.129 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7886e-05 2024-09-21T22:06:22.129 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:22.129 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:22.129 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:22.130 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.124535+0000", 2024-09-21T22:06:22.130 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000128749 2024-09-21T22:06:22.130 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:22.130 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:22.130 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op must wait for map", 2024-09-21T22:06:22.130 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.124547+0000", 2024-09-21T22:06:22.130 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.136e-05 2024-09-21T22:06:22.130 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:22.130 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:22.130 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:22.130 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.127774+0000", 2024-09-21T22:06:22.130 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0032275519999999999 2024-09-21T22:06:22.130 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:22.131 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:22.131 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:22.131 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.128014+0000", 2024-09-21T22:06:22.131 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023956800000000001 2024-09-21T22:06:22.131 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:22.131 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:22.131 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-09-21T22:06:22.131 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.128462+0000", 2024-09-21T22:06:22.131 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00044797700000000001 2024-09-21T22:06:22.131 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:22.131 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:22.131 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:22.131 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.130185+0000", 2024-09-21T22:06:22.131 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00172313 2024-09-21T22:06:22.132 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:22.132 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:22.132 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:22.132 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.130937+0000", 2024-09-21T22:06:22.132 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00075244799999999996 2024-09-21T22:06:22.132 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:22.132 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:22.132 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:22.132 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.131025+0000", 2024-09-21T22:06:22.132 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7998999999999996e-05 2024-09-21T22:06:22.132 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:22.132 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:22.132 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:22.132 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.131093+0000", 2024-09-21T22:06:22.132 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7883e-05 2024-09-21T22:06:22.133 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:22.133 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:22.133 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:22.133 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:22.133 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:22.133 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4224.0:7 2.2 e18/16 ondisk, result = 0)", 2024-09-21T22:06:22.133 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:07.130777+0000", 2024-09-21T22:06:22.133 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 14.966542962, 2024-09-21T22:06:22.133 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00032786799999999999, 2024-09-21T22:06:22.133 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:22.133 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:22.133 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:06:22.133 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:22.133 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:22.133 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:22.134 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.130777+0000", 2024-09-21T22:06:22.134 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:22.134 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:22.134 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:22.134 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:22.134 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.130777+0000", 2024-09-21T22:06:22.134 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:22.134 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:22.134 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:22.134 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:22.134 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.130779+0000", 2024-09-21T22:06:22.134 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7e-06 2024-09-21T22:06:22.134 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:22.135 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:22.135 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:22.135 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.130788+0000", 2024-09-21T22:06:22.135 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6670000000000002e-06 2024-09-21T22:06:22.135 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:22.135 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:22.135 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:22.135 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.130814+0000", 2024-09-21T22:06:22.135 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6590000000000001e-05 2024-09-21T22:06:22.135 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:22.135 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:22.135 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:22.135 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.130828+0000", 2024-09-21T22:06:22.135 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.415e-05 2024-09-21T22:06:22.136 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:22.136 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:22.136 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:22.136 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.130915+0000", 2024-09-21T22:06:22.136 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6759000000000001e-05 2024-09-21T22:06:22.136 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:22.136 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:22.136 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:22.136 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.130928+0000", 2024-09-21T22:06:22.136 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.328e-05 2024-09-21T22:06:22.136 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:22.136 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:22.136 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:22.136 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.131105+0000", 2024-09-21T22:06:22.137 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017672200000000001 2024-09-21T22:06:22.137 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:22.137 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:22.137 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:22.137 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:22.137 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:22.137 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:06:22.137 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T22:06:22.138 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T22:06:22.172 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-21T22:06:22.182 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:06:22.181+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:06:22.270 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:06:22.270 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:06:22.270 INFO:teuthology.orchestra.run.smithi039.stdout: "num_ops": 0 2024-09-21T22:06:22.270 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:06:22.273 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:06:22.268+0000 7fc8d39f7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25294) UID: 0 2024-09-21T22:06:22.288 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T22:06:22.308 INFO:teuthology.orchestra.run.smithi039.stdout:osd.0: {} 2024-09-21T22:06:22.308 INFO:teuthology.orchestra.run.smithi039.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T22:06:22.325 INFO:teuthology.orchestra.run.smithi039.stdout:osd.1: {} 2024-09-21T22:06:22.325 INFO:teuthology.orchestra.run.smithi039.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T22:06:22.344 INFO:teuthology.orchestra.run.smithi039.stdout:osd.2: {} 2024-09-21T22:06:22.344 INFO:teuthology.orchestra.run.smithi039.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T22:06:22.366 INFO:teuthology.orchestra.run.smithi039.stdout:osd.3: {} 2024-09-21T22:06:22.366 INFO:teuthology.orchestra.run.smithi039.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T22:06:22.373 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:06:22.369+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:06:22.387 INFO:teuthology.orchestra.run.smithi039.stdout:osd.4: {} 2024-09-21T22:06:22.387 INFO:teuthology.orchestra.run.smithi039.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T22:06:22.407 INFO:teuthology.orchestra.run.smithi039.stdout:osd.5: {} 2024-09-21T22:06:22.408 INFO:teuthology.orchestra.run.smithi039.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T22:06:22.411 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:22.411 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:22.428 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T22:06:22.473 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:06:22.468+0000 7fc8d39f7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25294) UID: 0 2024-09-21T22:06:22.574 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:06:22.568+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:06:22.599 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:22.599 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:22.614 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T22:06:22.675 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:06:22.668+0000 7fc8d39f7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25294) UID: 0 2024-09-21T22:06:22.741 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:22.741 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:22.756 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T22:06:22.802 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:06:22.801+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:06:22.877 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:06:22.872+0000 7fc8d39f7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25294) UID: 0 2024-09-21T22:06:22.882 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:22.882 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:22.897 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T22:06:22.982 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:06:22.981+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:06:23.057 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:23.058 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:23.071 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T22:06:23.114 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:06:23.113+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:06:23.179 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:06:23.177+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:06:23.188 INFO:teuthology.orchestra.run.smithi039.stderr:nodeep-scrub is unset 2024-09-21T22:06:23.240 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:23.240 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:23.254 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T22:06:23.281 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:06:23.280+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:06:23.382 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:06:23.381+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:06:23.417 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:23.417 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:23.431 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T22:06:23.482 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:06:23.480+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:06:23.583 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:06:23.580+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:06:23.588 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:23.589 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:23.602 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T22:06:23.684 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:06:23.680+0000 7fc8d39f7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25294) UID: 0 2024-09-21T22:06:23.736 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:23.736 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:23.751 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T22:06:23.795 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:06:23.792+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:06:23.885 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:06:23.880+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:06:23.924 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:23.924 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:23.938 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T22:06:23.986 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:06:23.985+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:06:24.087 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:06:24.085+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:06:24.106 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:24.106 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:24.121 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T22:06:24.188 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:06:24.185+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:06:24.242 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:24.242 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:24.257 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T22:06:24.303 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:06:24.300+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:06:24.384 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:24.384 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:24.390 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:06:24.384+0000 7fc8d39f7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25294) UID: 0 2024-09-21T22:06:24.399 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T22:06:24.428 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T22:06:24.491 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:06:24.489+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:06:24.575 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:24.575 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:24.588 INFO:teuthology.orchestra.run.smithi039.stdout:osd.0: {} 2024-09-21T22:06:24.588 INFO:teuthology.orchestra.run.smithi039.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T22:06:24.589 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T22:06:24.591 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:06:24.588+0000 7fc8d39f7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25294) UID: 0 2024-09-21T22:06:24.604 INFO:teuthology.orchestra.run.smithi039.stdout:osd.1: {} 2024-09-21T22:06:24.605 INFO:teuthology.orchestra.run.smithi039.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T22:06:24.621 INFO:teuthology.orchestra.run.smithi039.stdout:osd.2: {} 2024-09-21T22:06:24.621 INFO:teuthology.orchestra.run.smithi039.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T22:06:24.639 INFO:teuthology.orchestra.run.smithi039.stdout:osd.3: {} 2024-09-21T22:06:24.639 INFO:teuthology.orchestra.run.smithi039.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T22:06:24.657 INFO:teuthology.orchestra.run.smithi039.stdout:osd.4: {} 2024-09-21T22:06:24.657 INFO:teuthology.orchestra.run.smithi039.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T22:06:24.677 INFO:teuthology.orchestra.run.smithi039.stdout:osd.5: {} 2024-09-21T22:06:24.677 INFO:teuthology.orchestra.run.smithi039.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T22:06:24.691 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:06:24.689+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:06:24.717 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:06:24.717 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:06:24.718 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-09-21T22:06:24.718 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:06:24.732 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T22:06:24.792 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:06:24.788+0000 7fc8d39f7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25294) UID: 0 2024-09-21T22:06:24.857 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:06:24.857 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:06:24.858 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-21T22:06:24.858 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-21T22:06:24.858 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:06:24.872 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T22:06:24.915 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:06:24.912+0000 7fc8d39f7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25294) UID: 0 2024-09-21T22:06:24.995 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:06:24.992+0000 7fc8d39f7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25294) UID: 0 2024-09-21T22:06:25.043 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:06:25.043 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-21T22:06:25.043 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-21T22:06:25.043 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-21T22:06:25.043 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:25.043 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4224.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-21T22:06:25.043 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:05.105895+0000", 2024-09-21T22:06:25.043 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 19.936593039999998, 2024-09-21T22:06:25.043 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0034226500000000002, 2024-09-21T22:06:25.043 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:25.044 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:25.044 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:25.044 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:25.044 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4224", 2024-09-21T22:06:25.044 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.39:0/795178185", 2024-09-21T22:06:25.044 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1 2024-09-21T22:06:25.044 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:25.044 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:25.044 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:25.044 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:25.044 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.105895+0000", 2024-09-21T22:06:25.044 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:25.044 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:25.044 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:25.044 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:25.045 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.105895+0000", 2024-09-21T22:06:25.045 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:25.045 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:25.045 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:25.045 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:25.045 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.105905+0000", 2024-09-21T22:06:25.045 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8069999999999994e-06 2024-09-21T22:06:25.045 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:25.045 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:25.045 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:25.045 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.105919+0000", 2024-09-21T22:06:25.045 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3976e-05 2024-09-21T22:06:25.045 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:25.046 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:25.046 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:25.046 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.105955+0000", 2024-09-21T22:06:25.046 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6776999999999997e-05 2024-09-21T22:06:25.046 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:25.046 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:25.046 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:25.046 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.105983+0000", 2024-09-21T22:06:25.046 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7535999999999999e-05 2024-09-21T22:06:25.046 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:25.046 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:25.046 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:25.046 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.106162+0000", 2024-09-21T22:06:25.047 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000178839 2024-09-21T22:06:25.047 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:25.047 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:25.047 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:25.047 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.106437+0000", 2024-09-21T22:06:25.047 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027542099999999998 2024-09-21T22:06:25.047 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:25.047 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:25.047 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-09-21T22:06:25.047 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.106726+0000", 2024-09-21T22:06:25.047 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00028856800000000001 2024-09-21T22:06:25.047 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:25.047 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:25.048 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:25.048 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.108194+0000", 2024-09-21T22:06:25.048 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001468095 2024-09-21T22:06:25.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:25.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:25.048 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:25.048 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.109130+0000", 2024-09-21T22:06:25.048 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00093653599999999997 2024-09-21T22:06:25.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:25.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:25.048 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:25.048 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.109216+0000", 2024-09-21T22:06:25.048 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5191999999999993e-05 2024-09-21T22:06:25.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:25.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:25.049 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:25.049 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.109317+0000", 2024-09-21T22:06:25.049 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000101903 2024-09-21T22:06:25.049 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:25.049 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:25.049 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:25.049 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:25.049 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:25.049 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4224.0:1 2.3 e17/16 ondisk, result = 0)", 2024-09-21T22:06:25.049 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:05.108920+0000", 2024-09-21T22:06:25.049 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 19.933567533000002, 2024-09-21T22:06:25.049 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00041366599999999998, 2024-09-21T22:06:25.049 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:25.049 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:25.050 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:06:25.050 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:25.050 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:25.050 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:25.050 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.108920+0000", 2024-09-21T22:06:25.050 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:25.050 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:25.050 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:25.050 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:25.050 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.108920+0000", 2024-09-21T22:06:25.050 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:25.050 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:25.050 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:25.051 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:25.051 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.108922+0000", 2024-09-21T22:06:25.051 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.703e-06 2024-09-21T22:06:25.051 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:25.051 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:25.051 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:25.051 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.108938+0000", 2024-09-21T22:06:25.051 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6350000000000001e-05 2024-09-21T22:06:25.051 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:25.051 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:25.051 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:25.051 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.108974+0000", 2024-09-21T22:06:25.051 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5612999999999999e-05 2024-09-21T22:06:25.051 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:25.051 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:25.052 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:25.052 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.108989+0000", 2024-09-21T22:06:25.052 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4576000000000001e-05 2024-09-21T22:06:25.052 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:25.052 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:25.052 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:25.052 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.109108+0000", 2024-09-21T22:06:25.052 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011901899999999999 2024-09-21T22:06:25.052 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:25.052 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:25.052 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:25.052 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.109120+0000", 2024-09-21T22:06:25.052 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.293e-05 2024-09-21T22:06:25.052 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:25.053 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:25.053 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:25.053 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.109334+0000", 2024-09-21T22:06:25.053 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000213475 2024-09-21T22:06:25.053 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:25.053 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:25.053 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:25.053 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:25.053 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:25.053 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4224.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e17)", 2024-09-21T22:06:25.053 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:05.115572+0000", 2024-09-21T22:06:25.053 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 19.926915624999999, 2024-09-21T22:06:25.053 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00041313, 2024-09-21T22:06:25.053 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:25.054 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:25.054 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-21T22:06:25.054 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:25.054 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4224", 2024-09-21T22:06:25.054 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.39:0/795178185", 2024-09-21T22:06:25.054 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2 2024-09-21T22:06:25.054 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:25.054 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:25.054 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:25.054 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:25.054 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.115572+0000", 2024-09-21T22:06:25.054 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:25.054 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:25.054 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:25.055 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:25.055 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.115572+0000", 2024-09-21T22:06:25.055 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:25.055 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:25.055 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:25.055 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:25.055 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.115574+0000", 2024-09-21T22:06:25.055 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0530000000000001e-06 2024-09-21T22:06:25.055 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:25.055 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:25.055 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:25.055 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.115585+0000", 2024-09-21T22:06:25.055 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0947e-05 2024-09-21T22:06:25.055 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:25.056 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:25.056 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:25.056 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.115615+0000", 2024-09-21T22:06:25.056 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9408999999999999e-05 2024-09-21T22:06:25.056 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:25.056 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:25.056 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:25.056 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.115637+0000", 2024-09-21T22:06:25.056 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2073e-05 2024-09-21T22:06:25.056 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:25.056 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:25.056 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:25.056 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.115768+0000", 2024-09-21T22:06:25.056 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013153899999999999 2024-09-21T22:06:25.056 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:25.057 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:25.057 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:25.057 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.115985+0000", 2024-09-21T22:06:25.057 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000217109 2024-09-21T22:06:25.057 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:25.057 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:25.057 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:25.057 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:25.057 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:25.057 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4224.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e17)", 2024-09-21T22:06:25.057 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:05.116456+0000", 2024-09-21T22:06:25.057 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 19.926032078999999, 2024-09-21T22:06:25.057 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00030938500000000002, 2024-09-21T22:06:25.058 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:25.058 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:25.058 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-21T22:06:25.058 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:25.058 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4224", 2024-09-21T22:06:25.058 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.39:0/795178185", 2024-09-21T22:06:25.058 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3 2024-09-21T22:06:25.058 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:25.058 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:25.058 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:25.058 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:25.058 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.116456+0000", 2024-09-21T22:06:25.058 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:25.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:25.059 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:25.059 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:25.059 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.116456+0000", 2024-09-21T22:06:25.059 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:25.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:25.059 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:25.059 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:25.059 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.116458+0000", 2024-09-21T22:06:25.059 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.77e-06 2024-09-21T22:06:25.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:25.059 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:25.059 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:25.059 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.116466+0000", 2024-09-21T22:06:25.059 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1799999999999996e-06 2024-09-21T22:06:25.060 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:25.060 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:25.060 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:25.060 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.116492+0000", 2024-09-21T22:06:25.060 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6523999999999999e-05 2024-09-21T22:06:25.060 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:25.060 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:25.060 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:25.060 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.116513+0000", 2024-09-21T22:06:25.060 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0400000000000001e-05 2024-09-21T22:06:25.060 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:25.060 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:25.060 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:25.060 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.116616+0000", 2024-09-21T22:06:25.060 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000103862 2024-09-21T22:06:25.060 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:25.061 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:25.061 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:25.061 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.116765+0000", 2024-09-21T22:06:25.061 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014864899999999999 2024-09-21T22:06:25.061 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:25.061 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:25.061 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:25.061 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:25.061 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:25.061 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4224.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-21T22:06:25.061 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:06.109464+0000", 2024-09-21T22:06:25.061 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 18.933023401, 2024-09-21T22:06:25.061 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0070431790000000001, 2024-09-21T22:06:25.061 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:25.061 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:25.061 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:25.062 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:25.062 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4224", 2024-09-21T22:06:25.062 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.39:0/795178185", 2024-09-21T22:06:25.062 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 5 2024-09-21T22:06:25.062 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:25.062 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:25.062 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:25.062 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:25.062 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.109464+0000", 2024-09-21T22:06:25.062 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:25.062 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:25.062 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:25.062 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:25.062 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.109464+0000", 2024-09-21T22:06:25.062 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:25.063 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:25.063 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:25.063 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:25.063 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.109467+0000", 2024-09-21T22:06:25.063 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.066e-06 2024-09-21T22:06:25.063 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:25.063 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:25.063 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:25.063 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.109485+0000", 2024-09-21T22:06:25.063 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7464000000000001e-05 2024-09-21T22:06:25.063 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:25.063 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:25.063 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:25.063 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.109518+0000", 2024-09-21T22:06:25.063 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3343000000000002e-05 2024-09-21T22:06:25.063 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:25.064 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:25.064 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:25.064 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.109537+0000", 2024-09-21T22:06:25.064 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8743e-05 2024-09-21T22:06:25.064 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:25.064 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:25.064 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:25.064 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.109708+0000", 2024-09-21T22:06:25.064 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017073899999999999 2024-09-21T22:06:25.064 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:25.064 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:25.064 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op must wait for map", 2024-09-21T22:06:25.064 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.109725+0000", 2024-09-21T22:06:25.065 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7013000000000001e-05 2024-09-21T22:06:25.065 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:25.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:25.065 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:25.065 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.113449+0000", 2024-09-21T22:06:25.065 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0037244579999999999 2024-09-21T22:06:25.065 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:25.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:25.065 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:25.065 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.113659+0000", 2024-09-21T22:06:25.065 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020968199999999999 2024-09-21T22:06:25.065 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:25.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:25.065 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-09-21T22:06:25.065 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.113872+0000", 2024-09-21T22:06:25.065 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000213062 2024-09-21T22:06:25.066 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:25.066 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:25.066 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:25.066 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.115375+0000", 2024-09-21T22:06:25.066 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015028979999999999 2024-09-21T22:06:25.066 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:25.066 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:25.066 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:25.066 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.116356+0000", 2024-09-21T22:06:25.066 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00098098200000000003 2024-09-21T22:06:25.066 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:25.066 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:25.066 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:25.066 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.116442+0000", 2024-09-21T22:06:25.066 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6533000000000007e-05 2024-09-21T22:06:25.067 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:25.067 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:25.067 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:25.067 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.116508+0000", 2024-09-21T22:06:25.067 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5196000000000004e-05 2024-09-21T22:06:25.067 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:25.067 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:25.067 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:25.067 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:25.067 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:25.067 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4224.0:5 2.2 e17/16 ondisk, result = 0)", 2024-09-21T22:06:25.067 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:06.116133+0000", 2024-09-21T22:06:25.068 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 18.926354916000001, 2024-09-21T22:06:25.068 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00038649700000000002, 2024-09-21T22:06:25.068 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:25.068 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:25.068 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:06:25.068 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:25.068 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:25.068 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:25.068 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.116133+0000", 2024-09-21T22:06:25.068 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:25.068 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:25.068 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:25.068 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:25.068 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.116133+0000", 2024-09-21T22:06:25.068 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:25.069 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:25.069 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:25.069 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:25.069 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.116135+0000", 2024-09-21T22:06:25.069 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8870000000000001e-06 2024-09-21T22:06:25.069 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:25.069 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:25.069 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:25.069 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.116146+0000", 2024-09-21T22:06:25.069 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0953e-05 2024-09-21T22:06:25.069 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:25.069 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:25.069 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:25.069 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.116175+0000", 2024-09-21T22:06:25.070 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9383e-05 2024-09-21T22:06:25.070 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:25.070 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:25.070 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:25.070 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.116190+0000", 2024-09-21T22:06:25.070 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4613000000000001e-05 2024-09-21T22:06:25.070 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:25.070 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:25.070 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:25.070 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.116328+0000", 2024-09-21T22:06:25.070 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013848300000000001 2024-09-21T22:06:25.070 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:25.070 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:25.070 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:25.071 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.116345+0000", 2024-09-21T22:06:25.071 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6883000000000001e-05 2024-09-21T22:06:25.071 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:25.071 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:25.071 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:25.071 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.116519+0000", 2024-09-21T22:06:25.071 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017429500000000001 2024-09-21T22:06:25.071 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:25.071 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:25.071 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:25.071 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:25.071 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:25.071 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4224.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-21T22:06:25.071 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:07.124322+0000", 2024-09-21T22:06:25.072 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 17.918165610999999, 2024-09-21T22:06:25.072 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0067709010000000002, 2024-09-21T22:06:25.072 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:25.072 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:25.072 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:25.072 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:25.072 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4224", 2024-09-21T22:06:25.072 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.39:0/795178185", 2024-09-21T22:06:25.072 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 7 2024-09-21T22:06:25.072 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:25.072 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:25.072 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:25.072 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:25.072 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.124322+0000", 2024-09-21T22:06:25.073 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:25.073 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:25.073 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:25.073 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:25.073 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.124322+0000", 2024-09-21T22:06:25.073 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:25.073 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:25.073 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:25.073 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:25.073 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.124325+0000", 2024-09-21T22:06:25.073 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1489999999999998e-06 2024-09-21T22:06:25.073 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:25.073 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:25.073 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:25.073 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.124342+0000", 2024-09-21T22:06:25.074 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.666e-05 2024-09-21T22:06:25.074 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:25.074 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:25.074 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:25.074 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.124379+0000", 2024-09-21T22:06:25.074 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6539999999999999e-05 2024-09-21T22:06:25.074 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:25.074 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:25.074 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:25.074 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.124406+0000", 2024-09-21T22:06:25.074 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7886e-05 2024-09-21T22:06:25.074 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:25.074 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:25.074 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:25.075 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.124535+0000", 2024-09-21T22:06:25.075 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000128749 2024-09-21T22:06:25.075 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:25.075 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:25.075 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op must wait for map", 2024-09-21T22:06:25.075 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.124547+0000", 2024-09-21T22:06:25.075 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.136e-05 2024-09-21T22:06:25.075 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:25.075 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:25.075 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:25.075 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.127774+0000", 2024-09-21T22:06:25.075 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0032275519999999999 2024-09-21T22:06:25.075 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:25.075 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:25.076 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:25.076 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.128014+0000", 2024-09-21T22:06:25.076 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023956800000000001 2024-09-21T22:06:25.076 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:25.076 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:25.076 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-09-21T22:06:25.076 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.128462+0000", 2024-09-21T22:06:25.076 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00044797700000000001 2024-09-21T22:06:25.076 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:25.076 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:25.076 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:25.076 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.130185+0000", 2024-09-21T22:06:25.076 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00172313 2024-09-21T22:06:25.076 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:25.076 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:25.076 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:25.077 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.130937+0000", 2024-09-21T22:06:25.077 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00075244799999999996 2024-09-21T22:06:25.077 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:25.077 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:25.077 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:25.077 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.131025+0000", 2024-09-21T22:06:25.077 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7998999999999996e-05 2024-09-21T22:06:25.077 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:25.077 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:25.077 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:25.077 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.131093+0000", 2024-09-21T22:06:25.077 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7883e-05 2024-09-21T22:06:25.077 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:25.077 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:25.077 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:25.078 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:25.078 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:25.078 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4224.0:7 2.2 e18/16 ondisk, result = 0)", 2024-09-21T22:06:25.078 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:07.130777+0000", 2024-09-21T22:06:25.078 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 17.911710565, 2024-09-21T22:06:25.078 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00032786799999999999, 2024-09-21T22:06:25.078 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:25.078 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:25.078 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:06:25.078 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:25.078 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:25.078 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:25.078 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.130777+0000", 2024-09-21T22:06:25.078 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:25.078 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:25.079 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:25.079 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:25.079 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.130777+0000", 2024-09-21T22:06:25.079 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:25.079 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:25.079 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:25.079 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:25.079 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.130779+0000", 2024-09-21T22:06:25.079 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7e-06 2024-09-21T22:06:25.079 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:25.079 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:25.079 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:25.079 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.130788+0000", 2024-09-21T22:06:25.079 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6670000000000002e-06 2024-09-21T22:06:25.080 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:25.080 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:25.080 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:25.080 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.130814+0000", 2024-09-21T22:06:25.080 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6590000000000001e-05 2024-09-21T22:06:25.080 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:25.080 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:25.080 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:25.080 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.130828+0000", 2024-09-21T22:06:25.080 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.415e-05 2024-09-21T22:06:25.080 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:25.080 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:25.080 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:25.080 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.130915+0000", 2024-09-21T22:06:25.080 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6759000000000001e-05 2024-09-21T22:06:25.080 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:25.081 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:25.081 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:25.081 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.130928+0000", 2024-09-21T22:06:25.081 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.328e-05 2024-09-21T22:06:25.081 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:25.081 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:25.081 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:25.081 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.131105+0000", 2024-09-21T22:06:25.081 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017672200000000001 2024-09-21T22:06:25.081 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:25.081 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:25.081 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:25.081 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:25.081 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:25.081 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:06:25.083 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T22:06:25.131 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:06:25.129+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:06:25.173 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-09-21T22:06:25.173 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T22:06:25.173 INFO:tasks.thrashosds.thrasher:inject_pause on osd.2 2024-09-21T22:06:25.174 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-21T22:06:25.174 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-21T22:06:25.174 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok config set filestore_inject_stall 3 2024-09-21T22:06:25.197 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:06:25.192+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:06:25.210 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-21T22:06:25.216 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:06:25.216 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:06:25.216 INFO:teuthology.orchestra.run.smithi039.stdout: "num_ops": 0 2024-09-21T22:06:25.217 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:06:25.235 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T22:06:25.298 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:06:25.296+0000 7fc8d39f7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25294) UID: 0 2024-09-21T22:06:25.314 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:06:25.314 INFO:teuthology.orchestra.run.smithi039.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-21T22:06:25.314 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:06:25.398 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:06:25.397+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:06:25.403 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:06:25.403 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:06:25.403 INFO:teuthology.orchestra.run.smithi039.stdout: "complaint_time": 30, 2024-09-21T22:06:25.403 INFO:teuthology.orchestra.run.smithi039.stdout: "num_blocked_ops": 0 2024-09-21T22:06:25.403 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:06:25.418 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T22:06:25.499 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:06:25.496+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:06:25.582 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:06:25.582 INFO:teuthology.orchestra.run.smithi039.stdout: "size": 20, 2024-09-21T22:06:25.582 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 600, 2024-09-21T22:06:25.582 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [ 2024-09-21T22:06:25.582 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.582 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:20 1.0 e14/13 1:30306672:devicehealth::main.db.0000000000000000:head v 14'10, mlcod=14'8)", 2024-09-21T22:06:25.583 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.137639+0000", 2024-09-21T22:06:25.583 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 23.443715512000001, 2024-09-21T22:06:25.583 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0039902100000000001, 2024-09-21T22:06:25.583 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:25.583 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:25.583 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:25.583 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:25.583 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.583 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:25.583 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.137639+0000", 2024-09-21T22:06:25.583 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:25.583 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.583 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.584 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:25.584 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.137639+0000", 2024-09-21T22:06:25.584 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:25.584 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.584 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.584 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:25.584 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.137641+0000", 2024-09-21T22:06:25.584 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.686e-06 2024-09-21T22:06:25.584 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.584 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.584 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:25.584 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.137934+0000", 2024-09-21T22:06:25.584 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00029319099999999999 2024-09-21T22:06:25.585 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.585 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.585 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:25.585 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.137963+0000", 2024-09-21T22:06:25.585 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9502999999999999e-05 2024-09-21T22:06:25.585 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.585 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.585 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:25.585 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.137989+0000", 2024-09-21T22:06:25.585 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5573000000000001e-05 2024-09-21T22:06:25.585 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.585 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.585 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:25.585 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.138086+0000", 2024-09-21T22:06:25.586 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.6968999999999999e-05 2024-09-21T22:06:25.586 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.586 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.586 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:25.586 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.138120+0000", 2024-09-21T22:06:25.586 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.4560000000000001e-05 2024-09-21T22:06:25.586 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.586 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.586 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:25.586 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.141539+0000", 2024-09-21T22:06:25.586 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0034183289999999999 2024-09-21T22:06:25.586 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.587 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.587 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:25.587 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.141629+0000", 2024-09-21T22:06:25.587 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.0399e-05 2024-09-21T22:06:25.587 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:25.587 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:25.587 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:25.587 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.587 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.587 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:23 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'13, mlcod=14'11)", 2024-09-21T22:06:25.587 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.144914+0000", 2024-09-21T22:06:25.587 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 23.436440193999999, 2024-09-21T22:06:25.587 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0040223230000000004, 2024-09-21T22:06:25.587 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:25.588 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:25.588 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:25.588 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:25.588 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.588 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:25.588 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.144914+0000", 2024-09-21T22:06:25.588 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:25.588 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.588 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.588 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:25.588 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.144914+0000", 2024-09-21T22:06:25.588 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:25.589 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.589 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.589 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:25.589 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.144916+0000", 2024-09-21T22:06:25.589 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6470000000000001e-06 2024-09-21T22:06:25.589 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.589 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.589 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:25.589 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.144929+0000", 2024-09-21T22:06:25.589 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2903e-05 2024-09-21T22:06:25.589 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.589 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.589 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:25.589 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.144958+0000", 2024-09-21T22:06:25.589 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9280000000000001e-05 2024-09-21T22:06:25.590 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.590 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.590 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:25.590 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.144971+0000", 2024-09-21T22:06:25.590 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3407000000000001e-05 2024-09-21T22:06:25.590 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.590 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.590 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:25.590 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.145106+0000", 2024-09-21T22:06:25.590 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013440500000000001 2024-09-21T22:06:25.590 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.590 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.590 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:25.590 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.145170+0000", 2024-09-21T22:06:25.591 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.4597000000000005e-05 2024-09-21T22:06:25.591 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.591 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.591 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:25.591 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.148854+0000", 2024-09-21T22:06:25.591 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0036833249999999999 2024-09-21T22:06:25.591 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.591 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.591 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:25.591 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.148937+0000", 2024-09-21T22:06:25.591 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.2758999999999998e-05 2024-09-21T22:06:25.591 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:25.591 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:25.591 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:25.592 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.592 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.592 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:31 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'17, mlcod=14'14)", 2024-09-21T22:06:25.592 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.163958+0000", 2024-09-21T22:06:25.592 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 23.417395927000001, 2024-09-21T22:06:25.592 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0033891149999999998, 2024-09-21T22:06:25.592 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:25.592 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:25.592 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:25.592 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:25.592 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.592 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:25.592 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.163958+0000", 2024-09-21T22:06:25.592 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:25.593 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.593 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.593 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:25.593 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.163958+0000", 2024-09-21T22:06:25.593 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:25.593 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.593 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.593 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:25.593 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.163960+0000", 2024-09-21T22:06:25.593 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.64e-06 2024-09-21T22:06:25.593 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.593 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.593 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:25.593 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.163971+0000", 2024-09-21T22:06:25.594 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0797e-05 2024-09-21T22:06:25.594 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.594 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.594 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:25.594 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.163995+0000", 2024-09-21T22:06:25.594 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3876e-05 2024-09-21T22:06:25.594 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.594 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.594 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:25.594 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.164008+0000", 2024-09-21T22:06:25.594 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3674e-05 2024-09-21T22:06:25.594 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.594 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.594 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:25.595 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.164203+0000", 2024-09-21T22:06:25.595 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00019457799999999999 2024-09-21T22:06:25.595 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.595 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.595 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:25.595 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.164265+0000", 2024-09-21T22:06:25.595 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.2415999999999993e-05 2024-09-21T22:06:25.595 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.595 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.595 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:25.595 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.167209+0000", 2024-09-21T22:06:25.595 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0029436380000000002 2024-09-21T22:06:25.595 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.596 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.596 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:25.596 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.167348+0000", 2024-09-21T22:06:25.596 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000138496 2024-09-21T22:06:25.596 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:25.596 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:25.596 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:25.596 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.596 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.596 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:32 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'18, mlcod=14'14)", 2024-09-21T22:06:25.596 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.165319+0000", 2024-09-21T22:06:25.596 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 23.416035077, 2024-09-21T22:06:25.596 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0033722280000000001, 2024-09-21T22:06:25.596 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:25.596 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:25.597 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:25.597 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:25.597 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.597 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:25.597 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.165319+0000", 2024-09-21T22:06:25.597 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:25.597 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.597 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.597 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:25.597 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.165319+0000", 2024-09-21T22:06:25.597 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:25.597 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.597 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.597 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:25.598 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.165321+0000", 2024-09-21T22:06:25.598 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5999999999999999e-06 2024-09-21T22:06:25.598 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.598 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.598 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:25.598 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.165377+0000", 2024-09-21T22:06:25.598 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.5877000000000001e-05 2024-09-21T22:06:25.598 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.598 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.598 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:25.598 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.165407+0000", 2024-09-21T22:06:25.598 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.0263000000000001e-05 2024-09-21T22:06:25.598 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.598 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.599 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:25.599 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.165441+0000", 2024-09-21T22:06:25.599 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.3966e-05 2024-09-21T22:06:25.599 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.599 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.599 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:25.599 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.165574+0000", 2024-09-21T22:06:25.599 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013327599999999999 2024-09-21T22:06:25.599 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.599 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.599 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:25.599 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.165635+0000", 2024-09-21T22:06:25.599 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.0949000000000002e-05 2024-09-21T22:06:25.599 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.599 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.599 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:25.600 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.168619+0000", 2024-09-21T22:06:25.600 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0029836979999999999 2024-09-21T22:06:25.600 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.600 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.600 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:25.600 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.168692+0000", 2024-09-21T22:06:25.600 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.2599000000000001e-05 2024-09-21T22:06:25.600 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:25.600 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:25.600 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:25.600 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.600 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.600 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:38 1.0 e14/13 1:30306672:devicehealth::main.db.0000000000000000:head v 14'24, mlcod=14'22)", 2024-09-21T22:06:25.600 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.180244+0000", 2024-09-21T22:06:25.600 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 23.401110431999999, 2024-09-21T22:06:25.600 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0041712219999999996, 2024-09-21T22:06:25.601 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:25.601 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:25.601 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:25.601 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:25.601 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.601 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:25.601 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.180244+0000", 2024-09-21T22:06:25.601 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:25.601 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.601 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.601 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:25.601 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.180244+0000", 2024-09-21T22:06:25.601 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:25.601 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.601 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.601 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:25.602 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.180246+0000", 2024-09-21T22:06:25.602 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6959999999999999e-06 2024-09-21T22:06:25.602 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.602 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.602 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:25.602 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.180508+0000", 2024-09-21T22:06:25.602 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00026275199999999999 2024-09-21T22:06:25.602 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.602 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.602 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:25.602 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.180538+0000", 2024-09-21T22:06:25.602 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9142999999999999e-05 2024-09-21T22:06:25.602 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.602 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.602 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:25.603 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.180552+0000", 2024-09-21T22:06:25.603 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4466e-05 2024-09-21T22:06:25.603 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.603 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.603 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:25.603 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.180677+0000", 2024-09-21T22:06:25.603 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000125413 2024-09-21T22:06:25.603 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.603 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.603 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:25.603 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.180739+0000", 2024-09-21T22:06:25.603 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.1122000000000002e-05 2024-09-21T22:06:25.603 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.604 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.604 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:25.604 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.184291+0000", 2024-09-21T22:06:25.604 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0035525510000000001 2024-09-21T22:06:25.604 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.604 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.604 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:25.604 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.184415+0000", 2024-09-21T22:06:25.604 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000124079 2024-09-21T22:06:25.604 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:25.604 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:25.604 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:25.604 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.604 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.605 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:54 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'32, mlcod=14'29)", 2024-09-21T22:06:25.605 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.214326+0000", 2024-09-21T22:06:25.605 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 23.367028195, 2024-09-21T22:06:25.605 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0035226569999999998, 2024-09-21T22:06:25.605 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:25.605 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:25.605 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:25.605 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:25.605 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.605 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:25.605 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.214326+0000", 2024-09-21T22:06:25.605 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:25.605 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.605 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.605 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:25.606 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.214326+0000", 2024-09-21T22:06:25.606 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:25.606 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.606 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.606 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:25.606 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.214328+0000", 2024-09-21T22:06:25.606 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6270000000000001e-06 2024-09-21T22:06:25.606 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.606 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.606 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:25.606 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.214340+0000", 2024-09-21T22:06:25.606 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2286000000000001e-05 2024-09-21T22:06:25.606 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.607 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.607 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:25.607 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.214369+0000", 2024-09-21T22:06:25.607 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9297000000000001e-05 2024-09-21T22:06:25.607 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.607 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.607 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:25.607 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.214383+0000", 2024-09-21T22:06:25.607 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3773e-05 2024-09-21T22:06:25.607 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.607 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.607 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:25.607 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.214513+0000", 2024-09-21T22:06:25.607 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000130056 2024-09-21T22:06:25.607 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.608 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.608 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:25.608 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.214568+0000", 2024-09-21T22:06:25.608 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.4873000000000001e-05 2024-09-21T22:06:25.608 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.608 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.608 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:25.608 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.217739+0000", 2024-09-21T22:06:25.608 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0031712720000000002 2024-09-21T22:06:25.608 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.608 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.608 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:25.608 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.217849+0000", 2024-09-21T22:06:25.608 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000109473 2024-09-21T22:06:25.608 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:25.609 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:25.609 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:25.609 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.609 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.609 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:57 1.0 e14/13 1:30306672:devicehealth::main.db.0000000000000000:head v 14'35, mlcod=14'33)", 2024-09-21T22:06:25.609 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.221549+0000", 2024-09-21T22:06:25.609 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 23.359805695999999, 2024-09-21T22:06:25.609 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0041720860000000002, 2024-09-21T22:06:25.609 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:25.609 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:25.609 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:25.609 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:25.609 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.609 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:25.609 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.221549+0000", 2024-09-21T22:06:25.610 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:25.610 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.610 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.610 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:25.610 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.221549+0000", 2024-09-21T22:06:25.610 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:25.610 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.610 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.610 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:25.610 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.221550+0000", 2024-09-21T22:06:25.610 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.717e-06 2024-09-21T22:06:25.610 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.610 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.611 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:25.611 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.221673+0000", 2024-09-21T22:06:25.611 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000122612 2024-09-21T22:06:25.611 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.611 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.611 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:25.611 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.221703+0000", 2024-09-21T22:06:25.611 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9539999999999998e-05 2024-09-21T22:06:25.611 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.611 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.611 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:25.611 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.221716+0000", 2024-09-21T22:06:25.611 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3149999999999999e-05 2024-09-21T22:06:25.611 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.611 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.611 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:25.612 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.221848+0000", 2024-09-21T22:06:25.612 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013201900000000001 2024-09-21T22:06:25.612 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.612 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.612 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:25.612 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.221908+0000", 2024-09-21T22:06:25.612 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.0452999999999999e-05 2024-09-21T22:06:25.612 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.612 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.612 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:25.612 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.225573+0000", 2024-09-21T22:06:25.612 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.003664676 2024-09-21T22:06:25.612 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.612 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.612 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:25.613 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.225721+0000", 2024-09-21T22:06:25.613 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00014791899999999999 2024-09-21T22:06:25.613 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:25.613 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:25.613 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:25.613 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.613 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.613 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:68 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'42, mlcod=14'39)", 2024-09-21T22:06:25.613 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.247413+0000", 2024-09-21T22:06:25.613 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 23.333941347, 2024-09-21T22:06:25.613 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0039572339999999996, 2024-09-21T22:06:25.613 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:25.613 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:25.613 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:25.613 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:25.613 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.614 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:25.614 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.247413+0000", 2024-09-21T22:06:25.614 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:25.614 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.614 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.614 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:25.614 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.247413+0000", 2024-09-21T22:06:25.614 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:25.614 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.614 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.614 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:25.614 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.247415+0000", 2024-09-21T22:06:25.614 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.883e-06 2024-09-21T22:06:25.614 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.614 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.615 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:25.615 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.247427+0000", 2024-09-21T22:06:25.615 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2557e-05 2024-09-21T22:06:25.615 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.615 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.615 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:25.615 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.247457+0000", 2024-09-21T22:06:25.615 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9966000000000001e-05 2024-09-21T22:06:25.615 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.615 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.615 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:25.615 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.247471+0000", 2024-09-21T22:06:25.615 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3747000000000001e-05 2024-09-21T22:06:25.615 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.616 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.616 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:25.616 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.247603+0000", 2024-09-21T22:06:25.616 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013171600000000001 2024-09-21T22:06:25.616 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.616 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.616 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:25.616 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.247656+0000", 2024-09-21T22:06:25.616 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.3529999999999997e-05 2024-09-21T22:06:25.616 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.616 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.616 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:25.616 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.251225+0000", 2024-09-21T22:06:25.616 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00356865 2024-09-21T22:06:25.616 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.616 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.617 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:25.617 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.251370+0000", 2024-09-21T22:06:25.617 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00014518500000000001 2024-09-21T22:06:25.617 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:25.617 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:25.617 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:25.617 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.617 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.617 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:69 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'43, mlcod=14'39)", 2024-09-21T22:06:25.617 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.248839+0000", 2024-09-21T22:06:25.617 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 23.332515417, 2024-09-21T22:06:25.617 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0034954470000000001, 2024-09-21T22:06:25.617 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:25.617 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:25.618 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:25.618 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:25.618 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.618 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:25.618 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.248839+0000", 2024-09-21T22:06:25.618 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:25.618 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.618 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.618 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:25.618 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.248839+0000", 2024-09-21T22:06:25.618 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:25.618 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.618 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.618 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:25.619 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.248841+0000", 2024-09-21T22:06:25.619 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6759999999999999e-06 2024-09-21T22:06:25.619 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.619 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.619 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:25.619 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.249058+0000", 2024-09-21T22:06:25.619 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000217455 2024-09-21T22:06:25.619 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.619 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.619 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:25.619 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.249085+0000", 2024-09-21T22:06:25.619 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.7002999999999999e-05 2024-09-21T22:06:25.619 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.619 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.619 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:25.619 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.249099+0000", 2024-09-21T22:06:25.620 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.402e-05 2024-09-21T22:06:25.620 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.620 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.620 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:25.620 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.249241+0000", 2024-09-21T22:06:25.620 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00014194899999999999 2024-09-21T22:06:25.620 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.620 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.620 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:25.620 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.249298+0000", 2024-09-21T22:06:25.620 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.6687000000000001e-05 2024-09-21T22:06:25.620 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.620 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.621 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:25.621 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.252266+0000", 2024-09-21T22:06:25.621 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.002968348 2024-09-21T22:06:25.621 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.621 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.621 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:25.621 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.252334+0000", 2024-09-21T22:06:25.621 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.8308999999999999e-05 2024-09-21T22:06:25.621 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:25.621 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:25.621 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:25.621 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.621 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.621 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:71 1.0 e14/13 1:30306672:devicehealth::main.db.0000000000000000:head v 14'45, mlcod=14'43)", 2024-09-21T22:06:25.622 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.256068+0000", 2024-09-21T22:06:25.622 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 23.325285945000001, 2024-09-21T22:06:25.622 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0041712579999999997, 2024-09-21T22:06:25.622 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:25.622 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:25.622 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:25.622 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:25.622 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.622 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:25.622 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.256068+0000", 2024-09-21T22:06:25.622 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:25.622 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.622 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.622 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:25.623 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.256068+0000", 2024-09-21T22:06:25.623 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:25.623 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.623 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.623 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:25.623 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.256070+0000", 2024-09-21T22:06:25.623 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.61e-06 2024-09-21T22:06:25.623 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.623 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.623 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:25.623 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.256129+0000", 2024-09-21T22:06:25.623 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.8702999999999997e-05 2024-09-21T22:06:25.623 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.623 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.624 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:25.624 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.256158+0000", 2024-09-21T22:06:25.624 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9533e-05 2024-09-21T22:06:25.624 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.624 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.624 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:25.624 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.256173+0000", 2024-09-21T22:06:25.624 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4267000000000001e-05 2024-09-21T22:06:25.624 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.624 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.624 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:25.624 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.256304+0000", 2024-09-21T22:06:25.624 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000131352 2024-09-21T22:06:25.624 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.624 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.625 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:25.625 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.256357+0000", 2024-09-21T22:06:25.625 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.3492999999999998e-05 2024-09-21T22:06:25.625 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.625 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.625 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:25.625 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.260147+0000", 2024-09-21T22:06:25.625 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0037900410000000001 2024-09-21T22:06:25.625 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.625 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.625 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:25.625 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.260240+0000", 2024-09-21T22:06:25.625 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.2258999999999999e-05 2024-09-21T22:06:25.625 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:25.626 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:25.626 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:25.626 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.626 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.626 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:72 1.0 e14/13 1:30306672:devicehealth::main.db.0000000000000000:head v 14'46, mlcod=14'43)", 2024-09-21T22:06:25.626 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.258358+0000", 2024-09-21T22:06:25.626 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 23.322995951999999, 2024-09-21T22:06:25.626 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0041492919999999997, 2024-09-21T22:06:25.626 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:25.626 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:25.626 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:25.626 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:25.626 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.627 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:25.627 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.258358+0000", 2024-09-21T22:06:25.627 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:25.627 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.627 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.627 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:25.627 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.258358+0000", 2024-09-21T22:06:25.627 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:25.627 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.627 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.627 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:25.627 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.258360+0000", 2024-09-21T22:06:25.627 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.623e-06 2024-09-21T22:06:25.627 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.627 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.628 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:25.628 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.258415+0000", 2024-09-21T22:06:25.628 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.4756000000000002e-05 2024-09-21T22:06:25.628 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.628 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.628 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:25.628 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.258444+0000", 2024-09-21T22:06:25.628 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9164000000000001e-05 2024-09-21T22:06:25.628 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.628 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.628 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:25.628 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.258459+0000", 2024-09-21T22:06:25.628 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4996e-05 2024-09-21T22:06:25.628 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.628 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.629 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:25.629 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.258637+0000", 2024-09-21T22:06:25.629 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00017796500000000001 2024-09-21T22:06:25.629 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.629 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.629 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:25.629 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.258695+0000", 2024-09-21T22:06:25.629 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.8207e-05 2024-09-21T22:06:25.629 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.629 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.629 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:25.629 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.262421+0000", 2024-09-21T22:06:25.629 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0037261550000000001 2024-09-21T22:06:25.629 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.630 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.630 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:25.630 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.262508+0000", 2024-09-21T22:06:25.630 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.6426000000000002e-05 2024-09-21T22:06:25.630 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:25.630 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:25.630 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:25.630 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.630 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.630 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:87 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'57, mlcod=14'52)", 2024-09-21T22:06:25.630 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.290115+0000", 2024-09-21T22:06:25.630 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 23.291239507, 2024-09-21T22:06:25.630 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0030478200000000001, 2024-09-21T22:06:25.631 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:25.631 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:25.631 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:25.631 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:25.631 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.631 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:25.631 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.290115+0000", 2024-09-21T22:06:25.631 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:25.631 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.631 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.631 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:25.631 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.290115+0000", 2024-09-21T22:06:25.631 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:25.631 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.632 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.632 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:25.632 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.290116+0000", 2024-09-21T22:06:25.632 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3829999999999999e-06 2024-09-21T22:06:25.632 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.632 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.632 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:25.632 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.290191+0000", 2024-09-21T22:06:25.632 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.4562999999999997e-05 2024-09-21T22:06:25.632 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.632 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.632 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:25.632 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.290216+0000", 2024-09-21T22:06:25.633 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.565e-05 2024-09-21T22:06:25.633 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.633 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.633 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:25.633 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.290236+0000", 2024-09-21T22:06:25.633 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9326000000000001e-05 2024-09-21T22:06:25.633 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.633 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.633 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:25.633 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.290340+0000", 2024-09-21T22:06:25.633 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0001047 2024-09-21T22:06:25.633 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.633 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.634 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:25.634 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.290377+0000", 2024-09-21T22:06:25.634 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.6356000000000002e-05 2024-09-21T22:06:25.634 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.634 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.634 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:25.634 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.293096+0000", 2024-09-21T22:06:25.634 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.002718996 2024-09-21T22:06:25.634 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.634 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.634 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:25.634 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.293163+0000", 2024-09-21T22:06:25.634 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.6846000000000003e-05 2024-09-21T22:06:25.634 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:25.635 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:25.635 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:25.635 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.635 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.635 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:114 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'76, mlcod=14'73)", 2024-09-21T22:06:25.635 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.490030+0000", 2024-09-21T22:06:25.635 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 23.091324067999999, 2024-09-21T22:06:25.635 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.004227873, 2024-09-21T22:06:25.635 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:25.635 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:25.635 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:25.635 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:25.635 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.635 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:25.636 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.490030+0000", 2024-09-21T22:06:25.636 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:25.636 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.636 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.636 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:25.636 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.490030+0000", 2024-09-21T22:06:25.636 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:25.636 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.636 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.636 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:25.636 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.490031+0000", 2024-09-21T22:06:25.636 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.3300000000000001e-07 2024-09-21T22:06:25.636 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.636 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.637 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:25.637 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.490042+0000", 2024-09-21T22:06:25.637 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.1095e-05 2024-09-21T22:06:25.637 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.637 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.637 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:25.637 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.490053+0000", 2024-09-21T22:06:25.637 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0501e-05 2024-09-21T22:06:25.637 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.637 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.637 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:25.637 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.490060+0000", 2024-09-21T22:06:25.637 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.9829999999999999e-06 2024-09-21T22:06:25.637 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.638 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.638 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:25.638 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.490107+0000", 2024-09-21T22:06:25.638 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.7278999999999999e-05 2024-09-21T22:06:25.638 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.638 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.638 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:25.638 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.490124+0000", 2024-09-21T22:06:25.638 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7190999999999999e-05 2024-09-21T22:06:25.638 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.638 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.638 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:25.638 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.494229+0000", 2024-09-21T22:06:25.638 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0041050440000000004 2024-09-21T22:06:25.638 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.639 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.639 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:25.639 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.494258+0000", 2024-09-21T22:06:25.639 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9047000000000002e-05 2024-09-21T22:06:25.639 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:25.639 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:25.639 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:25.639 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.639 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.639 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:115 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'77, mlcod=14'73)", 2024-09-21T22:06:25.639 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.491380+0000", 2024-09-21T22:06:25.639 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 23.089974445999999, 2024-09-21T22:06:25.639 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0031550889999999998, 2024-09-21T22:06:25.639 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:25.639 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:25.640 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:25.640 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:25.640 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.640 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:25.640 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.491380+0000", 2024-09-21T22:06:25.640 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:25.640 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.640 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.640 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:25.640 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.491380+0000", 2024-09-21T22:06:25.640 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:25.640 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.640 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.640 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:25.641 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.491381+0000", 2024-09-21T22:06:25.641 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.1399999999999996e-07 2024-09-21T22:06:25.641 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.641 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.641 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:25.641 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.491435+0000", 2024-09-21T22:06:25.641 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.3797e-05 2024-09-21T22:06:25.641 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.641 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.641 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:25.641 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.491447+0000", 2024-09-21T22:06:25.641 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2642e-05 2024-09-21T22:06:25.641 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.641 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.642 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:25.642 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.491453+0000", 2024-09-21T22:06:25.642 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.4820000000000002e-06 2024-09-21T22:06:25.642 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.642 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.642 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:25.642 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.491502+0000", 2024-09-21T22:06:25.642 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.9038e-05 2024-09-21T22:06:25.642 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.642 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.642 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:25.642 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.491514+0000", 2024-09-21T22:06:25.642 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2483999999999999e-05 2024-09-21T22:06:25.642 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.643 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.643 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:25.643 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.494486+0000", 2024-09-21T22:06:25.643 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.002971338 2024-09-21T22:06:25.643 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.643 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.643 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:25.643 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.494535+0000", 2024-09-21T22:06:25.643 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.9493999999999997e-05 2024-09-21T22:06:25.643 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:25.643 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:25.643 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:25.644 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.644 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.644 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:118 1.0 e14/13 1:30306672:devicehealth::main.db.0000000000000000:head v 14'80, mlcod=14'78)", 2024-09-21T22:06:25.644 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.499045+0000", 2024-09-21T22:06:25.644 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 23.082309444, 2024-09-21T22:06:25.644 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0029889589999999998, 2024-09-21T22:06:25.644 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:25.644 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:25.644 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:25.644 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:25.644 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.644 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:25.644 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.499045+0000", 2024-09-21T22:06:25.644 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:25.644 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.645 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.645 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:25.645 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.499045+0000", 2024-09-21T22:06:25.645 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:25.645 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.645 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.645 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:25.645 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.499046+0000", 2024-09-21T22:06:25.645 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.0299999999999998e-07 2024-09-21T22:06:25.645 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.645 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.645 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:25.645 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.499110+0000", 2024-09-21T22:06:25.645 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.4688000000000001e-05 2024-09-21T22:06:25.645 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.646 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.646 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:25.646 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.499125+0000", 2024-09-21T22:06:25.646 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4387e-05 2024-09-21T22:06:25.646 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.646 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.646 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:25.646 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.499131+0000", 2024-09-21T22:06:25.646 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.2129999999999996e-06 2024-09-21T22:06:25.646 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.646 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.646 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:25.646 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.499255+0000", 2024-09-21T22:06:25.646 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00012424199999999999 2024-09-21T22:06:25.647 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.647 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.647 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:25.647 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.499312+0000", 2024-09-21T22:06:25.647 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.6709999999999997e-05 2024-09-21T22:06:25.647 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.647 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.647 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:25.647 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.502005+0000", 2024-09-21T22:06:25.647 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0026927169999999999 2024-09-21T22:06:25.647 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.647 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.647 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:25.647 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.502034+0000", 2024-09-21T22:06:25.648 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9198999999999999e-05 2024-09-21T22:06:25.648 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:25.648 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:25.648 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:25.648 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.648 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.648 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:136 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'94, mlcod=14'89)", 2024-09-21T22:06:25.648 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.903810+0000", 2024-09-21T22:06:25.648 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 22.677544473000001, 2024-09-21T22:06:25.648 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0062746829999999996, 2024-09-21T22:06:25.648 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:25.648 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:25.648 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:25.648 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:25.648 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.649 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:25.649 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.903810+0000", 2024-09-21T22:06:25.649 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:25.649 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.649 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.649 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:25.649 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.903810+0000", 2024-09-21T22:06:25.649 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:25.649 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.649 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.649 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:25.649 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.903810+0000", 2024-09-21T22:06:25.649 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.03e-07 2024-09-21T22:06:25.649 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.650 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.650 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:25.650 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.903814+0000", 2024-09-21T22:06:25.650 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.676e-06 2024-09-21T22:06:25.650 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.650 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.650 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:25.650 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.903822+0000", 2024-09-21T22:06:25.650 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.2879999999999993e-06 2024-09-21T22:06:25.650 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.650 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.650 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:25.650 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.903829+0000", 2024-09-21T22:06:25.650 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.2920000000000001e-06 2024-09-21T22:06:25.650 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.651 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.651 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:25.651 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.903879+0000", 2024-09-21T22:06:25.651 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.0253999999999997e-05 2024-09-21T22:06:25.651 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.651 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.651 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:25.651 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.903897+0000", 2024-09-21T22:06:25.651 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8468999999999999e-05 2024-09-21T22:06:25.651 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.651 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.651 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:25.651 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.910044+0000", 2024-09-21T22:06:25.651 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0061468950000000003 2024-09-21T22:06:25.652 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.652 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.652 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:25.652 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.910085+0000", 2024-09-21T22:06:25.652 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.0405999999999999e-05 2024-09-21T22:06:25.652 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:25.652 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:25.652 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:25.652 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.652 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.652 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:137 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'95, mlcod=14'90)", 2024-09-21T22:06:25.652 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.904439+0000", 2024-09-21T22:06:25.652 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 22.676915077, 2024-09-21T22:06:25.652 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0072625019999999997, 2024-09-21T22:06:25.653 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:25.653 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:25.653 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:25.653 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:25.653 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.653 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:25.653 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.904439+0000", 2024-09-21T22:06:25.653 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:25.653 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.653 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.653 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:25.653 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.904439+0000", 2024-09-21T22:06:25.653 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:25.653 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.653 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.654 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:25.654 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.904440+0000", 2024-09-21T22:06:25.654 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8400000000000005e-07 2024-09-21T22:06:25.654 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.654 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.654 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:25.654 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.904498+0000", 2024-09-21T22:06:25.654 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.8320999999999999e-05 2024-09-21T22:06:25.654 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.654 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.654 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:25.654 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.904507+0000", 2024-09-21T22:06:25.654 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.7790000000000001e-06 2024-09-21T22:06:25.654 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.655 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.655 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:25.655 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.904511+0000", 2024-09-21T22:06:25.655 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.9700000000000001e-06 2024-09-21T22:06:25.655 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.655 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.655 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:25.655 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.904551+0000", 2024-09-21T22:06:25.655 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.9910000000000002e-05 2024-09-21T22:06:25.655 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.655 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.655 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:25.655 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.904567+0000", 2024-09-21T22:06:25.655 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5903999999999999e-05 2024-09-21T22:06:25.656 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.656 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.656 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:25.656 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.911666+0000", 2024-09-21T22:06:25.656 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0070996929999999998 2024-09-21T22:06:25.656 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.656 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.656 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:25.656 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.911702+0000", 2024-09-21T22:06:25.656 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.5441000000000002e-05 2024-09-21T22:06:25.656 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:25.656 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:25.656 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:25.656 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.656 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.657 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:138 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'96, mlcod=14'90)", 2024-09-21T22:06:25.657 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.905402+0000", 2024-09-21T22:06:25.657 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 22.6759521, 2024-09-21T22:06:25.657 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0063366890000000004, 2024-09-21T22:06:25.657 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:25.657 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:25.657 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:25.657 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:25.657 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.657 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:25.657 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905402+0000", 2024-09-21T22:06:25.657 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:25.658 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.658 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.658 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:25.658 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905402+0000", 2024-09-21T22:06:25.658 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:25.658 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.658 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.658 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:25.658 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905403+0000", 2024-09-21T22:06:25.658 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.9200000000000002e-07 2024-09-21T22:06:25.658 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.658 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.658 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:25.658 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905406+0000", 2024-09-21T22:06:25.659 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.326e-06 2024-09-21T22:06:25.659 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.659 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.659 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:25.659 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905413+0000", 2024-09-21T22:06:25.659 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.3320000000000003e-06 2024-09-21T22:06:25.659 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.659 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.659 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:25.659 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905417+0000", 2024-09-21T22:06:25.659 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.7139999999999999e-06 2024-09-21T22:06:25.659 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.659 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.659 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:25.660 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.910115+0000", 2024-09-21T22:06:25.660 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0046983939999999998 2024-09-21T22:06:25.660 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.660 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.660 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:25.660 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.910140+0000", 2024-09-21T22:06:25.660 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4689999999999999e-05 2024-09-21T22:06:25.660 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.660 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.660 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:25.660 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.911706+0000", 2024-09-21T22:06:25.660 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0015658479999999999 2024-09-21T22:06:25.660 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.660 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.660 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:25.660 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.911739+0000", 2024-09-21T22:06:25.661 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.2993000000000001e-05 2024-09-21T22:06:25.661 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:25.661 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:25.661 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:25.661 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.661 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.661 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:139 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'97, mlcod=14'90)", 2024-09-21T22:06:25.661 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.905983+0000", 2024-09-21T22:06:25.661 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 22.675371181999999, 2024-09-21T22:06:25.661 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0057906030000000001, 2024-09-21T22:06:25.661 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:25.661 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:25.661 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:25.661 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:25.661 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.662 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:25.662 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905983+0000", 2024-09-21T22:06:25.662 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:25.662 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.662 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.662 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:25.662 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905983+0000", 2024-09-21T22:06:25.662 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:25.662 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.662 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.662 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:25.662 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905984+0000", 2024-09-21T22:06:25.662 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.15e-07 2024-09-21T22:06:25.663 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.663 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.663 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:25.663 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905986+0000", 2024-09-21T22:06:25.663 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.852e-06 2024-09-21T22:06:25.663 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.663 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.663 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:25.663 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905993+0000", 2024-09-21T22:06:25.663 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.652e-06 2024-09-21T22:06:25.663 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.663 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.663 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:25.663 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905996+0000", 2024-09-21T22:06:25.663 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.3519999999999998e-06 2024-09-21T22:06:25.664 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.664 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.664 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:25.664 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.910519+0000", 2024-09-21T22:06:25.664 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0045230820000000003 2024-09-21T22:06:25.664 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.664 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.664 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:25.664 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.910548+0000", 2024-09-21T22:06:25.664 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8399999999999999e-05 2024-09-21T22:06:25.664 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.664 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.664 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:25.664 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.911742+0000", 2024-09-21T22:06:25.664 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011939909999999999 2024-09-21T22:06:25.664 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.665 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.665 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:25.665 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.911774+0000", 2024-09-21T22:06:25.665 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.1958999999999997e-05 2024-09-21T22:06:25.665 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:25.665 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:25.665 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:25.665 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.665 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.665 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:140 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'98, mlcod=14'90)", 2024-09-21T22:06:25.665 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.906441+0000", 2024-09-21T22:06:25.665 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 22.674913053000001, 2024-09-21T22:06:25.665 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0057052989999999996, 2024-09-21T22:06:25.666 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:25.666 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:25.666 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:25.666 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:25.666 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.666 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:25.666 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.906441+0000", 2024-09-21T22:06:25.666 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:25.666 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.666 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.666 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:25.666 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.906441+0000", 2024-09-21T22:06:25.666 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:25.666 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.667 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.667 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:25.667 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.906442+0000", 2024-09-21T22:06:25.667 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.7e-07 2024-09-21T22:06:25.667 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.667 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.667 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:25.667 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.906511+0000", 2024-09-21T22:06:25.667 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.8857999999999997e-05 2024-09-21T22:06:25.667 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.667 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.667 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:25.667 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.906518+0000", 2024-09-21T22:06:25.668 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.1690000000000003e-06 2024-09-21T22:06:25.668 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.668 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.668 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:25.668 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.906521+0000", 2024-09-21T22:06:25.668 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.4829999999999999e-06 2024-09-21T22:06:25.668 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.668 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.668 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:25.668 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.910939+0000", 2024-09-21T22:06:25.668 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0044176809999999997 2024-09-21T22:06:25.668 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.668 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.668 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:25.669 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.910959+0000", 2024-09-21T22:06:25.669 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9898000000000002e-05 2024-09-21T22:06:25.669 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.669 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.669 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:25.669 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.912113+0000", 2024-09-21T22:06:25.669 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001153811 2024-09-21T22:06:25.669 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:25.669 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:25.669 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:25.669 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.912147+0000", 2024-09-21T22:06:25.669 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.4029000000000002e-05 2024-09-21T22:06:25.669 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:25.670 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:25.670 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:25.670 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:25.670 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:25.670 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:06:25.671 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T22:06:25.672 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:06:25.668+0000 7fc8d39f7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25294) UID: 0 2024-09-21T22:06:25.773 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:06:25.772+0000 7fc8d39f7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25294) UID: 0 2024-09-21T22:06:25.799 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:06:25.799 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:06:25.799 INFO:teuthology.orchestra.run.smithi039.stdout: "num_ops": 0 2024-09-21T22:06:25.799 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:06:25.813 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T22:06:25.873 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:06:25.869+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:06:25.931 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:06:25.931 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:06:25.931 INFO:teuthology.orchestra.run.smithi039.stdout: "complaint_time": 30, 2024-09-21T22:06:25.931 INFO:teuthology.orchestra.run.smithi039.stdout: "num_blocked_ops": 0 2024-09-21T22:06:25.931 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:06:25.946 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T22:06:25.974 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:06:25.968+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:06:26.075 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:06:26.068+0000 7fc8d39f7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25294) UID: 0 2024-09-21T22:06:26.109 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:06:26.109 INFO:teuthology.orchestra.run.smithi039.stdout: "size": 20, 2024-09-21T22:06:26.109 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 600, 2024-09-21T22:06:26.109 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [ 2024-09-21T22:06:26.109 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:26.109 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4224.0:5 2.2 e18/16 2:5cea7035:::rbd_info:head v 18'1, mlcod=0'0)", 2024-09-21T22:06:26.109 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:06.114955+0000", 2024-09-21T22:06:26.109 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 19.994979107999999, 2024-09-21T22:06:26.109 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0017716170000000001, 2024-09-21T22:06:26.109 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:26.110 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:26.110 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:26.110 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:26.110 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:26.110 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:26.110 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:06.114955+0000", 2024-09-21T22:06:26.110 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:26.110 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:26.110 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:26.110 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:26.110 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:06.114955+0000", 2024-09-21T22:06:26.110 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:26.110 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:26.110 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:26.110 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:26.111 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:06.114956+0000", 2024-09-21T22:06:26.111 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.9299999999999997e-07 2024-09-21T22:06:26.111 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:26.111 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:26.111 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:26.111 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:06.114964+0000", 2024-09-21T22:06:26.111 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.9840000000000001e-06 2024-09-21T22:06:26.111 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:26.111 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:26.111 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:26.111 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:06.115001+0000", 2024-09-21T22:06:26.111 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.7719999999999998e-05 2024-09-21T22:06:26.111 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:26.111 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:26.111 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:26.112 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:06.115019+0000", 2024-09-21T22:06:26.112 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7943000000000001e-05 2024-09-21T22:06:26.112 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:26.112 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:26.112 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:26.112 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:06.115168+0000", 2024-09-21T22:06:26.112 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000148589 2024-09-21T22:06:26.112 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:26.112 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:26.112 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:26.112 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:06.115225+0000", 2024-09-21T22:06:26.112 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.6975999999999998e-05 2024-09-21T22:06:26.112 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:26.112 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:26.112 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:26.113 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:06.116585+0000", 2024-09-21T22:06:26.113 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00136038 2024-09-21T22:06:26.113 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:26.113 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:26.113 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:26.113 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:06.116726+0000", 2024-09-21T22:06:26.113 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00014133200000000001 2024-09-21T22:06:26.113 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:26.113 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:26.113 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:26.113 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:26.113 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:26.113 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4224.0:7 2.2 e19/16 2:5cea7035:::rbd_info:head v 19'2, mlcod=0'0)", 2024-09-21T22:06:26.113 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:07.129567+0000", 2024-09-21T22:06:26.113 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 18.980367098999999, 2024-09-21T22:06:26.114 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0018460239999999999, 2024-09-21T22:06:26.114 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:26.114 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:26.114 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:26.114 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:26.114 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:26.114 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:26.114 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:07.129567+0000", 2024-09-21T22:06:26.114 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:26.114 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:26.114 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:26.114 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:26.114 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:07.129567+0000", 2024-09-21T22:06:26.114 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:26.114 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:26.115 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:26.115 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:26.115 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:07.129567+0000", 2024-09-21T22:06:26.115 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.9499999999999998e-07 2024-09-21T22:06:26.115 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:26.115 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:26.115 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:26.115 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:07.129580+0000", 2024-09-21T22:06:26.115 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2335e-05 2024-09-21T22:06:26.115 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:26.115 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:26.115 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:26.115 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:07.129606+0000", 2024-09-21T22:06:26.115 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.6528000000000001e-05 2024-09-21T22:06:26.115 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:26.115 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:26.116 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:26.116 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:07.129628+0000", 2024-09-21T22:06:26.116 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1549999999999999e-05 2024-09-21T22:06:26.116 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:26.116 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:26.116 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:26.116 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:07.129744+0000", 2024-09-21T22:06:26.116 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000116551 2024-09-21T22:06:26.116 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:26.116 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:26.116 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:26.116 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:07.129780+0000", 2024-09-21T22:06:26.116 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.5447000000000001e-05 2024-09-21T22:06:26.116 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:26.116 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:26.117 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:26.117 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:07.131348+0000", 2024-09-21T22:06:26.117 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001568631 2024-09-21T22:06:26.117 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:26.117 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:26.117 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:26.117 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:07.131413+0000", 2024-09-21T22:06:26.117 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.4586999999999997e-05 2024-09-21T22:06:26.117 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:26.117 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:26.117 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:26.117 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:26.117 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:26.117 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:06:26.123 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T22:06:26.176 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:06:26.172+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:06:26.244 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:06:26.244 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:06:26.245 INFO:teuthology.orchestra.run.smithi039.stdout: "num_ops": 0 2024-09-21T22:06:26.245 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:06:26.252 INFO:teuthology.orchestra.run.smithi039.stderr:noscrub is set 2024-09-21T22:06:26.261 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T22:06:26.318 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:06:26.317+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:06:26.378 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:06:26.372+0000 7fc8d39f7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25294) UID: 0 2024-09-21T22:06:26.379 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:06:26.379 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:06:26.379 INFO:teuthology.orchestra.run.smithi039.stdout: "complaint_time": 30, 2024-09-21T22:06:26.380 INFO:teuthology.orchestra.run.smithi039.stdout: "num_blocked_ops": 0 2024-09-21T22:06:26.380 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:06:26.394 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T22:06:26.478 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:06:26.476+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:06:26.564 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:06:26.564 INFO:teuthology.orchestra.run.smithi039.stdout: "size": 20, 2024-09-21T22:06:26.564 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 600, 2024-09-21T22:06:26.565 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [ 2024-09-21T22:06:26.565 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:26.565 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4224.0:1 2.3 e17/16 2:c4c92e5a:::rbd_trash:head v 17'1, mlcod=0'0)", 2024-09-21T22:06:26.565 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:05.107810+0000", 2024-09-21T22:06:26.565 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 21.457224760999999, 2024-09-21T22:06:26.565 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0017679200000000001, 2024-09-21T22:06:26.565 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:26.565 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:26.565 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:26.565 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:26.565 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:26.565 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:26.565 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.107810+0000", 2024-09-21T22:06:26.566 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:26.566 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:26.566 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:26.566 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:26.566 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.107810+0000", 2024-09-21T22:06:26.566 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:26.566 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:26.566 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:26.566 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:26.566 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.107811+0000", 2024-09-21T22:06:26.566 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.6700000000000003e-07 2024-09-21T22:06:26.566 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:26.566 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:26.566 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:26.567 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.107822+0000", 2024-09-21T22:06:26.567 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0443e-05 2024-09-21T22:06:26.567 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:26.567 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:26.567 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:26.567 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.107853+0000", 2024-09-21T22:06:26.567 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.1046000000000001e-05 2024-09-21T22:06:26.567 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:26.567 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:26.567 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:26.567 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.107876+0000", 2024-09-21T22:06:26.567 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3547e-05 2024-09-21T22:06:26.567 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:26.567 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:26.567 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:26.568 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.108002+0000", 2024-09-21T22:06:26.568 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000125599 2024-09-21T22:06:26.568 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:26.568 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:26.568 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:26.568 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.108069+0000", 2024-09-21T22:06:26.568 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.7162999999999994e-05 2024-09-21T22:06:26.568 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:26.568 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:26.568 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:26.568 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.109385+0000", 2024-09-21T22:06:26.568 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0013162899999999999 2024-09-21T22:06:26.568 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:26.568 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:26.569 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:26.569 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.109578+0000", 2024-09-21T22:06:26.569 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000193065 2024-09-21T22:06:26.569 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:26.569 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:26.569 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:26.569 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:26.569 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:26.569 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:06:26.578 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T22:06:26.579 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:06:26.572+0000 7fc8d39f7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25294) UID: 0 2024-09-21T22:06:26.679 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:06:26.676+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:06:26.695 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T22:06:26.713 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:06:26.713 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:06:26.713 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-09-21T22:06:26.713 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:06:26.727 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T22:06:26.780 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:06:26.777+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:06:26.848 INFO:teuthology.orchestra.run.smithi039.stdout:osd.0: {} 2024-09-21T22:06:26.849 INFO:teuthology.orchestra.run.smithi039.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T22:06:26.865 INFO:teuthology.orchestra.run.smithi039.stdout:osd.1: {} 2024-09-21T22:06:26.865 INFO:teuthology.orchestra.run.smithi039.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T22:06:26.881 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:06:26.876+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:06:26.882 INFO:teuthology.orchestra.run.smithi039.stdout:osd.2: {} 2024-09-21T22:06:26.882 INFO:teuthology.orchestra.run.smithi039.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T22:06:26.903 INFO:teuthology.orchestra.run.smithi039.stdout:osd.3: {} 2024-09-21T22:06:26.903 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:26.903 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:26.903 INFO:teuthology.orchestra.run.smithi039.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T22:06:26.917 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T22:06:26.921 INFO:teuthology.orchestra.run.smithi039.stdout:osd.4: {} 2024-09-21T22:06:26.922 INFO:teuthology.orchestra.run.smithi039.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T22:06:26.942 INFO:teuthology.orchestra.run.smithi039.stdout:osd.5: {} 2024-09-21T22:06:26.942 INFO:teuthology.orchestra.run.smithi039.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T22:06:27.003 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:06:27.000+0000 7fc8d39f7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25294) UID: 0 2024-09-21T22:06:27.083 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:06:27.081+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:06:27.087 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:27.088 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:27.102 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T22:06:27.184 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:06:27.181+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:06:27.272 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:27.272 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:27.286 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:06:27.280+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:06:27.287 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T22:06:27.386 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:06:27.385+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:06:27.460 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:27.460 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:27.474 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T22:06:27.519 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:06:27.516+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:06:27.587 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:06:27.584+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:06:27.602 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:27.602 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:27.617 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T22:06:27.688 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:06:27.685+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:06:27.780 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:27.780 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:27.789 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:06:27.785+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:06:27.795 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T22:06:27.890 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:06:27.885+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:06:27.914 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:27.914 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:27.929 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T22:06:27.990 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:06:27.984+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:06:28.053 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:28.053 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:28.068 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T22:06:28.114 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:06:28.113+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:06:28.192 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:06:28.189+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:06:28.203 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:28.204 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:28.218 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T22:06:28.274 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-21T22:06:28.293 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:06:28.292+0000 7fc8d39f7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25294) UID: 0 2024-09-21T22:06:28.391 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:28.391 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:28.394 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:06:28.389+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:06:28.407 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T22:06:28.494 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:06:28.493+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:06:28.565 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:28.566 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:28.582 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T22:06:28.626 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:06:28.625+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:06:28.670 INFO:teuthology.orchestra.run.smithi039.stderr:nodeep-scrub is set 2024-09-21T22:06:28.697 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:06:28.692+0000 7fc8d39f7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25294) UID: 0 2024-09-21T22:06:28.746 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:28.747 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:28.760 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T22:06:28.797 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:06:28.792+0000 7fc8d39f7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25294) UID: 0 2024-09-21T22:06:28.880 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:28.881 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:28.895 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T22:06:28.898 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:06:28.897+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:06:28.960 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T22:06:28.999 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:06:28.996+0000 7fc8d39f7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25294) UID: 0 2024-09-21T22:06:29.015 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:29.015 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:29.031 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T22:06:29.100 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:06:29.096+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:06:29.110 INFO:teuthology.orchestra.run.smithi039.stdout:osd.0: {} 2024-09-21T22:06:29.110 INFO:teuthology.orchestra.run.smithi039.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T22:06:29.127 INFO:teuthology.orchestra.run.smithi039.stdout:osd.1: {} 2024-09-21T22:06:29.127 INFO:teuthology.orchestra.run.smithi039.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T22:06:29.158 INFO:teuthology.orchestra.run.smithi039.stdout:osd.2: {} 2024-09-21T22:06:29.159 INFO:teuthology.orchestra.run.smithi039.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T22:06:29.164 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:06:29.164 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:06:29.164 INFO:teuthology.orchestra.run.smithi039.stdout: "num_ops": 0 2024-09-21T22:06:29.164 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:06:29.179 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T22:06:29.180 INFO:teuthology.orchestra.run.smithi039.stdout:osd.3: {} 2024-09-21T22:06:29.180 INFO:teuthology.orchestra.run.smithi039.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T22:06:29.201 INFO:teuthology.orchestra.run.smithi039.stdout:osd.4: {} 2024-09-21T22:06:29.201 INFO:teuthology.orchestra.run.smithi039.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T22:06:29.201 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:06:29.197+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:06:29.223 INFO:teuthology.orchestra.run.smithi039.stdout:osd.5: {} 2024-09-21T22:06:29.223 INFO:teuthology.orchestra.run.smithi039.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T22:06:29.299 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:06:29.299 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:06:29.299 INFO:teuthology.orchestra.run.smithi039.stdout: "complaint_time": 30, 2024-09-21T22:06:29.299 INFO:teuthology.orchestra.run.smithi039.stdout: "num_blocked_ops": 0 2024-09-21T22:06:29.299 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:06:29.301 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:06:29.297+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:06:29.314 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T22:06:29.403 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:06:29.396+0000 7fc8d39f7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25294) UID: 0 2024-09-21T22:06:29.483 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:06:29.484 INFO:teuthology.orchestra.run.smithi039.stdout: "size": 20, 2024-09-21T22:06:29.484 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 600, 2024-09-21T22:06:29.484 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [ 2024-09-21T22:06:29.484 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:29.484 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4224.0:1 2.3 e17/16 2:c4c92e5a:::rbd_trash:head v 17'1, mlcod=0'0)", 2024-09-21T22:06:29.484 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:05.107810+0000", 2024-09-21T22:06:29.484 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 24.376434111000002, 2024-09-21T22:06:29.484 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0017679200000000001, 2024-09-21T22:06:29.484 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:29.484 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:29.484 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:29.484 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:29.485 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:29.485 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:29.485 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.107810+0000", 2024-09-21T22:06:29.485 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:29.485 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:29.485 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:29.485 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:29.485 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.107810+0000", 2024-09-21T22:06:29.485 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:29.485 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:29.485 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:29.485 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:29.485 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.107811+0000", 2024-09-21T22:06:29.485 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.6700000000000003e-07 2024-09-21T22:06:29.485 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:29.486 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:29.486 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:29.486 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.107822+0000", 2024-09-21T22:06:29.486 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0443e-05 2024-09-21T22:06:29.486 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:29.486 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:29.486 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:29.486 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.107853+0000", 2024-09-21T22:06:29.486 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.1046000000000001e-05 2024-09-21T22:06:29.486 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:29.486 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:29.486 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:29.486 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.107876+0000", 2024-09-21T22:06:29.486 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3547e-05 2024-09-21T22:06:29.487 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:29.487 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:29.487 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:29.487 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.108002+0000", 2024-09-21T22:06:29.487 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000125599 2024-09-21T22:06:29.487 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:29.487 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:29.487 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:29.487 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.108069+0000", 2024-09-21T22:06:29.487 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.7162999999999994e-05 2024-09-21T22:06:29.487 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:29.487 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:29.487 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:29.488 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.109385+0000", 2024-09-21T22:06:29.488 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0013162899999999999 2024-09-21T22:06:29.488 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:29.488 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:29.488 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:29.488 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.109578+0000", 2024-09-21T22:06:29.488 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000193065 2024-09-21T22:06:29.488 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:29.488 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:29.488 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:29.488 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:29.488 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:29.488 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:06:29.499 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T22:06:29.503 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:06:29.501+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:06:29.604 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:06:29.600+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:06:29.634 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:06:29.634 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:06:29.634 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-09-21T22:06:29.634 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:06:29.648 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T22:06:29.705 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:06:29.701+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:06:29.806 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:06:29.805+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:06:29.820 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:06:29.820 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:06:29.820 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-21T22:06:29.820 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-21T22:06:29.820 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:06:29.834 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T22:06:29.908 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:06:29.904+0000 7fc8d39f7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25294) UID: 0 2024-09-21T22:06:29.960 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:06:29.960 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-21T22:06:29.960 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-21T22:06:29.961 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-21T22:06:29.961 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:29.961 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:06:29.961 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.036355+0000", 2024-09-21T22:06:29.961 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 27.919907383999998, 2024-09-21T22:06:29.961 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.062441934999999997, 2024-09-21T22:06:29.961 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:29.961 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:29.961 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:29.961 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:29.961 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:06:29.961 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:06:29.961 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1 2024-09-21T22:06:29.961 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:29.961 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:29.962 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:29.962 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:29.962 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036355+0000", 2024-09-21T22:06:29.962 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:29.962 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:29.962 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:29.962 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:29.962 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036355+0000", 2024-09-21T22:06:29.962 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:29.962 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:29.962 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:29.962 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:29.962 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036357+0000", 2024-09-21T22:06:29.963 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.446e-06 2024-09-21T22:06:29.963 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:29.963 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:29.963 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:29.963 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036373+0000", 2024-09-21T22:06:29.963 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5447e-05 2024-09-21T22:06:29.963 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:29.963 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:29.963 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:29.963 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036399+0000", 2024-09-21T22:06:29.963 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6423000000000001e-05 2024-09-21T22:06:29.963 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:29.963 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:29.963 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:29.963 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036420+0000", 2024-09-21T22:06:29.964 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1129999999999999e-05 2024-09-21T22:06:29.964 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:29.964 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:29.964 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:29.964 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036557+0000", 2024-09-21T22:06:29.964 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013686899999999999 2024-09-21T22:06:29.964 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:29.964 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:29.964 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-21T22:06:29.964 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036585+0000", 2024-09-21T22:06:29.964 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8362999999999999e-05 2024-09-21T22:06:29.964 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:29.964 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:29.964 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:29.964 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.095068+0000", 2024-09-21T22:06:29.964 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058482972000000001 2024-09-21T22:06:29.965 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:29.965 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:29.965 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:29.965 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.095516+0000", 2024-09-21T22:06:29.965 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00044802999999999998 2024-09-21T22:06:29.965 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:29.965 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:29.965 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:06:29.965 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.095869+0000", 2024-09-21T22:06:29.965 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000352104 2024-09-21T22:06:29.965 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:29.965 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:29.965 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:29.965 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.097055+0000", 2024-09-21T22:06:29.965 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011865599999999999 2024-09-21T22:06:29.965 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:29.966 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:29.966 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:29.966 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.098637+0000", 2024-09-21T22:06:29.966 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001581458 2024-09-21T22:06:29.966 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:29.966 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:29.966 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:29.966 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.098728+0000", 2024-09-21T22:06:29.966 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1043000000000002e-05 2024-09-21T22:06:29.966 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:29.966 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:29.966 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:29.966 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.098797+0000", 2024-09-21T22:06:29.966 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9090000000000004e-05 2024-09-21T22:06:29.966 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:29.967 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:29.967 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:29.967 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:29.967 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:29.967 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:06:29.967 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.161242+0000", 2024-09-21T22:06:29.967 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 27.795020472000001, 2024-09-21T22:06:29.967 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014477078000000001, 2024-09-21T22:06:29.967 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:29.967 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:29.967 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:29.967 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:29.967 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:06:29.967 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:06:29.967 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 35 2024-09-21T22:06:29.968 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:29.968 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:29.968 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:29.968 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:29.968 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161242+0000", 2024-09-21T22:06:29.968 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:29.968 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:29.968 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:29.968 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:29.968 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161242+0000", 2024-09-21T22:06:29.968 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:29.968 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:29.968 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:29.968 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:29.968 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161243+0000", 2024-09-21T22:06:29.969 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4729999999999999e-06 2024-09-21T22:06:29.969 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:29.969 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:29.969 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:29.969 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161385+0000", 2024-09-21T22:06:29.969 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014226899999999999 2024-09-21T22:06:29.969 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:29.969 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:29.969 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:29.969 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161417+0000", 2024-09-21T22:06:29.969 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1406999999999999e-05 2024-09-21T22:06:29.969 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:29.969 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:29.969 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:29.969 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161434+0000", 2024-09-21T22:06:29.969 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7623000000000001e-05 2024-09-21T22:06:29.970 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:29.970 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:29.970 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:29.970 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.169728+0000", 2024-09-21T22:06:29.970 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0082933090000000004 2024-09-21T22:06:29.970 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:29.970 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:29.970 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:29.970 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.169861+0000", 2024-09-21T22:06:29.970 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013367300000000001 2024-09-21T22:06:29.970 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:29.970 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:29.970 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:06:29.970 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.170186+0000", 2024-09-21T22:06:29.970 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00032466400000000001 2024-09-21T22:06:29.971 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:29.971 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:29.971 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:29.971 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.174759+0000", 2024-09-21T22:06:29.971 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0045727880000000004 2024-09-21T22:06:29.971 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:29.971 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:29.971 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:29.971 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.175497+0000", 2024-09-21T22:06:29.971 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00073803400000000004 2024-09-21T22:06:29.971 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:29.971 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:29.971 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:29.971 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.175607+0000", 2024-09-21T22:06:29.971 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010987199999999999 2024-09-21T22:06:29.971 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:29.972 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:29.972 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:29.972 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.175719+0000", 2024-09-21T22:06:29.972 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011196600000000001 2024-09-21T22:06:29.972 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:29.972 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:29.972 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:29.972 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:29.972 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:29.972 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:06:29.972 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.161483+0000", 2024-09-21T22:06:29.972 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 27.794778960999999, 2024-09-21T22:06:29.972 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016155732999999999, 2024-09-21T22:06:29.972 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:29.972 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:29.973 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:29.973 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:29.973 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:06:29.973 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:06:29.973 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 36 2024-09-21T22:06:29.973 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:29.973 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:29.973 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:29.973 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:29.973 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161483+0000", 2024-09-21T22:06:29.973 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:29.973 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:29.973 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:29.973 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:29.973 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161483+0000", 2024-09-21T22:06:29.973 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:29.974 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:29.974 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:29.974 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:29.974 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161485+0000", 2024-09-21T22:06:29.974 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5999999999999999e-06 2024-09-21T22:06:29.974 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:29.974 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:29.974 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:29.974 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161498+0000", 2024-09-21T22:06:29.974 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3166999999999999e-05 2024-09-21T22:06:29.974 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:29.974 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:29.974 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:29.974 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161526+0000", 2024-09-21T22:06:29.974 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7786000000000001e-05 2024-09-21T22:06:29.975 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:29.975 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:29.975 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:29.975 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161545+0000", 2024-09-21T22:06:29.975 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8899999999999999e-05 2024-09-21T22:06:29.975 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:29.975 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:29.975 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:29.975 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.172615+0000", 2024-09-21T22:06:29.975 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011070465 2024-09-21T22:06:29.975 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:29.975 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:29.975 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:29.975 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.172759+0000", 2024-09-21T22:06:29.975 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014416900000000001 2024-09-21T22:06:29.975 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:29.976 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:29.976 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:06:29.976 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.172967+0000", 2024-09-21T22:06:29.976 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020758500000000001 2024-09-21T22:06:29.976 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:29.976 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:29.976 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:29.976 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.174775+0000", 2024-09-21T22:06:29.976 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0018080990000000001 2024-09-21T22:06:29.976 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:29.976 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:29.976 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:29.976 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.177447+0000", 2024-09-21T22:06:29.976 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026717429999999999 2024-09-21T22:06:29.976 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:29.976 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:29.977 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:29.977 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.177525+0000", 2024-09-21T22:06:29.977 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7846e-05 2024-09-21T22:06:29.977 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:29.977 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:29.977 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:29.977 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.177639+0000", 2024-09-21T22:06:29.977 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000114373 2024-09-21T22:06:29.977 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:29.977 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:29.977 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:29.977 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:29.977 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:29.977 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:06:29.977 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.161577+0000", 2024-09-21T22:06:29.977 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 27.794685428000001, 2024-09-21T22:06:29.978 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016652991999999998, 2024-09-21T22:06:29.978 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:29.978 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:29.978 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:29.978 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:29.978 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:06:29.978 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:06:29.978 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 37 2024-09-21T22:06:29.978 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:29.978 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:29.978 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:29.978 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:29.978 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161577+0000", 2024-09-21T22:06:29.978 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:29.978 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:29.979 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:29.979 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:29.979 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161577+0000", 2024-09-21T22:06:29.979 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:29.979 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:29.979 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:29.979 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:29.979 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161578+0000", 2024-09-21T22:06:29.979 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5269999999999999e-06 2024-09-21T22:06:29.979 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:29.979 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:29.979 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:29.979 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161591+0000", 2024-09-21T22:06:29.979 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.296e-05 2024-09-21T22:06:29.979 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:29.980 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:29.980 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:29.980 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161629+0000", 2024-09-21T22:06:29.980 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7799000000000002e-05 2024-09-21T22:06:29.980 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:29.980 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:29.980 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:29.980 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161644+0000", 2024-09-21T22:06:29.980 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5333e-05 2024-09-21T22:06:29.980 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:29.980 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:29.980 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:29.980 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.173636+0000", 2024-09-21T22:06:29.980 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011992078 2024-09-21T22:06:29.980 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:29.980 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:29.980 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:29.981 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.173862+0000", 2024-09-21T22:06:29.981 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022556799999999999 2024-09-21T22:06:29.981 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:29.981 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:29.981 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:06:29.981 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.174193+0000", 2024-09-21T22:06:29.981 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000331088 2024-09-21T22:06:29.981 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:29.981 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:29.981 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:29.981 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.175740+0000", 2024-09-21T22:06:29.981 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015467910000000001 2024-09-21T22:06:29.981 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:29.981 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:29.981 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:29.982 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.178073+0000", 2024-09-21T22:06:29.982 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002333356 2024-09-21T22:06:29.982 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:29.982 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:29.982 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:29.982 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.178162+0000", 2024-09-21T22:06:29.982 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8306000000000005e-05 2024-09-21T22:06:29.982 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:29.982 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:29.982 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:29.982 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.178230+0000", 2024-09-21T22:06:29.982 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8186000000000001e-05 2024-09-21T22:06:29.982 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:29.982 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:29.982 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:29.983 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:29.983 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:29.983 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:06:29.983 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.280540+0000", 2024-09-21T22:06:29.983 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 27.675721855999999, 2024-09-21T22:06:29.983 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013246212, 2024-09-21T22:06:29.983 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:29.983 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:29.983 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:29.983 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:29.983 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:06:29.983 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:06:29.983 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 88 2024-09-21T22:06:29.983 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:29.983 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:29.983 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:29.984 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:29.984 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.280540+0000", 2024-09-21T22:06:29.984 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:29.984 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:29.984 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:29.984 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:29.984 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.280540+0000", 2024-09-21T22:06:29.984 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:29.984 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:29.984 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:29.984 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:29.984 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.280541+0000", 2024-09-21T22:06:29.984 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.54e-07 2024-09-21T22:06:29.984 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:29.984 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:29.984 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:29.985 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.280549+0000", 2024-09-21T22:06:29.985 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3329999999999997e-06 2024-09-21T22:06:29.985 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:29.985 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:29.985 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:29.985 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.280560+0000", 2024-09-21T22:06:29.985 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1003e-05 2024-09-21T22:06:29.985 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:29.985 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:29.985 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:29.985 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.280569+0000", 2024-09-21T22:06:29.985 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2430000000000005e-06 2024-09-21T22:06:29.985 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:29.985 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:29.985 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:29.985 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.291095+0000", 2024-09-21T22:06:29.986 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010526259 2024-09-21T22:06:29.986 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:29.986 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:29.986 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:29.986 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.291231+0000", 2024-09-21T22:06:29.986 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013561899999999999 2024-09-21T22:06:29.986 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:29.986 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:29.986 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:06:29.986 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.291457+0000", 2024-09-21T22:06:29.986 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022686200000000001 2024-09-21T22:06:29.986 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:29.986 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:29.986 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:29.987 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.293117+0000", 2024-09-21T22:06:29.987 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001659474 2024-09-21T22:06:29.987 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:29.987 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:29.987 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:29.987 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.293524+0000", 2024-09-21T22:06:29.987 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00040701300000000001 2024-09-21T22:06:29.987 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:29.987 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:29.987 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:29.987 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.293673+0000", 2024-09-21T22:06:29.987 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000148782 2024-09-21T22:06:29.987 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:29.987 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:29.987 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:29.987 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.293786+0000", 2024-09-21T22:06:29.988 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011377 2024-09-21T22:06:29.988 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:29.988 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:29.988 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:29.988 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:29.988 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:29.988 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:06:29.988 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.466485+0000", 2024-09-21T22:06:29.988 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 27.489777125, 2024-09-21T22:06:29.988 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014443981, 2024-09-21T22:06:29.988 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:29.988 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:29.988 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:29.988 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:29.988 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:06:29.988 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:06:29.989 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 107 2024-09-21T22:06:29.989 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:29.989 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:29.989 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:29.989 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:29.989 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.466485+0000", 2024-09-21T22:06:29.989 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:29.989 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:29.989 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:29.989 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:29.989 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.466485+0000", 2024-09-21T22:06:29.989 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:29.989 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:29.989 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:29.989 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:29.989 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.466486+0000", 2024-09-21T22:06:29.990 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6299999999999993e-07 2024-09-21T22:06:29.990 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:29.990 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:29.990 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:29.990 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.466493+0000", 2024-09-21T22:06:29.990 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9639999999999997e-06 2024-09-21T22:06:29.990 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:29.990 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:29.990 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:29.990 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.466507+0000", 2024-09-21T22:06:29.990 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4216000000000001e-05 2024-09-21T22:06:29.990 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:29.990 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:29.990 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:29.990 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.466515+0000", 2024-09-21T22:06:29.991 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.157e-06 2024-09-21T22:06:29.991 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:29.991 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:29.991 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:29.991 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.472033+0000", 2024-09-21T22:06:29.991 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0055175839999999999 2024-09-21T22:06:29.991 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:29.991 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:29.991 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:29.991 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.472174+0000", 2024-09-21T22:06:29.991 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000141612 2024-09-21T22:06:29.991 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:29.991 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:29.991 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:06:29.992 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.472381+0000", 2024-09-21T22:06:29.992 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000206961 2024-09-21T22:06:29.992 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:29.992 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:29.992 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:29.992 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.473924+0000", 2024-09-21T22:06:29.992 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015427119999999999 2024-09-21T22:06:29.992 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:29.992 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:29.992 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:29.992 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.480666+0000", 2024-09-21T22:06:29.992 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0067415410000000002 2024-09-21T22:06:29.992 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:29.992 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:29.992 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:29.993 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.480811+0000", 2024-09-21T22:06:29.993 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000145422 2024-09-21T22:06:29.993 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:29.993 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:29.993 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:29.993 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.480929+0000", 2024-09-21T22:06:29.993 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000117849 2024-09-21T22:06:29.993 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:29.993 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:29.993 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:29.993 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:29.993 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:29.993 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:06:29.993 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.467057+0000", 2024-09-21T22:06:29.993 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 27.489204866000001, 2024-09-21T22:06:29.994 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018350195, 2024-09-21T22:06:29.994 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:29.994 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:29.994 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:29.994 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:29.994 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:06:29.994 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:06:29.994 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 108 2024-09-21T22:06:29.994 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:29.994 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:29.994 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:29.994 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:29.994 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467057+0000", 2024-09-21T22:06:29.995 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:29.995 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:29.995 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:29.995 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:29.995 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467057+0000", 2024-09-21T22:06:29.995 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:29.995 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:29.995 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:29.995 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:29.995 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467059+0000", 2024-09-21T22:06:29.995 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.826e-06 2024-09-21T22:06:29.995 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:29.995 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:29.995 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:29.996 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467071+0000", 2024-09-21T22:06:29.996 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2224e-05 2024-09-21T22:06:29.996 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:29.996 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:29.996 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:29.996 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467098+0000", 2024-09-21T22:06:29.996 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6953000000000001e-05 2024-09-21T22:06:29.996 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:29.996 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:29.996 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:29.996 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467113+0000", 2024-09-21T22:06:29.996 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5119999999999999e-05 2024-09-21T22:06:29.996 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:29.996 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:29.996 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:29.997 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.474060+0000", 2024-09-21T22:06:29.997 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0069462919999999997 2024-09-21T22:06:29.997 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:29.997 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:29.997 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:29.997 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.474283+0000", 2024-09-21T22:06:29.997 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000223315 2024-09-21T22:06:29.997 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:29.997 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:29.997 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:06:29.997 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.474660+0000", 2024-09-21T22:06:29.997 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00037729400000000002 2024-09-21T22:06:29.997 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:29.997 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:29.998 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:29.998 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.480622+0000", 2024-09-21T22:06:29.998 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0059613369999999997 2024-09-21T22:06:29.998 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:29.998 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:29.998 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:29.998 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.485145+0000", 2024-09-21T22:06:29.998 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0045231960000000002 2024-09-21T22:06:29.998 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:29.998 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:29.998 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:29.998 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.485288+0000", 2024-09-21T22:06:29.998 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000143232 2024-09-21T22:06:29.998 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:29.999 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:29.999 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:29.999 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.485407+0000", 2024-09-21T22:06:29.999 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000119406 2024-09-21T22:06:29.999 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:29.999 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:29.999 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:29.999 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:29.999 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:29.999 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:06:29.999 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.467144+0000", 2024-09-21T22:06:29.999 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 27.489117647, 2024-09-21T22:06:29.999 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018907146999999999, 2024-09-21T22:06:29.999 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:30.000 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:30.000 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:30.000 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:30.000 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:06:30.000 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:06:30.000 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 109 2024-09-21T22:06:30.000 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.000 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:30.000 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.000 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:30.000 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467144+0000", 2024-09-21T22:06:30.000 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:30.000 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.000 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.001 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:30.001 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467144+0000", 2024-09-21T22:06:30.001 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:30.001 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.001 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.001 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:30.001 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467145+0000", 2024-09-21T22:06:30.001 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9999999999999996e-07 2024-09-21T22:06:30.001 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.001 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.001 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:30.001 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467331+0000", 2024-09-21T22:06:30.001 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018518899999999999 2024-09-21T22:06:30.001 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.001 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.001 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:30.002 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467350+0000", 2024-09-21T22:06:30.002 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8966000000000001e-05 2024-09-21T22:06:30.002 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.002 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.002 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:30.002 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467361+0000", 2024-09-21T22:06:30.002 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1233000000000001e-05 2024-09-21T22:06:30.002 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.002 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.002 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:30.002 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.475613+0000", 2024-09-21T22:06:30.002 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0082518629999999999 2024-09-21T22:06:30.002 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.002 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.003 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:30.003 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.475859+0000", 2024-09-21T22:06:30.003 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024633800000000003 2024-09-21T22:06:30.003 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.003 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.003 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:06:30.003 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.476223+0000", 2024-09-21T22:06:30.003 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00036408700000000003 2024-09-21T22:06:30.003 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.003 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.003 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:30.003 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.481046+0000", 2024-09-21T22:06:30.003 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.004823356 2024-09-21T22:06:30.003 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.004 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.004 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:30.004 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.485722+0000", 2024-09-21T22:06:30.004 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0046753640000000004 2024-09-21T22:06:30.004 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.004 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.004 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:30.004 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.485908+0000", 2024-09-21T22:06:30.004 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000185859 2024-09-21T22:06:30.004 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.004 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.004 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:30.004 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.486052+0000", 2024-09-21T22:06:30.004 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014399200000000001 2024-09-21T22:06:30.004 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:30.005 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:30.005 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:30.005 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.005 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.005 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:06:30.005 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.467384+0000", 2024-09-21T22:06:30.005 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 27.488877818999999, 2024-09-21T22:06:30.005 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.01894144, 2024-09-21T22:06:30.005 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:30.005 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:30.005 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:30.005 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:30.005 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:06:30.005 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:06:30.005 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 110 2024-09-21T22:06:30.006 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.006 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:30.006 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.006 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:30.006 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467384+0000", 2024-09-21T22:06:30.006 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:30.006 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.006 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.006 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:30.006 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467384+0000", 2024-09-21T22:06:30.006 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:30.006 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.006 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.006 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:30.007 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467385+0000", 2024-09-21T22:06:30.007 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0470000000000001e-06 2024-09-21T22:06:30.007 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.007 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.007 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:30.007 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467393+0000", 2024-09-21T22:06:30.007 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3869999999999997e-06 2024-09-21T22:06:30.007 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.007 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.007 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:30.007 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467413+0000", 2024-09-21T22:06:30.007 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9953e-05 2024-09-21T22:06:30.007 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.007 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.008 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:30.008 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467421+0000", 2024-09-21T22:06:30.008 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2400000000000007e-06 2024-09-21T22:06:30.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.008 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:30.008 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.479542+0000", 2024-09-21T22:06:30.008 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012120832 2024-09-21T22:06:30.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.008 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:30.008 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.479705+0000", 2024-09-21T22:06:30.008 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016362899999999999 2024-09-21T22:06:30.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.009 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:06:30.009 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.479914+0000", 2024-09-21T22:06:30.009 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020814499999999999 2024-09-21T22:06:30.009 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.009 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.009 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:30.009 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.485536+0000", 2024-09-21T22:06:30.009 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0056222800000000003 2024-09-21T22:06:30.009 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.009 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.009 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:30.009 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.486077+0000", 2024-09-21T22:06:30.009 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00054079900000000001 2024-09-21T22:06:30.009 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.009 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.010 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:30.010 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.486213+0000", 2024-09-21T22:06:30.010 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000136682 2024-09-21T22:06:30.010 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.010 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.010 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:30.010 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.486326+0000", 2024-09-21T22:06:30.010 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000112446 2024-09-21T22:06:30.010 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:30.010 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:30.010 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:30.010 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.010 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.010 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:06:30.011 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.467436+0000", 2024-09-21T22:06:30.011 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 27.488826319000001, 2024-09-21T22:06:30.011 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019152168000000001, 2024-09-21T22:06:30.011 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:30.011 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:30.011 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:30.011 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:30.011 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:06:30.011 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:06:30.011 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 111 2024-09-21T22:06:30.011 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.011 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:30.011 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.011 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:30.011 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467436+0000", 2024-09-21T22:06:30.012 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:30.012 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.012 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:30.012 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467436+0000", 2024-09-21T22:06:30.012 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:30.012 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.012 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:30.012 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467437+0000", 2024-09-21T22:06:30.012 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0699999999999996e-07 2024-09-21T22:06:30.012 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.012 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:30.013 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467445+0000", 2024-09-21T22:06:30.013 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.789e-06 2024-09-21T22:06:30.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.013 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.013 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:30.013 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467467+0000", 2024-09-21T22:06:30.013 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1253000000000001e-05 2024-09-21T22:06:30.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.013 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.013 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:30.013 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467475+0000", 2024-09-21T22:06:30.013 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0039999999999999e-06 2024-09-21T22:06:30.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.013 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.014 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:30.014 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.481164+0000", 2024-09-21T22:06:30.014 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013689114 2024-09-21T22:06:30.014 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.014 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.014 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:30.014 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.481394+0000", 2024-09-21T22:06:30.014 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000229931 2024-09-21T22:06:30.014 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.014 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.014 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:06:30.014 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.481779+0000", 2024-09-21T22:06:30.014 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00038545400000000003 2024-09-21T22:06:30.014 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.014 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.015 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:30.015 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.485675+0000", 2024-09-21T22:06:30.015 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0038958370000000001 2024-09-21T22:06:30.015 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.015 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.015 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:30.015 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.486346+0000", 2024-09-21T22:06:30.015 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00067119100000000002 2024-09-21T22:06:30.015 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.015 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.015 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:30.016 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.486478+0000", 2024-09-21T22:06:30.016 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013182600000000001 2024-09-21T22:06:30.016 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.016 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.016 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:30.016 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.486588+0000", 2024-09-21T22:06:30.016 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000110062 2024-09-21T22:06:30.016 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:30.016 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:30.016 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:30.016 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.016 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.016 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:06:30.016 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.467488+0000", 2024-09-21T22:06:30.016 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 27.488773692999999, 2024-09-21T22:06:30.017 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021038304000000001, 2024-09-21T22:06:30.017 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:30.017 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:30.017 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:30.017 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:30.017 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:06:30.017 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:06:30.017 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 112 2024-09-21T22:06:30.017 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.017 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:30.017 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.017 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:30.017 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467488+0000", 2024-09-21T22:06:30.017 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:30.017 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.017 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.018 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:30.018 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467488+0000", 2024-09-21T22:06:30.018 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:30.018 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.018 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.018 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:30.018 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467489+0000", 2024-09-21T22:06:30.018 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7999999999999995e-07 2024-09-21T22:06:30.018 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.018 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.018 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:30.018 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467687+0000", 2024-09-21T22:06:30.018 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019776200000000001 2024-09-21T22:06:30.018 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.018 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.019 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:30.019 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467704+0000", 2024-09-21T22:06:30.019 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7193e-05 2024-09-21T22:06:30.019 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.019 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.019 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:30.019 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467719+0000", 2024-09-21T22:06:30.019 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4703e-05 2024-09-21T22:06:30.019 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.019 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.019 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:30.019 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.482745+0000", 2024-09-21T22:06:30.019 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015025801 2024-09-21T22:06:30.019 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.019 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.020 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:30.020 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.483015+0000", 2024-09-21T22:06:30.020 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026999400000000002 2024-09-21T22:06:30.020 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.020 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.020 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:06:30.020 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.483440+0000", 2024-09-21T22:06:30.020 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000425697 2024-09-21T22:06:30.020 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.020 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.020 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:30.020 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.486701+0000", 2024-09-21T22:06:30.020 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0032611950000000002 2024-09-21T22:06:30.020 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.020 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.020 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:30.021 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.488262+0000", 2024-09-21T22:06:30.021 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015610439999999999 2024-09-21T22:06:30.021 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.021 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.021 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:30.021 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.488406+0000", 2024-09-21T22:06:30.021 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000143232 2024-09-21T22:06:30.021 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.021 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.021 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:30.021 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.488527+0000", 2024-09-21T22:06:30.021 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012100299999999999 2024-09-21T22:06:30.021 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:30.021 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:30.021 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:30.021 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.022 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.022 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:06:30.022 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.467786+0000", 2024-09-21T22:06:30.022 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 27.488476267999999, 2024-09-21T22:06:30.022 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026339741999999999, 2024-09-21T22:06:30.022 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:30.022 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:30.022 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:30.022 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:30.022 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:06:30.022 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:06:30.022 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 113 2024-09-21T22:06:30.022 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.022 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:30.022 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.023 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:30.023 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467786+0000", 2024-09-21T22:06:30.023 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:30.023 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.023 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.023 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:30.023 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467786+0000", 2024-09-21T22:06:30.023 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:30.023 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.023 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.023 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:30.023 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467787+0000", 2024-09-21T22:06:30.023 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.446e-06 2024-09-21T22:06:30.023 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.023 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.024 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:30.024 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467802+0000", 2024-09-21T22:06:30.024 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4769999999999999e-05 2024-09-21T22:06:30.024 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.024 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.024 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:30.024 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467826+0000", 2024-09-21T22:06:30.024 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3430000000000001e-05 2024-09-21T22:06:30.024 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.024 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.024 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:30.024 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467843+0000", 2024-09-21T22:06:30.024 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7065999999999999e-05 2024-09-21T22:06:30.024 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.024 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.025 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:30.025 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.486820+0000", 2024-09-21T22:06:30.025 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018977233999999999 2024-09-21T22:06:30.025 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.025 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.025 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:30.025 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.487065+0000", 2024-09-21T22:06:30.025 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024558799999999998 2024-09-21T22:06:30.025 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.025 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.025 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:06:30.025 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.487442+0000", 2024-09-21T22:06:30.025 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000376707 2024-09-21T22:06:30.025 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.025 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.025 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:30.026 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.492256+0000", 2024-09-21T22:06:30.026 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0048141620000000003 2024-09-21T22:06:30.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.026 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.026 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:30.026 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.493865+0000", 2024-09-21T22:06:30.026 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016090150000000001 2024-09-21T22:06:30.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.026 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.026 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:30.026 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.494007+0000", 2024-09-21T22:06:30.026 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000141359 2024-09-21T22:06:30.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.026 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.027 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:30.027 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.494126+0000", 2024-09-21T22:06:30.027 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000118965 2024-09-21T22:06:30.027 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:30.027 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:30.027 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:30.027 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.027 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.027 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:06:30.027 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.467878+0000", 2024-09-21T22:06:30.027 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 27.488383778999999, 2024-09-21T22:06:30.027 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.027783958000000001, 2024-09-21T22:06:30.027 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:30.027 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:30.028 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:30.028 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:30.028 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:06:30.028 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:06:30.028 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 114 2024-09-21T22:06:30.028 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.028 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:30.028 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.028 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:30.028 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467878+0000", 2024-09-21T22:06:30.028 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:30.028 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.028 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.028 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:30.029 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467878+0000", 2024-09-21T22:06:30.029 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:30.029 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.029 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.029 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:30.029 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467880+0000", 2024-09-21T22:06:30.029 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.223e-06 2024-09-21T22:06:30.029 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.029 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.029 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:30.029 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467891+0000", 2024-09-21T22:06:30.029 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1527e-05 2024-09-21T22:06:30.029 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.029 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.029 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:30.030 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467913+0000", 2024-09-21T22:06:30.030 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.143e-05 2024-09-21T22:06:30.030 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.030 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.030 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:30.030 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467930+0000", 2024-09-21T22:06:30.030 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7620000000000001e-05 2024-09-21T22:06:30.030 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.030 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.030 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:30.030 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.488690+0000", 2024-09-21T22:06:30.031 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020759588999999998 2024-09-21T22:06:30.031 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.031 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.031 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:30.031 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.488882+0000", 2024-09-21T22:06:30.031 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019236200000000001 2024-09-21T22:06:30.031 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.031 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.032 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:06:30.032 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.489093+0000", 2024-09-21T22:06:30.032 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000211318 2024-09-21T22:06:30.032 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.032 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.032 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:30.032 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.494152+0000", 2024-09-21T22:06:30.032 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0050582769999999999 2024-09-21T22:06:30.032 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.032 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.032 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:30.032 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.495373+0000", 2024-09-21T22:06:30.032 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001221747 2024-09-21T22:06:30.032 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.033 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.033 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:30.033 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.495543+0000", 2024-09-21T22:06:30.033 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000169259 2024-09-21T22:06:30.033 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.033 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.033 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:30.033 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.495662+0000", 2024-09-21T22:06:30.033 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011960600000000001 2024-09-21T22:06:30.033 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:30.033 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:30.033 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:30.033 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.033 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.034 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:06:30.034 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.468118+0000", 2024-09-21T22:06:30.034 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 27.488144038000002, 2024-09-21T22:06:30.034 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.027938877000000001, 2024-09-21T22:06:30.034 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:30.034 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:30.034 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:30.034 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:30.034 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:06:30.034 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:06:30.034 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 115 2024-09-21T22:06:30.034 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.034 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:30.034 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.035 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:30.035 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468118+0000", 2024-09-21T22:06:30.035 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:30.035 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.035 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:30.035 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468118+0000", 2024-09-21T22:06:30.035 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:30.035 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.035 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:30.035 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468120+0000", 2024-09-21T22:06:30.035 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7069999999999999e-06 2024-09-21T22:06:30.035 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.036 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:30.036 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468381+0000", 2024-09-21T22:06:30.036 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026135399999999999 2024-09-21T22:06:30.036 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.036 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.036 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:30.036 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468409+0000", 2024-09-21T22:06:30.036 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.781e-05 2024-09-21T22:06:30.036 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.036 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.036 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:30.036 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468433+0000", 2024-09-21T22:06:30.036 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4343999999999999e-05 2024-09-21T22:06:30.036 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.036 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.036 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:30.037 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.489785+0000", 2024-09-21T22:06:30.037 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021351221 2024-09-21T22:06:30.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.037 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.037 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:30.037 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.490018+0000", 2024-09-21T22:06:30.037 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023317800000000001 2024-09-21T22:06:30.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.037 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.037 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:06:30.037 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.490412+0000", 2024-09-21T22:06:30.037 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00039417400000000001 2024-09-21T22:06:30.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.037 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.037 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:30.038 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.495694+0000", 2024-09-21T22:06:30.038 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.005282335 2024-09-21T22:06:30.038 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.038 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.038 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:30.038 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.495852+0000", 2024-09-21T22:06:30.038 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015777200000000001 2024-09-21T22:06:30.038 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.038 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.038 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:30.038 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.495938+0000", 2024-09-21T22:06:30.038 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6493e-05 2024-09-21T22:06:30.038 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.038 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.039 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:30.039 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.496057+0000", 2024-09-21T22:06:30.039 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000118489 2024-09-21T22:06:30.039 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:30.039 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:30.039 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:30.039 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.039 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.039 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:06:30.039 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.468474+0000", 2024-09-21T22:06:30.039 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 27.487788277, 2024-09-21T22:06:30.039 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.028015895999999998, 2024-09-21T22:06:30.039 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:30.039 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:30.040 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:30.040 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:30.040 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:06:30.040 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:06:30.040 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 116 2024-09-21T22:06:30.040 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.040 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:30.040 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.040 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:30.040 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468474+0000", 2024-09-21T22:06:30.040 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:30.040 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.040 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.041 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:30.041 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468474+0000", 2024-09-21T22:06:30.041 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:30.041 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.041 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.041 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:30.041 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468476+0000", 2024-09-21T22:06:30.041 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7060000000000001e-06 2024-09-21T22:06:30.041 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.041 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.041 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:30.041 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468488+0000", 2024-09-21T22:06:30.041 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.234e-05 2024-09-21T22:06:30.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.042 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.042 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:30.042 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468517+0000", 2024-09-21T22:06:30.042 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.898e-05 2024-09-21T22:06:30.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.042 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.042 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:30.042 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468536+0000", 2024-09-21T22:06:30.042 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8899999999999999e-05 2024-09-21T22:06:30.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.042 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.042 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:30.042 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.492376+0000", 2024-09-21T22:06:30.043 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023840659 2024-09-21T22:06:30.043 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.043 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.043 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:30.043 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.492621+0000", 2024-09-21T22:06:30.043 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000244794 2024-09-21T22:06:30.043 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.043 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.043 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:06:30.043 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.493008+0000", 2024-09-21T22:06:30.043 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00038629099999999998 2024-09-21T22:06:30.043 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.043 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.043 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:30.044 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.496192+0000", 2024-09-21T22:06:30.044 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0031847220000000001 2024-09-21T22:06:30.044 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.044 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.044 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:30.044 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.496233+0000", 2024-09-21T22:06:30.044 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0410000000000001e-05 2024-09-21T22:06:30.044 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.044 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.044 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:30.044 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.496374+0000", 2024-09-21T22:06:30.044 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140872 2024-09-21T22:06:30.044 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.044 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.045 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:30.045 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.496490+0000", 2024-09-21T22:06:30.045 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000116222 2024-09-21T22:06:30.045 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:30.045 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:30.045 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:30.045 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.045 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.045 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:06:30.045 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.468568+0000", 2024-09-21T22:06:30.045 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 27.487694158, 2024-09-21T22:06:30.045 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.028181578999999998, 2024-09-21T22:06:30.045 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:30.045 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:30.046 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:30.046 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:30.046 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:06:30.046 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:06:30.046 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 117 2024-09-21T22:06:30.046 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.046 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:30.046 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.046 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:30.046 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468568+0000", 2024-09-21T22:06:30.046 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:30.046 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.046 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.047 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:30.047 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468568+0000", 2024-09-21T22:06:30.047 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:30.047 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.047 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.047 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:30.047 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468569+0000", 2024-09-21T22:06:30.047 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4139999999999999e-06 2024-09-21T22:06:30.047 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.047 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.047 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:30.047 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468583+0000", 2024-09-21T22:06:30.047 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3409e-05 2024-09-21T22:06:30.047 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.047 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.048 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:30.048 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468608+0000", 2024-09-21T22:06:30.048 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5559999999999999e-05 2024-09-21T22:06:30.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.048 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:30.048 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468626+0000", 2024-09-21T22:06:30.048 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7863000000000002e-05 2024-09-21T22:06:30.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.048 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:30.048 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.494260+0000", 2024-09-21T22:06:30.048 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025633744999999999 2024-09-21T22:06:30.049 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.049 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.049 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:30.049 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.494519+0000", 2024-09-21T22:06:30.049 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025924800000000001 2024-09-21T22:06:30.049 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.049 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.049 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:06:30.049 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.494747+0000", 2024-09-21T22:06:30.049 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022792799999999999 2024-09-21T22:06:30.049 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.049 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.049 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:30.049 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.496519+0000", 2024-09-21T22:06:30.050 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017720030000000001 2024-09-21T22:06:30.050 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.050 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.050 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:30.050 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.496607+0000", 2024-09-21T22:06:30.050 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7873000000000004e-05 2024-09-21T22:06:30.050 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.050 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.050 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:30.050 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.496691+0000", 2024-09-21T22:06:30.050 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4009000000000002e-05 2024-09-21T22:06:30.050 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.050 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.051 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:30.051 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.496750+0000", 2024-09-21T22:06:30.051 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8526999999999997e-05 2024-09-21T22:06:30.051 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:30.051 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:30.051 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:30.051 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.051 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.051 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:06:30.051 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.898244+0000", 2024-09-21T22:06:30.051 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 27.058017854999999, 2024-09-21T22:06:30.051 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014749464, 2024-09-21T22:06:30.051 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:30.051 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:30.052 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:30.052 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:30.052 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:06:30.052 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:06:30.052 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 140 2024-09-21T22:06:30.052 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.052 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:30.052 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.052 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:30.052 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898244+0000", 2024-09-21T22:06:30.052 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:30.052 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.052 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.052 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:30.053 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898244+0000", 2024-09-21T22:06:30.053 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:30.053 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.053 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.053 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:30.053 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898245+0000", 2024-09-21T22:06:30.053 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3000000000000002e-07 2024-09-21T22:06:30.053 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.053 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.053 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:30.053 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898308+0000", 2024-09-21T22:06:30.053 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3468999999999998e-05 2024-09-21T22:06:30.053 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.054 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.054 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:30.054 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898315+0000", 2024-09-21T22:06:30.054 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.725e-06 2024-09-21T22:06:30.054 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.054 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.054 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:30.054 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898321+0000", 2024-09-21T22:06:30.054 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2090000000000002e-06 2024-09-21T22:06:30.054 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.054 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.054 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:30.054 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.905452+0000", 2024-09-21T22:06:30.055 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0071305989999999996 2024-09-21T22:06:30.055 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.055 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.055 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:30.055 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.905504+0000", 2024-09-21T22:06:30.055 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2638e-05 2024-09-21T22:06:30.055 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.055 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.055 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:06:30.055 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.905590+0000", 2024-09-21T22:06:30.055 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5333000000000004e-05 2024-09-21T22:06:30.055 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.055 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.055 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:30.056 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.911542+0000", 2024-09-21T22:06:30.056 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0059519819999999998 2024-09-21T22:06:30.056 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.056 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.056 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:30.056 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.912908+0000", 2024-09-21T22:06:30.056 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0013666640000000001 2024-09-21T22:06:30.056 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.056 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.056 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:30.056 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.912957+0000", 2024-09-21T22:06:30.056 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9203000000000003e-05 2024-09-21T22:06:30.056 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.057 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.057 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:30.057 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.912994+0000", 2024-09-21T22:06:30.057 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6312000000000001e-05 2024-09-21T22:06:30.057 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:30.057 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:30.057 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:30.057 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.057 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.057 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:06:30.057 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.898342+0000", 2024-09-21T22:06:30.057 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 27.057919717000001, 2024-09-21T22:06:30.057 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014787918000000001, 2024-09-21T22:06:30.057 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:30.057 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:30.058 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:30.058 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:30.058 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:06:30.058 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:06:30.058 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 141 2024-09-21T22:06:30.058 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.058 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:30.058 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.058 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:30.058 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898342+0000", 2024-09-21T22:06:30.058 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:30.058 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.058 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.058 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:30.059 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898342+0000", 2024-09-21T22:06:30.059 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:30.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.059 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.059 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:30.059 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898343+0000", 2024-09-21T22:06:30.059 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3000000000000001e-07 2024-09-21T22:06:30.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.059 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.059 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:30.059 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898346+0000", 2024-09-21T22:06:30.059 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6009999999999999e-06 2024-09-21T22:06:30.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.059 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.059 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:30.059 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898352+0000", 2024-09-21T22:06:30.060 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8089999999999998e-06 2024-09-21T22:06:30.060 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.060 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.060 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:30.060 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898358+0000", 2024-09-21T22:06:30.060 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4349999999999996e-06 2024-09-21T22:06:30.060 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.060 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.060 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:30.060 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.909387+0000", 2024-09-21T22:06:30.060 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011029105000000001 2024-09-21T22:06:30.060 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.060 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.060 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:30.061 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.909462+0000", 2024-09-21T22:06:30.061 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5192999999999996e-05 2024-09-21T22:06:30.061 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.061 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.061 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:06:30.061 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.909551+0000", 2024-09-21T22:06:30.061 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9046000000000001e-05 2024-09-21T22:06:30.061 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.061 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.061 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:30.061 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.911852+0000", 2024-09-21T22:06:30.061 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0023005579999999999 2024-09-21T22:06:30.061 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.061 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.061 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:30.061 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.913002+0000", 2024-09-21T22:06:30.062 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011502929999999999 2024-09-21T22:06:30.062 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.062 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.062 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:30.062 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.913060+0000", 2024-09-21T22:06:30.062 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8427000000000002e-05 2024-09-21T22:06:30.062 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.062 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.062 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:30.062 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.913130+0000", 2024-09-21T22:06:30.062 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0020999999999999e-05 2024-09-21T22:06:30.062 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:30.062 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:30.062 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:30.062 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.063 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.063 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:158 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T22:06:30.063 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:03.511201+0000", 2024-09-21T22:06:30.063 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 26.445061234000001, 2024-09-21T22:06:30.063 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013353538999999999, 2024-09-21T22:06:30.063 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:30.063 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:30.063 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:30.063 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:30.063 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:06:30.063 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:06:30.063 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 158 2024-09-21T22:06:30.063 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.064 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:30.064 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.064 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:30.064 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511201+0000", 2024-09-21T22:06:30.064 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:30.064 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.064 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.064 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:30.064 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511201+0000", 2024-09-21T22:06:30.064 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:30.064 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.064 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.064 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:30.064 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511202+0000", 2024-09-21T22:06:30.064 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9299999999999997e-07 2024-09-21T22:06:30.064 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.065 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:30.065 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511315+0000", 2024-09-21T22:06:30.065 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000113909 2024-09-21T22:06:30.065 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.065 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:30.065 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511333+0000", 2024-09-21T22:06:30.065 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7062999999999999e-05 2024-09-21T22:06:30.065 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.065 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:30.065 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511344+0000", 2024-09-21T22:06:30.065 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1177e-05 2024-09-21T22:06:30.065 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.066 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:30.066 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.517415+0000", 2024-09-21T22:06:30.066 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0060709830000000003 2024-09-21T22:06:30.066 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.066 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.066 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:30.066 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.517539+0000", 2024-09-21T22:06:30.066 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000124432 2024-09-21T22:06:30.066 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.066 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.066 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:06:30.066 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.517732+0000", 2024-09-21T22:06:30.066 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019262200000000001 2024-09-21T22:06:30.066 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.066 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.067 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:30.067 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.520709+0000", 2024-09-21T22:06:30.067 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0029767470000000001 2024-09-21T22:06:30.067 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.067 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.067 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:30.067 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.524389+0000", 2024-09-21T22:06:30.067 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003680975 2024-09-21T22:06:30.067 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.067 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.067 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:30.067 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.524486+0000", 2024-09-21T22:06:30.067 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6575000000000003e-05 2024-09-21T22:06:30.067 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.067 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.068 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:30.068 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.524554+0000", 2024-09-21T22:06:30.068 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8362999999999996e-05 2024-09-21T22:06:30.068 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:30.068 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:30.068 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:30.068 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.068 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.068 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T22:06:30.068 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:03.511368+0000", 2024-09-21T22:06:30.068 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 26.444894332000001, 2024-09-21T22:06:30.068 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013312133, 2024-09-21T22:06:30.068 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:30.068 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:30.068 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:30.069 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:30.069 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:06:30.069 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:06:30.069 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 159 2024-09-21T22:06:30.069 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.069 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:30.069 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.069 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:30.069 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511368+0000", 2024-09-21T22:06:30.069 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:30.069 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.069 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.069 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:30.069 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511368+0000", 2024-09-21T22:06:30.069 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:30.069 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.070 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.070 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:30.070 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511369+0000", 2024-09-21T22:06:30.070 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9599999999999998e-07 2024-09-21T22:06:30.070 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.070 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.070 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:30.070 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511382+0000", 2024-09-21T22:06:30.070 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3487e-05 2024-09-21T22:06:30.070 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.070 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.070 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:30.070 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511394+0000", 2024-09-21T22:06:30.071 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1337e-05 2024-09-21T22:06:30.071 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.071 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.071 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:30.071 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511402+0000", 2024-09-21T22:06:30.071 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0059999999999992e-06 2024-09-21T22:06:30.071 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.071 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.071 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:30.071 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.519376+0000", 2024-09-21T22:06:30.071 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0079746520000000005 2024-09-21T22:06:30.071 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.071 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.071 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:30.072 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.519498+0000", 2024-09-21T22:06:30.072 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012212900000000001 2024-09-21T22:06:30.072 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.072 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.072 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:06:30.072 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.519728+0000", 2024-09-21T22:06:30.072 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022985799999999999 2024-09-21T22:06:30.072 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.072 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.072 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:30.072 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.521344+0000", 2024-09-21T22:06:30.072 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016163169999999999 2024-09-21T22:06:30.072 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.072 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.073 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:30.073 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.524568+0000", 2024-09-21T22:06:30.073 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0032234220000000001 2024-09-21T22:06:30.073 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.073 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.073 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:30.073 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.524633+0000", 2024-09-21T22:06:30.073 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5126e-05 2024-09-21T22:06:30.073 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:30.073 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:30.073 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:30.073 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.524680+0000", 2024-09-21T22:06:30.073 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6903000000000001e-05 2024-09-21T22:06:30.073 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:30.074 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:30.074 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:30.074 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:30.074 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:30.074 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:06:30.075 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T22:06:30.076 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:06:30.073+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:06:30.176 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:06:30.173+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:06:30.248 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:06:30.248 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:06:30.248 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-09-21T22:06:30.248 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:06:30.262 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T22:06:30.307 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:06:30.304+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:06:30.331 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-09-21T22:06:30.331 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T22:06:30.331 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-21T22:06:30.331 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-09-21T22:06:30.378 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:06:30.377+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:06:30.381 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:06:30.381 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:06:30.381 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-21T22:06:30.381 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-21T22:06:30.381 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:06:30.395 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T22:06:30.477 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:06:30.473+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:06:30.562 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:06:30.562 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-21T22:06:30.562 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-21T22:06:30.562 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [] 2024-09-21T22:06:30.562 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:06:30.577 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T22:06:30.579 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:06:30.577+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:06:30.647 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T22:06:30.647 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:06:30.647 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 29, 2024-09-21T22:06:30.647 INFO:teuthology.orchestra.run.smithi039.stdout: "fsid": "00bb09a3-e410-4e18-a2b2-08eaf61978e6", 2024-09-21T22:06:30.647 INFO:teuthology.orchestra.run.smithi039.stdout: "created": "2024-09-21T22:05:45.303341+0000", 2024-09-21T22:06:30.647 INFO:teuthology.orchestra.run.smithi039.stdout: "modified": "2024-09-21T22:06:28.665546+0000", 2024-09-21T22:06:30.647 INFO:teuthology.orchestra.run.smithi039.stdout: "last_up_change": "2024-09-21T22:05:58.958630+0000", 2024-09-21T22:06:30.647 INFO:teuthology.orchestra.run.smithi039.stdout: "last_in_change": "2024-09-21T22:06:20.134002+0000", 2024-09-21T22:06:30.648 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-21T22:06:30.648 INFO:teuthology.orchestra.run.smithi039.stdout: "flags_num": 5806080, 2024-09-21T22:06:30.648 INFO:teuthology.orchestra.run.smithi039.stdout: "flags_set": [ 2024-09-21T22:06:30.648 INFO:teuthology.orchestra.run.smithi039.stdout: "nodeep-scrub", 2024-09-21T22:06:30.648 INFO:teuthology.orchestra.run.smithi039.stdout: "noscrub", 2024-09-21T22:06:30.648 INFO:teuthology.orchestra.run.smithi039.stdout: "pglog_hardlimit", 2024-09-21T22:06:30.648 INFO:teuthology.orchestra.run.smithi039.stdout: "purged_snapdirs", 2024-09-21T22:06:30.648 INFO:teuthology.orchestra.run.smithi039.stdout: "recovery_deletes", 2024-09-21T22:06:30.648 INFO:teuthology.orchestra.run.smithi039.stdout: "sortbitwise" 2024-09-21T22:06:30.648 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T22:06:30.648 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_version": 4, 2024-09-21T22:06:30.648 INFO:teuthology.orchestra.run.smithi039.stdout: "full_ratio": 0.94999998807907104, 2024-09-21T22:06:30.648 INFO:teuthology.orchestra.run.smithi039.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-21T22:06:30.648 INFO:teuthology.orchestra.run.smithi039.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-21T22:06:30.649 INFO:teuthology.orchestra.run.smithi039.stdout: "cluster_snapshot": "", 2024-09-21T22:06:30.649 INFO:teuthology.orchestra.run.smithi039.stdout: "pool_max": 2, 2024-09-21T22:06:30.649 INFO:teuthology.orchestra.run.smithi039.stdout: "max_osd": 6, 2024-09-21T22:06:30.649 INFO:teuthology.orchestra.run.smithi039.stdout: "require_min_compat_client": "luminous", 2024-09-21T22:06:30.649 INFO:teuthology.orchestra.run.smithi039.stdout: "min_compat_client": "jewel", 2024-09-21T22:06:30.649 INFO:teuthology.orchestra.run.smithi039.stdout: "require_osd_release": "squid", 2024-09-21T22:06:30.649 INFO:teuthology.orchestra.run.smithi039.stdout: "allow_crimson": false, 2024-09-21T22:06:30.649 INFO:teuthology.orchestra.run.smithi039.stdout: "pools": [ 2024-09-21T22:06:30.649 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:30.649 INFO:teuthology.orchestra.run.smithi039.stdout: "pool": 1, 2024-09-21T22:06:30.649 INFO:teuthology.orchestra.run.smithi039.stdout: "pool_name": ".mgr", 2024-09-21T22:06:30.649 INFO:teuthology.orchestra.run.smithi039.stdout: "create_time": "2024-09-21T22:06:00.868591+0000", 2024-09-21T22:06:30.649 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T22:06:30.650 INFO:teuthology.orchestra.run.smithi039.stdout: "flags_names": "hashpspool", 2024-09-21T22:06:30.650 INFO:teuthology.orchestra.run.smithi039.stdout: "type": 1, 2024-09-21T22:06:30.650 INFO:teuthology.orchestra.run.smithi039.stdout: "size": 2, 2024-09-21T22:06:30.650 INFO:teuthology.orchestra.run.smithi039.stdout: "min_size": 1, 2024-09-21T22:06:30.650 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_rule": 0, 2024-09-21T22:06:30.650 INFO:teuthology.orchestra.run.smithi039.stdout: "peering_crush_bucket_count": 0, 2024-09-21T22:06:30.650 INFO:teuthology.orchestra.run.smithi039.stdout: "peering_crush_bucket_target": 0, 2024-09-21T22:06:30.650 INFO:teuthology.orchestra.run.smithi039.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T22:06:30.650 INFO:teuthology.orchestra.run.smithi039.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T22:06:30.650 INFO:teuthology.orchestra.run.smithi039.stdout: "is_stretch_pool": false, 2024-09-21T22:06:30.650 INFO:teuthology.orchestra.run.smithi039.stdout: "object_hash": 2, 2024-09-21T22:06:30.650 INFO:teuthology.orchestra.run.smithi039.stdout: "pg_autoscale_mode": "off", 2024-09-21T22:06:30.650 INFO:teuthology.orchestra.run.smithi039.stdout: "pg_num": 1, 2024-09-21T22:06:30.651 INFO:teuthology.orchestra.run.smithi039.stdout: "pg_placement_num": 1, 2024-09-21T22:06:30.651 INFO:teuthology.orchestra.run.smithi039.stdout: "pg_placement_num_target": 1, 2024-09-21T22:06:30.651 INFO:teuthology.orchestra.run.smithi039.stdout: "pg_num_target": 1, 2024-09-21T22:06:30.651 INFO:teuthology.orchestra.run.smithi039.stdout: "pg_num_pending": 1, 2024-09-21T22:06:30.651 INFO:teuthology.orchestra.run.smithi039.stdout: "last_pg_merge_meta": { 2024-09-21T22:06:30.651 INFO:teuthology.orchestra.run.smithi039.stdout: "source_pgid": "0.0", 2024-09-21T22:06:30.651 INFO:teuthology.orchestra.run.smithi039.stdout: "ready_epoch": 0, 2024-09-21T22:06:30.651 INFO:teuthology.orchestra.run.smithi039.stdout: "last_epoch_started": 0, 2024-09-21T22:06:30.651 INFO:teuthology.orchestra.run.smithi039.stdout: "last_epoch_clean": 0, 2024-09-21T22:06:30.651 INFO:teuthology.orchestra.run.smithi039.stdout: "source_version": "0'0", 2024-09-21T22:06:30.651 INFO:teuthology.orchestra.run.smithi039.stdout: "target_version": "0'0" 2024-09-21T22:06:30.651 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:30.651 INFO:teuthology.orchestra.run.smithi039.stdout: "last_change": "15", 2024-09-21T22:06:30.652 INFO:teuthology.orchestra.run.smithi039.stdout: "last_force_op_resend": "0", 2024-09-21T22:06:30.652 INFO:teuthology.orchestra.run.smithi039.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T22:06:30.652 INFO:teuthology.orchestra.run.smithi039.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T22:06:30.652 INFO:teuthology.orchestra.run.smithi039.stdout: "auid": 0, 2024-09-21T22:06:30.652 INFO:teuthology.orchestra.run.smithi039.stdout: "snap_mode": "selfmanaged", 2024-09-21T22:06:30.652 INFO:teuthology.orchestra.run.smithi039.stdout: "snap_seq": 0, 2024-09-21T22:06:30.652 INFO:teuthology.orchestra.run.smithi039.stdout: "snap_epoch": 0, 2024-09-21T22:06:30.652 INFO:teuthology.orchestra.run.smithi039.stdout: "pool_snaps": [], 2024-09-21T22:06:30.652 INFO:teuthology.orchestra.run.smithi039.stdout: "removed_snaps": "[]", 2024-09-21T22:06:30.652 INFO:teuthology.orchestra.run.smithi039.stdout: "quota_max_bytes": 0, 2024-09-21T22:06:30.652 INFO:teuthology.orchestra.run.smithi039.stdout: "quota_max_objects": 0, 2024-09-21T22:06:30.652 INFO:teuthology.orchestra.run.smithi039.stdout: "tiers": [], 2024-09-21T22:06:30.653 INFO:teuthology.orchestra.run.smithi039.stdout: "tier_of": -1, 2024-09-21T22:06:30.653 INFO:teuthology.orchestra.run.smithi039.stdout: "read_tier": -1, 2024-09-21T22:06:30.653 INFO:teuthology.orchestra.run.smithi039.stdout: "write_tier": -1, 2024-09-21T22:06:30.653 INFO:teuthology.orchestra.run.smithi039.stdout: "cache_mode": "none", 2024-09-21T22:06:30.653 INFO:teuthology.orchestra.run.smithi039.stdout: "target_max_bytes": 0, 2024-09-21T22:06:30.653 INFO:teuthology.orchestra.run.smithi039.stdout: "target_max_objects": 0, 2024-09-21T22:06:30.653 INFO:teuthology.orchestra.run.smithi039.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T22:06:30.653 INFO:teuthology.orchestra.run.smithi039.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T22:06:30.653 INFO:teuthology.orchestra.run.smithi039.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T22:06:30.653 INFO:teuthology.orchestra.run.smithi039.stdout: "cache_min_flush_age": 0, 2024-09-21T22:06:30.653 INFO:teuthology.orchestra.run.smithi039.stdout: "cache_min_evict_age": 0, 2024-09-21T22:06:30.653 INFO:teuthology.orchestra.run.smithi039.stdout: "erasure_code_profile": "", 2024-09-21T22:06:30.653 INFO:teuthology.orchestra.run.smithi039.stdout: "hit_set_params": { 2024-09-21T22:06:30.654 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "none" 2024-09-21T22:06:30.654 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:30.654 INFO:teuthology.orchestra.run.smithi039.stdout: "hit_set_period": 0, 2024-09-21T22:06:30.654 INFO:teuthology.orchestra.run.smithi039.stdout: "hit_set_count": 0, 2024-09-21T22:06:30.654 INFO:teuthology.orchestra.run.smithi039.stdout: "use_gmt_hitset": true, 2024-09-21T22:06:30.654 INFO:teuthology.orchestra.run.smithi039.stdout: "min_read_recency_for_promote": 0, 2024-09-21T22:06:30.654 INFO:teuthology.orchestra.run.smithi039.stdout: "min_write_recency_for_promote": 0, 2024-09-21T22:06:30.654 INFO:teuthology.orchestra.run.smithi039.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T22:06:30.654 INFO:teuthology.orchestra.run.smithi039.stdout: "hit_set_search_last_n": 0, 2024-09-21T22:06:30.654 INFO:teuthology.orchestra.run.smithi039.stdout: "grade_table": [], 2024-09-21T22:06:30.654 INFO:teuthology.orchestra.run.smithi039.stdout: "stripe_width": 0, 2024-09-21T22:06:30.654 INFO:teuthology.orchestra.run.smithi039.stdout: "expected_num_objects": 0, 2024-09-21T22:06:30.655 INFO:teuthology.orchestra.run.smithi039.stdout: "fast_read": false, 2024-09-21T22:06:30.655 INFO:teuthology.orchestra.run.smithi039.stdout: "options": { 2024-09-21T22:06:30.655 INFO:teuthology.orchestra.run.smithi039.stdout: "pg_num_max": 32, 2024-09-21T22:06:30.655 INFO:teuthology.orchestra.run.smithi039.stdout: "pg_num_min": 1 2024-09-21T22:06:30.655 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:30.655 INFO:teuthology.orchestra.run.smithi039.stdout: "application_metadata": { 2024-09-21T22:06:30.655 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": {} 2024-09-21T22:06:30.655 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:30.655 INFO:teuthology.orchestra.run.smithi039.stdout: "read_balance": { 2024-09-21T22:06:30.655 INFO:teuthology.orchestra.run.smithi039.stdout: "score_type": "Fair distribution", 2024-09-21T22:06:30.655 INFO:teuthology.orchestra.run.smithi039.stdout: "score_acting": 6.059999942779541, 2024-09-21T22:06:30.655 INFO:teuthology.orchestra.run.smithi039.stdout: "score_stable": 6.059999942779541, 2024-09-21T22:06:30.655 INFO:teuthology.orchestra.run.smithi039.stdout: "optimal_score": 0.33000001311302185, 2024-09-21T22:06:30.656 INFO:teuthology.orchestra.run.smithi039.stdout: "raw_score_acting": 2, 2024-09-21T22:06:30.656 INFO:teuthology.orchestra.run.smithi039.stdout: "raw_score_stable": 2, 2024-09-21T22:06:30.656 INFO:teuthology.orchestra.run.smithi039.stdout: "primary_affinity_weighted": 1, 2024-09-21T22:06:30.656 INFO:teuthology.orchestra.run.smithi039.stdout: "average_primary_affinity": 1, 2024-09-21T22:06:30.656 INFO:teuthology.orchestra.run.smithi039.stdout: "average_primary_affinity_weighted": 1 2024-09-21T22:06:30.656 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:30.656 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:30.656 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:30.656 INFO:teuthology.orchestra.run.smithi039.stdout: "pool": 2, 2024-09-21T22:06:30.656 INFO:teuthology.orchestra.run.smithi039.stdout: "pool_name": "rbd", 2024-09-21T22:06:30.656 INFO:teuthology.orchestra.run.smithi039.stdout: "create_time": "2024-09-21T22:06:03.535857+0000", 2024-09-21T22:06:30.656 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 8193, 2024-09-21T22:06:30.656 INFO:teuthology.orchestra.run.smithi039.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-21T22:06:30.657 INFO:teuthology.orchestra.run.smithi039.stdout: "type": 1, 2024-09-21T22:06:30.657 INFO:teuthology.orchestra.run.smithi039.stdout: "size": 2, 2024-09-21T22:06:30.657 INFO:teuthology.orchestra.run.smithi039.stdout: "min_size": 1, 2024-09-21T22:06:30.657 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_rule": 0, 2024-09-21T22:06:30.657 INFO:teuthology.orchestra.run.smithi039.stdout: "peering_crush_bucket_count": 0, 2024-09-21T22:06:30.657 INFO:teuthology.orchestra.run.smithi039.stdout: "peering_crush_bucket_target": 0, 2024-09-21T22:06:30.657 INFO:teuthology.orchestra.run.smithi039.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T22:06:30.657 INFO:teuthology.orchestra.run.smithi039.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T22:06:30.657 INFO:teuthology.orchestra.run.smithi039.stdout: "is_stretch_pool": false, 2024-09-21T22:06:30.657 INFO:teuthology.orchestra.run.smithi039.stdout: "object_hash": 2, 2024-09-21T22:06:30.657 INFO:teuthology.orchestra.run.smithi039.stdout: "pg_autoscale_mode": "off", 2024-09-21T22:06:30.657 INFO:teuthology.orchestra.run.smithi039.stdout: "pg_num": 8, 2024-09-21T22:06:30.657 INFO:teuthology.orchestra.run.smithi039.stdout: "pg_placement_num": 8, 2024-09-21T22:06:30.657 INFO:teuthology.orchestra.run.smithi039.stdout: "pg_placement_num_target": 8, 2024-09-21T22:06:30.657 INFO:teuthology.orchestra.run.smithi039.stdout: "pg_num_target": 8, 2024-09-21T22:06:30.657 INFO:teuthology.orchestra.run.smithi039.stdout: "pg_num_pending": 8, 2024-09-21T22:06:30.658 INFO:teuthology.orchestra.run.smithi039.stdout: "last_pg_merge_meta": { 2024-09-21T22:06:30.658 INFO:teuthology.orchestra.run.smithi039.stdout: "source_pgid": "0.0", 2024-09-21T22:06:30.658 INFO:teuthology.orchestra.run.smithi039.stdout: "ready_epoch": 0, 2024-09-21T22:06:30.658 INFO:teuthology.orchestra.run.smithi039.stdout: "last_epoch_started": 0, 2024-09-21T22:06:30.658 INFO:teuthology.orchestra.run.smithi039.stdout: "last_epoch_clean": 0, 2024-09-21T22:06:30.658 INFO:teuthology.orchestra.run.smithi039.stdout: "source_version": "0'0", 2024-09-21T22:06:30.658 INFO:teuthology.orchestra.run.smithi039.stdout: "target_version": "0'0" 2024-09-21T22:06:30.658 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:30.658 INFO:teuthology.orchestra.run.smithi039.stdout: "last_change": "19", 2024-09-21T22:06:30.658 INFO:teuthology.orchestra.run.smithi039.stdout: "last_force_op_resend": "0", 2024-09-21T22:06:30.658 INFO:teuthology.orchestra.run.smithi039.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T22:06:30.658 INFO:teuthology.orchestra.run.smithi039.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T22:06:30.658 INFO:teuthology.orchestra.run.smithi039.stdout: "auid": 0, 2024-09-21T22:06:30.659 INFO:teuthology.orchestra.run.smithi039.stdout: "snap_mode": "selfmanaged", 2024-09-21T22:06:30.659 INFO:teuthology.orchestra.run.smithi039.stdout: "snap_seq": 2, 2024-09-21T22:06:30.659 INFO:teuthology.orchestra.run.smithi039.stdout: "snap_epoch": 19, 2024-09-21T22:06:30.659 INFO:teuthology.orchestra.run.smithi039.stdout: "pool_snaps": [], 2024-09-21T22:06:30.659 INFO:teuthology.orchestra.run.smithi039.stdout: "removed_snaps": "[]", 2024-09-21T22:06:30.659 INFO:teuthology.orchestra.run.smithi039.stdout: "quota_max_bytes": 0, 2024-09-21T22:06:30.659 INFO:teuthology.orchestra.run.smithi039.stdout: "quota_max_objects": 0, 2024-09-21T22:06:30.659 INFO:teuthology.orchestra.run.smithi039.stdout: "tiers": [], 2024-09-21T22:06:30.659 INFO:teuthology.orchestra.run.smithi039.stdout: "tier_of": -1, 2024-09-21T22:06:30.659 INFO:teuthology.orchestra.run.smithi039.stdout: "read_tier": -1, 2024-09-21T22:06:30.659 INFO:teuthology.orchestra.run.smithi039.stdout: "write_tier": -1, 2024-09-21T22:06:30.659 INFO:teuthology.orchestra.run.smithi039.stdout: "cache_mode": "none", 2024-09-21T22:06:30.659 INFO:teuthology.orchestra.run.smithi039.stdout: "target_max_bytes": 0, 2024-09-21T22:06:30.659 INFO:teuthology.orchestra.run.smithi039.stdout: "target_max_objects": 0, 2024-09-21T22:06:30.659 INFO:teuthology.orchestra.run.smithi039.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T22:06:30.660 INFO:teuthology.orchestra.run.smithi039.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T22:06:30.660 INFO:teuthology.orchestra.run.smithi039.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T22:06:30.660 INFO:teuthology.orchestra.run.smithi039.stdout: "cache_min_flush_age": 0, 2024-09-21T22:06:30.660 INFO:teuthology.orchestra.run.smithi039.stdout: "cache_min_evict_age": 0, 2024-09-21T22:06:30.660 INFO:teuthology.orchestra.run.smithi039.stdout: "erasure_code_profile": "", 2024-09-21T22:06:30.660 INFO:teuthology.orchestra.run.smithi039.stdout: "hit_set_params": { 2024-09-21T22:06:30.660 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "none" 2024-09-21T22:06:30.660 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:30.660 INFO:teuthology.orchestra.run.smithi039.stdout: "hit_set_period": 0, 2024-09-21T22:06:30.660 INFO:teuthology.orchestra.run.smithi039.stdout: "hit_set_count": 0, 2024-09-21T22:06:30.660 INFO:teuthology.orchestra.run.smithi039.stdout: "use_gmt_hitset": true, 2024-09-21T22:06:30.660 INFO:teuthology.orchestra.run.smithi039.stdout: "min_read_recency_for_promote": 0, 2024-09-21T22:06:30.660 INFO:teuthology.orchestra.run.smithi039.stdout: "min_write_recency_for_promote": 0, 2024-09-21T22:06:30.660 INFO:teuthology.orchestra.run.smithi039.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T22:06:30.660 INFO:teuthology.orchestra.run.smithi039.stdout: "hit_set_search_last_n": 0, 2024-09-21T22:06:30.660 INFO:teuthology.orchestra.run.smithi039.stdout: "grade_table": [], 2024-09-21T22:06:30.660 INFO:teuthology.orchestra.run.smithi039.stdout: "stripe_width": 0, 2024-09-21T22:06:30.661 INFO:teuthology.orchestra.run.smithi039.stdout: "expected_num_objects": 0, 2024-09-21T22:06:30.661 INFO:teuthology.orchestra.run.smithi039.stdout: "fast_read": false, 2024-09-21T22:06:30.661 INFO:teuthology.orchestra.run.smithi039.stdout: "options": {}, 2024-09-21T22:06:30.661 INFO:teuthology.orchestra.run.smithi039.stdout: "application_metadata": { 2024-09-21T22:06:30.661 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd": {} 2024-09-21T22:06:30.661 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:30.661 INFO:teuthology.orchestra.run.smithi039.stdout: "read_balance": { 2024-09-21T22:06:30.661 INFO:teuthology.orchestra.run.smithi039.stdout: "score_type": "Fair distribution", 2024-09-21T22:06:30.661 INFO:teuthology.orchestra.run.smithi039.stdout: "score_acting": 2.2699999809265137, 2024-09-21T22:06:30.661 INFO:teuthology.orchestra.run.smithi039.stdout: "score_stable": 2.2699999809265137, 2024-09-21T22:06:30.661 INFO:teuthology.orchestra.run.smithi039.stdout: "optimal_score": 0.82999998331069946, 2024-09-21T22:06:30.661 INFO:teuthology.orchestra.run.smithi039.stdout: "raw_score_acting": 1.8799999952316284, 2024-09-21T22:06:30.661 INFO:teuthology.orchestra.run.smithi039.stdout: "raw_score_stable": 1.8799999952316284, 2024-09-21T22:06:30.661 INFO:teuthology.orchestra.run.smithi039.stdout: "primary_affinity_weighted": 1, 2024-09-21T22:06:30.661 INFO:teuthology.orchestra.run.smithi039.stdout: "average_primary_affinity": 1, 2024-09-21T22:06:30.661 INFO:teuthology.orchestra.run.smithi039.stdout: "average_primary_affinity_weighted": 1 2024-09-21T22:06:30.662 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:30.662 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:30.662 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T22:06:30.662 INFO:teuthology.orchestra.run.smithi039.stdout: "osds": [ 2024-09-21T22:06:30.662 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:30.662 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": 0, 2024-09-21T22:06:30.662 INFO:teuthology.orchestra.run.smithi039.stdout: "uuid": "8336c397-2018-4c75-ab31-2e5f6ad564b5", 2024-09-21T22:06:30.662 INFO:teuthology.orchestra.run.smithi039.stdout: "up": 1, 2024-09-21T22:06:30.662 INFO:teuthology.orchestra.run.smithi039.stdout: "in": 0, 2024-09-21T22:06:30.662 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2024-09-21T22:06:30.662 INFO:teuthology.orchestra.run.smithi039.stdout: "primary_affinity": 1, 2024-09-21T22:06:30.662 INFO:teuthology.orchestra.run.smithi039.stdout: "last_clean_begin": 0, 2024-09-21T22:06:30.662 INFO:teuthology.orchestra.run.smithi039.stdout: "last_clean_end": 0, 2024-09-21T22:06:30.662 INFO:teuthology.orchestra.run.smithi039.stdout: "up_from": 11, 2024-09-21T22:06:30.662 INFO:teuthology.orchestra.run.smithi039.stdout: "up_thru": 0, 2024-09-21T22:06:30.663 INFO:teuthology.orchestra.run.smithi039.stdout: "down_at": 0, 2024-09-21T22:06:30.663 INFO:teuthology.orchestra.run.smithi039.stdout: "lost_at": 0, 2024-09-21T22:06:30.663 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2024-09-21T22:06:30.663 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2024-09-21T22:06:30.663 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:30.663 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2024-09-21T22:06:30.663 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6800", 2024-09-21T22:06:30.663 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 3054061468 2024-09-21T22:06:30.663 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:30.663 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:30.663 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-09-21T22:06:30.663 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6801", 2024-09-21T22:06:30.663 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 3054061468 2024-09-21T22:06:30.663 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:30.663 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:30.664 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:30.664 INFO:teuthology.orchestra.run.smithi039.stdout: "cluster_addrs": { 2024-09-21T22:06:30.664 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2024-09-21T22:06:30.664 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:30.664 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2024-09-21T22:06:30.664 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6802", 2024-09-21T22:06:30.664 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 3054061468 2024-09-21T22:06:30.664 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:30.664 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:30.664 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-09-21T22:06:30.664 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6803", 2024-09-21T22:06:30.664 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 3054061468 2024-09-21T22:06:30.664 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:30.664 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:30.665 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:30.665 INFO:teuthology.orchestra.run.smithi039.stdout: "heartbeat_back_addrs": { 2024-09-21T22:06:30.665 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2024-09-21T22:06:30.665 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:30.665 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2024-09-21T22:06:30.665 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6808", 2024-09-21T22:06:30.665 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 3054061468 2024-09-21T22:06:30.665 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:30.665 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:30.665 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-09-21T22:06:30.665 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6810", 2024-09-21T22:06:30.665 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 3054061468 2024-09-21T22:06:30.665 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:30.665 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:30.665 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:30.666 INFO:teuthology.orchestra.run.smithi039.stdout: "heartbeat_front_addrs": { 2024-09-21T22:06:30.666 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2024-09-21T22:06:30.666 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:30.666 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2024-09-21T22:06:30.666 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6804", 2024-09-21T22:06:30.666 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 3054061468 2024-09-21T22:06:30.666 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:30.666 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:30.666 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-09-21T22:06:30.666 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6806", 2024-09-21T22:06:30.666 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 3054061468 2024-09-21T22:06:30.666 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:30.666 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:30.666 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:30.667 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6801/3054061468", 2024-09-21T22:06:30.667 INFO:teuthology.orchestra.run.smithi039.stdout: "cluster_addr": "172.21.15.39:6803/3054061468", 2024-09-21T22:06:30.667 INFO:teuthology.orchestra.run.smithi039.stdout: "heartbeat_back_addr": "172.21.15.39:6810/3054061468", 2024-09-21T22:06:30.667 INFO:teuthology.orchestra.run.smithi039.stdout: "heartbeat_front_addr": "172.21.15.39:6806/3054061468", 2024-09-21T22:06:30.667 INFO:teuthology.orchestra.run.smithi039.stdout: "state": [ 2024-09-21T22:06:30.667 INFO:teuthology.orchestra.run.smithi039.stdout: "exists", 2024-09-21T22:06:30.667 INFO:teuthology.orchestra.run.smithi039.stdout: "up" 2024-09-21T22:06:30.667 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:30.667 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:30.667 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:30.667 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": 1, 2024-09-21T22:06:30.667 INFO:teuthology.orchestra.run.smithi039.stdout: "uuid": "bca64af6-7427-4c5d-97a4-a183b1f3a7d9", 2024-09-21T22:06:30.667 INFO:teuthology.orchestra.run.smithi039.stdout: "up": 1, 2024-09-21T22:06:30.667 INFO:teuthology.orchestra.run.smithi039.stdout: "in": 1, 2024-09-21T22:06:30.668 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 1, 2024-09-21T22:06:30.668 INFO:teuthology.orchestra.run.smithi039.stdout: "primary_affinity": 1, 2024-09-21T22:06:30.668 INFO:teuthology.orchestra.run.smithi039.stdout: "last_clean_begin": 0, 2024-09-21T22:06:30.668 INFO:teuthology.orchestra.run.smithi039.stdout: "last_clean_end": 0, 2024-09-21T22:06:30.668 INFO:teuthology.orchestra.run.smithi039.stdout: "up_from": 11, 2024-09-21T22:06:30.668 INFO:teuthology.orchestra.run.smithi039.stdout: "up_thru": 23, 2024-09-21T22:06:30.668 INFO:teuthology.orchestra.run.smithi039.stdout: "down_at": 0, 2024-09-21T22:06:30.668 INFO:teuthology.orchestra.run.smithi039.stdout: "lost_at": 0, 2024-09-21T22:06:30.668 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2024-09-21T22:06:30.668 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2024-09-21T22:06:30.668 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:30.668 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2024-09-21T22:06:30.668 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6816", 2024-09-21T22:06:30.668 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 2876495139 2024-09-21T22:06:30.668 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:30.668 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:30.669 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-09-21T22:06:30.669 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6817", 2024-09-21T22:06:30.669 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 2876495139 2024-09-21T22:06:30.669 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:30.669 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:30.669 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:30.669 INFO:teuthology.orchestra.run.smithi039.stdout: "cluster_addrs": { 2024-09-21T22:06:30.669 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2024-09-21T22:06:30.669 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:30.669 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2024-09-21T22:06:30.669 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6818", 2024-09-21T22:06:30.669 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 2876495139 2024-09-21T22:06:30.669 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:30.669 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:30.669 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-09-21T22:06:30.669 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6819", 2024-09-21T22:06:30.670 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 2876495139 2024-09-21T22:06:30.670 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:30.670 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:30.670 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:30.670 INFO:teuthology.orchestra.run.smithi039.stdout: "heartbeat_back_addrs": { 2024-09-21T22:06:30.670 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2024-09-21T22:06:30.670 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:30.670 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2024-09-21T22:06:30.670 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6822", 2024-09-21T22:06:30.670 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 2876495139 2024-09-21T22:06:30.670 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:30.670 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:30.670 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-09-21T22:06:30.670 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6823", 2024-09-21T22:06:30.670 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 2876495139 2024-09-21T22:06:30.671 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:30.671 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:30.671 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:30.671 INFO:teuthology.orchestra.run.smithi039.stdout: "heartbeat_front_addrs": { 2024-09-21T22:06:30.671 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2024-09-21T22:06:30.671 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:30.671 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2024-09-21T22:06:30.671 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6820", 2024-09-21T22:06:30.671 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 2876495139 2024-09-21T22:06:30.671 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:30.671 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:30.671 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-09-21T22:06:30.671 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6821", 2024-09-21T22:06:30.671 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 2876495139 2024-09-21T22:06:30.671 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:30.671 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:30.672 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:30.672 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6817/2876495139", 2024-09-21T22:06:30.672 INFO:teuthology.orchestra.run.smithi039.stdout: "cluster_addr": "172.21.15.39:6819/2876495139", 2024-09-21T22:06:30.672 INFO:teuthology.orchestra.run.smithi039.stdout: "heartbeat_back_addr": "172.21.15.39:6823/2876495139", 2024-09-21T22:06:30.672 INFO:teuthology.orchestra.run.smithi039.stdout: "heartbeat_front_addr": "172.21.15.39:6821/2876495139", 2024-09-21T22:06:30.672 INFO:teuthology.orchestra.run.smithi039.stdout: "state": [ 2024-09-21T22:06:30.672 INFO:teuthology.orchestra.run.smithi039.stdout: "exists", 2024-09-21T22:06:30.672 INFO:teuthology.orchestra.run.smithi039.stdout: "up" 2024-09-21T22:06:30.672 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:30.672 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:30.672 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:30.672 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": 2, 2024-09-21T22:06:30.672 INFO:teuthology.orchestra.run.smithi039.stdout: "uuid": "5f2678ed-6a23-4a67-96a4-c0ca91254bd7", 2024-09-21T22:06:30.672 INFO:teuthology.orchestra.run.smithi039.stdout: "up": 1, 2024-09-21T22:06:30.672 INFO:teuthology.orchestra.run.smithi039.stdout: "in": 1, 2024-09-21T22:06:30.672 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 1, 2024-09-21T22:06:30.673 INFO:teuthology.orchestra.run.smithi039.stdout: "primary_affinity": 1, 2024-09-21T22:06:30.673 INFO:teuthology.orchestra.run.smithi039.stdout: "last_clean_begin": 0, 2024-09-21T22:06:30.673 INFO:teuthology.orchestra.run.smithi039.stdout: "last_clean_end": 0, 2024-09-21T22:06:30.673 INFO:teuthology.orchestra.run.smithi039.stdout: "up_from": 11, 2024-09-21T22:06:30.673 INFO:teuthology.orchestra.run.smithi039.stdout: "up_thru": 16, 2024-09-21T22:06:30.673 INFO:teuthology.orchestra.run.smithi039.stdout: "down_at": 0, 2024-09-21T22:06:30.673 INFO:teuthology.orchestra.run.smithi039.stdout: "lost_at": 0, 2024-09-21T22:06:30.673 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2024-09-21T22:06:30.673 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2024-09-21T22:06:30.673 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:30.673 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2024-09-21T22:06:30.673 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6805", 2024-09-21T22:06:30.673 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 1262038957 2024-09-21T22:06:30.673 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:30.673 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:30.674 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-09-21T22:06:30.674 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6807", 2024-09-21T22:06:30.674 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 1262038957 2024-09-21T22:06:30.674 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:30.674 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:30.674 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:30.674 INFO:teuthology.orchestra.run.smithi039.stdout: "cluster_addrs": { 2024-09-21T22:06:30.674 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2024-09-21T22:06:30.674 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:30.674 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2024-09-21T22:06:30.674 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6809", 2024-09-21T22:06:30.674 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 1262038957 2024-09-21T22:06:30.674 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:30.674 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:30.674 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-09-21T22:06:30.675 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6811", 2024-09-21T22:06:30.675 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 1262038957 2024-09-21T22:06:30.675 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:30.675 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:30.675 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:30.675 INFO:teuthology.orchestra.run.smithi039.stdout: "heartbeat_back_addrs": { 2024-09-21T22:06:30.675 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2024-09-21T22:06:30.675 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:30.675 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2024-09-21T22:06:30.675 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6814", 2024-09-21T22:06:30.675 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 1262038957 2024-09-21T22:06:30.675 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:30.675 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:30.675 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-09-21T22:06:30.675 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6815", 2024-09-21T22:06:30.676 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 1262038957 2024-09-21T22:06:30.676 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:30.676 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:30.676 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:30.676 INFO:teuthology.orchestra.run.smithi039.stdout: "heartbeat_front_addrs": { 2024-09-21T22:06:30.676 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2024-09-21T22:06:30.676 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:30.676 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2024-09-21T22:06:30.676 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6812", 2024-09-21T22:06:30.676 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 1262038957 2024-09-21T22:06:30.676 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:30.676 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:30.676 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-09-21T22:06:30.676 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6813", 2024-09-21T22:06:30.677 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 1262038957 2024-09-21T22:06:30.677 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:30.677 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:30.677 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:30.677 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6807/1262038957", 2024-09-21T22:06:30.677 INFO:teuthology.orchestra.run.smithi039.stdout: "cluster_addr": "172.21.15.39:6811/1262038957", 2024-09-21T22:06:30.677 INFO:teuthology.orchestra.run.smithi039.stdout: "heartbeat_back_addr": "172.21.15.39:6815/1262038957", 2024-09-21T22:06:30.677 INFO:teuthology.orchestra.run.smithi039.stdout: "heartbeat_front_addr": "172.21.15.39:6813/1262038957", 2024-09-21T22:06:30.677 INFO:teuthology.orchestra.run.smithi039.stdout: "state": [ 2024-09-21T22:06:30.677 INFO:teuthology.orchestra.run.smithi039.stdout: "exists", 2024-09-21T22:06:30.677 INFO:teuthology.orchestra.run.smithi039.stdout: "up" 2024-09-21T22:06:30.677 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:30.677 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:30.677 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:30.678 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": 3, 2024-09-21T22:06:30.678 INFO:teuthology.orchestra.run.smithi039.stdout: "uuid": "aba30555-ef9e-47c9-a38b-ee1a4b8092f5", 2024-09-21T22:06:30.678 INFO:teuthology.orchestra.run.smithi039.stdout: "up": 1, 2024-09-21T22:06:30.678 INFO:teuthology.orchestra.run.smithi039.stdout: "in": 1, 2024-09-21T22:06:30.678 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 1, 2024-09-21T22:06:30.678 INFO:teuthology.orchestra.run.smithi039.stdout: "primary_affinity": 1, 2024-09-21T22:06:30.678 INFO:teuthology.orchestra.run.smithi039.stdout: "last_clean_begin": 0, 2024-09-21T22:06:30.678 INFO:teuthology.orchestra.run.smithi039.stdout: "last_clean_end": 0, 2024-09-21T22:06:30.678 INFO:teuthology.orchestra.run.smithi039.stdout: "up_from": 11, 2024-09-21T22:06:30.678 INFO:teuthology.orchestra.run.smithi039.stdout: "up_thru": 26, 2024-09-21T22:06:30.678 INFO:teuthology.orchestra.run.smithi039.stdout: "down_at": 0, 2024-09-21T22:06:30.678 INFO:teuthology.orchestra.run.smithi039.stdout: "lost_at": 0, 2024-09-21T22:06:30.678 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2024-09-21T22:06:30.678 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2024-09-21T22:06:30.679 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:30.679 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2024-09-21T22:06:30.679 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.64:6800", 2024-09-21T22:06:30.679 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 1132198485 2024-09-21T22:06:30.679 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:30.679 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:30.679 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-09-21T22:06:30.679 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.64:6801", 2024-09-21T22:06:30.679 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 1132198485 2024-09-21T22:06:30.679 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:30.679 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:30.679 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:30.679 INFO:teuthology.orchestra.run.smithi039.stdout: "cluster_addrs": { 2024-09-21T22:06:30.679 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2024-09-21T22:06:30.680 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:30.680 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2024-09-21T22:06:30.680 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.64:6802", 2024-09-21T22:06:30.680 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 1132198485 2024-09-21T22:06:30.680 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:30.680 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:30.680 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-09-21T22:06:30.680 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.64:6803", 2024-09-21T22:06:30.680 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 1132198485 2024-09-21T22:06:30.680 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:30.680 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:30.680 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:30.680 INFO:teuthology.orchestra.run.smithi039.stdout: "heartbeat_back_addrs": { 2024-09-21T22:06:30.680 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2024-09-21T22:06:30.681 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:30.681 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2024-09-21T22:06:30.681 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.64:6806", 2024-09-21T22:06:30.681 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 1132198485 2024-09-21T22:06:30.681 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:30.681 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:30.681 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-09-21T22:06:30.681 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.64:6807", 2024-09-21T22:06:30.681 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 1132198485 2024-09-21T22:06:30.681 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:30.681 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:30.681 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:30.681 INFO:teuthology.orchestra.run.smithi039.stdout: "heartbeat_front_addrs": { 2024-09-21T22:06:30.681 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2024-09-21T22:06:30.682 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:30.682 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2024-09-21T22:06:30.682 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.64:6804", 2024-09-21T22:06:30.682 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 1132198485 2024-09-21T22:06:30.682 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:30.682 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:30.682 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-09-21T22:06:30.682 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.64:6805", 2024-09-21T22:06:30.682 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 1132198485 2024-09-21T22:06:30.682 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:30.682 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:30.682 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:30.682 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.64:6801/1132198485", 2024-09-21T22:06:30.682 INFO:teuthology.orchestra.run.smithi039.stdout: "cluster_addr": "172.21.15.64:6803/1132198485", 2024-09-21T22:06:30.682 INFO:teuthology.orchestra.run.smithi039.stdout: "heartbeat_back_addr": "172.21.15.64:6807/1132198485", 2024-09-21T22:06:30.683 INFO:teuthology.orchestra.run.smithi039.stdout: "heartbeat_front_addr": "172.21.15.64:6805/1132198485", 2024-09-21T22:06:30.683 INFO:teuthology.orchestra.run.smithi039.stdout: "state": [ 2024-09-21T22:06:30.683 INFO:teuthology.orchestra.run.smithi039.stdout: "exists", 2024-09-21T22:06:30.683 INFO:teuthology.orchestra.run.smithi039.stdout: "up" 2024-09-21T22:06:30.683 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:30.683 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:30.683 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:30.683 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": 4, 2024-09-21T22:06:30.683 INFO:teuthology.orchestra.run.smithi039.stdout: "uuid": "2a1288d7-618d-4f44-8346-930cf56d4e80", 2024-09-21T22:06:30.683 INFO:teuthology.orchestra.run.smithi039.stdout: "up": 1, 2024-09-21T22:06:30.683 INFO:teuthology.orchestra.run.smithi039.stdout: "in": 1, 2024-09-21T22:06:30.683 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 1, 2024-09-21T22:06:30.683 INFO:teuthology.orchestra.run.smithi039.stdout: "primary_affinity": 1, 2024-09-21T22:06:30.683 INFO:teuthology.orchestra.run.smithi039.stdout: "last_clean_begin": 0, 2024-09-21T22:06:30.684 INFO:teuthology.orchestra.run.smithi039.stdout: "last_clean_end": 0, 2024-09-21T22:06:30.684 INFO:teuthology.orchestra.run.smithi039.stdout: "up_from": 11, 2024-09-21T22:06:30.684 INFO:teuthology.orchestra.run.smithi039.stdout: "up_thru": 0, 2024-09-21T22:06:30.684 INFO:teuthology.orchestra.run.smithi039.stdout: "down_at": 0, 2024-09-21T22:06:30.684 INFO:teuthology.orchestra.run.smithi039.stdout: "lost_at": 0, 2024-09-21T22:06:30.684 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2024-09-21T22:06:30.684 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2024-09-21T22:06:30.684 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:30.684 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2024-09-21T22:06:30.684 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.64:6808", 2024-09-21T22:06:30.684 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 3500786459 2024-09-21T22:06:30.684 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:30.684 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:30.684 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-09-21T22:06:30.684 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.64:6809", 2024-09-21T22:06:30.684 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 3500786459 2024-09-21T22:06:30.685 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:30.685 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:30.685 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:30.685 INFO:teuthology.orchestra.run.smithi039.stdout: "cluster_addrs": { 2024-09-21T22:06:30.685 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2024-09-21T22:06:30.685 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:30.685 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2024-09-21T22:06:30.685 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.64:6810", 2024-09-21T22:06:30.685 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 3500786459 2024-09-21T22:06:30.685 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:30.685 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:30.685 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-09-21T22:06:30.685 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.64:6811", 2024-09-21T22:06:30.685 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 3500786459 2024-09-21T22:06:30.686 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:30.686 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:30.686 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:30.686 INFO:teuthology.orchestra.run.smithi039.stdout: "heartbeat_back_addrs": { 2024-09-21T22:06:30.686 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2024-09-21T22:06:30.686 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:30.686 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2024-09-21T22:06:30.686 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.64:6814", 2024-09-21T22:06:30.686 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 3500786459 2024-09-21T22:06:30.686 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:30.686 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:30.686 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-09-21T22:06:30.686 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.64:6815", 2024-09-21T22:06:30.686 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 3500786459 2024-09-21T22:06:30.686 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:30.686 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:30.687 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:30.687 INFO:teuthology.orchestra.run.smithi039.stdout: "heartbeat_front_addrs": { 2024-09-21T22:06:30.687 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2024-09-21T22:06:30.687 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:30.687 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2024-09-21T22:06:30.687 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.64:6812", 2024-09-21T22:06:30.687 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 3500786459 2024-09-21T22:06:30.687 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:30.687 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:30.687 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-09-21T22:06:30.687 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.64:6813", 2024-09-21T22:06:30.687 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 3500786459 2024-09-21T22:06:30.687 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:30.687 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:30.687 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:30.688 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.64:6809/3500786459", 2024-09-21T22:06:30.688 INFO:teuthology.orchestra.run.smithi039.stdout: "cluster_addr": "172.21.15.64:6811/3500786459", 2024-09-21T22:06:30.688 INFO:teuthology.orchestra.run.smithi039.stdout: "heartbeat_back_addr": "172.21.15.64:6815/3500786459", 2024-09-21T22:06:30.688 INFO:teuthology.orchestra.run.smithi039.stdout: "heartbeat_front_addr": "172.21.15.64:6813/3500786459", 2024-09-21T22:06:30.688 INFO:teuthology.orchestra.run.smithi039.stdout: "state": [ 2024-09-21T22:06:30.688 INFO:teuthology.orchestra.run.smithi039.stdout: "exists", 2024-09-21T22:06:30.688 INFO:teuthology.orchestra.run.smithi039.stdout: "up" 2024-09-21T22:06:30.688 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:30.688 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:30.688 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:30.688 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": 5, 2024-09-21T22:06:30.688 INFO:teuthology.orchestra.run.smithi039.stdout: "uuid": "3ce719a8-34e9-4890-bd40-d22201e7bad8", 2024-09-21T22:06:30.688 INFO:teuthology.orchestra.run.smithi039.stdout: "up": 1, 2024-09-21T22:06:30.688 INFO:teuthology.orchestra.run.smithi039.stdout: "in": 1, 2024-09-21T22:06:30.688 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 1, 2024-09-21T22:06:30.688 INFO:teuthology.orchestra.run.smithi039.stdout: "primary_affinity": 1, 2024-09-21T22:06:30.689 INFO:teuthology.orchestra.run.smithi039.stdout: "last_clean_begin": 0, 2024-09-21T22:06:30.689 INFO:teuthology.orchestra.run.smithi039.stdout: "last_clean_end": 0, 2024-09-21T22:06:30.689 INFO:teuthology.orchestra.run.smithi039.stdout: "up_from": 11, 2024-09-21T22:06:30.689 INFO:teuthology.orchestra.run.smithi039.stdout: "up_thru": 16, 2024-09-21T22:06:30.689 INFO:teuthology.orchestra.run.smithi039.stdout: "down_at": 0, 2024-09-21T22:06:30.689 INFO:teuthology.orchestra.run.smithi039.stdout: "lost_at": 0, 2024-09-21T22:06:30.689 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2024-09-21T22:06:30.689 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2024-09-21T22:06:30.689 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:30.689 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2024-09-21T22:06:30.689 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.64:6816", 2024-09-21T22:06:30.689 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 2604542492 2024-09-21T22:06:30.689 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:30.689 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:30.689 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-09-21T22:06:30.689 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.64:6817", 2024-09-21T22:06:30.690 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 2604542492 2024-09-21T22:06:30.690 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:30.690 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:30.690 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:30.690 INFO:teuthology.orchestra.run.smithi039.stdout: "cluster_addrs": { 2024-09-21T22:06:30.690 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2024-09-21T22:06:30.690 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:30.690 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2024-09-21T22:06:30.690 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.64:6818", 2024-09-21T22:06:30.690 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 2604542492 2024-09-21T22:06:30.690 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:30.690 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:30.690 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-09-21T22:06:30.690 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.64:6819", 2024-09-21T22:06:30.690 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 2604542492 2024-09-21T22:06:30.691 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:30.691 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:30.691 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:30.691 INFO:teuthology.orchestra.run.smithi039.stdout: "heartbeat_back_addrs": { 2024-09-21T22:06:30.691 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2024-09-21T22:06:30.691 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:30.691 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2024-09-21T22:06:30.691 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.64:6822", 2024-09-21T22:06:30.691 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 2604542492 2024-09-21T22:06:30.691 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:30.691 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:30.691 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-09-21T22:06:30.691 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.64:6823", 2024-09-21T22:06:30.691 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 2604542492 2024-09-21T22:06:30.692 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:30.692 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:30.692 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:30.692 INFO:teuthology.orchestra.run.smithi039.stdout: "heartbeat_front_addrs": { 2024-09-21T22:06:30.692 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2024-09-21T22:06:30.692 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:30.692 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2024-09-21T22:06:30.692 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.64:6820", 2024-09-21T22:06:30.692 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 2604542492 2024-09-21T22:06:30.692 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:30.692 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:30.692 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-09-21T22:06:30.692 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.64:6821", 2024-09-21T22:06:30.692 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 2604542492 2024-09-21T22:06:30.692 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:30.692 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:30.693 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:30.693 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.64:6817/2604542492", 2024-09-21T22:06:30.693 INFO:teuthology.orchestra.run.smithi039.stdout: "cluster_addr": "172.21.15.64:6819/2604542492", 2024-09-21T22:06:30.693 INFO:teuthology.orchestra.run.smithi039.stdout: "heartbeat_back_addr": "172.21.15.64:6823/2604542492", 2024-09-21T22:06:30.693 INFO:teuthology.orchestra.run.smithi039.stdout: "heartbeat_front_addr": "172.21.15.64:6821/2604542492", 2024-09-21T22:06:30.693 INFO:teuthology.orchestra.run.smithi039.stdout: "state": [ 2024-09-21T22:06:30.693 INFO:teuthology.orchestra.run.smithi039.stdout: "exists", 2024-09-21T22:06:30.693 INFO:teuthology.orchestra.run.smithi039.stdout: "up" 2024-09-21T22:06:30.693 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:30.693 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:30.693 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T22:06:30.693 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_xinfo": [ 2024-09-21T22:06:30.693 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:30.693 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": 0, 2024-09-21T22:06:30.693 INFO:teuthology.orchestra.run.smithi039.stdout: "down_stamp": "0.000000", 2024-09-21T22:06:30.694 INFO:teuthology.orchestra.run.smithi039.stdout: "laggy_probability": 0, 2024-09-21T22:06:30.694 INFO:teuthology.orchestra.run.smithi039.stdout: "laggy_interval": 0, 2024-09-21T22:06:30.694 INFO:teuthology.orchestra.run.smithi039.stdout: "features": 4540701547738038271, 2024-09-21T22:06:30.694 INFO:teuthology.orchestra.run.smithi039.stdout: "old_weight": 65536, 2024-09-21T22:06:30.694 INFO:teuthology.orchestra.run.smithi039.stdout: "last_purged_snaps_scrub": "2024-09-21T22:05:54.224881+0000", 2024-09-21T22:06:30.694 INFO:teuthology.orchestra.run.smithi039.stdout: "dead_epoch": 0 2024-09-21T22:06:30.694 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:30.694 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:30.694 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": 1, 2024-09-21T22:06:30.694 INFO:teuthology.orchestra.run.smithi039.stdout: "down_stamp": "0.000000", 2024-09-21T22:06:30.694 INFO:teuthology.orchestra.run.smithi039.stdout: "laggy_probability": 0, 2024-09-21T22:06:30.694 INFO:teuthology.orchestra.run.smithi039.stdout: "laggy_interval": 0, 2024-09-21T22:06:30.694 INFO:teuthology.orchestra.run.smithi039.stdout: "features": 4540701547738038271, 2024-09-21T22:06:30.694 INFO:teuthology.orchestra.run.smithi039.stdout: "old_weight": 0, 2024-09-21T22:06:30.695 INFO:teuthology.orchestra.run.smithi039.stdout: "last_purged_snaps_scrub": "2024-09-21T22:05:54.668978+0000", 2024-09-21T22:06:30.695 INFO:teuthology.orchestra.run.smithi039.stdout: "dead_epoch": 0 2024-09-21T22:06:30.695 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:30.695 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:30.695 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": 2, 2024-09-21T22:06:30.695 INFO:teuthology.orchestra.run.smithi039.stdout: "down_stamp": "0.000000", 2024-09-21T22:06:30.695 INFO:teuthology.orchestra.run.smithi039.stdout: "laggy_probability": 0, 2024-09-21T22:06:30.695 INFO:teuthology.orchestra.run.smithi039.stdout: "laggy_interval": 0, 2024-09-21T22:06:30.695 INFO:teuthology.orchestra.run.smithi039.stdout: "features": 4540701547738038271, 2024-09-21T22:06:30.695 INFO:teuthology.orchestra.run.smithi039.stdout: "old_weight": 0, 2024-09-21T22:06:30.695 INFO:teuthology.orchestra.run.smithi039.stdout: "last_purged_snaps_scrub": "2024-09-21T22:05:54.315876+0000", 2024-09-21T22:06:30.695 INFO:teuthology.orchestra.run.smithi039.stdout: "dead_epoch": 0 2024-09-21T22:06:30.695 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:30.695 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:30.696 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": 3, 2024-09-21T22:06:30.696 INFO:teuthology.orchestra.run.smithi039.stdout: "down_stamp": "0.000000", 2024-09-21T22:06:30.696 INFO:teuthology.orchestra.run.smithi039.stdout: "laggy_probability": 0, 2024-09-21T22:06:30.696 INFO:teuthology.orchestra.run.smithi039.stdout: "laggy_interval": 0, 2024-09-21T22:06:30.696 INFO:teuthology.orchestra.run.smithi039.stdout: "features": 4540701547738038271, 2024-09-21T22:06:30.696 INFO:teuthology.orchestra.run.smithi039.stdout: "old_weight": 0, 2024-09-21T22:06:30.696 INFO:teuthology.orchestra.run.smithi039.stdout: "last_purged_snaps_scrub": "2024-09-21T22:05:54.314795+0000", 2024-09-21T22:06:30.696 INFO:teuthology.orchestra.run.smithi039.stdout: "dead_epoch": 0 2024-09-21T22:06:30.696 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:30.696 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:30.696 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": 4, 2024-09-21T22:06:30.696 INFO:teuthology.orchestra.run.smithi039.stdout: "down_stamp": "0.000000", 2024-09-21T22:06:30.696 INFO:teuthology.orchestra.run.smithi039.stdout: "laggy_probability": 0, 2024-09-21T22:06:30.696 INFO:teuthology.orchestra.run.smithi039.stdout: "laggy_interval": 0, 2024-09-21T22:06:30.696 INFO:teuthology.orchestra.run.smithi039.stdout: "features": 4540701547738038271, 2024-09-21T22:06:30.697 INFO:teuthology.orchestra.run.smithi039.stdout: "old_weight": 0, 2024-09-21T22:06:30.697 INFO:teuthology.orchestra.run.smithi039.stdout: "last_purged_snaps_scrub": "2024-09-21T22:05:54.322403+0000", 2024-09-21T22:06:30.697 INFO:teuthology.orchestra.run.smithi039.stdout: "dead_epoch": 0 2024-09-21T22:06:30.697 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:30.697 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:30.697 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": 5, 2024-09-21T22:06:30.697 INFO:teuthology.orchestra.run.smithi039.stdout: "down_stamp": "0.000000", 2024-09-21T22:06:30.697 INFO:teuthology.orchestra.run.smithi039.stdout: "laggy_probability": 0, 2024-09-21T22:06:30.697 INFO:teuthology.orchestra.run.smithi039.stdout: "laggy_interval": 0, 2024-09-21T22:06:30.697 INFO:teuthology.orchestra.run.smithi039.stdout: "features": 4540701547738038271, 2024-09-21T22:06:30.697 INFO:teuthology.orchestra.run.smithi039.stdout: "old_weight": 0, 2024-09-21T22:06:30.697 INFO:teuthology.orchestra.run.smithi039.stdout: "last_purged_snaps_scrub": "2024-09-21T22:05:54.750633+0000", 2024-09-21T22:06:30.697 INFO:teuthology.orchestra.run.smithi039.stdout: "dead_epoch": 0 2024-09-21T22:06:30.697 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:30.697 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T22:06:30.697 INFO:teuthology.orchestra.run.smithi039.stdout: "pg_upmap": [], 2024-09-21T22:06:30.698 INFO:teuthology.orchestra.run.smithi039.stdout: "pg_upmap_items": [], 2024-09-21T22:06:30.698 INFO:teuthology.orchestra.run.smithi039.stdout: "pg_upmap_primaries": [], 2024-09-21T22:06:30.698 INFO:teuthology.orchestra.run.smithi039.stdout: "pg_temp": [], 2024-09-21T22:06:30.698 INFO:teuthology.orchestra.run.smithi039.stdout: "primary_temp": [], 2024-09-21T22:06:30.698 INFO:teuthology.orchestra.run.smithi039.stdout: "blocklist": {}, 2024-09-21T22:06:30.698 INFO:teuthology.orchestra.run.smithi039.stdout: "range_blocklist": {}, 2024-09-21T22:06:30.698 INFO:teuthology.orchestra.run.smithi039.stdout: "erasure_code_profiles": { 2024-09-21T22:06:30.698 INFO:teuthology.orchestra.run.smithi039.stdout: "default": { 2024-09-21T22:06:30.698 INFO:teuthology.orchestra.run.smithi039.stdout: "crush-failure-domain": "osd", 2024-09-21T22:06:30.698 INFO:teuthology.orchestra.run.smithi039.stdout: "k": "2", 2024-09-21T22:06:30.698 INFO:teuthology.orchestra.run.smithi039.stdout: "m": "1", 2024-09-21T22:06:30.698 INFO:teuthology.orchestra.run.smithi039.stdout: "plugin": "jerasure", 2024-09-21T22:06:30.698 INFO:teuthology.orchestra.run.smithi039.stdout: "technique": "reed_sol_van" 2024-09-21T22:06:30.698 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:30.699 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:30.699 INFO:teuthology.orchestra.run.smithi039.stdout: "removed_snaps_queue": [], 2024-09-21T22:06:30.699 INFO:teuthology.orchestra.run.smithi039.stdout: "new_removed_snaps": [], 2024-09-21T22:06:30.699 INFO:teuthology.orchestra.run.smithi039.stdout: "new_purged_snaps": [], 2024-09-21T22:06:30.699 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_node_flags": {}, 2024-09-21T22:06:30.699 INFO:teuthology.orchestra.run.smithi039.stdout: "device_class_flags": {}, 2024-09-21T22:06:30.699 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": { 2024-09-21T22:06:30.699 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode_enabled": false, 2024-09-21T22:06:30.699 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_bucket_count": 0, 2024-09-21T22:06:30.699 INFO:teuthology.orchestra.run.smithi039.stdout: "degraded_stretch_mode": 0, 2024-09-21T22:06:30.699 INFO:teuthology.orchestra.run.smithi039.stdout: "recovering_stretch_mode": 0, 2024-09-21T22:06:30.699 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode_bucket": 0 2024-09-21T22:06:30.699 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:30.699 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:06:30.700 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-21T22:06:30.701 INFO:tasks.thrashosds.thrasher:j is {'epoch': 29, 'fsid': '00bb09a3-e410-4e18-a2b2-08eaf61978e6', 'created': '2024-09-21T22:05:45.303341+0000', 'modified': '2024-09-21T22:06:28.665546+0000', 'last_up_change': '2024-09-21T22:05:58.958630+0000', 'last_in_change': '2024-09-21T22:06:20.134002+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 4, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 2, 'max_osd': 6, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-21T22:06:00.868591+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6.059999942779541, 'score_stable': 6.059999942779541, 'optimal_score': 0.33000001311302185, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-21T22:06:03.535857+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 19, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.2699999809265137, 'score_stable': 2.2699999809265137, 'optimal_score': 0.8299999833106995, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '8336c397-2018-4c75-ab31-2e5f6ad564b5', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6800', 'nonce': 3054061468}, {'type': 'v1', 'addr': '172.21.15.39:6801', 'nonce': 3054061468}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6802', 'nonce': 3054061468}, {'type': 'v1', 'addr': '172.21.15.39:6803', 'nonce': 3054061468}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6808', 'nonce': 3054061468}, {'type': 'v1', 'addr': '172.21.15.39:6810', 'nonce': 3054061468}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6804', 'nonce': 3054061468}, {'type': 'v1', 'addr': '172.21.15.39:6806', 'nonce': 3054061468}]}, 'public_addr': '172.21.15.39:6801/3054061468', 'cluster_addr': '172.21.15.39:6803/3054061468', 'heartbeat_back_addr': '172.21.15.39:6810/3054061468', 'heartbeat_front_addr': '172.21.15.39:6806/3054061468', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'bca64af6-7427-4c5d-97a4-a183b1f3a7d9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 23, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6816', 'nonce': 2876495139}, {'type': 'v1', 'addr': '172.21.15.39:6817', 'nonce': 2876495139}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6818', 'nonce': 2876495139}, {'type': 'v1', 'addr': '172.21.15.39:6819', 'nonce': 2876495139}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6822', 'nonce': 2876495139}, {'type': 'v1', 'addr': '172.21.15.39:6823', 'nonce': 2876495139}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6820', 'nonce': 2876495139}, {'type': 'v1', 'addr': '172.21.15.39:6821', 'nonce': 2876495139}]}, 'public_addr': '172.21.15.39:6817/2876495139', 'cluster_addr': '172.21.15.39:6819/2876495139', 'heartbeat_back_addr': '172.21.15.39:6823/2876495139', 'heartbeat_front_addr': '172.21.15.39:6821/2876495139', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '5f2678ed-6a23-4a67-96a4-c0ca91254bd7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 16, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6805', 'nonce': 1262038957}, {'type': 'v1', 'addr': '172.21.15.39:6807', 'nonce': 1262038957}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6809', 'nonce': 1262038957}, {'type': 'v1', 'addr': '172.21.15.39:6811', 'nonce': 1262038957}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6814', 'nonce': 1262038957}, {'type': 'v1', 'addr': '172.21.15.39:6815', 'nonce': 1262038957}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6812', 'nonce': 1262038957}, {'type': 'v1', 'addr': '172.21.15.39:6813', 'nonce': 1262038957}]}, 'public_addr': '172.21.15.39:6807/1262038957', 'cluster_addr': '172.21.15.39:6811/1262038957', 'heartbeat_back_addr': '172.21.15.39:6815/1262038957', 'heartbeat_front_addr': '172.21.15.39:6813/1262038957', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'aba30555-ef9e-47c9-a38b-ee1a4b8092f5', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 26, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6800', 'nonce': 1132198485}, {'type': 'v1', 'addr': '172.21.15.64:6801', 'nonce': 1132198485}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6802', 'nonce': 1132198485}, {'type': 'v1', 'addr': '172.21.15.64:6803', 'nonce': 1132198485}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6806', 'nonce': 1132198485}, {'type': 'v1', 'addr': '172.21.15.64:6807', 'nonce': 1132198485}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6804', 'nonce': 1132198485}, {'type': 'v1', 'addr': '172.21.15.64:6805', 'nonce': 1132198485}]}, 'public_addr': '172.21.15.64:6801/1132198485', 'cluster_addr': '172.21.15.64:6803/1132198485', 'heartbeat_back_addr': '172.21.15.64:6807/1132198485', 'heartbeat_front_addr': '172.21.15.64:6805/1132198485', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '2a1288d7-618d-4f44-8346-930cf56d4e80', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6808', 'nonce': 3500786459}, {'type': 'v1', 'addr': '172.21.15.64:6809', 'nonce': 3500786459}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6810', 'nonce': 3500786459}, {'type': 'v1', 'addr': '172.21.15.64:6811', 'nonce': 3500786459}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6814', 'nonce': 3500786459}, {'type': 'v1', 'addr': '172.21.15.64:6815', 'nonce': 3500786459}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6812', 'nonce': 3500786459}, {'type': 'v1', 'addr': '172.21.15.64:6813', 'nonce': 3500786459}]}, 'public_addr': '172.21.15.64:6809/3500786459', 'cluster_addr': '172.21.15.64:6811/3500786459', 'heartbeat_back_addr': '172.21.15.64:6815/3500786459', 'heartbeat_front_addr': '172.21.15.64:6813/3500786459', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '3ce719a8-34e9-4890-bd40-d22201e7bad8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 16, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6816', 'nonce': 2604542492}, {'type': 'v1', 'addr': '172.21.15.64:6817', 'nonce': 2604542492}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6818', 'nonce': 2604542492}, {'type': 'v1', 'addr': '172.21.15.64:6819', 'nonce': 2604542492}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6822', 'nonce': 2604542492}, {'type': 'v1', 'addr': '172.21.15.64:6823', 'nonce': 2604542492}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6820', 'nonce': 2604542492}, {'type': 'v1', 'addr': '172.21.15.64:6821', 'nonce': 2604542492}]}, 'public_addr': '172.21.15.64:6817/2604542492', 'cluster_addr': '172.21.15.64:6819/2604542492', 'heartbeat_back_addr': '172.21.15.64:6823/2604542492', 'heartbeat_front_addr': '172.21.15.64:6821/2604542492', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-21T22:05:54.224881+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T22:05:54.668978+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T22:05:54.315876+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T22:05:54.314795+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T22:05:54.322403+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T22:05:54.750633+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-21T22:06:30.702 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T22:06:30.702 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:06:30.697+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:06:30.702 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:06:30.702 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:06:30.702 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-09-21T22:06:30.703 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:06:30.718 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T22:06:30.801 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:06:30.796+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:06:30.840 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:06:30.840 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:06:30.840 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-21T22:06:30.840 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-21T22:06:30.840 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:06:30.854 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T22:06:30.903 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:06:30.901+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:06:31.004 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:06:31.000+0000 7fc8d39f7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25294) UID: 0 2024-09-21T22:06:31.021 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:06:31.021 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-21T22:06:31.022 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-21T22:06:31.022 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-21T22:06:31.022 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:31.022 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4224.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-21T22:06:31.022 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:05.105895+0000", 2024-09-21T22:06:31.022 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 25.9145371, 2024-09-21T22:06:31.022 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0034226500000000002, 2024-09-21T22:06:31.022 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:31.022 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:31.022 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:31.022 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:31.022 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4224", 2024-09-21T22:06:31.022 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.39:0/795178185", 2024-09-21T22:06:31.023 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1 2024-09-21T22:06:31.023 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:31.023 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:31.023 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:31.023 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:31.023 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.105895+0000", 2024-09-21T22:06:31.023 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:31.023 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:31.023 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:31.023 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:31.023 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.105895+0000", 2024-09-21T22:06:31.023 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:31.023 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:31.024 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:31.024 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:31.024 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.105905+0000", 2024-09-21T22:06:31.024 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8069999999999994e-06 2024-09-21T22:06:31.024 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:31.024 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:31.024 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:31.024 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.105919+0000", 2024-09-21T22:06:31.024 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3976e-05 2024-09-21T22:06:31.024 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:31.024 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:31.024 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:31.024 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.105955+0000", 2024-09-21T22:06:31.024 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6776999999999997e-05 2024-09-21T22:06:31.024 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:31.025 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:31.025 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:31.025 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.105983+0000", 2024-09-21T22:06:31.025 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7535999999999999e-05 2024-09-21T22:06:31.025 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:31.025 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:31.025 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:31.025 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.106162+0000", 2024-09-21T22:06:31.025 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000178839 2024-09-21T22:06:31.025 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:31.025 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:31.025 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:31.025 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.106437+0000", 2024-09-21T22:06:31.026 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027542099999999998 2024-09-21T22:06:31.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:31.026 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:31.026 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-09-21T22:06:31.026 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.106726+0000", 2024-09-21T22:06:31.026 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00028856800000000001 2024-09-21T22:06:31.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:31.026 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:31.026 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:31.026 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.108194+0000", 2024-09-21T22:06:31.026 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001468095 2024-09-21T22:06:31.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:31.026 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:31.026 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:31.027 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.109130+0000", 2024-09-21T22:06:31.027 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00093653599999999997 2024-09-21T22:06:31.027 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:31.027 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:31.027 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:31.027 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.109216+0000", 2024-09-21T22:06:31.027 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5191999999999993e-05 2024-09-21T22:06:31.027 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:31.027 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:31.027 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:31.027 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.109317+0000", 2024-09-21T22:06:31.027 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000101903 2024-09-21T22:06:31.027 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:31.027 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:31.027 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:31.028 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:31.028 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:31.028 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4224.0:1 2.3 e17/16 ondisk, result = 0)", 2024-09-21T22:06:31.028 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:05.108920+0000", 2024-09-21T22:06:31.028 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 25.911511593, 2024-09-21T22:06:31.028 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00041366599999999998, 2024-09-21T22:06:31.028 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:31.028 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:31.028 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:06:31.028 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:31.028 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:31.028 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:31.028 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.108920+0000", 2024-09-21T22:06:31.028 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:31.029 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:31.029 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:31.029 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:31.029 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.108920+0000", 2024-09-21T22:06:31.029 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:31.029 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:31.029 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:31.029 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:31.029 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.108922+0000", 2024-09-21T22:06:31.029 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.703e-06 2024-09-21T22:06:31.029 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:31.029 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:31.029 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:31.029 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.108938+0000", 2024-09-21T22:06:31.029 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6350000000000001e-05 2024-09-21T22:06:31.030 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:31.030 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:31.030 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:31.030 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.108974+0000", 2024-09-21T22:06:31.030 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5612999999999999e-05 2024-09-21T22:06:31.030 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:31.030 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:31.030 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:31.030 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.108989+0000", 2024-09-21T22:06:31.030 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4576000000000001e-05 2024-09-21T22:06:31.030 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:31.030 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:31.037 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:31.037 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.109108+0000", 2024-09-21T22:06:31.037 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011901899999999999 2024-09-21T22:06:31.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:31.038 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:31.038 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:31.038 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.109120+0000", 2024-09-21T22:06:31.038 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.293e-05 2024-09-21T22:06:31.038 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:31.038 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:31.038 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:31.038 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.109334+0000", 2024-09-21T22:06:31.038 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000213475 2024-09-21T22:06:31.038 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:31.038 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:31.038 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:31.038 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:31.038 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:31.039 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4224.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e17)", 2024-09-21T22:06:31.039 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:05.115572+0000", 2024-09-21T22:06:31.039 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 25.904859685000002, 2024-09-21T22:06:31.039 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00041313, 2024-09-21T22:06:31.039 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:31.039 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:31.039 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-21T22:06:31.039 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:31.039 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4224", 2024-09-21T22:06:31.039 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.39:0/795178185", 2024-09-21T22:06:31.039 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2 2024-09-21T22:06:31.039 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:31.039 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:31.039 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:31.040 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:31.040 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.115572+0000", 2024-09-21T22:06:31.040 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:31.040 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:31.040 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:31.040 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:31.040 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.115572+0000", 2024-09-21T22:06:31.040 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:31.040 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:31.040 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:31.040 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:31.040 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.115574+0000", 2024-09-21T22:06:31.040 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0530000000000001e-06 2024-09-21T22:06:31.040 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:31.040 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:31.041 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:31.041 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.115585+0000", 2024-09-21T22:06:31.041 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0947e-05 2024-09-21T22:06:31.041 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:31.041 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:31.041 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:31.041 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.115615+0000", 2024-09-21T22:06:31.041 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9408999999999999e-05 2024-09-21T22:06:31.041 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:31.041 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:31.041 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:31.041 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.115637+0000", 2024-09-21T22:06:31.041 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2073e-05 2024-09-21T22:06:31.041 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:31.041 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:31.042 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:31.042 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.115768+0000", 2024-09-21T22:06:31.042 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013153899999999999 2024-09-21T22:06:31.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:31.042 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:31.042 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:31.042 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.115985+0000", 2024-09-21T22:06:31.042 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000217109 2024-09-21T22:06:31.042 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:31.042 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:31.042 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:31.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:31.042 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:31.043 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4224.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e17)", 2024-09-21T22:06:31.043 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:05.116456+0000", 2024-09-21T22:06:31.043 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 25.903976139000001, 2024-09-21T22:06:31.043 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00030938500000000002, 2024-09-21T22:06:31.043 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:31.043 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:31.043 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-21T22:06:31.043 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:31.043 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4224", 2024-09-21T22:06:31.043 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.39:0/795178185", 2024-09-21T22:06:31.043 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3 2024-09-21T22:06:31.043 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:31.044 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:31.044 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:31.044 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:31.044 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.116456+0000", 2024-09-21T22:06:31.044 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:31.044 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:31.044 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:31.044 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:31.044 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.116456+0000", 2024-09-21T22:06:31.044 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:31.044 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:31.044 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:31.044 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:31.044 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.116458+0000", 2024-09-21T22:06:31.045 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.77e-06 2024-09-21T22:06:31.045 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:31.045 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:31.045 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:31.045 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.116466+0000", 2024-09-21T22:06:31.045 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1799999999999996e-06 2024-09-21T22:06:31.045 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:31.045 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:31.045 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:31.045 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.116492+0000", 2024-09-21T22:06:31.045 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6523999999999999e-05 2024-09-21T22:06:31.045 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:31.045 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:31.045 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:31.045 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.116513+0000", 2024-09-21T22:06:31.046 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0400000000000001e-05 2024-09-21T22:06:31.046 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:31.046 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:31.046 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:31.046 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.116616+0000", 2024-09-21T22:06:31.046 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000103862 2024-09-21T22:06:31.046 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:31.046 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:31.046 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:31.046 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.116765+0000", 2024-09-21T22:06:31.046 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014864899999999999 2024-09-21T22:06:31.046 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:31.046 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:31.046 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:31.047 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:31.047 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:31.047 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4224.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-21T22:06:31.047 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:06.109464+0000", 2024-09-21T22:06:31.047 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 24.910967460999998, 2024-09-21T22:06:31.047 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0070431790000000001, 2024-09-21T22:06:31.047 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:31.047 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:31.047 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:31.047 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:31.047 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4224", 2024-09-21T22:06:31.047 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.39:0/795178185", 2024-09-21T22:06:31.047 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 5 2024-09-21T22:06:31.047 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:31.048 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:31.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:31.048 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:31.048 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.109464+0000", 2024-09-21T22:06:31.048 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:31.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:31.049 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:31.049 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:31.049 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.109464+0000", 2024-09-21T22:06:31.049 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:31.049 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:31.049 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:31.050 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:31.050 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.109467+0000", 2024-09-21T22:06:31.050 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.066e-06 2024-09-21T22:06:31.050 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:31.050 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:31.050 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:31.051 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.109485+0000", 2024-09-21T22:06:31.051 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7464000000000001e-05 2024-09-21T22:06:31.051 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:31.051 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:31.051 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:31.051 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.109518+0000", 2024-09-21T22:06:31.052 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3343000000000002e-05 2024-09-21T22:06:31.052 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:31.052 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:31.052 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:31.052 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.109537+0000", 2024-09-21T22:06:31.052 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8743e-05 2024-09-21T22:06:31.052 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:31.052 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:31.052 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:31.052 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.109708+0000", 2024-09-21T22:06:31.052 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017073899999999999 2024-09-21T22:06:31.052 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:31.052 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:31.053 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op must wait for map", 2024-09-21T22:06:31.053 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.109725+0000", 2024-09-21T22:06:31.053 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7013000000000001e-05 2024-09-21T22:06:31.053 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:31.053 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:31.053 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:31.053 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.113449+0000", 2024-09-21T22:06:31.053 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0037244579999999999 2024-09-21T22:06:31.053 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:31.053 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:31.053 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:31.053 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.113659+0000", 2024-09-21T22:06:31.053 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020968199999999999 2024-09-21T22:06:31.054 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:31.054 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:31.054 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-09-21T22:06:31.054 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.113872+0000", 2024-09-21T22:06:31.054 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000213062 2024-09-21T22:06:31.054 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:31.054 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:31.054 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:31.054 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.115375+0000", 2024-09-21T22:06:31.054 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015028979999999999 2024-09-21T22:06:31.054 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:31.054 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:31.054 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:31.055 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.116356+0000", 2024-09-21T22:06:31.055 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00098098200000000003 2024-09-21T22:06:31.055 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:31.055 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:31.055 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:31.055 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.116442+0000", 2024-09-21T22:06:31.055 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6533000000000007e-05 2024-09-21T22:06:31.055 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:31.055 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:31.055 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:31.055 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.116508+0000", 2024-09-21T22:06:31.055 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5196000000000004e-05 2024-09-21T22:06:31.055 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:31.055 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:31.055 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:31.056 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:31.056 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:31.056 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4224.0:5 2.2 e17/16 ondisk, result = 0)", 2024-09-21T22:06:31.056 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:06.116133+0000", 2024-09-21T22:06:31.056 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 24.904298976, 2024-09-21T22:06:31.056 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00038649700000000002, 2024-09-21T22:06:31.056 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:31.056 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:31.056 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:06:31.056 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:31.056 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:31.056 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:31.056 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.116133+0000", 2024-09-21T22:06:31.056 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:31.057 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:31.057 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:31.057 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:31.057 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.116133+0000", 2024-09-21T22:06:31.057 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:31.057 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:31.057 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:31.057 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:31.057 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.116135+0000", 2024-09-21T22:06:31.057 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8870000000000001e-06 2024-09-21T22:06:31.057 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:31.057 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:31.057 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:31.057 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.116146+0000", 2024-09-21T22:06:31.057 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0953e-05 2024-09-21T22:06:31.058 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:31.058 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:31.058 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:31.058 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.116175+0000", 2024-09-21T22:06:31.058 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9383e-05 2024-09-21T22:06:31.058 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:31.058 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:31.058 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:31.058 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.116190+0000", 2024-09-21T22:06:31.058 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4613000000000001e-05 2024-09-21T22:06:31.058 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:31.058 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:31.058 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:31.058 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.116328+0000", 2024-09-21T22:06:31.059 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013848300000000001 2024-09-21T22:06:31.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:31.059 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:31.059 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:31.059 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.116345+0000", 2024-09-21T22:06:31.059 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6883000000000001e-05 2024-09-21T22:06:31.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:31.059 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:31.059 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:31.059 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.116519+0000", 2024-09-21T22:06:31.059 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017429500000000001 2024-09-21T22:06:31.059 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:31.059 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:31.060 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:31.060 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:31.060 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:31.060 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4224.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-21T22:06:31.060 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:07.124322+0000", 2024-09-21T22:06:31.060 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 23.896109671000001, 2024-09-21T22:06:31.060 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0067709010000000002, 2024-09-21T22:06:31.060 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:31.060 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:31.060 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:31.060 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:31.060 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4224", 2024-09-21T22:06:31.060 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.39:0/795178185", 2024-09-21T22:06:31.060 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 7 2024-09-21T22:06:31.060 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:31.061 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:31.061 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:31.061 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:31.061 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.124322+0000", 2024-09-21T22:06:31.061 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:31.061 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:31.061 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:31.061 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:31.061 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.124322+0000", 2024-09-21T22:06:31.061 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:31.061 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:31.061 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:31.061 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:31.061 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.124325+0000", 2024-09-21T22:06:31.062 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1489999999999998e-06 2024-09-21T22:06:31.062 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:31.062 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:31.062 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:31.062 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.124342+0000", 2024-09-21T22:06:31.062 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.666e-05 2024-09-21T22:06:31.062 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:31.062 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:31.062 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:31.062 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.124379+0000", 2024-09-21T22:06:31.062 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6539999999999999e-05 2024-09-21T22:06:31.062 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:31.062 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:31.062 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:31.063 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.124406+0000", 2024-09-21T22:06:31.063 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7886e-05 2024-09-21T22:06:31.063 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:31.063 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:31.063 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:31.063 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.124535+0000", 2024-09-21T22:06:31.063 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000128749 2024-09-21T22:06:31.063 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:31.063 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:31.063 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op must wait for map", 2024-09-21T22:06:31.063 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.124547+0000", 2024-09-21T22:06:31.063 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.136e-05 2024-09-21T22:06:31.063 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:31.063 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:31.064 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:31.064 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.127774+0000", 2024-09-21T22:06:31.064 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0032275519999999999 2024-09-21T22:06:31.064 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:31.064 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:31.064 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:31.064 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.128014+0000", 2024-09-21T22:06:31.064 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023956800000000001 2024-09-21T22:06:31.064 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:31.064 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:31.064 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-09-21T22:06:31.064 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.128462+0000", 2024-09-21T22:06:31.064 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00044797700000000001 2024-09-21T22:06:31.064 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:31.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:31.065 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:31.065 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.130185+0000", 2024-09-21T22:06:31.065 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00172313 2024-09-21T22:06:31.065 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:31.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:31.065 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:31.065 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.130937+0000", 2024-09-21T22:06:31.065 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00075244799999999996 2024-09-21T22:06:31.065 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:31.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:31.065 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:31.065 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.131025+0000", 2024-09-21T22:06:31.065 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7998999999999996e-05 2024-09-21T22:06:31.066 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:31.066 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:31.066 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:31.066 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.131093+0000", 2024-09-21T22:06:31.066 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7883e-05 2024-09-21T22:06:31.066 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:31.066 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:31.066 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:31.066 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:31.066 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:31.066 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4224.0:7 2.2 e18/16 ondisk, result = 0)", 2024-09-21T22:06:31.066 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:07.130777+0000", 2024-09-21T22:06:31.066 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 23.889654624999999, 2024-09-21T22:06:31.067 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00032786799999999999, 2024-09-21T22:06:31.067 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:31.067 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:31.067 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:06:31.067 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:31.067 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:31.067 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:31.067 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.130777+0000", 2024-09-21T22:06:31.067 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:31.067 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:31.067 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:31.067 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:31.067 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.130777+0000", 2024-09-21T22:06:31.067 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:31.067 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:31.068 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:31.068 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:31.068 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.130779+0000", 2024-09-21T22:06:31.068 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7e-06 2024-09-21T22:06:31.068 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:31.068 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:31.068 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:31.068 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.130788+0000", 2024-09-21T22:06:31.068 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6670000000000002e-06 2024-09-21T22:06:31.068 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:31.068 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:31.068 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:31.068 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.130814+0000", 2024-09-21T22:06:31.068 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6590000000000001e-05 2024-09-21T22:06:31.069 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:31.069 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:31.069 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:31.069 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.130828+0000", 2024-09-21T22:06:31.069 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.415e-05 2024-09-21T22:06:31.069 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:31.069 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:31.069 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:31.069 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.130915+0000", 2024-09-21T22:06:31.069 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6759000000000001e-05 2024-09-21T22:06:31.069 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:31.069 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:31.069 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:31.069 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.130928+0000", 2024-09-21T22:06:31.070 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.328e-05 2024-09-21T22:06:31.070 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:31.070 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:31.070 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:31.070 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.131105+0000", 2024-09-21T22:06:31.070 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017672200000000001 2024-09-21T22:06:31.070 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:31.070 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:31.070 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:31.070 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:31.070 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:31.070 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:06:31.071 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T22:06:31.072 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T22:06:31.073 INFO:teuthology.orchestra.run.smithi039.stderr:dumped all 2024-09-21T22:06:31.073 INFO:teuthology.orchestra.run.smithi039.stdout:{"pg_ready":true,"pg_map":{"version":40,"stamp":"2024-09-21T22:06:30.793943+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":471859200,"kb_used":145048,"kb_used_data":1584,"kb_used_omap":7,"kb_used_meta":143352,"kb_avail":471714152,"statfs":{"total":483183820800,"available":483035291648,"internally_reserved":0,"allocated":1622016,"data_stored":1712619,"data_compressed":13990,"data_compressed_allocated":73728,"data_compressed_original":1163264,"omap_allocated":7973,"internal_metadata":146792667},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.580914"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":39,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-21T22:06:28.674973+0000","last_change":"2024-09-21T22:06:07.122328+0000","last_active":"2024-09-21T22:06:28.674973+0000","last_peered":"2024-09-21T22:06:28.674973+0000","last_clean":"2024-09-21T22:06:28.674973+0000","last_became_active":"2024-09-21T22:06:05.220996+0000","last_became_peered":"2024-09-21T22:06:05.220996+0000","last_unstale":"2024-09-21T22:06:28.674973+0000","last_undegraded":"2024-09-21T22:06:28.674973+0000","last_fullsized":"2024-09-21T22:06:28.674973+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T23:45:49.178013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001473755,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":39,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-21T22:06:28.817197+0000","last_change":"2024-09-21T22:06:07.133757+0000","last_active":"2024-09-21T22:06:28.817197+0000","last_peered":"2024-09-21T22:06:28.817197+0000","last_clean":"2024-09-21T22:06:28.817197+0000","last_became_active":"2024-09-21T22:06:05.109065+0000","last_became_peered":"2024-09-21T22:06:05.109065+0000","last_unstale":"2024-09-21T22:06:28.817197+0000","last_undegraded":"2024-09-21T22:06:28.817197+0000","last_fullsized":"2024-09-21T22:06:28.817197+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:31:12.654893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00088865700000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":44,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-21T22:06:28.674926+0000","last_change":"2024-09-21T22:06:21.313951+0000","last_active":"2024-09-21T22:06:28.674926+0000","last_peered":"2024-09-21T22:06:28.674926+0000","last_clean":"2024-09-21T22:06:28.674926+0000","last_became_active":"2024-09-21T22:06:21.313514+0000","last_became_peered":"2024-09-21T22:06:21.313514+0000","last_unstale":"2024-09-21T22:06:28.674926+0000","last_undegraded":"2024-09-21T22:06:28.674926+0000","last_fullsized":"2024-09-21T22:06:28.674926+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:07:37.870228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091949900000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":44,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-21T22:06:28.817180+0000","last_change":"2024-09-21T22:06:21.314973+0000","last_active":"2024-09-21T22:06:28.817180+0000","last_peered":"2024-09-21T22:06:28.817180+0000","last_clean":"2024-09-21T22:06:28.817180+0000","last_became_active":"2024-09-21T22:06:21.313946+0000","last_became_peered":"2024-09-21T22:06:21.313946+0000","last_unstale":"2024-09-21T22:06:28.817180+0000","last_undegraded":"2024-09-21T22:06:28.817180+0000","last_fullsized":"2024-09-21T22:06:28.817180+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:14:03.108117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00090834100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"19'2","reported_seq":41,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-21T22:06:28.817743+0000","last_change":"2024-09-21T22:06:07.130034+0000","last_active":"2024-09-21T22:06:28.817743+0000","last_peered":"2024-09-21T22:06:28.817743+0000","last_clean":"2024-09-21T22:06:28.817743+0000","last_became_active":"2024-09-21T22:06:05.102579+0000","last_became_peered":"2024-09-21T22:06:05.102579+0000","last_unstale":"2024-09-21T22:06:28.817743+0000","last_undegraded":"2024-09-21T22:06:28.817743+0000","last_fullsized":"2024-09-21T22:06:28.817743+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T05:18:35.468678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026694230000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":39,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-21T22:06:28.678189+0000","last_change":"2024-09-21T22:06:07.123496+0000","last_active":"2024-09-21T22:06:28.678189+0000","last_peered":"2024-09-21T22:06:28.678189+0000","last_clean":"2024-09-21T22:06:28.678189+0000","last_became_active":"2024-09-21T22:06:05.104941+0000","last_became_peered":"2024-09-21T22:06:05.104941+0000","last_unstale":"2024-09-21T22:06:28.678189+0000","last_undegraded":"2024-09-21T22:06:28.678189+0000","last_fullsized":"2024-09-21T22:06:28.678189+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:46:29.901819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010052629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":39,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-21T22:06:28.674796+0000","last_change":"2024-09-21T22:06:07.121573+0000","last_active":"2024-09-21T22:06:28.674796+0000","last_peered":"2024-09-21T22:06:28.674796+0000","last_clean":"2024-09-21T22:06:28.674796+0000","last_became_active":"2024-09-21T22:06:05.108187+0000","last_became_peered":"2024-09-21T22:06:05.108187+0000","last_unstale":"2024-09-21T22:06:28.674796+0000","last_undegraded":"2024-09-21T22:06:28.674796+0000","last_fullsized":"2024-09-21T22:06:28.674796+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T05:05:08.108341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079478100000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"17'1","reported_seq":40,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-21T22:06:28.817791+0000","last_change":"2024-09-21T22:06:07.128525+0000","last_active":"2024-09-21T22:06:28.817791+0000","last_peered":"2024-09-21T22:06:28.817791+0000","last_clean":"2024-09-21T22:06:28.817791+0000","last_became_active":"2024-09-21T22:06:05.101507+0000","last_became_peered":"2024-09-21T22:06:05.101507+0000","last_unstale":"2024-09-21T22:06:28.817791+0000","last_undegraded":"2024-09-21T22:06:28.817791+0000","last_fullsized":"2024-09-21T22:06:28.817791+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:43:06.377355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001125968,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":203,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-21T22:06:28.674902+0000","last_change":"2024-09-21T22:06:24.228963+0000","last_active":"2024-09-21T22:06:28.674902+0000","last_peered":"2024-09-21T22:06:28.674902+0000","last_clean":"2024-09-21T22:06:28.674902+0000","last_became_active":"2024-09-21T22:06:24.227898+0000","last_became_peered":"2024-09-21T22:06:24.227898+0000","last_unstale":"2024-09-21T22:06:28.674902+0000","last_undegraded":"2024-09-21T22:06:28.674902+0000","last_fullsized":"2024-09-21T22:06:28.674902+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:00.990331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:00.990331+0000","last_clean_scrub_stamp":"2024-09-21T22:06:00.990331+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T09:06:54.254990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":1771104,"data_compressed":20551,"data_compressed_allocated":110592,"data_compressed_original":1736704,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640267,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27332,"kb_used_data":300,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344508,"statfs":{"total":96636764160,"available":96608776192,"internally_reserved":0,"allocated":307200,"data_stored":106388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640266,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27324,"kb_used_data":296,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344516,"statfs":{"total":96636764160,"available":96608784384,"internally_reserved":0,"allocated":303104,"data_stored":106369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640269,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27380,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344460,"statfs":{"total":96636764160,"available":96608727040,"internally_reserved":0,"allocated":360448,"data_stored":696737,"data_compressed":6561,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640268,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31484,"kb_used_data":296,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340356,"statfs":{"total":96636764160,"available":96604524544,"internally_reserved":0,"allocated":303104,"data_stored":106369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640268,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31528,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340312,"statfs":{"total":96636764160,"available":96604479488,"internally_reserved":0,"allocated":348160,"data_stored":696756,"data_compressed":7429,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":0,"seq":47244640269,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6561,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":7429,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6561,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T22:06:31.074 INFO:tasks.thrashosds.thrasher:Setting 2.7 to [3, 0, 1, 4] 2024-09-21T22:06:31.074 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '2.7', '3', '0', '1', '4'] 2024-09-21T22:06:31.074 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap-items 2.7 3 0 1 4 2024-09-21T22:06:31.116 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:06:31.113+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:06:31.205 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:06:31.201+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:06:31.239 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:06:31.240 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:06:31.240 INFO:teuthology.orchestra.run.smithi039.stdout: "num_ops": 0 2024-09-21T22:06:31.240 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:06:31.241 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T22:06:31.255 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T22:06:31.306 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:06:31.300+0000 7fc8d39f7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25294) UID: 0 2024-09-21T22:06:31.376 INFO:teuthology.orchestra.run.smithi039.stdout:osd.0: {} 2024-09-21T22:06:31.376 INFO:teuthology.orchestra.run.smithi039.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T22:06:31.394 INFO:teuthology.orchestra.run.smithi039.stdout:osd.1: {} 2024-09-21T22:06:31.394 INFO:teuthology.orchestra.run.smithi039.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T22:06:31.407 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:06:31.405+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:06:31.410 INFO:teuthology.orchestra.run.smithi039.stdout:osd.2: {} 2024-09-21T22:06:31.410 INFO:teuthology.orchestra.run.smithi039.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T22:06:31.413 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:31.414 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:31.427 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T22:06:31.430 INFO:teuthology.orchestra.run.smithi039.stdout:osd.3: {} 2024-09-21T22:06:31.431 INFO:teuthology.orchestra.run.smithi039.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T22:06:31.509 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:06:31.504+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:06:31.538 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:31.538 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:31.551 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T22:06:31.610 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:06:31.609+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:06:31.655 INFO:teuthology.orchestra.run.smithi039.stdout:osd.4: {} 2024-09-21T22:06:31.655 INFO:teuthology.orchestra.run.smithi039.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T22:06:31.672 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:31.672 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:31.684 INFO:teuthology.orchestra.run.smithi039.stdout:osd.5: {} 2024-09-21T22:06:31.684 INFO:teuthology.orchestra.run.smithi039.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T22:06:31.685 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T22:06:31.710 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:06:31.704+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:06:31.737 INFO:teuthology.orchestra.run.smithi039.stderr:set 2.7 pg_upmap_items mapping to [3->0,1->4] 2024-09-21T22:06:31.737 INFO:teuthology.orchestra.run.smithi039.stderr:noscrub is unset 2024-09-21T22:06:31.811 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:06:31.809+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:06:31.850 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:31.850 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:31.863 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T22:06:31.950 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:06:31.949+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:06:32.013 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:06:32.009+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:06:32.025 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:32.025 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:32.038 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T22:06:32.126 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:06:32.125+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:06:32.207 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:32.207 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:32.214 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:06:32.208+0000 7fc8d39f7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25294) UID: 0 2024-09-21T22:06:32.221 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T22:06:32.315 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:06:32.313+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:06:32.390 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:32.390 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:32.404 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T22:06:32.446 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:06:32.445+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:06:32.516 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:06:32.513+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:06:32.575 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:32.575 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:32.591 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T22:06:32.635 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:06:32.632+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:06:32.717 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:06:32.712+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:06:32.726 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:32.726 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:32.741 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T22:06:32.817 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:06:32.817+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:06:32.908 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:32.908 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:32.918 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:06:32.912+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:06:32.922 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T22:06:33.019 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:06:33.017+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:06:33.091 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:33.091 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:33.105 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T22:06:33.120 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:06:33.117+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:06:33.220 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:06:33.216+0000 7fc8d39f7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25294) UID: 0 2024-09-21T22:06:33.235 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:33.236 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:33.251 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T22:06:33.339 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:06:33.336+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:06:33.422 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:06:33.416+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:06:33.423 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:33.423 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:33.438 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T22:06:33.523 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:06:33.520+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:06:33.599 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:33.599 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:33.614 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T22:06:33.623 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:06:33.621+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:06:33.701 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T22:06:33.724 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:06:33.720+0000 7fc8d39f7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25294) UID: 0 2024-09-21T22:06:33.757 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-21T22:06:33.775 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:33.775 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:33.789 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T22:06:33.835 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:06:33.832+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:06:33.857 INFO:teuthology.orchestra.run.smithi039.stdout:osd.0: {} 2024-09-21T22:06:33.857 INFO:teuthology.orchestra.run.smithi039.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T22:06:33.874 INFO:teuthology.orchestra.run.smithi039.stdout:osd.1: {} 2024-09-21T22:06:33.874 INFO:teuthology.orchestra.run.smithi039.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T22:06:33.891 INFO:teuthology.orchestra.run.smithi039.stdout:osd.2: {} 2024-09-21T22:06:33.891 INFO:teuthology.orchestra.run.smithi039.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T22:06:33.911 INFO:teuthology.orchestra.run.smithi039.stdout:osd.3: {} 2024-09-21T22:06:33.911 INFO:teuthology.orchestra.run.smithi039.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T22:06:33.926 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:06:33.920+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:06:33.933 INFO:teuthology.orchestra.run.smithi039.stdout:osd.4: {} 2024-09-21T22:06:33.933 INFO:teuthology.orchestra.run.smithi039.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T22:06:33.953 INFO:teuthology.orchestra.run.smithi039.stdout:osd.5: {} 2024-09-21T22:06:33.953 INFO:teuthology.orchestra.run.smithi039.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T22:06:33.962 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:06:33.962 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:06:33.962 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-21T22:06:33.962 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-21T22:06:33.962 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:06:33.976 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T22:06:34.027 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:06:34.025+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:06:34.127 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:06:34.125+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:06:34.143 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:06:34.143 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-21T22:06:34.143 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-21T22:06:34.143 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-21T22:06:34.143 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:34.143 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4224.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-21T22:06:34.144 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:05.105895+0000", 2024-09-21T22:06:34.144 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 29.036866922000002, 2024-09-21T22:06:34.144 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0034226500000000002, 2024-09-21T22:06:34.144 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:34.144 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:34.144 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:34.144 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:34.144 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4224", 2024-09-21T22:06:34.144 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.39:0/795178185", 2024-09-21T22:06:34.144 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1 2024-09-21T22:06:34.144 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:34.144 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:34.144 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:34.144 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:34.144 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.105895+0000", 2024-09-21T22:06:34.144 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:34.145 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:34.145 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:34.145 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:34.145 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.105895+0000", 2024-09-21T22:06:34.145 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:34.145 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:34.145 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:34.145 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:34.145 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.105905+0000", 2024-09-21T22:06:34.145 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8069999999999994e-06 2024-09-21T22:06:34.145 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:34.145 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:34.145 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:34.146 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.105919+0000", 2024-09-21T22:06:34.146 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3976e-05 2024-09-21T22:06:34.146 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:34.146 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:34.146 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:34.146 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.105955+0000", 2024-09-21T22:06:34.146 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6776999999999997e-05 2024-09-21T22:06:34.146 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:34.146 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:34.146 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:34.146 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.105983+0000", 2024-09-21T22:06:34.146 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7535999999999999e-05 2024-09-21T22:06:34.147 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:34.147 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:34.147 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:34.147 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.106162+0000", 2024-09-21T22:06:34.147 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000178839 2024-09-21T22:06:34.147 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:34.147 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:34.147 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:34.147 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.106437+0000", 2024-09-21T22:06:34.147 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027542099999999998 2024-09-21T22:06:34.147 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:34.147 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:34.147 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-09-21T22:06:34.147 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.106726+0000", 2024-09-21T22:06:34.147 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00028856800000000001 2024-09-21T22:06:34.148 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:34.148 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:34.148 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:34.148 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.108194+0000", 2024-09-21T22:06:34.148 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001468095 2024-09-21T22:06:34.148 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:34.148 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:34.148 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:34.148 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.109130+0000", 2024-09-21T22:06:34.148 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00093653599999999997 2024-09-21T22:06:34.149 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:34.149 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:34.149 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:34.149 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.109216+0000", 2024-09-21T22:06:34.149 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5191999999999993e-05 2024-09-21T22:06:34.149 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:34.149 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:34.149 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:34.149 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.109317+0000", 2024-09-21T22:06:34.149 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000101903 2024-09-21T22:06:34.149 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:34.149 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:34.149 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:34.149 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:34.149 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:34.150 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4224.0:1 2.3 e17/16 ondisk, result = 0)", 2024-09-21T22:06:34.150 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:05.108920+0000", 2024-09-21T22:06:34.150 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 29.033841415000001, 2024-09-21T22:06:34.150 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00041366599999999998, 2024-09-21T22:06:34.150 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:34.150 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:34.150 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:06:34.150 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:34.150 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:34.150 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:34.150 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.108920+0000", 2024-09-21T22:06:34.150 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:34.150 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:34.151 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:34.151 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:34.151 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.108920+0000", 2024-09-21T22:06:34.151 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:34.151 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:34.151 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:34.151 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:34.151 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.108922+0000", 2024-09-21T22:06:34.151 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.703e-06 2024-09-21T22:06:34.151 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:34.152 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:34.152 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:34.152 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.108938+0000", 2024-09-21T22:06:34.152 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6350000000000001e-05 2024-09-21T22:06:34.152 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:34.152 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:34.152 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:34.152 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.108974+0000", 2024-09-21T22:06:34.152 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5612999999999999e-05 2024-09-21T22:06:34.152 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:34.152 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:34.152 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:34.152 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.108989+0000", 2024-09-21T22:06:34.152 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4576000000000001e-05 2024-09-21T22:06:34.152 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:34.153 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:34.153 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:34.153 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.109108+0000", 2024-09-21T22:06:34.153 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011901899999999999 2024-09-21T22:06:34.153 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:34.153 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:34.153 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:34.153 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.109120+0000", 2024-09-21T22:06:34.153 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.293e-05 2024-09-21T22:06:34.153 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:34.153 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:34.153 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:34.153 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.109334+0000", 2024-09-21T22:06:34.153 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000213475 2024-09-21T22:06:34.154 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:34.154 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:34.154 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:34.154 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:34.154 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:34.154 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4224.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e17)", 2024-09-21T22:06:34.154 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:05.115572+0000", 2024-09-21T22:06:34.154 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 29.027189506999999, 2024-09-21T22:06:34.154 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00041313, 2024-09-21T22:06:34.154 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:34.154 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:34.154 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-21T22:06:34.154 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:34.154 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4224", 2024-09-21T22:06:34.155 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.39:0/795178185", 2024-09-21T22:06:34.155 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2 2024-09-21T22:06:34.155 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:34.155 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:34.155 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:34.155 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:34.155 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.115572+0000", 2024-09-21T22:06:34.155 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:34.155 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:34.155 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:34.155 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:34.155 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.115572+0000", 2024-09-21T22:06:34.155 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:34.155 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:34.155 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:34.156 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:34.156 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.115574+0000", 2024-09-21T22:06:34.156 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0530000000000001e-06 2024-09-21T22:06:34.156 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:34.156 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:34.156 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:34.156 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.115585+0000", 2024-09-21T22:06:34.156 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0947e-05 2024-09-21T22:06:34.156 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:34.156 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:34.156 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:34.156 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.115615+0000", 2024-09-21T22:06:34.156 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9408999999999999e-05 2024-09-21T22:06:34.156 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:34.156 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:34.156 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:34.157 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.115637+0000", 2024-09-21T22:06:34.157 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2073e-05 2024-09-21T22:06:34.157 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:34.157 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:34.157 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:34.157 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.115768+0000", 2024-09-21T22:06:34.157 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013153899999999999 2024-09-21T22:06:34.157 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:34.157 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:34.157 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:34.157 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.115985+0000", 2024-09-21T22:06:34.157 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000217109 2024-09-21T22:06:34.157 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:34.157 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:34.157 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:34.158 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:34.158 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:34.158 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4224.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e17)", 2024-09-21T22:06:34.158 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:05.116456+0000", 2024-09-21T22:06:34.158 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 29.026305960999998, 2024-09-21T22:06:34.158 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00030938500000000002, 2024-09-21T22:06:34.158 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:34.158 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:34.158 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-21T22:06:34.158 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:34.158 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4224", 2024-09-21T22:06:34.158 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.39:0/795178185", 2024-09-21T22:06:34.158 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3 2024-09-21T22:06:34.159 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:34.159 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:34.159 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:34.159 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:34.159 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.116456+0000", 2024-09-21T22:06:34.159 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:34.159 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:34.159 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:34.159 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:34.159 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.116456+0000", 2024-09-21T22:06:34.159 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:34.159 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:34.159 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:34.159 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:34.159 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.116458+0000", 2024-09-21T22:06:34.160 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.77e-06 2024-09-21T22:06:34.160 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:34.160 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:34.160 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:34.160 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.116466+0000", 2024-09-21T22:06:34.160 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1799999999999996e-06 2024-09-21T22:06:34.160 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:34.160 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:34.160 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:34.160 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.116492+0000", 2024-09-21T22:06:34.160 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6523999999999999e-05 2024-09-21T22:06:34.160 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:34.160 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:34.160 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:34.160 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.116513+0000", 2024-09-21T22:06:34.161 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0400000000000001e-05 2024-09-21T22:06:34.161 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:34.161 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:34.161 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:34.161 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.116616+0000", 2024-09-21T22:06:34.161 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000103862 2024-09-21T22:06:34.161 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:34.161 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:34.161 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:34.161 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.116765+0000", 2024-09-21T22:06:34.161 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014864899999999999 2024-09-21T22:06:34.161 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:34.161 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:34.161 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:34.162 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:34.162 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:34.162 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4224.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-21T22:06:34.162 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:06.109464+0000", 2024-09-21T22:06:34.162 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 28.033297283, 2024-09-21T22:06:34.162 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0070431790000000001, 2024-09-21T22:06:34.162 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:34.162 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:34.162 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:34.162 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:34.162 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4224", 2024-09-21T22:06:34.162 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.39:0/795178185", 2024-09-21T22:06:34.162 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 5 2024-09-21T22:06:34.162 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:34.162 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:34.163 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:34.163 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:34.163 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.109464+0000", 2024-09-21T22:06:34.163 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:34.163 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:34.163 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:34.163 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:34.163 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.109464+0000", 2024-09-21T22:06:34.163 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:34.163 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:34.163 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:34.163 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:34.163 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.109467+0000", 2024-09-21T22:06:34.163 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.066e-06 2024-09-21T22:06:34.164 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:34.164 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:34.164 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:34.164 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.109485+0000", 2024-09-21T22:06:34.164 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7464000000000001e-05 2024-09-21T22:06:34.164 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:34.164 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:34.164 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:34.164 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.109518+0000", 2024-09-21T22:06:34.164 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3343000000000002e-05 2024-09-21T22:06:34.164 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:34.164 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:34.164 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:34.164 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.109537+0000", 2024-09-21T22:06:34.165 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8743e-05 2024-09-21T22:06:34.165 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:34.165 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:34.165 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:34.165 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.109708+0000", 2024-09-21T22:06:34.165 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017073899999999999 2024-09-21T22:06:34.165 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:34.165 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:34.165 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op must wait for map", 2024-09-21T22:06:34.165 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.109725+0000", 2024-09-21T22:06:34.165 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7013000000000001e-05 2024-09-21T22:06:34.165 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:34.165 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:34.165 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:34.165 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.113449+0000", 2024-09-21T22:06:34.166 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0037244579999999999 2024-09-21T22:06:34.166 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:34.166 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:34.166 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:34.166 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.113659+0000", 2024-09-21T22:06:34.166 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020968199999999999 2024-09-21T22:06:34.166 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:34.166 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:34.166 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-09-21T22:06:34.166 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.113872+0000", 2024-09-21T22:06:34.166 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000213062 2024-09-21T22:06:34.166 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:34.166 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:34.167 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:34.167 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.115375+0000", 2024-09-21T22:06:34.167 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015028979999999999 2024-09-21T22:06:34.167 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:34.167 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:34.167 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:34.167 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.116356+0000", 2024-09-21T22:06:34.167 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00098098200000000003 2024-09-21T22:06:34.167 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:34.167 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:34.167 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:34.167 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.116442+0000", 2024-09-21T22:06:34.167 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6533000000000007e-05 2024-09-21T22:06:34.167 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:34.167 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:34.168 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:34.168 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.116508+0000", 2024-09-21T22:06:34.168 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5196000000000004e-05 2024-09-21T22:06:34.168 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:34.168 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:34.168 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:34.168 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:34.168 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:34.168 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4224.0:5 2.2 e17/16 ondisk, result = 0)", 2024-09-21T22:06:34.168 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:06.116133+0000", 2024-09-21T22:06:34.168 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 28.026628798000001, 2024-09-21T22:06:34.168 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00038649700000000002, 2024-09-21T22:06:34.168 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:34.168 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:34.169 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:06:34.169 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:34.169 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:34.169 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:34.169 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.116133+0000", 2024-09-21T22:06:34.169 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:34.169 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:34.169 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:34.169 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:34.169 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.116133+0000", 2024-09-21T22:06:34.169 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:34.169 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:34.169 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:34.169 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:34.169 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.116135+0000", 2024-09-21T22:06:34.170 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8870000000000001e-06 2024-09-21T22:06:34.170 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:34.170 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:34.170 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:34.170 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.116146+0000", 2024-09-21T22:06:34.170 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0953e-05 2024-09-21T22:06:34.170 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:34.170 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:34.170 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:34.170 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.116175+0000", 2024-09-21T22:06:34.170 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9383e-05 2024-09-21T22:06:34.170 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:34.170 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:34.170 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:34.171 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.116190+0000", 2024-09-21T22:06:34.171 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4613000000000001e-05 2024-09-21T22:06:34.171 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:34.171 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:34.171 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:34.171 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.116328+0000", 2024-09-21T22:06:34.171 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013848300000000001 2024-09-21T22:06:34.171 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:34.171 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:34.171 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:34.171 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.116345+0000", 2024-09-21T22:06:34.171 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6883000000000001e-05 2024-09-21T22:06:34.171 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:34.171 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:34.172 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:34.172 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.116519+0000", 2024-09-21T22:06:34.172 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017429500000000001 2024-09-21T22:06:34.172 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:34.172 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:34.172 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:34.172 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:34.172 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:34.172 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4224.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-21T22:06:34.172 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:07.124322+0000", 2024-09-21T22:06:34.172 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 27.018439492999999, 2024-09-21T22:06:34.172 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0067709010000000002, 2024-09-21T22:06:34.172 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:34.172 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:34.172 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:34.173 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:34.173 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4224", 2024-09-21T22:06:34.173 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.39:0/795178185", 2024-09-21T22:06:34.173 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 7 2024-09-21T22:06:34.173 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:34.173 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:34.173 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:34.173 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:34.173 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.124322+0000", 2024-09-21T22:06:34.173 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:34.173 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:34.173 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:34.173 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:34.173 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.124322+0000", 2024-09-21T22:06:34.174 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:34.174 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:34.174 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:34.174 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:34.174 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.124325+0000", 2024-09-21T22:06:34.174 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1489999999999998e-06 2024-09-21T22:06:34.174 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:34.174 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:34.174 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:34.174 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.124342+0000", 2024-09-21T22:06:34.174 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.666e-05 2024-09-21T22:06:34.174 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:34.174 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:34.175 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:34.175 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.124379+0000", 2024-09-21T22:06:34.175 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6539999999999999e-05 2024-09-21T22:06:34.175 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:34.175 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:34.175 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:34.175 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.124406+0000", 2024-09-21T22:06:34.175 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7886e-05 2024-09-21T22:06:34.175 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:34.175 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:34.175 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:34.175 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.124535+0000", 2024-09-21T22:06:34.175 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000128749 2024-09-21T22:06:34.175 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:34.176 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:34.176 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op must wait for map", 2024-09-21T22:06:34.176 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.124547+0000", 2024-09-21T22:06:34.176 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.136e-05 2024-09-21T22:06:34.176 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:34.176 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:34.176 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:34.176 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.127774+0000", 2024-09-21T22:06:34.176 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0032275519999999999 2024-09-21T22:06:34.176 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:34.176 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:34.176 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:34.176 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.128014+0000", 2024-09-21T22:06:34.176 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023956800000000001 2024-09-21T22:06:34.177 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:34.177 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:34.177 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-09-21T22:06:34.177 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.128462+0000", 2024-09-21T22:06:34.177 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00044797700000000001 2024-09-21T22:06:34.177 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:34.177 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:34.177 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:34.177 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.130185+0000", 2024-09-21T22:06:34.177 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00172313 2024-09-21T22:06:34.177 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:34.178 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:34.178 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:34.178 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.130937+0000", 2024-09-21T22:06:34.178 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00075244799999999996 2024-09-21T22:06:34.178 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:34.178 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:34.178 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:34.178 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.131025+0000", 2024-09-21T22:06:34.178 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7998999999999996e-05 2024-09-21T22:06:34.178 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:34.178 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:34.178 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:34.178 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.131093+0000", 2024-09-21T22:06:34.178 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7883e-05 2024-09-21T22:06:34.179 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:34.179 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:34.179 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:34.179 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:34.179 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:34.179 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4224.0:7 2.2 e18/16 ondisk, result = 0)", 2024-09-21T22:06:34.179 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:07.130777+0000", 2024-09-21T22:06:34.179 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 27.011984447, 2024-09-21T22:06:34.179 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00032786799999999999, 2024-09-21T22:06:34.179 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:34.179 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:34.179 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:06:34.179 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:34.179 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:34.180 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:34.180 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.130777+0000", 2024-09-21T22:06:34.180 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:34.180 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:34.180 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:34.180 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:34.180 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.130777+0000", 2024-09-21T22:06:34.180 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:34.180 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:34.180 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:34.180 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:34.180 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.130779+0000", 2024-09-21T22:06:34.180 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7e-06 2024-09-21T22:06:34.180 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:34.181 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:34.181 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:34.181 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.130788+0000", 2024-09-21T22:06:34.181 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6670000000000002e-06 2024-09-21T22:06:34.181 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:34.181 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:34.181 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:34.181 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.130814+0000", 2024-09-21T22:06:34.181 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6590000000000001e-05 2024-09-21T22:06:34.181 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:34.181 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:34.181 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:34.181 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.130828+0000", 2024-09-21T22:06:34.181 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.415e-05 2024-09-21T22:06:34.181 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:34.182 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:34.182 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:34.182 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.130915+0000", 2024-09-21T22:06:34.182 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6759000000000001e-05 2024-09-21T22:06:34.182 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:34.182 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:34.182 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:34.182 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.130928+0000", 2024-09-21T22:06:34.182 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.328e-05 2024-09-21T22:06:34.182 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:34.182 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:34.182 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:34.182 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.131105+0000", 2024-09-21T22:06:34.182 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017672200000000001 2024-09-21T22:06:34.183 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:34.183 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:34.183 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:34.183 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:34.183 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:34.183 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:06:34.184 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T22:06:34.230 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:06:34.229+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:06:34.315 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:06:34.315 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:06:34.315 INFO:teuthology.orchestra.run.smithi039.stdout: "num_ops": 0 2024-09-21T22:06:34.315 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:06:34.329 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:06:34.328+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:06:34.330 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T22:06:34.429 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:06:34.425+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:06:34.501 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:06:34.501 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:06:34.501 INFO:teuthology.orchestra.run.smithi039.stdout: "complaint_time": 30, 2024-09-21T22:06:34.501 INFO:teuthology.orchestra.run.smithi039.stdout: "num_blocked_ops": 0 2024-09-21T22:06:34.501 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:06:34.515 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T22:06:34.530 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:06:34.528+0000 7fc8d39f7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25294) UID: 0 2024-09-21T22:06:34.630 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:06:34.629+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:06:34.688 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:06:34.688 INFO:teuthology.orchestra.run.smithi039.stdout: "size": 20, 2024-09-21T22:06:34.688 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 600, 2024-09-21T22:06:34.688 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [ 2024-09-21T22:06:34.688 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.689 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:20 1.0 e14/13 1:30306672:devicehealth::main.db.0000000000000000:head v 14'10, mlcod=14'8)", 2024-09-21T22:06:34.689 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.137639+0000", 2024-09-21T22:06:34.689 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 32.548657857999999, 2024-09-21T22:06:34.689 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0039902100000000001, 2024-09-21T22:06:34.689 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:34.689 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:34.689 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:34.689 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:34.689 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.689 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:34.689 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.137639+0000", 2024-09-21T22:06:34.689 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:34.690 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.690 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.690 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:34.690 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.137639+0000", 2024-09-21T22:06:34.690 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:34.690 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.690 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.690 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:34.690 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.137641+0000", 2024-09-21T22:06:34.690 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.686e-06 2024-09-21T22:06:34.690 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.690 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.690 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:34.690 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.137934+0000", 2024-09-21T22:06:34.691 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00029319099999999999 2024-09-21T22:06:34.691 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.691 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.691 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:34.691 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.137963+0000", 2024-09-21T22:06:34.691 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9502999999999999e-05 2024-09-21T22:06:34.691 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.691 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.691 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:34.691 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.137989+0000", 2024-09-21T22:06:34.691 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5573000000000001e-05 2024-09-21T22:06:34.691 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.691 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.692 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:34.692 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.138086+0000", 2024-09-21T22:06:34.692 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.6968999999999999e-05 2024-09-21T22:06:34.692 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.692 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.692 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:34.692 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.138120+0000", 2024-09-21T22:06:34.692 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.4560000000000001e-05 2024-09-21T22:06:34.692 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.692 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.692 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:34.692 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.141539+0000", 2024-09-21T22:06:34.692 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0034183289999999999 2024-09-21T22:06:34.692 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.693 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.693 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:34.693 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.141629+0000", 2024-09-21T22:06:34.693 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.0399e-05 2024-09-21T22:06:34.693 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:34.693 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:34.693 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:34.693 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.693 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.693 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:23 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'13, mlcod=14'11)", 2024-09-21T22:06:34.693 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.144914+0000", 2024-09-21T22:06:34.693 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 32.541382540000001, 2024-09-21T22:06:34.693 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0040223230000000004, 2024-09-21T22:06:34.694 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:34.694 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:34.694 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:34.694 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:34.694 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.694 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:34.694 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.144914+0000", 2024-09-21T22:06:34.694 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:34.694 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.694 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.694 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:34.694 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.144914+0000", 2024-09-21T22:06:34.694 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:34.695 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.695 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.695 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:34.695 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.144916+0000", 2024-09-21T22:06:34.695 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6470000000000001e-06 2024-09-21T22:06:34.695 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.695 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.695 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:34.695 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.144929+0000", 2024-09-21T22:06:34.695 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2903e-05 2024-09-21T22:06:34.695 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.695 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.695 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:34.696 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.144958+0000", 2024-09-21T22:06:34.696 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9280000000000001e-05 2024-09-21T22:06:34.696 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.696 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.696 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:34.696 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.144971+0000", 2024-09-21T22:06:34.696 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3407000000000001e-05 2024-09-21T22:06:34.696 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.696 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.696 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:34.696 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.145106+0000", 2024-09-21T22:06:34.696 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013440500000000001 2024-09-21T22:06:34.696 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.696 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.697 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:34.697 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.145170+0000", 2024-09-21T22:06:34.697 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.4597000000000005e-05 2024-09-21T22:06:34.697 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.697 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.697 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:34.697 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.148854+0000", 2024-09-21T22:06:34.697 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0036833249999999999 2024-09-21T22:06:34.697 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.697 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.697 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:34.697 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.148937+0000", 2024-09-21T22:06:34.697 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.2758999999999998e-05 2024-09-21T22:06:34.697 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:34.698 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:34.698 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:34.698 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.698 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.698 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:31 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'17, mlcod=14'14)", 2024-09-21T22:06:34.698 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.163958+0000", 2024-09-21T22:06:34.698 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 32.522338273000003, 2024-09-21T22:06:34.698 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0033891149999999998, 2024-09-21T22:06:34.698 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:34.698 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:34.698 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:34.698 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:34.699 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.699 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:34.699 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.163958+0000", 2024-09-21T22:06:34.699 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:34.699 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.699 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.699 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:34.699 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.163958+0000", 2024-09-21T22:06:34.699 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:34.699 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.699 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.699 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:34.699 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.163960+0000", 2024-09-21T22:06:34.699 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.64e-06 2024-09-21T22:06:34.700 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.700 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.700 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:34.700 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.163971+0000", 2024-09-21T22:06:34.700 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0797e-05 2024-09-21T22:06:34.700 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.700 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.700 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:34.700 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.163995+0000", 2024-09-21T22:06:34.700 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3876e-05 2024-09-21T22:06:34.700 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.700 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.700 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:34.700 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.164008+0000", 2024-09-21T22:06:34.701 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3674e-05 2024-09-21T22:06:34.701 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.701 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.701 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:34.701 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.164203+0000", 2024-09-21T22:06:34.701 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00019457799999999999 2024-09-21T22:06:34.701 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.701 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.701 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:34.701 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.164265+0000", 2024-09-21T22:06:34.701 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.2415999999999993e-05 2024-09-21T22:06:34.701 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.701 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.701 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:34.701 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.167209+0000", 2024-09-21T22:06:34.702 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0029436380000000002 2024-09-21T22:06:34.702 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.702 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.702 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:34.702 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.167348+0000", 2024-09-21T22:06:34.702 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000138496 2024-09-21T22:06:34.702 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:34.702 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:34.702 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:34.702 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.702 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.702 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:32 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'18, mlcod=14'14)", 2024-09-21T22:06:34.702 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.165319+0000", 2024-09-21T22:06:34.702 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 32.520977422999998, 2024-09-21T22:06:34.703 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0033722280000000001, 2024-09-21T22:06:34.703 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:34.703 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:34.703 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:34.703 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:34.703 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.703 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:34.703 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.165319+0000", 2024-09-21T22:06:34.703 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:34.703 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.703 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.703 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:34.703 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.165319+0000", 2024-09-21T22:06:34.703 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:34.704 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.704 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.704 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:34.704 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.165321+0000", 2024-09-21T22:06:34.704 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5999999999999999e-06 2024-09-21T22:06:34.704 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.704 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.704 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:34.704 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.165377+0000", 2024-09-21T22:06:34.704 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.5877000000000001e-05 2024-09-21T22:06:34.704 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.704 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.704 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:34.705 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.165407+0000", 2024-09-21T22:06:34.705 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.0263000000000001e-05 2024-09-21T22:06:34.705 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.705 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.705 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:34.705 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.165441+0000", 2024-09-21T22:06:34.705 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.3966e-05 2024-09-21T22:06:34.705 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.705 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.705 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:34.705 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.165574+0000", 2024-09-21T22:06:34.705 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013327599999999999 2024-09-21T22:06:34.705 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.705 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.705 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:34.706 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.165635+0000", 2024-09-21T22:06:34.706 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.0949000000000002e-05 2024-09-21T22:06:34.706 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.706 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.706 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:34.706 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.168619+0000", 2024-09-21T22:06:34.706 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0029836979999999999 2024-09-21T22:06:34.706 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.706 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.706 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:34.706 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.168692+0000", 2024-09-21T22:06:34.706 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.2599000000000001e-05 2024-09-21T22:06:34.706 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:34.707 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:34.707 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:34.707 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.707 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.707 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:38 1.0 e14/13 1:30306672:devicehealth::main.db.0000000000000000:head v 14'24, mlcod=14'22)", 2024-09-21T22:06:34.707 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.180244+0000", 2024-09-21T22:06:34.707 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 32.506052777999997, 2024-09-21T22:06:34.707 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0041712219999999996, 2024-09-21T22:06:34.707 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:34.707 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:34.707 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:34.707 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:34.707 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.707 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:34.707 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.180244+0000", 2024-09-21T22:06:34.708 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:34.708 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.708 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.708 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:34.708 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.180244+0000", 2024-09-21T22:06:34.708 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:34.708 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.708 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.708 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:34.708 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.180246+0000", 2024-09-21T22:06:34.708 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6959999999999999e-06 2024-09-21T22:06:34.708 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.708 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.708 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:34.709 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.180508+0000", 2024-09-21T22:06:34.709 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00026275199999999999 2024-09-21T22:06:34.709 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.709 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.709 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:34.709 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.180538+0000", 2024-09-21T22:06:34.709 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9142999999999999e-05 2024-09-21T22:06:34.709 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.709 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.709 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:34.709 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.180552+0000", 2024-09-21T22:06:34.709 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4466e-05 2024-09-21T22:06:34.709 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.709 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.710 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:34.710 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.180677+0000", 2024-09-21T22:06:34.710 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000125413 2024-09-21T22:06:34.710 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.710 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.710 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:34.710 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.180739+0000", 2024-09-21T22:06:34.710 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.1122000000000002e-05 2024-09-21T22:06:34.710 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.710 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.710 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:34.710 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.184291+0000", 2024-09-21T22:06:34.710 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0035525510000000001 2024-09-21T22:06:34.710 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.711 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.711 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:34.711 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.184415+0000", 2024-09-21T22:06:34.711 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000124079 2024-09-21T22:06:34.711 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:34.711 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:34.711 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:34.711 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.711 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.711 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:54 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'32, mlcod=14'29)", 2024-09-21T22:06:34.711 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.214326+0000", 2024-09-21T22:06:34.711 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 32.471970540999997, 2024-09-21T22:06:34.711 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0035226569999999998, 2024-09-21T22:06:34.711 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:34.712 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:34.712 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:34.712 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:34.712 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.712 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:34.712 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.214326+0000", 2024-09-21T22:06:34.712 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:34.712 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.712 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.712 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:34.712 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.214326+0000", 2024-09-21T22:06:34.712 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:34.712 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.712 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.713 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:34.713 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.214328+0000", 2024-09-21T22:06:34.713 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6270000000000001e-06 2024-09-21T22:06:34.713 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.713 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.713 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:34.713 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.214340+0000", 2024-09-21T22:06:34.713 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2286000000000001e-05 2024-09-21T22:06:34.713 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.713 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.713 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:34.713 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.214369+0000", 2024-09-21T22:06:34.713 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9297000000000001e-05 2024-09-21T22:06:34.713 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.714 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.714 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:34.714 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.214383+0000", 2024-09-21T22:06:34.714 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3773e-05 2024-09-21T22:06:34.714 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.714 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.714 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:34.714 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.214513+0000", 2024-09-21T22:06:34.714 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000130056 2024-09-21T22:06:34.714 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.714 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.714 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:34.715 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.214568+0000", 2024-09-21T22:06:34.715 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.4873000000000001e-05 2024-09-21T22:06:34.715 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.715 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.715 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:34.715 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.217739+0000", 2024-09-21T22:06:34.715 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0031712720000000002 2024-09-21T22:06:34.715 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.715 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.715 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:34.715 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.217849+0000", 2024-09-21T22:06:34.715 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000109473 2024-09-21T22:06:34.715 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:34.715 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:34.716 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:34.716 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.716 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.716 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:57 1.0 e14/13 1:30306672:devicehealth::main.db.0000000000000000:head v 14'35, mlcod=14'33)", 2024-09-21T22:06:34.716 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.221549+0000", 2024-09-21T22:06:34.716 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 32.464748041999997, 2024-09-21T22:06:34.716 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0041720860000000002, 2024-09-21T22:06:34.716 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:34.716 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:34.716 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:34.716 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:34.716 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.716 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:34.716 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.221549+0000", 2024-09-21T22:06:34.716 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:34.717 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.717 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.717 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:34.717 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.221549+0000", 2024-09-21T22:06:34.717 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:34.717 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.717 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.717 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:34.717 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.221550+0000", 2024-09-21T22:06:34.717 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.717e-06 2024-09-21T22:06:34.717 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.717 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.717 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:34.717 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.221673+0000", 2024-09-21T22:06:34.718 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000122612 2024-09-21T22:06:34.718 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.718 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.718 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:34.718 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.221703+0000", 2024-09-21T22:06:34.718 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9539999999999998e-05 2024-09-21T22:06:34.718 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.718 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.718 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:34.718 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.221716+0000", 2024-09-21T22:06:34.718 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3149999999999999e-05 2024-09-21T22:06:34.718 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.718 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.718 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:34.719 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.221848+0000", 2024-09-21T22:06:34.719 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013201900000000001 2024-09-21T22:06:34.719 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.719 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.719 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:34.719 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.221908+0000", 2024-09-21T22:06:34.719 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.0452999999999999e-05 2024-09-21T22:06:34.719 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.719 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.719 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:34.719 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.225573+0000", 2024-09-21T22:06:34.719 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.003664676 2024-09-21T22:06:34.719 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.720 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.720 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:34.720 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.225721+0000", 2024-09-21T22:06:34.720 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00014791899999999999 2024-09-21T22:06:34.720 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:34.720 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:34.720 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:34.720 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.720 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.720 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:68 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'42, mlcod=14'39)", 2024-09-21T22:06:34.720 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.247413+0000", 2024-09-21T22:06:34.720 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 32.438883693000001, 2024-09-21T22:06:34.720 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0039572339999999996, 2024-09-21T22:06:34.720 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:34.721 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:34.721 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:34.721 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:34.721 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.721 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:34.721 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.247413+0000", 2024-09-21T22:06:34.721 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:34.721 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.721 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.721 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:34.721 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.247413+0000", 2024-09-21T22:06:34.721 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:34.721 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.722 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.722 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:34.722 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.247415+0000", 2024-09-21T22:06:34.722 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.883e-06 2024-09-21T22:06:34.722 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.722 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.722 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:34.722 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.247427+0000", 2024-09-21T22:06:34.722 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2557e-05 2024-09-21T22:06:34.722 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.722 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.722 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:34.722 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.247457+0000", 2024-09-21T22:06:34.722 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9966000000000001e-05 2024-09-21T22:06:34.723 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.723 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.723 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:34.723 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.247471+0000", 2024-09-21T22:06:34.723 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3747000000000001e-05 2024-09-21T22:06:34.723 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.723 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.723 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:34.723 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.247603+0000", 2024-09-21T22:06:34.723 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013171600000000001 2024-09-21T22:06:34.723 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.723 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.723 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:34.723 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.247656+0000", 2024-09-21T22:06:34.724 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.3529999999999997e-05 2024-09-21T22:06:34.724 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.724 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.724 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:34.724 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.251225+0000", 2024-09-21T22:06:34.724 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00356865 2024-09-21T22:06:34.724 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.724 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.724 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:34.724 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.251370+0000", 2024-09-21T22:06:34.724 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00014518500000000001 2024-09-21T22:06:34.724 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:34.724 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:34.724 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:34.725 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.725 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.725 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:69 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'43, mlcod=14'39)", 2024-09-21T22:06:34.725 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.248839+0000", 2024-09-21T22:06:34.725 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 32.437457762999998, 2024-09-21T22:06:34.725 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0034954470000000001, 2024-09-21T22:06:34.725 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:34.725 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:34.725 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:34.725 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:34.725 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.725 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:34.725 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.248839+0000", 2024-09-21T22:06:34.726 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:34.726 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.726 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.726 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:34.726 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.248839+0000", 2024-09-21T22:06:34.726 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:34.726 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.726 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.726 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:34.726 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.248841+0000", 2024-09-21T22:06:34.726 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6759999999999999e-06 2024-09-21T22:06:34.726 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.726 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.726 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:34.726 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.249058+0000", 2024-09-21T22:06:34.727 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000217455 2024-09-21T22:06:34.727 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.727 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.727 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:34.727 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.249085+0000", 2024-09-21T22:06:34.727 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.7002999999999999e-05 2024-09-21T22:06:34.727 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.727 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.727 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:34.727 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.249099+0000", 2024-09-21T22:06:34.727 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.402e-05 2024-09-21T22:06:34.727 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.727 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.727 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:34.728 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.249241+0000", 2024-09-21T22:06:34.728 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00014194899999999999 2024-09-21T22:06:34.728 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.728 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.728 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:34.728 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.249298+0000", 2024-09-21T22:06:34.728 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.6687000000000001e-05 2024-09-21T22:06:34.728 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.728 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.728 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:34.728 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.252266+0000", 2024-09-21T22:06:34.728 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.002968348 2024-09-21T22:06:34.728 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.728 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.729 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:34.729 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.252334+0000", 2024-09-21T22:06:34.729 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.8308999999999999e-05 2024-09-21T22:06:34.729 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:34.729 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:34.729 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:34.729 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.729 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.729 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:71 1.0 e14/13 1:30306672:devicehealth::main.db.0000000000000000:head v 14'45, mlcod=14'43)", 2024-09-21T22:06:34.729 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.256068+0000", 2024-09-21T22:06:34.729 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 32.430228290999999, 2024-09-21T22:06:34.729 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0041712579999999997, 2024-09-21T22:06:34.729 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:34.729 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:34.730 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:34.730 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:34.730 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.730 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:34.730 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.256068+0000", 2024-09-21T22:06:34.730 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:34.730 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.730 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.730 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:34.730 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.256068+0000", 2024-09-21T22:06:34.730 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:34.730 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.730 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.731 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:34.731 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.256070+0000", 2024-09-21T22:06:34.731 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.61e-06 2024-09-21T22:06:34.731 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.731 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.731 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:34.731 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.256129+0000", 2024-09-21T22:06:34.731 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.8702999999999997e-05 2024-09-21T22:06:34.731 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.731 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.731 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:34.731 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.256158+0000", 2024-09-21T22:06:34.731 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9533e-05 2024-09-21T22:06:34.732 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.732 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.732 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:34.732 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.256173+0000", 2024-09-21T22:06:34.732 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4267000000000001e-05 2024-09-21T22:06:34.732 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.732 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.732 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:34.732 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.256304+0000", 2024-09-21T22:06:34.732 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000131352 2024-09-21T22:06:34.732 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.732 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.732 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:34.733 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.256357+0000", 2024-09-21T22:06:34.733 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.3492999999999998e-05 2024-09-21T22:06:34.733 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.733 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.733 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:34.733 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.260147+0000", 2024-09-21T22:06:34.733 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0037900410000000001 2024-09-21T22:06:34.733 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.733 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.733 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:34.733 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.260240+0000", 2024-09-21T22:06:34.733 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.2258999999999999e-05 2024-09-21T22:06:34.733 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:34.733 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:34.734 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:34.734 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.734 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.734 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:72 1.0 e14/13 1:30306672:devicehealth::main.db.0000000000000000:head v 14'46, mlcod=14'43)", 2024-09-21T22:06:34.734 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.258358+0000", 2024-09-21T22:06:34.734 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 32.427938298000001, 2024-09-21T22:06:34.734 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0041492919999999997, 2024-09-21T22:06:34.734 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:34.734 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:34.734 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:34.734 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:34.734 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.735 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:34.735 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.258358+0000", 2024-09-21T22:06:34.735 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:34.735 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.735 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.735 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:34.735 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.258358+0000", 2024-09-21T22:06:34.735 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:34.735 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.735 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.735 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:34.735 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.258360+0000", 2024-09-21T22:06:34.735 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.623e-06 2024-09-21T22:06:34.735 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.735 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.736 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:34.736 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.258415+0000", 2024-09-21T22:06:34.736 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.4756000000000002e-05 2024-09-21T22:06:34.736 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.736 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.736 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:34.736 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.258444+0000", 2024-09-21T22:06:34.736 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9164000000000001e-05 2024-09-21T22:06:34.736 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.736 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.736 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:34.736 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.258459+0000", 2024-09-21T22:06:34.736 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4996e-05 2024-09-21T22:06:34.736 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.736 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.737 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:34.737 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.258637+0000", 2024-09-21T22:06:34.737 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00017796500000000001 2024-09-21T22:06:34.737 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.737 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.737 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:34.737 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.258695+0000", 2024-09-21T22:06:34.737 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.8207e-05 2024-09-21T22:06:34.737 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.737 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.737 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:34.737 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.262421+0000", 2024-09-21T22:06:34.737 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0037261550000000001 2024-09-21T22:06:34.738 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.738 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.738 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:34.738 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.262508+0000", 2024-09-21T22:06:34.738 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.6426000000000002e-05 2024-09-21T22:06:34.738 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:34.738 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:34.738 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:34.738 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.738 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.738 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:87 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'57, mlcod=14'52)", 2024-09-21T22:06:34.738 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.290115+0000", 2024-09-21T22:06:34.738 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 32.396181853000002, 2024-09-21T22:06:34.739 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0030478200000000001, 2024-09-21T22:06:34.739 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:34.739 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:34.739 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:34.739 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:34.739 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.739 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:34.739 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.290115+0000", 2024-09-21T22:06:34.739 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:34.739 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.739 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.739 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:34.739 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.290115+0000", 2024-09-21T22:06:34.739 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:34.740 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.740 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.740 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:34.740 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.290116+0000", 2024-09-21T22:06:34.740 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3829999999999999e-06 2024-09-21T22:06:34.740 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.740 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.740 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:34.740 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.290191+0000", 2024-09-21T22:06:34.740 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.4562999999999997e-05 2024-09-21T22:06:34.740 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.740 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.740 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:34.741 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.290216+0000", 2024-09-21T22:06:34.741 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.565e-05 2024-09-21T22:06:34.741 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.741 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.741 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:34.741 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.290236+0000", 2024-09-21T22:06:34.741 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9326000000000001e-05 2024-09-21T22:06:34.741 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.741 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.741 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:34.741 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.290340+0000", 2024-09-21T22:06:34.741 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0001047 2024-09-21T22:06:34.741 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.742 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.742 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:34.742 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.290377+0000", 2024-09-21T22:06:34.742 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.6356000000000002e-05 2024-09-21T22:06:34.742 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.742 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.742 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:34.742 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.293096+0000", 2024-09-21T22:06:34.742 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.002718996 2024-09-21T22:06:34.742 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.742 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.742 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:34.742 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.293163+0000", 2024-09-21T22:06:34.743 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.6846000000000003e-05 2024-09-21T22:06:34.743 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:34.743 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:34.743 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:34.743 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.743 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.743 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:114 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'76, mlcod=14'73)", 2024-09-21T22:06:34.743 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.490030+0000", 2024-09-21T22:06:34.743 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 32.196266414, 2024-09-21T22:06:34.743 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.004227873, 2024-09-21T22:06:34.743 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:34.743 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:34.743 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:34.744 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:34.744 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.744 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:34.744 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.490030+0000", 2024-09-21T22:06:34.744 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:34.744 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.744 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.744 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:34.744 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.490030+0000", 2024-09-21T22:06:34.744 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:34.744 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.744 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.744 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:34.744 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.490031+0000", 2024-09-21T22:06:34.745 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.3300000000000001e-07 2024-09-21T22:06:34.745 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.745 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.745 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:34.745 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.490042+0000", 2024-09-21T22:06:34.745 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.1095e-05 2024-09-21T22:06:34.745 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.745 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.745 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:34.745 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.490053+0000", 2024-09-21T22:06:34.745 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0501e-05 2024-09-21T22:06:34.745 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.745 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.745 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:34.746 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.490060+0000", 2024-09-21T22:06:34.746 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.9829999999999999e-06 2024-09-21T22:06:34.746 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.746 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.746 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:34.746 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.490107+0000", 2024-09-21T22:06:34.746 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.7278999999999999e-05 2024-09-21T22:06:34.746 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.746 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.746 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:34.746 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.490124+0000", 2024-09-21T22:06:34.746 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7190999999999999e-05 2024-09-21T22:06:34.747 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.747 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.747 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:34.747 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.494229+0000", 2024-09-21T22:06:34.747 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0041050440000000004 2024-09-21T22:06:34.747 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.747 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.747 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:34.747 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.494258+0000", 2024-09-21T22:06:34.747 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9047000000000002e-05 2024-09-21T22:06:34.747 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:34.747 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:34.747 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:34.748 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.748 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.748 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:115 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'77, mlcod=14'73)", 2024-09-21T22:06:34.748 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.491380+0000", 2024-09-21T22:06:34.748 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 32.194916792000001, 2024-09-21T22:06:34.748 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0031550889999999998, 2024-09-21T22:06:34.748 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:34.748 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:34.748 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:34.748 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:34.748 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.748 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:34.748 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.491380+0000", 2024-09-21T22:06:34.748 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:34.749 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.749 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.749 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:34.749 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.491380+0000", 2024-09-21T22:06:34.749 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:34.749 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.749 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.749 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:34.749 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.491381+0000", 2024-09-21T22:06:34.749 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.1399999999999996e-07 2024-09-21T22:06:34.749 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.749 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.749 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:34.750 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.491435+0000", 2024-09-21T22:06:34.750 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.3797e-05 2024-09-21T22:06:34.750 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.750 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.750 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:34.750 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.491447+0000", 2024-09-21T22:06:34.750 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2642e-05 2024-09-21T22:06:34.750 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.750 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.750 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:34.750 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.491453+0000", 2024-09-21T22:06:34.750 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.4820000000000002e-06 2024-09-21T22:06:34.751 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.751 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.751 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:34.751 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.491502+0000", 2024-09-21T22:06:34.751 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.9038e-05 2024-09-21T22:06:34.751 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.751 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.751 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:34.751 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.491514+0000", 2024-09-21T22:06:34.751 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2483999999999999e-05 2024-09-21T22:06:34.751 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.751 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.751 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:34.751 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.494486+0000", 2024-09-21T22:06:34.752 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.002971338 2024-09-21T22:06:34.752 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.752 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.752 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:34.752 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.494535+0000", 2024-09-21T22:06:34.752 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.9493999999999997e-05 2024-09-21T22:06:34.752 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:34.752 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:34.752 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:34.752 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.752 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.753 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:118 1.0 e14/13 1:30306672:devicehealth::main.db.0000000000000000:head v 14'80, mlcod=14'78)", 2024-09-21T22:06:34.753 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.499045+0000", 2024-09-21T22:06:34.753 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 32.187251789999998, 2024-09-21T22:06:34.753 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0029889589999999998, 2024-09-21T22:06:34.753 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:34.753 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:34.753 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:34.753 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:34.753 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.753 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:34.753 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.499045+0000", 2024-09-21T22:06:34.753 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:34.754 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.754 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.754 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:34.754 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.499045+0000", 2024-09-21T22:06:34.754 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:34.754 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.754 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.754 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:34.754 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.499046+0000", 2024-09-21T22:06:34.754 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.0299999999999998e-07 2024-09-21T22:06:34.754 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.754 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.754 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:34.755 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.499110+0000", 2024-09-21T22:06:34.755 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.4688000000000001e-05 2024-09-21T22:06:34.755 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.755 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.755 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:34.755 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.499125+0000", 2024-09-21T22:06:34.755 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4387e-05 2024-09-21T22:06:34.755 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.755 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.755 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:34.755 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.499131+0000", 2024-09-21T22:06:34.755 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.2129999999999996e-06 2024-09-21T22:06:34.755 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.755 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.756 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:34.756 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.499255+0000", 2024-09-21T22:06:34.756 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00012424199999999999 2024-09-21T22:06:34.756 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.756 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.756 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:34.756 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.499312+0000", 2024-09-21T22:06:34.756 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.6709999999999997e-05 2024-09-21T22:06:34.756 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.756 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.756 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:34.756 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.502005+0000", 2024-09-21T22:06:34.756 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0026927169999999999 2024-09-21T22:06:34.757 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.757 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.757 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:34.757 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.502034+0000", 2024-09-21T22:06:34.757 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9198999999999999e-05 2024-09-21T22:06:34.757 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:34.757 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:34.757 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:34.757 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.757 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.757 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:136 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'94, mlcod=14'89)", 2024-09-21T22:06:34.757 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.903810+0000", 2024-09-21T22:06:34.757 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 31.782486818999999, 2024-09-21T22:06:34.758 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0062746829999999996, 2024-09-21T22:06:34.758 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:34.758 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:34.758 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:34.758 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:34.758 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.758 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:34.758 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.903810+0000", 2024-09-21T22:06:34.758 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:34.758 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.758 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.758 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:34.758 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.903810+0000", 2024-09-21T22:06:34.759 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:34.759 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.759 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.759 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:34.759 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.903810+0000", 2024-09-21T22:06:34.759 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.03e-07 2024-09-21T22:06:34.759 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.759 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.759 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:34.759 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.903814+0000", 2024-09-21T22:06:34.759 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.676e-06 2024-09-21T22:06:34.759 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.759 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.759 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:34.759 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.903822+0000", 2024-09-21T22:06:34.760 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.2879999999999993e-06 2024-09-21T22:06:34.760 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.760 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.760 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:34.760 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.903829+0000", 2024-09-21T22:06:34.760 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.2920000000000001e-06 2024-09-21T22:06:34.760 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.760 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.760 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:34.760 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.903879+0000", 2024-09-21T22:06:34.760 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.0253999999999997e-05 2024-09-21T22:06:34.760 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.760 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.760 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:34.761 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.903897+0000", 2024-09-21T22:06:34.761 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8468999999999999e-05 2024-09-21T22:06:34.761 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.761 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.761 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:34.761 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.910044+0000", 2024-09-21T22:06:34.761 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0061468950000000003 2024-09-21T22:06:34.761 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.761 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.761 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:34.761 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.910085+0000", 2024-09-21T22:06:34.761 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.0405999999999999e-05 2024-09-21T22:06:34.761 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:34.761 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:34.761 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:34.762 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.762 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.762 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:137 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'95, mlcod=14'90)", 2024-09-21T22:06:34.762 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.904439+0000", 2024-09-21T22:06:34.762 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 31.781857423000002, 2024-09-21T22:06:34.762 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0072625019999999997, 2024-09-21T22:06:34.762 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:34.762 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:34.762 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:34.762 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:34.762 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.762 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:34.762 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.904439+0000", 2024-09-21T22:06:34.763 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:34.763 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.763 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.763 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:34.763 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.904439+0000", 2024-09-21T22:06:34.763 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:34.763 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.763 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.763 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:34.763 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.904440+0000", 2024-09-21T22:06:34.763 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8400000000000005e-07 2024-09-21T22:06:34.763 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.763 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.763 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:34.764 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.904498+0000", 2024-09-21T22:06:34.764 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.8320999999999999e-05 2024-09-21T22:06:34.764 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.764 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.764 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:34.764 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.904507+0000", 2024-09-21T22:06:34.764 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.7790000000000001e-06 2024-09-21T22:06:34.764 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.764 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.764 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:34.764 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.904511+0000", 2024-09-21T22:06:34.765 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.9700000000000001e-06 2024-09-21T22:06:34.765 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.765 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.765 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:34.765 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.904551+0000", 2024-09-21T22:06:34.765 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.9910000000000002e-05 2024-09-21T22:06:34.765 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.765 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.765 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:34.765 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.904567+0000", 2024-09-21T22:06:34.765 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5903999999999999e-05 2024-09-21T22:06:34.765 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.765 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.766 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:34.766 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.911666+0000", 2024-09-21T22:06:34.766 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0070996929999999998 2024-09-21T22:06:34.766 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.766 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.766 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:34.766 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.911702+0000", 2024-09-21T22:06:34.766 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.5441000000000002e-05 2024-09-21T22:06:34.766 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:34.766 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:34.766 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:34.766 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.766 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.767 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:138 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'96, mlcod=14'90)", 2024-09-21T22:06:34.767 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.905402+0000", 2024-09-21T22:06:34.767 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 31.780894446000001, 2024-09-21T22:06:34.767 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0063366890000000004, 2024-09-21T22:06:34.767 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:34.767 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:34.767 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:34.767 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:34.767 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.767 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:34.767 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905402+0000", 2024-09-21T22:06:34.767 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:34.767 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.767 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.767 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:34.768 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905402+0000", 2024-09-21T22:06:34.768 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:34.768 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.768 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.768 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:34.768 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905403+0000", 2024-09-21T22:06:34.768 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.9200000000000002e-07 2024-09-21T22:06:34.768 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.768 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.768 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:34.768 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905406+0000", 2024-09-21T22:06:34.768 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.326e-06 2024-09-21T22:06:34.768 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.769 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.769 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:34.769 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905413+0000", 2024-09-21T22:06:34.769 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.3320000000000003e-06 2024-09-21T22:06:34.769 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.769 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.769 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:34.769 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905417+0000", 2024-09-21T22:06:34.769 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.7139999999999999e-06 2024-09-21T22:06:34.769 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.769 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.769 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:34.769 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.910115+0000", 2024-09-21T22:06:34.769 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0046983939999999998 2024-09-21T22:06:34.770 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.770 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.770 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:34.770 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.910140+0000", 2024-09-21T22:06:34.770 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4689999999999999e-05 2024-09-21T22:06:34.770 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.770 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.770 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:34.770 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.911706+0000", 2024-09-21T22:06:34.770 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0015658479999999999 2024-09-21T22:06:34.770 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.770 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.770 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:34.770 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.911739+0000", 2024-09-21T22:06:34.771 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.2993000000000001e-05 2024-09-21T22:06:34.771 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:34.771 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:34.771 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:34.771 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.771 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.771 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:139 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'97, mlcod=14'90)", 2024-09-21T22:06:34.771 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.905983+0000", 2024-09-21T22:06:34.771 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 31.780313528000001, 2024-09-21T22:06:34.771 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0057906030000000001, 2024-09-21T22:06:34.771 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:34.771 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:34.771 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:34.771 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:34.771 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.772 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:34.772 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905983+0000", 2024-09-21T22:06:34.772 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:34.772 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.772 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.772 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:34.772 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905983+0000", 2024-09-21T22:06:34.772 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:34.772 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.772 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.772 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:34.772 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905984+0000", 2024-09-21T22:06:34.772 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.15e-07 2024-09-21T22:06:34.772 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.773 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.773 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:34.773 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905986+0000", 2024-09-21T22:06:34.773 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.852e-06 2024-09-21T22:06:34.773 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.773 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.773 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:34.773 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905993+0000", 2024-09-21T22:06:34.773 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.652e-06 2024-09-21T22:06:34.773 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.773 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.773 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:34.773 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905996+0000", 2024-09-21T22:06:34.773 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.3519999999999998e-06 2024-09-21T22:06:34.774 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.774 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.774 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:34.774 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.910519+0000", 2024-09-21T22:06:34.774 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0045230820000000003 2024-09-21T22:06:34.774 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.774 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.774 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:34.774 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.910548+0000", 2024-09-21T22:06:34.774 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8399999999999999e-05 2024-09-21T22:06:34.774 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.774 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.774 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:34.774 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.911742+0000", 2024-09-21T22:06:34.775 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011939909999999999 2024-09-21T22:06:34.775 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.775 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.775 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:34.775 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.911774+0000", 2024-09-21T22:06:34.775 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.1958999999999997e-05 2024-09-21T22:06:34.775 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:34.775 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:34.775 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:34.775 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.775 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.775 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:140 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'98, mlcod=14'90)", 2024-09-21T22:06:34.775 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.906441+0000", 2024-09-21T22:06:34.775 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 31.779855398999999, 2024-09-21T22:06:34.776 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0057052989999999996, 2024-09-21T22:06:34.776 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:34.776 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:34.776 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:34.776 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:34.776 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.776 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:34.776 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.906441+0000", 2024-09-21T22:06:34.776 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:34.776 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.776 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.776 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:34.776 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.906441+0000", 2024-09-21T22:06:34.776 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:34.777 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.777 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.777 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:34.777 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.906442+0000", 2024-09-21T22:06:34.777 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.7e-07 2024-09-21T22:06:34.777 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.777 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.777 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:34.777 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.906511+0000", 2024-09-21T22:06:34.777 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.8857999999999997e-05 2024-09-21T22:06:34.777 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.777 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.777 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:34.778 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.906518+0000", 2024-09-21T22:06:34.778 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.1690000000000003e-06 2024-09-21T22:06:34.778 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.778 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.778 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:34.778 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.906521+0000", 2024-09-21T22:06:34.778 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.4829999999999999e-06 2024-09-21T22:06:34.778 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.778 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.778 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:34.778 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.910939+0000", 2024-09-21T22:06:34.778 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0044176809999999997 2024-09-21T22:06:34.778 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.779 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.779 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:34.779 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.910959+0000", 2024-09-21T22:06:34.779 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9898000000000002e-05 2024-09-21T22:06:34.779 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.779 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.779 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:34.779 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.912113+0000", 2024-09-21T22:06:34.779 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001153811 2024-09-21T22:06:34.779 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:34.779 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:34.779 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:34.779 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.912147+0000", 2024-09-21T22:06:34.779 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.4029000000000002e-05 2024-09-21T22:06:34.780 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:34.780 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:34.780 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:34.780 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:34.780 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:34.780 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:06:34.782 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T22:06:34.782 INFO:teuthology.orchestra.run.smithi039.stderr:nodeep-scrub is unset 2024-09-21T22:06:34.783 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:06:34.777+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:06:34.882 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:06:34.876+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:06:34.948 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:06:34.949 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:06:34.949 INFO:teuthology.orchestra.run.smithi039.stdout: "num_ops": 0 2024-09-21T22:06:34.949 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:06:34.963 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T22:06:35.010 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:06:35.009+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:06:35.085 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:06:35.080+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:06:35.092 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:06:35.092 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:06:35.092 INFO:teuthology.orchestra.run.smithi039.stdout: "complaint_time": 30, 2024-09-21T22:06:35.092 INFO:teuthology.orchestra.run.smithi039.stdout: "num_blocked_ops": 0 2024-09-21T22:06:35.092 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:06:35.107 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T22:06:35.186 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:06:35.180+0000 7fc8d39f7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25294) UID: 0 2024-09-21T22:06:35.277 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:06:35.277 INFO:teuthology.orchestra.run.smithi039.stdout: "size": 20, 2024-09-21T22:06:35.277 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 600, 2024-09-21T22:06:35.277 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [ 2024-09-21T22:06:35.277 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:35.277 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4224.0:5 2.2 e18/16 2:5cea7035:::rbd_info:head v 18'1, mlcod=0'0)", 2024-09-21T22:06:35.277 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:06.114955+0000", 2024-09-21T22:06:35.277 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 29.162611674000001, 2024-09-21T22:06:35.278 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0017716170000000001, 2024-09-21T22:06:35.278 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:35.278 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:35.278 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:35.278 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:35.278 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:35.278 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:35.278 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:06.114955+0000", 2024-09-21T22:06:35.278 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:35.278 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:35.278 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:35.279 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:35.279 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:06.114955+0000", 2024-09-21T22:06:35.279 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:35.279 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:35.279 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:35.279 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:35.279 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:06.114956+0000", 2024-09-21T22:06:35.279 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.9299999999999997e-07 2024-09-21T22:06:35.279 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:35.279 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:35.279 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:35.279 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:06.114964+0000", 2024-09-21T22:06:35.280 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.9840000000000001e-06 2024-09-21T22:06:35.280 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:35.280 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:35.280 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:35.280 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:06.115001+0000", 2024-09-21T22:06:35.280 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.7719999999999998e-05 2024-09-21T22:06:35.280 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:35.280 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:35.280 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:35.280 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:06.115019+0000", 2024-09-21T22:06:35.280 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7943000000000001e-05 2024-09-21T22:06:35.280 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:35.280 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:35.280 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:35.281 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:06.115168+0000", 2024-09-21T22:06:35.281 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000148589 2024-09-21T22:06:35.281 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:35.281 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:35.281 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:35.281 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:06.115225+0000", 2024-09-21T22:06:35.281 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.6975999999999998e-05 2024-09-21T22:06:35.281 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:35.281 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:35.281 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:35.281 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:06.116585+0000", 2024-09-21T22:06:35.281 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00136038 2024-09-21T22:06:35.281 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:35.281 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:35.282 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:35.282 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:06.116726+0000", 2024-09-21T22:06:35.282 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00014133200000000001 2024-09-21T22:06:35.282 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:35.282 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:35.282 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:35.282 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:35.282 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:35.282 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4224.0:7 2.2 e19/16 2:5cea7035:::rbd_info:head v 19'2, mlcod=0'0)", 2024-09-21T22:06:35.282 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:07.129567+0000", 2024-09-21T22:06:35.282 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 28.147999665, 2024-09-21T22:06:35.282 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0018460239999999999, 2024-09-21T22:06:35.282 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:35.282 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:35.283 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:35.283 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:35.283 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:35.283 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:35.283 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:07.129567+0000", 2024-09-21T22:06:35.283 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:35.283 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:35.283 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:35.283 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:35.283 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:07.129567+0000", 2024-09-21T22:06:35.283 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:35.283 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:35.283 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:35.283 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:35.283 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:07.129567+0000", 2024-09-21T22:06:35.284 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.9499999999999998e-07 2024-09-21T22:06:35.284 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:35.284 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:35.284 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:35.284 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:07.129580+0000", 2024-09-21T22:06:35.284 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2335e-05 2024-09-21T22:06:35.284 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:35.284 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:35.284 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:35.284 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:07.129606+0000", 2024-09-21T22:06:35.284 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.6528000000000001e-05 2024-09-21T22:06:35.284 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:35.284 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:35.284 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:35.284 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:07.129628+0000", 2024-09-21T22:06:35.285 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1549999999999999e-05 2024-09-21T22:06:35.285 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:35.285 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:35.285 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:35.285 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:07.129744+0000", 2024-09-21T22:06:35.285 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000116551 2024-09-21T22:06:35.285 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:35.285 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:35.285 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:35.285 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:07.129780+0000", 2024-09-21T22:06:35.285 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.5447000000000001e-05 2024-09-21T22:06:35.286 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:35.286 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:35.286 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:35.286 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:07.131348+0000", 2024-09-21T22:06:35.286 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001568631 2024-09-21T22:06:35.286 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:35.286 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:35.286 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:35.286 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:07.131413+0000", 2024-09-21T22:06:35.286 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.4586999999999997e-05 2024-09-21T22:06:35.286 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:35.286 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:35.286 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:35.286 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:35.287 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:35.287 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:06:35.288 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:06:35.285+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:06:35.293 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T22:06:35.389 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:06:35.384+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:06:35.460 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:06:35.461 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:06:35.461 INFO:teuthology.orchestra.run.smithi039.stdout: "num_ops": 0 2024-09-21T22:06:35.461 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:06:35.475 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T22:06:35.526 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:06:35.525+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:06:35.591 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:06:35.588+0000 7fc8d39f7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25294) UID: 0 2024-09-21T22:06:35.602 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:06:35.603 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:06:35.603 INFO:teuthology.orchestra.run.smithi039.stdout: "complaint_time": 30, 2024-09-21T22:06:35.603 INFO:teuthology.orchestra.run.smithi039.stdout: "num_blocked_ops": 0 2024-09-21T22:06:35.603 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:06:35.617 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T22:06:35.692 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:06:35.688+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:06:35.778 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:06:35.778 INFO:teuthology.orchestra.run.smithi039.stdout: "size": 20, 2024-09-21T22:06:35.778 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 600, 2024-09-21T22:06:35.778 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [ 2024-09-21T22:06:35.778 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:35.778 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4224.0:1 2.3 e17/16 2:c4c92e5a:::rbd_trash:head v 17'1, mlcod=0'0)", 2024-09-21T22:06:35.778 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:05.107810+0000", 2024-09-21T22:06:35.778 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 30.671093936999998, 2024-09-21T22:06:35.779 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0017679200000000001, 2024-09-21T22:06:35.779 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:35.779 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:35.779 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:35.779 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:35.779 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:35.779 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:35.779 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.107810+0000", 2024-09-21T22:06:35.779 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:35.779 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:35.779 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:35.779 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:35.779 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.107810+0000", 2024-09-21T22:06:35.779 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:35.780 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:35.780 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:35.780 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:35.780 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.107811+0000", 2024-09-21T22:06:35.780 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.6700000000000003e-07 2024-09-21T22:06:35.780 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:35.780 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:35.780 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:35.780 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.107822+0000", 2024-09-21T22:06:35.780 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0443e-05 2024-09-21T22:06:35.780 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:35.780 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:35.780 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:35.780 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.107853+0000", 2024-09-21T22:06:35.780 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.1046000000000001e-05 2024-09-21T22:06:35.781 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:35.781 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:35.781 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:35.781 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.107876+0000", 2024-09-21T22:06:35.781 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3547e-05 2024-09-21T22:06:35.781 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:35.781 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:35.781 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:35.781 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.108002+0000", 2024-09-21T22:06:35.781 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000125599 2024-09-21T22:06:35.781 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:35.781 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:35.781 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:35.781 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.108069+0000", 2024-09-21T22:06:35.781 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.7162999999999994e-05 2024-09-21T22:06:35.782 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:35.782 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:35.782 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:35.782 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.109385+0000", 2024-09-21T22:06:35.782 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0013162899999999999 2024-09-21T22:06:35.782 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:35.782 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:35.782 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:35.782 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.109578+0000", 2024-09-21T22:06:35.782 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000193065 2024-09-21T22:06:35.782 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:35.782 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:35.782 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:35.782 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:35.782 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:35.783 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:06:35.793 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T22:06:35.794 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:06:35.788+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:06:35.894 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:06:35.893+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:06:35.930 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:06:35.930 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:06:35.930 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-09-21T22:06:35.930 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:06:35.944 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T22:06:35.971 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T22:06:35.995 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:06:35.992+0000 7fc8d39f7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25294) UID: 0 2024-09-21T22:06:36.062 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:06:36.062 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:06:36.062 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-21T22:06:36.062 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-21T22:06:36.062 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:06:36.076 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T22:06:36.119 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:06:36.116+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:06:36.129 INFO:teuthology.orchestra.run.smithi039.stdout:osd.0: {} 2024-09-21T22:06:36.129 INFO:teuthology.orchestra.run.smithi039.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T22:06:36.146 INFO:teuthology.orchestra.run.smithi039.stdout:osd.1: {} 2024-09-21T22:06:36.146 INFO:teuthology.orchestra.run.smithi039.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T22:06:36.162 INFO:teuthology.orchestra.run.smithi039.stdout:osd.2: {} 2024-09-21T22:06:36.163 INFO:teuthology.orchestra.run.smithi039.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T22:06:36.181 INFO:teuthology.orchestra.run.smithi039.stdout:osd.3: {} 2024-09-21T22:06:36.181 INFO:teuthology.orchestra.run.smithi039.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T22:06:36.197 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:06:36.192+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:06:36.200 INFO:teuthology.orchestra.run.smithi039.stdout:osd.4: {} 2024-09-21T22:06:36.200 INFO:teuthology.orchestra.run.smithi039.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T22:06:36.220 INFO:teuthology.orchestra.run.smithi039.stdout:osd.5: {} 2024-09-21T22:06:36.221 INFO:teuthology.orchestra.run.smithi039.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T22:06:36.250 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:36.250 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:36.264 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T22:06:36.298 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:06:36.297+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:06:36.399 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:06:36.397+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:06:36.435 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:36.435 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:36.450 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T22:06:36.499 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:06:36.497+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:06:36.576 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:36.576 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:36.590 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T22:06:36.635 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:06:36.632+0000 7fc8d39f7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25294) UID: 0 2024-09-21T22:06:36.702 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:06:36.696+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:06:36.717 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:36.718 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:36.732 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T22:06:36.758 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-09-21T22:06:36.758 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-09-21T22:06:36.784 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-21T22:06:36.806 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:06:36.805+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:06:36.901 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:36.901 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:36.903 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:06:36.900+0000 7fc8d39f7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25294) UID: 0 2024-09-21T22:06:36.915 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T22:06:37.004 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:06:37.000+0000 7fc8d39f7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25294) UID: 0 2024-09-21T22:06:37.083 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:37.083 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:37.098 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T22:06:37.139 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:06:37.136+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:06:37.205 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:06:37.201+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:06:37.270 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:37.270 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:37.285 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T22:06:37.305 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:06:37.300+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:06:37.406 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:06:37.400+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:06:37.413 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:37.414 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:37.427 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T22:06:37.507 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:06:37.504+0000 7fc8d39f7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25294) UID: 0 2024-09-21T22:06:37.549 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:37.549 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:37.564 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T22:06:37.614 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:06:37.613+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:06:37.698 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:37.699 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:37.710 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:06:37.705+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:06:37.713 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T22:06:37.754 INFO:teuthology.orchestra.run.smithi039.stderr:reweighted osd.1 to 1 (10000) 2024-09-21T22:06:37.755 INFO:teuthology.orchestra.run.smithi039.stderr:noscrub is set 2024-09-21T22:06:37.780 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-09-21T22:06:37.826 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:06:37.825+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:06:37.882 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:37.882 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:37.898 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T22:06:37.946 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:06:37.945+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:06:38.012 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:06:38.008+0000 7fc8d39f7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25294) UID: 0 2024-09-21T22:06:38.019 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:38.019 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:38.035 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T22:06:38.113 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:06:38.112+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:06:38.210 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:38.210 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:38.214 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:06:38.208+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:06:38.224 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T22:06:38.239 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T22:06:38.316 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:06:38.312+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:06:38.348 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:38.348 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:38.363 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T22:06:38.409 INFO:teuthology.orchestra.run.smithi039.stdout:osd.0: {} 2024-09-21T22:06:38.409 INFO:teuthology.orchestra.run.smithi039.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T22:06:38.416 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:06:38.413+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:06:38.426 INFO:teuthology.orchestra.run.smithi039.stdout:osd.1: {} 2024-09-21T22:06:38.426 INFO:teuthology.orchestra.run.smithi039.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T22:06:38.443 INFO:teuthology.orchestra.run.smithi039.stdout:osd.2: {} 2024-09-21T22:06:38.443 INFO:teuthology.orchestra.run.smithi039.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T22:06:38.464 INFO:teuthology.orchestra.run.smithi039.stdout:osd.3: {} 2024-09-21T22:06:38.464 INFO:teuthology.orchestra.run.smithi039.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T22:06:38.482 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:06:38.482 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:06:38.482 INFO:teuthology.orchestra.run.smithi039.stdout: "complaint_time": 30, 2024-09-21T22:06:38.482 INFO:teuthology.orchestra.run.smithi039.stdout: "num_blocked_ops": 0 2024-09-21T22:06:38.482 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:06:38.485 INFO:teuthology.orchestra.run.smithi039.stdout:osd.4: {} 2024-09-21T22:06:38.485 INFO:teuthology.orchestra.run.smithi039.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T22:06:38.496 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T22:06:38.507 INFO:teuthology.orchestra.run.smithi039.stdout:osd.5: {} 2024-09-21T22:06:38.507 INFO:teuthology.orchestra.run.smithi039.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T22:06:38.517 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:06:38.512+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:06:38.618 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:06:38.613+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:06:38.623 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:06:38.624 INFO:teuthology.orchestra.run.smithi039.stdout: "size": 20, 2024-09-21T22:06:38.624 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 600, 2024-09-21T22:06:38.624 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [ 2024-09-21T22:06:38.624 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:38.624 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4224.0:1 2.3 e17/16 2:c4c92e5a:::rbd_trash:head v 17'1, mlcod=0'0)", 2024-09-21T22:06:38.624 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:05.107810+0000", 2024-09-21T22:06:38.624 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 33.516846207999997, 2024-09-21T22:06:38.624 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0017679200000000001, 2024-09-21T22:06:38.624 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:38.624 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:38.624 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:38.625 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:38.625 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:38.625 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:38.625 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.107810+0000", 2024-09-21T22:06:38.625 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:38.625 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:38.625 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:38.625 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:38.625 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.107810+0000", 2024-09-21T22:06:38.625 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:38.625 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:38.625 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:38.625 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:38.625 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.107811+0000", 2024-09-21T22:06:38.626 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.6700000000000003e-07 2024-09-21T22:06:38.626 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:38.626 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:38.626 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:38.626 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.107822+0000", 2024-09-21T22:06:38.626 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0443e-05 2024-09-21T22:06:38.626 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:38.626 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:38.626 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:38.626 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.107853+0000", 2024-09-21T22:06:38.626 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.1046000000000001e-05 2024-09-21T22:06:38.626 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:38.626 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:38.626 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:38.627 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.107876+0000", 2024-09-21T22:06:38.627 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3547e-05 2024-09-21T22:06:38.627 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:38.627 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:38.627 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:38.627 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.108002+0000", 2024-09-21T22:06:38.627 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000125599 2024-09-21T22:06:38.627 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:38.627 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:38.627 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:38.627 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.108069+0000", 2024-09-21T22:06:38.627 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.7162999999999994e-05 2024-09-21T22:06:38.627 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:38.627 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:38.627 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:38.628 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.109385+0000", 2024-09-21T22:06:38.628 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0013162899999999999 2024-09-21T22:06:38.628 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:38.628 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:38.628 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:38.628 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.109578+0000", 2024-09-21T22:06:38.628 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000193065 2024-09-21T22:06:38.628 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:38.628 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:38.628 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:38.628 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:38.628 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:38.628 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:06:38.638 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T22:06:38.718 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:06:38.712+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:06:38.763 INFO:teuthology.orchestra.run.smithi039.stderr:reweighted osd.2 to 1 (10000) 2024-09-21T22:06:38.776 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:06:38.776 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:06:38.776 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-09-21T22:06:38.776 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:06:38.791 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-09-21T22:06:38.792 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T22:06:38.835 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:06:38.832+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:06:38.920 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:06:38.916+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:06:38.957 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:06:38.957 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:06:38.958 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-21T22:06:38.958 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-21T22:06:38.958 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:06:38.972 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T22:06:39.063 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:06:39.060+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:06:39.122 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:06:39.116+0000 7fc8d39f7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25294) UID: 0 2024-09-21T22:06:39.148 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:06:39.148 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-21T22:06:39.149 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-21T22:06:39.149 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-21T22:06:39.149 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.149 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:06:39.149 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.036355+0000", 2024-09-21T22:06:39.149 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 37.108428332000003, 2024-09-21T22:06:39.149 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.062441934999999997, 2024-09-21T22:06:39.149 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:39.149 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:39.149 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:39.149 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:39.149 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:06:39.149 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:06:39.149 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1 2024-09-21T22:06:39.150 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.150 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:39.150 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.150 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:39.150 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036355+0000", 2024-09-21T22:06:39.150 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:39.150 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.150 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.150 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:39.150 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036355+0000", 2024-09-21T22:06:39.150 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:39.150 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.150 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.150 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:39.151 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036357+0000", 2024-09-21T22:06:39.151 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.446e-06 2024-09-21T22:06:39.151 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.151 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.151 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:39.151 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036373+0000", 2024-09-21T22:06:39.151 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5447e-05 2024-09-21T22:06:39.151 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.151 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.151 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:39.151 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036399+0000", 2024-09-21T22:06:39.151 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6423000000000001e-05 2024-09-21T22:06:39.151 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.151 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.151 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:39.152 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036420+0000", 2024-09-21T22:06:39.152 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1129999999999999e-05 2024-09-21T22:06:39.152 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.152 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.152 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:39.152 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036557+0000", 2024-09-21T22:06:39.152 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013686899999999999 2024-09-21T22:06:39.152 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.152 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.152 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-21T22:06:39.152 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036585+0000", 2024-09-21T22:06:39.152 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8362999999999999e-05 2024-09-21T22:06:39.152 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.152 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.152 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:39.153 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.095068+0000", 2024-09-21T22:06:39.153 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058482972000000001 2024-09-21T22:06:39.153 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.153 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.153 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:39.153 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.095516+0000", 2024-09-21T22:06:39.153 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00044802999999999998 2024-09-21T22:06:39.153 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.153 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.153 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:06:39.153 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.095869+0000", 2024-09-21T22:06:39.153 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000352104 2024-09-21T22:06:39.153 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.153 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.154 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:39.154 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.097055+0000", 2024-09-21T22:06:39.154 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011865599999999999 2024-09-21T22:06:39.154 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.154 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.154 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:39.154 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.098637+0000", 2024-09-21T22:06:39.154 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001581458 2024-09-21T22:06:39.154 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.154 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.154 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:39.154 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.098728+0000", 2024-09-21T22:06:39.154 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1043000000000002e-05 2024-09-21T22:06:39.155 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.155 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.155 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:39.155 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.098797+0000", 2024-09-21T22:06:39.155 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9090000000000004e-05 2024-09-21T22:06:39.155 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:39.155 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:39.155 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:39.155 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.155 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.155 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:06:39.155 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.161242+0000", 2024-09-21T22:06:39.155 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 36.983541420000002, 2024-09-21T22:06:39.155 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014477078000000001, 2024-09-21T22:06:39.155 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:39.156 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:39.156 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:39.156 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:39.156 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:06:39.156 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:06:39.156 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 35 2024-09-21T22:06:39.156 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.156 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:39.156 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.156 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:39.156 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161242+0000", 2024-09-21T22:06:39.156 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:39.156 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.157 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.157 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:39.157 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161242+0000", 2024-09-21T22:06:39.157 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:39.157 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.157 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.157 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:39.157 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161243+0000", 2024-09-21T22:06:39.157 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4729999999999999e-06 2024-09-21T22:06:39.157 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.157 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.157 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:39.157 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161385+0000", 2024-09-21T22:06:39.157 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014226899999999999 2024-09-21T22:06:39.158 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.158 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.158 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:39.158 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161417+0000", 2024-09-21T22:06:39.158 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1406999999999999e-05 2024-09-21T22:06:39.158 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.158 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.158 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:39.158 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161434+0000", 2024-09-21T22:06:39.158 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7623000000000001e-05 2024-09-21T22:06:39.158 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.158 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.158 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:39.158 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.169728+0000", 2024-09-21T22:06:39.159 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0082933090000000004 2024-09-21T22:06:39.159 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.159 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.159 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:39.159 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.169861+0000", 2024-09-21T22:06:39.159 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013367300000000001 2024-09-21T22:06:39.159 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.159 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.159 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:06:39.159 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.170186+0000", 2024-09-21T22:06:39.159 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00032466400000000001 2024-09-21T22:06:39.159 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.159 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.159 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:39.159 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.174759+0000", 2024-09-21T22:06:39.160 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0045727880000000004 2024-09-21T22:06:39.160 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.160 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.160 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:39.160 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.175497+0000", 2024-09-21T22:06:39.160 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00073803400000000004 2024-09-21T22:06:39.160 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.160 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.160 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:39.160 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.175607+0000", 2024-09-21T22:06:39.160 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010987199999999999 2024-09-21T22:06:39.160 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.160 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.160 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:39.160 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.175719+0000", 2024-09-21T22:06:39.161 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011196600000000001 2024-09-21T22:06:39.161 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:39.161 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:39.161 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:39.161 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.161 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.161 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:06:39.161 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.161483+0000", 2024-09-21T22:06:39.161 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 36.983299909000003, 2024-09-21T22:06:39.161 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016155732999999999, 2024-09-21T22:06:39.161 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:39.161 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:39.161 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:39.161 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:39.161 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:06:39.162 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:06:39.162 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 36 2024-09-21T22:06:39.162 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.162 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:39.162 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.162 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:39.162 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161483+0000", 2024-09-21T22:06:39.162 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:39.162 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.162 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.162 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:39.162 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161483+0000", 2024-09-21T22:06:39.162 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:39.162 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.163 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.163 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:39.163 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161485+0000", 2024-09-21T22:06:39.163 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5999999999999999e-06 2024-09-21T22:06:39.163 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.163 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.163 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:39.163 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161498+0000", 2024-09-21T22:06:39.163 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3166999999999999e-05 2024-09-21T22:06:39.163 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.163 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.163 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:39.163 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161526+0000", 2024-09-21T22:06:39.163 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7786000000000001e-05 2024-09-21T22:06:39.164 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.164 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.164 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:39.164 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161545+0000", 2024-09-21T22:06:39.164 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8899999999999999e-05 2024-09-21T22:06:39.164 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.164 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.164 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:39.164 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.172615+0000", 2024-09-21T22:06:39.164 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011070465 2024-09-21T22:06:39.164 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.164 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.164 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:39.164 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.172759+0000", 2024-09-21T22:06:39.164 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014416900000000001 2024-09-21T22:06:39.165 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.165 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.165 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:06:39.165 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.172967+0000", 2024-09-21T22:06:39.165 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020758500000000001 2024-09-21T22:06:39.165 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.165 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.165 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:39.165 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.174775+0000", 2024-09-21T22:06:39.165 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0018080990000000001 2024-09-21T22:06:39.165 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.165 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.165 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:39.165 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.177447+0000", 2024-09-21T22:06:39.166 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026717429999999999 2024-09-21T22:06:39.166 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.166 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.166 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:39.166 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.177525+0000", 2024-09-21T22:06:39.166 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7846e-05 2024-09-21T22:06:39.166 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.166 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.166 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:39.166 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.177639+0000", 2024-09-21T22:06:39.166 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000114373 2024-09-21T22:06:39.166 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:39.166 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:39.166 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:39.167 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.167 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.167 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:06:39.167 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.161577+0000", 2024-09-21T22:06:39.167 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 36.983206375999998, 2024-09-21T22:06:39.167 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016652991999999998, 2024-09-21T22:06:39.167 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:39.167 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:39.167 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:39.167 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:39.167 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:06:39.167 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:06:39.167 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 37 2024-09-21T22:06:39.167 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.167 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:39.168 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.168 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:39.168 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161577+0000", 2024-09-21T22:06:39.168 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:39.168 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.168 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.168 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:39.168 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161577+0000", 2024-09-21T22:06:39.168 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:39.168 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.168 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.168 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:39.169 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161578+0000", 2024-09-21T22:06:39.169 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5269999999999999e-06 2024-09-21T22:06:39.169 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.169 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.169 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:39.169 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161591+0000", 2024-09-21T22:06:39.169 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.296e-05 2024-09-21T22:06:39.169 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.169 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.169 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:39.169 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161629+0000", 2024-09-21T22:06:39.169 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7799000000000002e-05 2024-09-21T22:06:39.169 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.169 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.169 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:39.170 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161644+0000", 2024-09-21T22:06:39.170 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5333e-05 2024-09-21T22:06:39.170 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.170 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.170 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:39.170 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.173636+0000", 2024-09-21T22:06:39.170 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011992078 2024-09-21T22:06:39.170 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.170 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.170 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:39.170 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.173862+0000", 2024-09-21T22:06:39.170 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022556799999999999 2024-09-21T22:06:39.170 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.170 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.171 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:06:39.171 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.174193+0000", 2024-09-21T22:06:39.171 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000331088 2024-09-21T22:06:39.171 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.171 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.171 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:39.171 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.175740+0000", 2024-09-21T22:06:39.171 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015467910000000001 2024-09-21T22:06:39.171 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.171 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.171 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:39.171 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.178073+0000", 2024-09-21T22:06:39.171 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002333356 2024-09-21T22:06:39.171 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.171 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.171 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:39.172 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.178162+0000", 2024-09-21T22:06:39.172 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8306000000000005e-05 2024-09-21T22:06:39.172 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.172 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.172 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:39.172 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.178230+0000", 2024-09-21T22:06:39.172 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8186000000000001e-05 2024-09-21T22:06:39.172 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:39.172 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:39.172 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:39.172 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.172 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.172 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:06:39.172 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.280540+0000", 2024-09-21T22:06:39.172 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 36.864242804, 2024-09-21T22:06:39.173 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013246212, 2024-09-21T22:06:39.173 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:39.173 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:39.173 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:39.173 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:39.173 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:06:39.173 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:06:39.173 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 88 2024-09-21T22:06:39.173 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.173 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:39.173 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.173 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:39.173 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.280540+0000", 2024-09-21T22:06:39.173 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:39.173 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.174 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.174 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:39.174 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.280540+0000", 2024-09-21T22:06:39.174 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:39.174 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.174 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.174 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:39.174 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.280541+0000", 2024-09-21T22:06:39.174 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.54e-07 2024-09-21T22:06:39.174 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.174 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.174 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:39.174 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.280549+0000", 2024-09-21T22:06:39.175 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3329999999999997e-06 2024-09-21T22:06:39.175 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.175 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.175 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:39.175 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.280560+0000", 2024-09-21T22:06:39.175 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1003e-05 2024-09-21T22:06:39.175 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.175 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.175 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:39.175 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.280569+0000", 2024-09-21T22:06:39.175 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2430000000000005e-06 2024-09-21T22:06:39.175 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.175 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.175 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:39.175 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.291095+0000", 2024-09-21T22:06:39.176 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010526259 2024-09-21T22:06:39.176 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.176 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.176 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:39.176 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.291231+0000", 2024-09-21T22:06:39.176 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013561899999999999 2024-09-21T22:06:39.176 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.176 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.176 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:06:39.176 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.291457+0000", 2024-09-21T22:06:39.176 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022686200000000001 2024-09-21T22:06:39.176 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.176 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.176 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:39.176 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.293117+0000", 2024-09-21T22:06:39.177 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001659474 2024-09-21T22:06:39.177 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.177 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.177 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:39.177 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.293524+0000", 2024-09-21T22:06:39.177 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00040701300000000001 2024-09-21T22:06:39.177 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.177 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.177 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:39.177 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.293673+0000", 2024-09-21T22:06:39.177 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000148782 2024-09-21T22:06:39.177 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.177 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.177 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:39.177 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.293786+0000", 2024-09-21T22:06:39.178 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011377 2024-09-21T22:06:39.178 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:39.178 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:39.178 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:39.178 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.178 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.178 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:06:39.178 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.466485+0000", 2024-09-21T22:06:39.178 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 36.678298073000001, 2024-09-21T22:06:39.178 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014443981, 2024-09-21T22:06:39.178 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:39.178 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:39.178 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:39.179 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:39.179 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:06:39.179 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:06:39.179 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 107 2024-09-21T22:06:39.179 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.179 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:39.179 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.179 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:39.179 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.466485+0000", 2024-09-21T22:06:39.179 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:39.179 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.179 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.179 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:39.179 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.466485+0000", 2024-09-21T22:06:39.179 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:39.180 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.180 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.180 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:39.180 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.466486+0000", 2024-09-21T22:06:39.180 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6299999999999993e-07 2024-09-21T22:06:39.180 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.180 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.180 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:39.180 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.466493+0000", 2024-09-21T22:06:39.180 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9639999999999997e-06 2024-09-21T22:06:39.180 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.180 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.180 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:39.180 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.466507+0000", 2024-09-21T22:06:39.180 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4216000000000001e-05 2024-09-21T22:06:39.180 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.181 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.181 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:39.181 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.466515+0000", 2024-09-21T22:06:39.181 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.157e-06 2024-09-21T22:06:39.181 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.181 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.181 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:39.181 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.472033+0000", 2024-09-21T22:06:39.181 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0055175839999999999 2024-09-21T22:06:39.181 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.181 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.181 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:39.181 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.472174+0000", 2024-09-21T22:06:39.181 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000141612 2024-09-21T22:06:39.182 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.182 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.182 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:06:39.182 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.472381+0000", 2024-09-21T22:06:39.182 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000206961 2024-09-21T22:06:39.182 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.182 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.182 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:39.182 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.473924+0000", 2024-09-21T22:06:39.182 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015427119999999999 2024-09-21T22:06:39.182 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.182 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.182 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:39.182 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.480666+0000", 2024-09-21T22:06:39.183 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0067415410000000002 2024-09-21T22:06:39.183 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.183 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.183 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:39.183 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.480811+0000", 2024-09-21T22:06:39.183 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000145422 2024-09-21T22:06:39.183 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.183 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.183 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:39.183 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.480929+0000", 2024-09-21T22:06:39.183 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000117849 2024-09-21T22:06:39.183 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:39.183 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:39.183 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:39.184 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.184 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.184 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:06:39.184 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.467057+0000", 2024-09-21T22:06:39.184 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 36.677725813999999, 2024-09-21T22:06:39.184 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018350195, 2024-09-21T22:06:39.184 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:39.184 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:39.184 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:39.184 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:39.184 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:06:39.184 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:06:39.184 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 108 2024-09-21T22:06:39.184 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.184 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:39.185 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.185 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:39.185 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467057+0000", 2024-09-21T22:06:39.185 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:39.185 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.185 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.185 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:39.185 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467057+0000", 2024-09-21T22:06:39.185 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:39.185 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.185 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.185 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:39.185 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467059+0000", 2024-09-21T22:06:39.185 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.826e-06 2024-09-21T22:06:39.185 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.186 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.186 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:39.186 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467071+0000", 2024-09-21T22:06:39.186 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2224e-05 2024-09-21T22:06:39.186 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.186 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.186 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:39.186 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467098+0000", 2024-09-21T22:06:39.186 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6953000000000001e-05 2024-09-21T22:06:39.186 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.186 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.186 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:39.186 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467113+0000", 2024-09-21T22:06:39.186 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5119999999999999e-05 2024-09-21T22:06:39.187 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.187 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.187 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:39.187 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.474060+0000", 2024-09-21T22:06:39.187 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0069462919999999997 2024-09-21T22:06:39.187 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.187 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.187 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:39.187 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.474283+0000", 2024-09-21T22:06:39.187 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000223315 2024-09-21T22:06:39.187 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.187 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.187 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:06:39.187 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.474660+0000", 2024-09-21T22:06:39.187 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00037729400000000002 2024-09-21T22:06:39.188 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.188 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.188 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:39.188 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.480622+0000", 2024-09-21T22:06:39.188 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0059613369999999997 2024-09-21T22:06:39.188 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.188 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.188 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:39.188 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.485145+0000", 2024-09-21T22:06:39.188 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0045231960000000002 2024-09-21T22:06:39.188 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.188 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.188 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:39.188 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.485288+0000", 2024-09-21T22:06:39.188 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000143232 2024-09-21T22:06:39.189 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.189 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.189 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:39.189 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.485407+0000", 2024-09-21T22:06:39.189 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000119406 2024-09-21T22:06:39.189 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:39.189 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:39.189 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:39.189 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.189 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.189 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:06:39.189 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.467144+0000", 2024-09-21T22:06:39.189 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 36.677638594999998, 2024-09-21T22:06:39.189 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018907146999999999, 2024-09-21T22:06:39.189 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:39.190 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:39.190 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:39.190 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:39.190 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:06:39.190 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:06:39.190 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 109 2024-09-21T22:06:39.190 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.190 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:39.190 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.190 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:39.190 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467144+0000", 2024-09-21T22:06:39.190 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:39.190 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.191 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.191 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:39.191 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467144+0000", 2024-09-21T22:06:39.191 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:39.191 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.191 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.191 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:39.191 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467145+0000", 2024-09-21T22:06:39.191 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9999999999999996e-07 2024-09-21T22:06:39.191 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.191 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.191 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:39.191 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467331+0000", 2024-09-21T22:06:39.191 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018518899999999999 2024-09-21T22:06:39.192 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.192 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.192 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:39.192 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467350+0000", 2024-09-21T22:06:39.192 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8966000000000001e-05 2024-09-21T22:06:39.192 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.192 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.192 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:39.192 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467361+0000", 2024-09-21T22:06:39.192 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1233000000000001e-05 2024-09-21T22:06:39.192 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.192 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.192 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:39.192 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.475613+0000", 2024-09-21T22:06:39.193 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0082518629999999999 2024-09-21T22:06:39.193 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.193 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.193 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:39.193 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.475859+0000", 2024-09-21T22:06:39.193 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024633800000000003 2024-09-21T22:06:39.193 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.193 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.193 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:06:39.193 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.476223+0000", 2024-09-21T22:06:39.193 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00036408700000000003 2024-09-21T22:06:39.193 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.193 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.193 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:39.194 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.481046+0000", 2024-09-21T22:06:39.194 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.004823356 2024-09-21T22:06:39.194 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.194 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.194 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:39.194 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.485722+0000", 2024-09-21T22:06:39.194 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0046753640000000004 2024-09-21T22:06:39.194 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.194 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.194 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:39.194 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.485908+0000", 2024-09-21T22:06:39.194 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000185859 2024-09-21T22:06:39.194 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.195 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.195 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:39.195 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.486052+0000", 2024-09-21T22:06:39.195 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014399200000000001 2024-09-21T22:06:39.195 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:39.195 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:39.195 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:39.195 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.195 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.195 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:06:39.195 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.467384+0000", 2024-09-21T22:06:39.195 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 36.677398767, 2024-09-21T22:06:39.195 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.01894144, 2024-09-21T22:06:39.195 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:39.195 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:39.196 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:39.196 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:39.196 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:06:39.196 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:06:39.196 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 110 2024-09-21T22:06:39.196 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.196 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:39.196 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.196 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:39.196 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467384+0000", 2024-09-21T22:06:39.196 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:39.196 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.196 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.196 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:39.197 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467384+0000", 2024-09-21T22:06:39.197 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:39.197 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.197 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.197 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:39.197 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467385+0000", 2024-09-21T22:06:39.197 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0470000000000001e-06 2024-09-21T22:06:39.197 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.197 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.197 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:39.197 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467393+0000", 2024-09-21T22:06:39.197 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3869999999999997e-06 2024-09-21T22:06:39.197 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.197 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.197 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:39.198 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467413+0000", 2024-09-21T22:06:39.198 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9953e-05 2024-09-21T22:06:39.198 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.198 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.198 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:39.198 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467421+0000", 2024-09-21T22:06:39.198 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2400000000000007e-06 2024-09-21T22:06:39.198 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.198 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.198 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:39.198 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.479542+0000", 2024-09-21T22:06:39.198 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012120832 2024-09-21T22:06:39.198 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.198 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.199 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:39.199 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.479705+0000", 2024-09-21T22:06:39.199 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016362899999999999 2024-09-21T22:06:39.199 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.199 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.199 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:06:39.199 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.479914+0000", 2024-09-21T22:06:39.199 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020814499999999999 2024-09-21T22:06:39.199 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.199 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.199 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:39.199 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.485536+0000", 2024-09-21T22:06:39.199 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0056222800000000003 2024-09-21T22:06:39.199 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.199 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.200 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:39.200 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.486077+0000", 2024-09-21T22:06:39.200 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00054079900000000001 2024-09-21T22:06:39.200 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.200 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.200 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:39.200 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.486213+0000", 2024-09-21T22:06:39.200 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000136682 2024-09-21T22:06:39.200 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.200 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.200 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:39.200 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.486326+0000", 2024-09-21T22:06:39.200 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000112446 2024-09-21T22:06:39.200 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:39.200 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:39.200 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:39.201 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.201 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.201 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:06:39.201 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.467436+0000", 2024-09-21T22:06:39.201 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 36.677347267000002, 2024-09-21T22:06:39.201 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019152168000000001, 2024-09-21T22:06:39.201 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:39.201 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:39.201 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:39.201 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:39.201 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:06:39.201 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:06:39.201 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 111 2024-09-21T22:06:39.201 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.202 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:39.202 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.202 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:39.202 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467436+0000", 2024-09-21T22:06:39.202 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:39.202 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.202 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.202 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:39.202 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467436+0000", 2024-09-21T22:06:39.202 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:39.202 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.202 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.202 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:39.203 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467437+0000", 2024-09-21T22:06:39.203 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0699999999999996e-07 2024-09-21T22:06:39.203 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.203 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.203 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:39.203 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467445+0000", 2024-09-21T22:06:39.203 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.789e-06 2024-09-21T22:06:39.203 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.203 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.203 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:39.203 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467467+0000", 2024-09-21T22:06:39.203 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1253000000000001e-05 2024-09-21T22:06:39.203 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.203 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.203 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:39.204 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467475+0000", 2024-09-21T22:06:39.204 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0039999999999999e-06 2024-09-21T22:06:39.204 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.204 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.204 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:39.204 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.481164+0000", 2024-09-21T22:06:39.204 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013689114 2024-09-21T22:06:39.204 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.204 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.204 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:39.204 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.481394+0000", 2024-09-21T22:06:39.204 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000229931 2024-09-21T22:06:39.204 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.204 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.204 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:06:39.205 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.481779+0000", 2024-09-21T22:06:39.205 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00038545400000000003 2024-09-21T22:06:39.205 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.205 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.205 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:39.205 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.485675+0000", 2024-09-21T22:06:39.205 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0038958370000000001 2024-09-21T22:06:39.205 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.205 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.205 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:39.205 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.486346+0000", 2024-09-21T22:06:39.205 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00067119100000000002 2024-09-21T22:06:39.205 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.205 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.206 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:39.206 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.486478+0000", 2024-09-21T22:06:39.206 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013182600000000001 2024-09-21T22:06:39.206 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.206 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.206 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:39.206 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.486588+0000", 2024-09-21T22:06:39.206 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000110062 2024-09-21T22:06:39.206 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:39.206 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:39.206 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:39.206 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.206 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.206 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:06:39.206 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.467488+0000", 2024-09-21T22:06:39.207 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 36.677294641000003, 2024-09-21T22:06:39.207 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021038304000000001, 2024-09-21T22:06:39.207 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:39.207 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:39.207 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:39.207 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:39.207 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:06:39.207 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:06:39.207 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 112 2024-09-21T22:06:39.207 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.207 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:39.207 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.207 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:39.207 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467488+0000", 2024-09-21T22:06:39.208 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:39.208 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.208 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.208 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:39.208 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467488+0000", 2024-09-21T22:06:39.208 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:39.208 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.208 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.208 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:39.208 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467489+0000", 2024-09-21T22:06:39.208 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7999999999999995e-07 2024-09-21T22:06:39.208 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.208 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.208 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:39.209 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467687+0000", 2024-09-21T22:06:39.209 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019776200000000001 2024-09-21T22:06:39.209 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.209 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.209 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:39.209 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467704+0000", 2024-09-21T22:06:39.209 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7193e-05 2024-09-21T22:06:39.209 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.209 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.209 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:39.209 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467719+0000", 2024-09-21T22:06:39.209 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4703e-05 2024-09-21T22:06:39.209 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.209 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.209 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:39.210 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.482745+0000", 2024-09-21T22:06:39.210 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015025801 2024-09-21T22:06:39.210 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.210 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.210 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:39.210 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.483015+0000", 2024-09-21T22:06:39.210 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026999400000000002 2024-09-21T22:06:39.210 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.210 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.210 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:06:39.210 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.483440+0000", 2024-09-21T22:06:39.210 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000425697 2024-09-21T22:06:39.210 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.211 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.211 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:39.211 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.486701+0000", 2024-09-21T22:06:39.211 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0032611950000000002 2024-09-21T22:06:39.211 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.211 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.211 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:39.211 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.488262+0000", 2024-09-21T22:06:39.211 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015610439999999999 2024-09-21T22:06:39.211 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.211 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.211 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:39.211 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.488406+0000", 2024-09-21T22:06:39.211 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000143232 2024-09-21T22:06:39.211 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.212 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.212 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:39.212 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.488527+0000", 2024-09-21T22:06:39.212 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012100299999999999 2024-09-21T22:06:39.212 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:39.212 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:39.212 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:39.212 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.212 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.212 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:06:39.212 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.467786+0000", 2024-09-21T22:06:39.212 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 36.676997215999997, 2024-09-21T22:06:39.212 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026339741999999999, 2024-09-21T22:06:39.212 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:39.212 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:39.213 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:39.213 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:39.213 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:06:39.213 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:06:39.213 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 113 2024-09-21T22:06:39.213 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.213 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:39.213 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.213 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:39.213 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467786+0000", 2024-09-21T22:06:39.213 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:39.213 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.213 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.213 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:39.213 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467786+0000", 2024-09-21T22:06:39.214 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:39.214 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.214 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.214 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:39.214 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467787+0000", 2024-09-21T22:06:39.214 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.446e-06 2024-09-21T22:06:39.214 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.214 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.214 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:39.214 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467802+0000", 2024-09-21T22:06:39.214 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4769999999999999e-05 2024-09-21T22:06:39.214 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.214 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.214 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:39.215 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467826+0000", 2024-09-21T22:06:39.215 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3430000000000001e-05 2024-09-21T22:06:39.215 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.215 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.215 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:39.215 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467843+0000", 2024-09-21T22:06:39.215 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7065999999999999e-05 2024-09-21T22:06:39.215 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.215 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.215 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:39.215 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.486820+0000", 2024-09-21T22:06:39.215 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018977233999999999 2024-09-21T22:06:39.215 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.215 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.216 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:39.216 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.487065+0000", 2024-09-21T22:06:39.216 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024558799999999998 2024-09-21T22:06:39.216 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.216 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.216 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:06:39.216 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.487442+0000", 2024-09-21T22:06:39.216 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000376707 2024-09-21T22:06:39.218 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.218 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.218 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:39.218 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.492256+0000", 2024-09-21T22:06:39.218 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0048141620000000003 2024-09-21T22:06:39.218 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.218 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.218 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:39.218 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.493865+0000", 2024-09-21T22:06:39.218 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016090150000000001 2024-09-21T22:06:39.218 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.218 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.218 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:39.219 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.494007+0000", 2024-09-21T22:06:39.219 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000141359 2024-09-21T22:06:39.219 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.219 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.219 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:39.219 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.494126+0000", 2024-09-21T22:06:39.219 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000118965 2024-09-21T22:06:39.219 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:39.219 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:39.219 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:39.219 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.219 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.219 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:06:39.219 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.467878+0000", 2024-09-21T22:06:39.220 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 36.676904727, 2024-09-21T22:06:39.220 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.027783958000000001, 2024-09-21T22:06:39.220 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:39.220 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:39.220 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:39.220 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:39.220 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:06:39.220 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:06:39.220 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 114 2024-09-21T22:06:39.220 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.220 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:39.220 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.220 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:39.220 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467878+0000", 2024-09-21T22:06:39.221 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:39.221 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.221 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.221 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:39.221 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467878+0000", 2024-09-21T22:06:39.221 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:39.221 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.221 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.221 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:39.221 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467880+0000", 2024-09-21T22:06:39.222 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.223e-06 2024-09-21T22:06:39.222 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.222 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.222 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:39.222 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467891+0000", 2024-09-21T22:06:39.222 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1527e-05 2024-09-21T22:06:39.222 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.222 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.222 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:39.222 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467913+0000", 2024-09-21T22:06:39.222 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.143e-05 2024-09-21T22:06:39.222 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.222 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.222 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:39.223 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467930+0000", 2024-09-21T22:06:39.223 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7620000000000001e-05 2024-09-21T22:06:39.223 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.223 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.223 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:39.223 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.488690+0000", 2024-09-21T22:06:39.223 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020759588999999998 2024-09-21T22:06:39.223 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.223 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.223 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:39.223 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.488882+0000", 2024-09-21T22:06:39.223 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019236200000000001 2024-09-21T22:06:39.223 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.224 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.224 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:06:39.224 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.489093+0000", 2024-09-21T22:06:39.224 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000211318 2024-09-21T22:06:39.224 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.224 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.224 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:39.224 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.494152+0000", 2024-09-21T22:06:39.224 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0050582769999999999 2024-09-21T22:06:39.224 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.224 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.224 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:39.225 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.495373+0000", 2024-09-21T22:06:39.225 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001221747 2024-09-21T22:06:39.225 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.225 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.225 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:39.225 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.495543+0000", 2024-09-21T22:06:39.225 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000169259 2024-09-21T22:06:39.225 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.225 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.225 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:39.225 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.495662+0000", 2024-09-21T22:06:39.225 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011960600000000001 2024-09-21T22:06:39.226 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:39.226 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:39.226 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:39.226 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.226 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.226 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:06:39.226 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.468118+0000", 2024-09-21T22:06:39.226 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 36.676664985999999, 2024-09-21T22:06:39.226 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.027938877000000001, 2024-09-21T22:06:39.226 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:39.226 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:39.226 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:39.226 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:39.226 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:06:39.227 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:06:39.227 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 115 2024-09-21T22:06:39.227 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.227 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:39.227 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.227 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:39.227 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468118+0000", 2024-09-21T22:06:39.227 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:39.227 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.227 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.227 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:39.227 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468118+0000", 2024-09-21T22:06:39.227 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:39.227 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.228 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.228 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:39.228 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468120+0000", 2024-09-21T22:06:39.228 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7069999999999999e-06 2024-09-21T22:06:39.228 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.228 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.228 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:39.228 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468381+0000", 2024-09-21T22:06:39.228 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026135399999999999 2024-09-21T22:06:39.228 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.228 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.228 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:39.228 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468409+0000", 2024-09-21T22:06:39.228 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.781e-05 2024-09-21T22:06:39.228 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.229 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.229 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:39.229 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468433+0000", 2024-09-21T22:06:39.229 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4343999999999999e-05 2024-09-21T22:06:39.229 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.229 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.229 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:39.229 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.489785+0000", 2024-09-21T22:06:39.229 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021351221 2024-09-21T22:06:39.229 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.229 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.229 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:39.229 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.490018+0000", 2024-09-21T22:06:39.229 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023317800000000001 2024-09-21T22:06:39.229 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.230 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.230 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:06:39.230 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.490412+0000", 2024-09-21T22:06:39.230 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00039417400000000001 2024-09-21T22:06:39.230 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.230 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.230 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:39.230 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.495694+0000", 2024-09-21T22:06:39.230 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.005282335 2024-09-21T22:06:39.230 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.230 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.230 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:39.230 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.495852+0000", 2024-09-21T22:06:39.230 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015777200000000001 2024-09-21T22:06:39.231 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.231 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.231 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:39.231 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.495938+0000", 2024-09-21T22:06:39.231 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6493e-05 2024-09-21T22:06:39.231 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.231 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.231 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:39.231 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.496057+0000", 2024-09-21T22:06:39.231 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000118489 2024-09-21T22:06:39.231 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:39.231 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:39.231 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:39.231 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.231 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.232 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:06:39.232 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.468474+0000", 2024-09-21T22:06:39.232 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 36.676309224999997, 2024-09-21T22:06:39.232 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.028015895999999998, 2024-09-21T22:06:39.232 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:39.232 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:39.232 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:39.232 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:39.232 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:06:39.232 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:06:39.232 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 116 2024-09-21T22:06:39.232 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.232 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:39.232 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.232 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:39.233 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468474+0000", 2024-09-21T22:06:39.233 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:39.233 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.233 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.233 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:39.233 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468474+0000", 2024-09-21T22:06:39.233 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:39.233 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.233 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.233 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:39.233 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468476+0000", 2024-09-21T22:06:39.233 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7060000000000001e-06 2024-09-21T22:06:39.233 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.233 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.234 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:39.234 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468488+0000", 2024-09-21T22:06:39.234 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.234e-05 2024-09-21T22:06:39.234 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.234 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.234 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:39.234 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468517+0000", 2024-09-21T22:06:39.234 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.898e-05 2024-09-21T22:06:39.234 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.234 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.234 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:39.234 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468536+0000", 2024-09-21T22:06:39.234 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8899999999999999e-05 2024-09-21T22:06:39.234 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.234 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.235 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:39.235 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.492376+0000", 2024-09-21T22:06:39.235 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023840659 2024-09-21T22:06:39.235 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.235 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.235 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:39.235 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.492621+0000", 2024-09-21T22:06:39.235 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000244794 2024-09-21T22:06:39.235 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.235 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.235 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:06:39.235 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.493008+0000", 2024-09-21T22:06:39.235 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00038629099999999998 2024-09-21T22:06:39.235 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.235 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.236 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:39.236 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.496192+0000", 2024-09-21T22:06:39.236 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0031847220000000001 2024-09-21T22:06:39.236 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.236 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.236 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:39.236 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.496233+0000", 2024-09-21T22:06:39.236 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0410000000000001e-05 2024-09-21T22:06:39.236 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.236 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.236 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:39.236 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.496374+0000", 2024-09-21T22:06:39.236 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140872 2024-09-21T22:06:39.236 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.237 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.237 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:39.237 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.496490+0000", 2024-09-21T22:06:39.237 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000116222 2024-09-21T22:06:39.237 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:39.237 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:39.237 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:39.237 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.237 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.237 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:06:39.237 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.468568+0000", 2024-09-21T22:06:39.237 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 36.676215106000001, 2024-09-21T22:06:39.237 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.028181578999999998, 2024-09-21T22:06:39.237 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:39.237 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:39.238 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:39.238 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:39.238 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:06:39.238 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:06:39.238 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 117 2024-09-21T22:06:39.238 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.238 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:39.238 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.238 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:39.238 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468568+0000", 2024-09-21T22:06:39.238 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:39.238 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.238 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.238 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:39.239 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468568+0000", 2024-09-21T22:06:39.239 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:39.239 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.239 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.239 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:39.239 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468569+0000", 2024-09-21T22:06:39.239 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4139999999999999e-06 2024-09-21T22:06:39.239 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.239 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.239 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:39.239 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468583+0000", 2024-09-21T22:06:39.239 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3409e-05 2024-09-21T22:06:39.239 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.239 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.239 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:39.240 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468608+0000", 2024-09-21T22:06:39.240 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5559999999999999e-05 2024-09-21T22:06:39.240 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.240 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.240 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:39.240 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468626+0000", 2024-09-21T22:06:39.240 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7863000000000002e-05 2024-09-21T22:06:39.240 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.240 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.240 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:39.240 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.494260+0000", 2024-09-21T22:06:39.240 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025633744999999999 2024-09-21T22:06:39.240 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.240 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.240 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:39.241 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.494519+0000", 2024-09-21T22:06:39.241 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025924800000000001 2024-09-21T22:06:39.241 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.241 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.241 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:06:39.241 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.494747+0000", 2024-09-21T22:06:39.241 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022792799999999999 2024-09-21T22:06:39.241 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.241 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.241 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:39.241 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.496519+0000", 2024-09-21T22:06:39.241 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017720030000000001 2024-09-21T22:06:39.241 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.241 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.242 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:39.242 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.496607+0000", 2024-09-21T22:06:39.242 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7873000000000004e-05 2024-09-21T22:06:39.242 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.242 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.242 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:39.242 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.496691+0000", 2024-09-21T22:06:39.242 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4009000000000002e-05 2024-09-21T22:06:39.242 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.242 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.242 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:39.242 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.496750+0000", 2024-09-21T22:06:39.242 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8526999999999997e-05 2024-09-21T22:06:39.242 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:39.242 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:39.243 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:39.243 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.243 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.243 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:06:39.243 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.898244+0000", 2024-09-21T22:06:39.243 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 36.246538803, 2024-09-21T22:06:39.243 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014749464, 2024-09-21T22:06:39.243 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:39.243 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:39.243 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:39.243 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:39.243 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:06:39.243 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:06:39.243 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 140 2024-09-21T22:06:39.244 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.244 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:39.244 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.244 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:39.244 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898244+0000", 2024-09-21T22:06:39.244 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:39.244 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.244 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.244 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:39.244 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898244+0000", 2024-09-21T22:06:39.244 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:39.244 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.244 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.244 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:39.244 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898245+0000", 2024-09-21T22:06:39.245 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3000000000000002e-07 2024-09-21T22:06:39.245 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.245 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.245 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:39.245 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898308+0000", 2024-09-21T22:06:39.245 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3468999999999998e-05 2024-09-21T22:06:39.245 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.245 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.245 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:39.245 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898315+0000", 2024-09-21T22:06:39.245 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.725e-06 2024-09-21T22:06:39.245 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.245 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.245 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:39.245 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898321+0000", 2024-09-21T22:06:39.246 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2090000000000002e-06 2024-09-21T22:06:39.246 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.246 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.246 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:39.246 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.905452+0000", 2024-09-21T22:06:39.246 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0071305989999999996 2024-09-21T22:06:39.246 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.246 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.246 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:39.246 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.905504+0000", 2024-09-21T22:06:39.246 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2638e-05 2024-09-21T22:06:39.246 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.246 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.246 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:06:39.247 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.905590+0000", 2024-09-21T22:06:39.247 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5333000000000004e-05 2024-09-21T22:06:39.247 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.247 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.247 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:39.247 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.911542+0000", 2024-09-21T22:06:39.247 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0059519819999999998 2024-09-21T22:06:39.247 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.247 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.247 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:39.247 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.912908+0000", 2024-09-21T22:06:39.247 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0013666640000000001 2024-09-21T22:06:39.247 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.247 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.247 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:39.248 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.912957+0000", 2024-09-21T22:06:39.248 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9203000000000003e-05 2024-09-21T22:06:39.248 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.248 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.248 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:39.248 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.912994+0000", 2024-09-21T22:06:39.248 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6312000000000001e-05 2024-09-21T22:06:39.248 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:39.248 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:39.248 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:39.248 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.248 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.248 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:06:39.248 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.898342+0000", 2024-09-21T22:06:39.249 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 36.246440665000002, 2024-09-21T22:06:39.249 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014787918000000001, 2024-09-21T22:06:39.249 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:39.249 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:39.249 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:39.249 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:39.249 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:06:39.249 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:06:39.249 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 141 2024-09-21T22:06:39.249 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.249 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:39.249 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.249 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:39.249 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898342+0000", 2024-09-21T22:06:39.249 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:39.250 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.250 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.250 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:39.250 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898342+0000", 2024-09-21T22:06:39.250 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:39.250 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.250 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.250 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:39.250 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898343+0000", 2024-09-21T22:06:39.250 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3000000000000001e-07 2024-09-21T22:06:39.250 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.250 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.250 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:39.250 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898346+0000", 2024-09-21T22:06:39.251 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6009999999999999e-06 2024-09-21T22:06:39.251 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.251 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.251 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:39.251 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898352+0000", 2024-09-21T22:06:39.251 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8089999999999998e-06 2024-09-21T22:06:39.251 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.251 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.251 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:39.251 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898358+0000", 2024-09-21T22:06:39.251 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4349999999999996e-06 2024-09-21T22:06:39.251 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.251 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.251 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:39.251 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.909387+0000", 2024-09-21T22:06:39.252 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011029105000000001 2024-09-21T22:06:39.252 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.252 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.252 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:39.252 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.909462+0000", 2024-09-21T22:06:39.252 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5192999999999996e-05 2024-09-21T22:06:39.252 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.252 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.252 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:06:39.252 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.909551+0000", 2024-09-21T22:06:39.252 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9046000000000001e-05 2024-09-21T22:06:39.252 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.252 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.252 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:39.253 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.911852+0000", 2024-09-21T22:06:39.253 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0023005579999999999 2024-09-21T22:06:39.253 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.253 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.253 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:39.253 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.913002+0000", 2024-09-21T22:06:39.253 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011502929999999999 2024-09-21T22:06:39.253 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.253 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.253 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:39.253 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.913060+0000", 2024-09-21T22:06:39.253 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8427000000000002e-05 2024-09-21T22:06:39.253 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.253 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.253 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:39.254 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.913130+0000", 2024-09-21T22:06:39.254 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0020999999999999e-05 2024-09-21T22:06:39.254 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:39.254 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:39.254 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:39.254 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.254 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.254 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:158 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T22:06:39.254 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:03.511201+0000", 2024-09-21T22:06:39.254 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 35.633582181999998, 2024-09-21T22:06:39.254 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013353538999999999, 2024-09-21T22:06:39.254 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:39.254 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:39.254 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:39.255 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:39.255 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:06:39.255 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:06:39.255 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 158 2024-09-21T22:06:39.255 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.255 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:39.255 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.255 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:39.255 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511201+0000", 2024-09-21T22:06:39.255 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:39.255 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.255 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.255 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:39.255 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511201+0000", 2024-09-21T22:06:39.255 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:39.256 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.256 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.256 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:39.256 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511202+0000", 2024-09-21T22:06:39.256 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9299999999999997e-07 2024-09-21T22:06:39.256 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.256 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.256 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:39.256 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511315+0000", 2024-09-21T22:06:39.256 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000113909 2024-09-21T22:06:39.256 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.256 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.256 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:39.256 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511333+0000", 2024-09-21T22:06:39.256 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7062999999999999e-05 2024-09-21T22:06:39.257 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.257 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.257 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:39.257 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511344+0000", 2024-09-21T22:06:39.257 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1177e-05 2024-09-21T22:06:39.257 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.257 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.257 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:39.257 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.517415+0000", 2024-09-21T22:06:39.257 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0060709830000000003 2024-09-21T22:06:39.257 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.257 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.257 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:39.257 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.517539+0000", 2024-09-21T22:06:39.257 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000124432 2024-09-21T22:06:39.258 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.258 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.258 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:06:39.258 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.517732+0000", 2024-09-21T22:06:39.258 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019262200000000001 2024-09-21T22:06:39.258 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.258 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.258 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:39.258 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.520709+0000", 2024-09-21T22:06:39.258 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0029767470000000001 2024-09-21T22:06:39.258 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.258 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.258 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:39.258 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.524389+0000", 2024-09-21T22:06:39.259 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003680975 2024-09-21T22:06:39.259 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.259 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.259 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:39.259 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.524486+0000", 2024-09-21T22:06:39.259 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6575000000000003e-05 2024-09-21T22:06:39.259 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.259 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.259 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:39.259 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.524554+0000", 2024-09-21T22:06:39.259 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8362999999999996e-05 2024-09-21T22:06:39.259 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:39.259 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:39.259 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:39.259 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.260 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.260 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T22:06:39.260 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:03.511368+0000", 2024-09-21T22:06:39.260 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 35.633415280000001, 2024-09-21T22:06:39.260 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013312133, 2024-09-21T22:06:39.260 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:39.260 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:39.260 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:39.260 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:39.260 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:06:39.260 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:06:39.260 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 159 2024-09-21T22:06:39.260 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.260 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:39.260 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.261 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:39.261 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511368+0000", 2024-09-21T22:06:39.261 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:39.261 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.261 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.261 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:39.261 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511368+0000", 2024-09-21T22:06:39.261 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:39.261 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.261 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.261 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:39.261 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511369+0000", 2024-09-21T22:06:39.261 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9599999999999998e-07 2024-09-21T22:06:39.261 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.261 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.262 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:39.262 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511382+0000", 2024-09-21T22:06:39.262 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3487e-05 2024-09-21T22:06:39.262 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.262 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.262 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:39.262 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511394+0000", 2024-09-21T22:06:39.262 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1337e-05 2024-09-21T22:06:39.262 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.262 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.262 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:39.262 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511402+0000", 2024-09-21T22:06:39.262 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0059999999999992e-06 2024-09-21T22:06:39.262 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.263 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.263 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:39.263 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.519376+0000", 2024-09-21T22:06:39.263 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0079746520000000005 2024-09-21T22:06:39.263 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.263 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.263 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:39.263 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.519498+0000", 2024-09-21T22:06:39.263 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012212900000000001 2024-09-21T22:06:39.263 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.263 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.263 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:06:39.263 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.519728+0000", 2024-09-21T22:06:39.263 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022985799999999999 2024-09-21T22:06:39.263 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.264 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.264 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:39.264 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.521344+0000", 2024-09-21T22:06:39.264 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016163169999999999 2024-09-21T22:06:39.264 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.264 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.264 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:39.264 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.524568+0000", 2024-09-21T22:06:39.264 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0032234220000000001 2024-09-21T22:06:39.264 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.264 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.264 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:39.264 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.524633+0000", 2024-09-21T22:06:39.264 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5126e-05 2024-09-21T22:06:39.264 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:39.265 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:39.265 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:39.265 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.524680+0000", 2024-09-21T22:06:39.265 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6903000000000001e-05 2024-09-21T22:06:39.265 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:39.265 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:39.265 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:39.265 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:39.265 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:39.265 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:06:39.266 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T22:06:39.267 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:06:39.265+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:06:39.367 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:06:39.365+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:06:39.443 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:06:39.443 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:06:39.443 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-09-21T22:06:39.443 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:06:39.457 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T22:06:39.503 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:06:39.500+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:06:39.569 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:06:39.564+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:06:39.584 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:06:39.585 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:06:39.585 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-21T22:06:39.585 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-21T22:06:39.585 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:06:39.599 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T22:06:39.670 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:06:39.669+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:06:39.771 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:06:39.771 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-21T22:06:39.771 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-21T22:06:39.771 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [] 2024-09-21T22:06:39.771 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:06:39.771 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:06:39.769+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:06:39.782 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-21T22:06:39.786 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T22:06:39.787 INFO:teuthology.orchestra.run.smithi039.stderr:reweighted osd.3 to 1 (10000) 2024-09-21T22:06:39.813 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-09-21T22:06:39.898 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:06:39.897+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:06:39.915 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:06:39.915 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:06:39.915 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-09-21T22:06:39.915 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:06:39.930 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T22:06:39.972 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:06:39.969+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:06:40.056 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:06:40.057 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:06:40.057 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-21T22:06:40.057 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-21T22:06:40.057 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:06:40.071 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T22:06:40.072 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:06:40.069+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:06:40.173 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:06:40.169+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:06:40.191 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:06:40.191 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-21T22:06:40.191 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-21T22:06:40.192 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-21T22:06:40.192 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:40.192 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4224.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-21T22:06:40.192 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:05.105895+0000", 2024-09-21T22:06:40.192 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 35.084969929000003, 2024-09-21T22:06:40.192 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0034226500000000002, 2024-09-21T22:06:40.192 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:40.192 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:40.192 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:40.192 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:40.192 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4224", 2024-09-21T22:06:40.192 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.39:0/795178185", 2024-09-21T22:06:40.193 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1 2024-09-21T22:06:40.193 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:40.193 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:40.193 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:40.193 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:40.193 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.105895+0000", 2024-09-21T22:06:40.193 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:40.193 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:40.193 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:40.193 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:40.193 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.105895+0000", 2024-09-21T22:06:40.193 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:40.193 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:40.194 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:40.194 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:40.194 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.105905+0000", 2024-09-21T22:06:40.194 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8069999999999994e-06 2024-09-21T22:06:40.194 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:40.194 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:40.194 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:40.194 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.105919+0000", 2024-09-21T22:06:40.194 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3976e-05 2024-09-21T22:06:40.194 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:40.194 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:40.194 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:40.194 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.105955+0000", 2024-09-21T22:06:40.194 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6776999999999997e-05 2024-09-21T22:06:40.195 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:40.195 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:40.195 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:40.195 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.105983+0000", 2024-09-21T22:06:40.195 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7535999999999999e-05 2024-09-21T22:06:40.195 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:40.195 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:40.195 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:40.195 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.106162+0000", 2024-09-21T22:06:40.195 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000178839 2024-09-21T22:06:40.195 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:40.195 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:40.195 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:40.195 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.106437+0000", 2024-09-21T22:06:40.195 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027542099999999998 2024-09-21T22:06:40.196 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:40.196 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:40.196 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-09-21T22:06:40.196 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.106726+0000", 2024-09-21T22:06:40.196 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00028856800000000001 2024-09-21T22:06:40.196 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:40.196 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:40.196 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:40.196 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.108194+0000", 2024-09-21T22:06:40.196 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001468095 2024-09-21T22:06:40.196 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:40.196 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:40.196 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:40.196 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.109130+0000", 2024-09-21T22:06:40.196 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00093653599999999997 2024-09-21T22:06:40.197 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:40.197 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:40.197 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:40.197 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.109216+0000", 2024-09-21T22:06:40.197 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5191999999999993e-05 2024-09-21T22:06:40.197 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:40.197 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:40.197 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:40.197 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.109317+0000", 2024-09-21T22:06:40.197 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000101903 2024-09-21T22:06:40.197 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:40.197 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:40.197 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:40.197 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:40.198 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:40.198 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4224.0:1 2.3 e17/16 ondisk, result = 0)", 2024-09-21T22:06:40.198 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:05.108920+0000", 2024-09-21T22:06:40.198 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 35.081944421999999, 2024-09-21T22:06:40.198 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00041366599999999998, 2024-09-21T22:06:40.198 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:40.198 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:40.198 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:06:40.198 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:40.198 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:40.198 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:40.198 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.108920+0000", 2024-09-21T22:06:40.198 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:40.199 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:40.199 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:40.199 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:40.199 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.108920+0000", 2024-09-21T22:06:40.199 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:40.199 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:40.199 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:40.199 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:40.199 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.108922+0000", 2024-09-21T22:06:40.199 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.703e-06 2024-09-21T22:06:40.199 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:40.199 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:40.199 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:40.199 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.108938+0000", 2024-09-21T22:06:40.200 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6350000000000001e-05 2024-09-21T22:06:40.200 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:40.200 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:40.200 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:40.200 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.108974+0000", 2024-09-21T22:06:40.200 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5612999999999999e-05 2024-09-21T22:06:40.200 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:40.200 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:40.200 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:40.200 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.108989+0000", 2024-09-21T22:06:40.200 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4576000000000001e-05 2024-09-21T22:06:40.200 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:40.200 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:40.200 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:40.201 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.109108+0000", 2024-09-21T22:06:40.201 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011901899999999999 2024-09-21T22:06:40.201 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:40.201 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:40.201 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:40.201 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.109120+0000", 2024-09-21T22:06:40.201 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.293e-05 2024-09-21T22:06:40.201 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:40.201 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:40.201 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:40.201 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.109334+0000", 2024-09-21T22:06:40.201 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000213475 2024-09-21T22:06:40.201 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:40.201 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:40.201 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:40.202 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:40.202 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:40.202 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4224.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e17)", 2024-09-21T22:06:40.202 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:05.115572+0000", 2024-09-21T22:06:40.202 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 35.075292513999997, 2024-09-21T22:06:40.202 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00041313, 2024-09-21T22:06:40.202 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:40.202 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:40.202 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-21T22:06:40.202 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:40.202 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4224", 2024-09-21T22:06:40.202 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.39:0/795178185", 2024-09-21T22:06:40.202 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2 2024-09-21T22:06:40.202 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:40.203 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:40.203 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:40.203 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:40.203 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.115572+0000", 2024-09-21T22:06:40.203 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:40.203 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:40.203 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:40.203 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:40.203 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.115572+0000", 2024-09-21T22:06:40.203 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:40.203 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:40.203 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:40.203 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:40.203 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.115574+0000", 2024-09-21T22:06:40.203 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0530000000000001e-06 2024-09-21T22:06:40.204 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:40.204 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:40.204 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:40.204 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.115585+0000", 2024-09-21T22:06:40.204 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0947e-05 2024-09-21T22:06:40.204 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:40.204 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:40.204 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:40.204 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.115615+0000", 2024-09-21T22:06:40.204 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9408999999999999e-05 2024-09-21T22:06:40.204 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:40.204 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:40.204 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:40.204 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.115637+0000", 2024-09-21T22:06:40.204 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2073e-05 2024-09-21T22:06:40.204 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:40.205 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:40.205 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:40.205 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.115768+0000", 2024-09-21T22:06:40.205 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013153899999999999 2024-09-21T22:06:40.205 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:40.205 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:40.205 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:40.205 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.115985+0000", 2024-09-21T22:06:40.205 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000217109 2024-09-21T22:06:40.205 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:40.205 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:40.205 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:40.205 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:40.205 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:40.206 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4224.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e17)", 2024-09-21T22:06:40.206 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:05.116456+0000", 2024-09-21T22:06:40.206 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 35.074408968, 2024-09-21T22:06:40.206 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00030938500000000002, 2024-09-21T22:06:40.206 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:40.206 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:40.206 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-21T22:06:40.206 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:40.206 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4224", 2024-09-21T22:06:40.206 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.39:0/795178185", 2024-09-21T22:06:40.206 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3 2024-09-21T22:06:40.206 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:40.206 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:40.207 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:40.207 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:40.207 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.116456+0000", 2024-09-21T22:06:40.207 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:40.207 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:40.207 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:40.207 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:40.207 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.116456+0000", 2024-09-21T22:06:40.207 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:40.207 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:40.207 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:40.207 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:40.207 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.116458+0000", 2024-09-21T22:06:40.207 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.77e-06 2024-09-21T22:06:40.208 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:40.208 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:40.208 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:40.208 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.116466+0000", 2024-09-21T22:06:40.208 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1799999999999996e-06 2024-09-21T22:06:40.208 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:40.208 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:40.208 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:40.208 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.116492+0000", 2024-09-21T22:06:40.208 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6523999999999999e-05 2024-09-21T22:06:40.208 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:40.208 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:40.208 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:40.208 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.116513+0000", 2024-09-21T22:06:40.208 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0400000000000001e-05 2024-09-21T22:06:40.209 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:40.209 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:40.209 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:40.209 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.116616+0000", 2024-09-21T22:06:40.209 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000103862 2024-09-21T22:06:40.209 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:40.209 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:40.209 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:40.209 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.116765+0000", 2024-09-21T22:06:40.209 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014864899999999999 2024-09-21T22:06:40.209 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:40.209 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:40.209 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:40.209 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:40.209 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:40.210 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4224.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-21T22:06:40.210 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:06.109464+0000", 2024-09-21T22:06:40.210 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 34.081400289999998, 2024-09-21T22:06:40.210 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0070431790000000001, 2024-09-21T22:06:40.210 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:40.210 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:40.210 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:40.210 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:40.210 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4224", 2024-09-21T22:06:40.210 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.39:0/795178185", 2024-09-21T22:06:40.210 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 5 2024-09-21T22:06:40.210 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:40.210 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:40.210 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:40.211 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:40.211 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.109464+0000", 2024-09-21T22:06:40.211 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:40.211 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:40.211 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:40.211 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:40.211 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.109464+0000", 2024-09-21T22:06:40.211 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:40.211 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:40.211 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:40.211 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:40.211 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.109467+0000", 2024-09-21T22:06:40.211 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.066e-06 2024-09-21T22:06:40.211 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:40.211 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:40.212 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:40.212 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.109485+0000", 2024-09-21T22:06:40.212 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7464000000000001e-05 2024-09-21T22:06:40.212 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:40.212 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:40.212 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:40.212 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.109518+0000", 2024-09-21T22:06:40.212 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3343000000000002e-05 2024-09-21T22:06:40.212 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:40.212 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:40.212 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:40.212 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.109537+0000", 2024-09-21T22:06:40.212 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8743e-05 2024-09-21T22:06:40.212 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:40.212 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:40.213 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:40.213 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.109708+0000", 2024-09-21T22:06:40.213 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017073899999999999 2024-09-21T22:06:40.213 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:40.213 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:40.213 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op must wait for map", 2024-09-21T22:06:40.213 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.109725+0000", 2024-09-21T22:06:40.213 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7013000000000001e-05 2024-09-21T22:06:40.213 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:40.213 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:40.213 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:40.213 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.113449+0000", 2024-09-21T22:06:40.213 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0037244579999999999 2024-09-21T22:06:40.214 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:40.214 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:40.214 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:40.214 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.113659+0000", 2024-09-21T22:06:40.214 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020968199999999999 2024-09-21T22:06:40.214 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:40.214 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:40.214 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-09-21T22:06:40.214 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.113872+0000", 2024-09-21T22:06:40.214 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000213062 2024-09-21T22:06:40.214 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:40.214 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:40.214 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:40.214 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.115375+0000", 2024-09-21T22:06:40.215 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015028979999999999 2024-09-21T22:06:40.215 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:40.215 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:40.215 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:40.215 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.116356+0000", 2024-09-21T22:06:40.215 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00098098200000000003 2024-09-21T22:06:40.215 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:40.215 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:40.215 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:40.215 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.116442+0000", 2024-09-21T22:06:40.215 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6533000000000007e-05 2024-09-21T22:06:40.215 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:40.215 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:40.215 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:40.216 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.116508+0000", 2024-09-21T22:06:40.216 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5196000000000004e-05 2024-09-21T22:06:40.216 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:40.216 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:40.216 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:40.216 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:40.216 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:40.216 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4224.0:5 2.2 e17/16 ondisk, result = 0)", 2024-09-21T22:06:40.216 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:06.116133+0000", 2024-09-21T22:06:40.216 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 34.074731804999999, 2024-09-21T22:06:40.216 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00038649700000000002, 2024-09-21T22:06:40.216 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:40.216 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:40.216 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:06:40.216 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:40.217 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:40.217 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:40.217 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.116133+0000", 2024-09-21T22:06:40.217 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:40.217 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:40.217 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:40.217 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:40.217 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.116133+0000", 2024-09-21T22:06:40.217 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:40.217 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:40.217 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:40.217 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:40.217 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.116135+0000", 2024-09-21T22:06:40.217 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8870000000000001e-06 2024-09-21T22:06:40.218 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:40.218 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:40.218 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:40.218 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.116146+0000", 2024-09-21T22:06:40.218 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0953e-05 2024-09-21T22:06:40.218 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:40.218 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:40.218 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:40.218 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.116175+0000", 2024-09-21T22:06:40.218 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9383e-05 2024-09-21T22:06:40.218 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:40.218 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:40.218 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:40.218 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.116190+0000", 2024-09-21T22:06:40.219 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4613000000000001e-05 2024-09-21T22:06:40.219 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:40.219 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:40.219 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:40.219 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.116328+0000", 2024-09-21T22:06:40.219 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013848300000000001 2024-09-21T22:06:40.219 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:40.219 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:40.219 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:40.219 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.116345+0000", 2024-09-21T22:06:40.219 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6883000000000001e-05 2024-09-21T22:06:40.219 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:40.219 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:40.220 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:40.220 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.116519+0000", 2024-09-21T22:06:40.220 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017429500000000001 2024-09-21T22:06:40.220 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:40.220 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:40.220 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:40.220 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:40.220 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:40.220 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4224.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-21T22:06:40.220 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:07.124322+0000", 2024-09-21T22:06:40.220 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 33.066542499999997, 2024-09-21T22:06:40.220 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0067709010000000002, 2024-09-21T22:06:40.221 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:40.221 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:40.221 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:40.221 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:40.221 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4224", 2024-09-21T22:06:40.221 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.39:0/795178185", 2024-09-21T22:06:40.221 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 7 2024-09-21T22:06:40.221 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:40.221 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:40.221 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:40.221 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:40.221 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.124322+0000", 2024-09-21T22:06:40.221 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:40.221 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:40.222 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:40.222 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:40.222 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.124322+0000", 2024-09-21T22:06:40.222 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:40.222 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:40.222 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:40.222 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:40.222 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.124325+0000", 2024-09-21T22:06:40.222 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1489999999999998e-06 2024-09-21T22:06:40.222 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:40.222 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:40.222 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:40.222 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.124342+0000", 2024-09-21T22:06:40.223 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.666e-05 2024-09-21T22:06:40.223 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:40.223 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:40.223 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:40.223 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.124379+0000", 2024-09-21T22:06:40.223 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6539999999999999e-05 2024-09-21T22:06:40.223 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:40.223 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:40.223 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:40.223 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.124406+0000", 2024-09-21T22:06:40.223 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7886e-05 2024-09-21T22:06:40.223 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:40.223 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:40.223 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:40.224 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.124535+0000", 2024-09-21T22:06:40.224 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000128749 2024-09-21T22:06:40.224 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:40.224 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:40.224 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op must wait for map", 2024-09-21T22:06:40.224 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.124547+0000", 2024-09-21T22:06:40.224 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.136e-05 2024-09-21T22:06:40.224 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:40.224 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:40.224 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:40.224 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.127774+0000", 2024-09-21T22:06:40.224 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0032275519999999999 2024-09-21T22:06:40.224 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:40.224 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:40.225 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:40.225 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.128014+0000", 2024-09-21T22:06:40.225 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023956800000000001 2024-09-21T22:06:40.225 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:40.225 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:40.225 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-09-21T22:06:40.225 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.128462+0000", 2024-09-21T22:06:40.225 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00044797700000000001 2024-09-21T22:06:40.225 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:40.225 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:40.225 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:40.225 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.130185+0000", 2024-09-21T22:06:40.225 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00172313 2024-09-21T22:06:40.226 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:40.226 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:40.226 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:40.226 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.130937+0000", 2024-09-21T22:06:40.226 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00075244799999999996 2024-09-21T22:06:40.226 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:40.226 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:40.226 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:40.226 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.131025+0000", 2024-09-21T22:06:40.226 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7998999999999996e-05 2024-09-21T22:06:40.226 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:40.226 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:40.226 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:40.227 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.131093+0000", 2024-09-21T22:06:40.227 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7883e-05 2024-09-21T22:06:40.227 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:40.227 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:40.227 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:40.227 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:40.227 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:40.227 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4224.0:7 2.2 e18/16 ondisk, result = 0)", 2024-09-21T22:06:40.227 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:07.130777+0000", 2024-09-21T22:06:40.227 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 33.060087453999998, 2024-09-21T22:06:40.227 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00032786799999999999, 2024-09-21T22:06:40.227 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:40.227 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:40.227 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:06:40.228 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:40.228 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:40.228 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:40.228 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.130777+0000", 2024-09-21T22:06:40.228 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:40.228 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:40.228 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:40.228 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:40.228 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.130777+0000", 2024-09-21T22:06:40.228 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:40.228 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:40.228 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:40.228 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:40.229 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.130779+0000", 2024-09-21T22:06:40.229 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7e-06 2024-09-21T22:06:40.229 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:40.229 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:40.229 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:40.229 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.130788+0000", 2024-09-21T22:06:40.229 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6670000000000002e-06 2024-09-21T22:06:40.229 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:40.229 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:40.229 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:40.229 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.130814+0000", 2024-09-21T22:06:40.229 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6590000000000001e-05 2024-09-21T22:06:40.229 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:40.229 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:40.230 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:40.230 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.130828+0000", 2024-09-21T22:06:40.230 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.415e-05 2024-09-21T22:06:40.230 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:40.230 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:40.230 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:40.230 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.130915+0000", 2024-09-21T22:06:40.230 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6759000000000001e-05 2024-09-21T22:06:40.230 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:40.230 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:40.230 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:40.230 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.130928+0000", 2024-09-21T22:06:40.230 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.328e-05 2024-09-21T22:06:40.230 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:40.230 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:40.231 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:40.231 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.131105+0000", 2024-09-21T22:06:40.231 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017672200000000001 2024-09-21T22:06:40.231 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:40.231 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:40.231 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:40.231 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:40.231 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:40.231 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:06:40.232 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T22:06:40.278 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:06:40.277+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:06:40.360 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:06:40.360 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:06:40.360 INFO:teuthology.orchestra.run.smithi039.stdout: "num_ops": 0 2024-09-21T22:06:40.360 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:06:40.374 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T22:06:40.375 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:06:40.372+0000 7fc8d39f7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25294) UID: 0 2024-09-21T22:06:40.475 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:06:40.472+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:06:40.525 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T22:06:40.546 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:06:40.546 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:06:40.547 INFO:teuthology.orchestra.run.smithi039.stdout: "complaint_time": 30, 2024-09-21T22:06:40.547 INFO:teuthology.orchestra.run.smithi039.stdout: "num_blocked_ops": 0 2024-09-21T22:06:40.547 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:06:40.562 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T22:06:40.610 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:06:40.609+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:06:40.667 INFO:teuthology.orchestra.run.smithi039.stdout:osd.0: {} 2024-09-21T22:06:40.667 INFO:teuthology.orchestra.run.smithi039.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T22:06:40.676 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:06:40.672+0000 7fc8d39f7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25294) UID: 0 2024-09-21T22:06:40.684 INFO:teuthology.orchestra.run.smithi039.stdout:osd.1: {} 2024-09-21T22:06:40.685 INFO:teuthology.orchestra.run.smithi039.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T22:06:40.702 INFO:teuthology.orchestra.run.smithi039.stdout:osd.2: {} 2024-09-21T22:06:40.702 INFO:teuthology.orchestra.run.smithi039.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T22:06:40.723 INFO:teuthology.orchestra.run.smithi039.stdout:osd.3: {} 2024-09-21T22:06:40.723 INFO:teuthology.orchestra.run.smithi039.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T22:06:40.729 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:40.729 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:40.743 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T22:06:40.744 INFO:teuthology.orchestra.run.smithi039.stdout:osd.4: {} 2024-09-21T22:06:40.744 INFO:teuthology.orchestra.run.smithi039.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T22:06:40.766 INFO:teuthology.orchestra.run.smithi039.stdout:osd.5: {} 2024-09-21T22:06:40.766 INFO:teuthology.orchestra.run.smithi039.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T22:06:40.778 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:06:40.772+0000 7fc8d39f7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25294) UID: 0 2024-09-21T22:06:40.811 INFO:teuthology.orchestra.run.smithi039.stderr:nodeep-scrub is set 2024-09-21T22:06:40.811 INFO:teuthology.orchestra.run.smithi039.stderr:reweighted osd.4 to 1 (10000) 2024-09-21T22:06:40.843 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-09-21T22:06:40.875 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:40.875 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:40.890 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T22:06:40.890 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:06:40.885+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:06:40.982 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:06:40.981+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:06:41.065 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:41.065 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:41.081 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T22:06:41.081 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:06:41.077+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:06:41.182 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:06:41.176+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:06:41.254 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:41.254 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:41.268 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T22:06:41.314 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:06:41.313+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:06:41.384 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:06:41.380+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:06:41.391 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:41.392 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:41.406 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T22:06:41.485 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:06:41.484+0000 7fc8d39f7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25294) UID: 0 2024-09-21T22:06:41.570 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:41.571 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:41.584 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T22:06:41.586 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:06:41.585+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:06:41.687 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:06:41.685+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:06:41.711 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:41.711 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:41.725 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T22:06:41.788 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:06:41.785+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:06:41.815 INFO:teuthology.orchestra.run.smithi039.stderr:reweighted osd.5 to 1 (10000) 2024-09-21T22:06:41.841 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-09-21T22:06:41.841 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T22:06:41.863 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:41.863 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:41.877 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T22:06:41.927 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:06:41.925+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:06:41.989 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:06:41.984+0000 7fc8d39f7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25294) UID: 0 2024-09-21T22:06:42.002 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:42.002 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:42.017 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T22:06:42.090 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:06:42.089+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:06:42.184 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T22:06:42.185 INFO:teuthology.orchestra.run.smithi039.stderr:dumped all 2024-09-21T22:06:42.187 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:42.187 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:42.190 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:06:42.189+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:06:42.202 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T22:06:42.203 INFO:teuthology.orchestra.run.smithi039.stdout:{"pg_ready":true,"pg_map":{"version":52,"stamp":"2024-09-21T22:06:41.816456+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":471859200,"kb_used":145288,"kb_used_data":1824,"kb_used_omap":7,"kb_used_meta":143352,"kb_avail":471713912,"statfs":{"total":483183820800,"available":483035045888,"internally_reserved":0,"allocated":1867776,"data_stored":1803704,"data_compressed":13990,"data_compressed_allocated":73728,"data_compressed_original":1163264,"omap_allocated":7973,"internal_metadata":146792667},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.060119"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":48,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-21T22:06:39.781540+0000","last_change":"2024-09-21T22:06:07.122328+0000","last_active":"2024-09-21T22:06:39.781540+0000","last_peered":"2024-09-21T22:06:39.781540+0000","last_clean":"2024-09-21T22:06:39.781540+0000","last_became_active":"2024-09-21T22:06:05.220996+0000","last_became_peered":"2024-09-21T22:06:05.220996+0000","last_unstale":"2024-09-21T22:06:39.781540+0000","last_undegraded":"2024-09-21T22:06:39.781540+0000","last_fullsized":"2024-09-21T22:06:39.781540+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T23:45:49.178013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001473755,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":49,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-21T22:06:39.818712+0000","last_change":"2024-09-21T22:06:07.133757+0000","last_active":"2024-09-21T22:06:39.818712+0000","last_peered":"2024-09-21T22:06:39.818712+0000","last_clean":"2024-09-21T22:06:39.818712+0000","last_became_active":"2024-09-21T22:06:05.109065+0000","last_became_peered":"2024-09-21T22:06:05.109065+0000","last_unstale":"2024-09-21T22:06:39.818712+0000","last_undegraded":"2024-09-21T22:06:39.818712+0000","last_fullsized":"2024-09-21T22:06:39.818712+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:31:12.654893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00088865700000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":53,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-21T22:06:39.781475+0000","last_change":"2024-09-21T22:06:21.313951+0000","last_active":"2024-09-21T22:06:39.781475+0000","last_peered":"2024-09-21T22:06:39.781475+0000","last_clean":"2024-09-21T22:06:39.781475+0000","last_became_active":"2024-09-21T22:06:21.313514+0000","last_became_peered":"2024-09-21T22:06:21.313514+0000","last_unstale":"2024-09-21T22:06:39.781475+0000","last_undegraded":"2024-09-21T22:06:39.781475+0000","last_fullsized":"2024-09-21T22:06:39.781475+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:07:37.870228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091949900000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":54,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-21T22:06:39.818700+0000","last_change":"2024-09-21T22:06:21.314973+0000","last_active":"2024-09-21T22:06:39.818700+0000","last_peered":"2024-09-21T22:06:39.818700+0000","last_clean":"2024-09-21T22:06:39.818700+0000","last_became_active":"2024-09-21T22:06:21.313946+0000","last_became_peered":"2024-09-21T22:06:21.313946+0000","last_unstale":"2024-09-21T22:06:39.818700+0000","last_undegraded":"2024-09-21T22:06:39.818700+0000","last_fullsized":"2024-09-21T22:06:39.818700+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:14:03.108117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00090834100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"19'2","reported_seq":51,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-21T22:06:39.818270+0000","last_change":"2024-09-21T22:06:07.130034+0000","last_active":"2024-09-21T22:06:39.818270+0000","last_peered":"2024-09-21T22:06:39.818270+0000","last_clean":"2024-09-21T22:06:39.818270+0000","last_became_active":"2024-09-21T22:06:05.102579+0000","last_became_peered":"2024-09-21T22:06:05.102579+0000","last_unstale":"2024-09-21T22:06:39.818270+0000","last_undegraded":"2024-09-21T22:06:39.818270+0000","last_fullsized":"2024-09-21T22:06:39.818270+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T05:18:35.468678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026694230000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":49,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-21T22:06:39.785976+0000","last_change":"2024-09-21T22:06:07.123496+0000","last_active":"2024-09-21T22:06:39.785976+0000","last_peered":"2024-09-21T22:06:39.785976+0000","last_clean":"2024-09-21T22:06:39.785976+0000","last_became_active":"2024-09-21T22:06:05.104941+0000","last_became_peered":"2024-09-21T22:06:05.104941+0000","last_unstale":"2024-09-21T22:06:39.785976+0000","last_undegraded":"2024-09-21T22:06:39.785976+0000","last_fullsized":"2024-09-21T22:06:39.785976+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:46:29.901819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010052629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":48,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-21T22:06:39.781532+0000","last_change":"2024-09-21T22:06:07.121573+0000","last_active":"2024-09-21T22:06:39.781532+0000","last_peered":"2024-09-21T22:06:39.781532+0000","last_clean":"2024-09-21T22:06:39.781532+0000","last_became_active":"2024-09-21T22:06:05.108187+0000","last_became_peered":"2024-09-21T22:06:05.108187+0000","last_unstale":"2024-09-21T22:06:39.781532+0000","last_undegraded":"2024-09-21T22:06:39.781532+0000","last_fullsized":"2024-09-21T22:06:39.781532+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T05:05:08.108341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079478100000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"17'1","reported_seq":50,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-21T22:06:39.818270+0000","last_change":"2024-09-21T22:06:07.128525+0000","last_active":"2024-09-21T22:06:39.818270+0000","last_peered":"2024-09-21T22:06:39.818270+0000","last_clean":"2024-09-21T22:06:39.818270+0000","last_became_active":"2024-09-21T22:06:05.101507+0000","last_became_peered":"2024-09-21T22:06:05.101507+0000","last_unstale":"2024-09-21T22:06:39.818270+0000","last_undegraded":"2024-09-21T22:06:39.818270+0000","last_fullsized":"2024-09-21T22:06:39.818270+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:43:06.377355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001125968,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":212,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-21T22:06:39.781559+0000","last_change":"2024-09-21T22:06:24.228963+0000","last_active":"2024-09-21T22:06:39.781559+0000","last_peered":"2024-09-21T22:06:39.781559+0000","last_clean":"2024-09-21T22:06:39.781559+0000","last_became_active":"2024-09-21T22:06:24.227898+0000","last_became_peered":"2024-09-21T22:06:24.227898+0000","last_unstale":"2024-09-21T22:06:39.781559+0000","last_undegraded":"2024-09-21T22:06:39.781559+0000","last_fullsized":"2024-09-21T22:06:39.781559+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:00.990331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:00.990331+0000","last_clean_scrub_stamp":"2024-09-21T22:06:00.990331+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T09:06:54.254990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":1771104,"data_compressed":20551,"data_compressed_allocated":110592,"data_compressed_original":1736704,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640269,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27376,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344464,"statfs":{"total":96636764160,"available":96608731136,"internally_reserved":0,"allocated":356352,"data_stored":124605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640268,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27372,"kb_used_data":344,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344468,"statfs":{"total":96636764160,"available":96608735232,"internally_reserved":0,"allocated":352256,"data_stored":124586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640271,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27428,"kb_used_data":400,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344412,"statfs":{"total":96636764160,"available":96608677888,"internally_reserved":0,"allocated":409600,"data_stored":714954,"data_compressed":6561,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640270,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31532,"kb_used_data":344,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340308,"statfs":{"total":96636764160,"available":96604475392,"internally_reserved":0,"allocated":352256,"data_stored":124586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640270,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31580,"kb_used_data":388,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340260,"statfs":{"total":96636764160,"available":96604426240,"internally_reserved":0,"allocated":397312,"data_stored":714973,"data_compressed":7429,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":0,"seq":47244640271,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6561,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":7429,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6561,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T22:06:42.204 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T22:06:42.292 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:06:42.289+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:06:42.328 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:42.329 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:42.344 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T22:06:42.393 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:06:42.389+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:06:42.471 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:42.471 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:42.486 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T22:06:42.531 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:06:42.528+0000 7fc8d39f7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25294) UID: 0 2024-09-21T22:06:42.540 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T22:06:42.541 INFO:teuthology.orchestra.run.smithi039.stderr:dumped all 2024-09-21T22:06:42.559 INFO:teuthology.orchestra.run.smithi039.stdout:{"pg_ready":true,"pg_map":{"version":52,"stamp":"2024-09-21T22:06:41.816456+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":471859200,"kb_used":145288,"kb_used_data":1824,"kb_used_omap":7,"kb_used_meta":143352,"kb_avail":471713912,"statfs":{"total":483183820800,"available":483035045888,"internally_reserved":0,"allocated":1867776,"data_stored":1803704,"data_compressed":13990,"data_compressed_allocated":73728,"data_compressed_original":1163264,"omap_allocated":7973,"internal_metadata":146792667},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.060119"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":48,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-21T22:06:39.781540+0000","last_change":"2024-09-21T22:06:07.122328+0000","last_active":"2024-09-21T22:06:39.781540+0000","last_peered":"2024-09-21T22:06:39.781540+0000","last_clean":"2024-09-21T22:06:39.781540+0000","last_became_active":"2024-09-21T22:06:05.220996+0000","last_became_peered":"2024-09-21T22:06:05.220996+0000","last_unstale":"2024-09-21T22:06:39.781540+0000","last_undegraded":"2024-09-21T22:06:39.781540+0000","last_fullsized":"2024-09-21T22:06:39.781540+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T23:45:49.178013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001473755,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":49,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-21T22:06:39.818712+0000","last_change":"2024-09-21T22:06:07.133757+0000","last_active":"2024-09-21T22:06:39.818712+0000","last_peered":"2024-09-21T22:06:39.818712+0000","last_clean":"2024-09-21T22:06:39.818712+0000","last_became_active":"2024-09-21T22:06:05.109065+0000","last_became_peered":"2024-09-21T22:06:05.109065+0000","last_unstale":"2024-09-21T22:06:39.818712+0000","last_undegraded":"2024-09-21T22:06:39.818712+0000","last_fullsized":"2024-09-21T22:06:39.818712+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:31:12.654893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00088865700000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":53,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-21T22:06:39.781475+0000","last_change":"2024-09-21T22:06:21.313951+0000","last_active":"2024-09-21T22:06:39.781475+0000","last_peered":"2024-09-21T22:06:39.781475+0000","last_clean":"2024-09-21T22:06:39.781475+0000","last_became_active":"2024-09-21T22:06:21.313514+0000","last_became_peered":"2024-09-21T22:06:21.313514+0000","last_unstale":"2024-09-21T22:06:39.781475+0000","last_undegraded":"2024-09-21T22:06:39.781475+0000","last_fullsized":"2024-09-21T22:06:39.781475+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:07:37.870228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091949900000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":54,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-21T22:06:39.818700+0000","last_change":"2024-09-21T22:06:21.314973+0000","last_active":"2024-09-21T22:06:39.818700+0000","last_peered":"2024-09-21T22:06:39.818700+0000","last_clean":"2024-09-21T22:06:39.818700+0000","last_became_active":"2024-09-21T22:06:21.313946+0000","last_became_peered":"2024-09-21T22:06:21.313946+0000","last_unstale":"2024-09-21T22:06:39.818700+0000","last_undegraded":"2024-09-21T22:06:39.818700+0000","last_fullsized":"2024-09-21T22:06:39.818700+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:14:03.108117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00090834100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"19'2","reported_seq":51,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-21T22:06:39.818270+0000","last_change":"2024-09-21T22:06:07.130034+0000","last_active":"2024-09-21T22:06:39.818270+0000","last_peered":"2024-09-21T22:06:39.818270+0000","last_clean":"2024-09-21T22:06:39.818270+0000","last_became_active":"2024-09-21T22:06:05.102579+0000","last_became_peered":"2024-09-21T22:06:05.102579+0000","last_unstale":"2024-09-21T22:06:39.818270+0000","last_undegraded":"2024-09-21T22:06:39.818270+0000","last_fullsized":"2024-09-21T22:06:39.818270+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T05:18:35.468678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026694230000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":49,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-21T22:06:39.785976+0000","last_change":"2024-09-21T22:06:07.123496+0000","last_active":"2024-09-21T22:06:39.785976+0000","last_peered":"2024-09-21T22:06:39.785976+0000","last_clean":"2024-09-21T22:06:39.785976+0000","last_became_active":"2024-09-21T22:06:05.104941+0000","last_became_peered":"2024-09-21T22:06:05.104941+0000","last_unstale":"2024-09-21T22:06:39.785976+0000","last_undegraded":"2024-09-21T22:06:39.785976+0000","last_fullsized":"2024-09-21T22:06:39.785976+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:46:29.901819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010052629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":48,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-21T22:06:39.781532+0000","last_change":"2024-09-21T22:06:07.121573+0000","last_active":"2024-09-21T22:06:39.781532+0000","last_peered":"2024-09-21T22:06:39.781532+0000","last_clean":"2024-09-21T22:06:39.781532+0000","last_became_active":"2024-09-21T22:06:05.108187+0000","last_became_peered":"2024-09-21T22:06:05.108187+0000","last_unstale":"2024-09-21T22:06:39.781532+0000","last_undegraded":"2024-09-21T22:06:39.781532+0000","last_fullsized":"2024-09-21T22:06:39.781532+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T05:05:08.108341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079478100000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"17'1","reported_seq":50,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-21T22:06:39.818270+0000","last_change":"2024-09-21T22:06:07.128525+0000","last_active":"2024-09-21T22:06:39.818270+0000","last_peered":"2024-09-21T22:06:39.818270+0000","last_clean":"2024-09-21T22:06:39.818270+0000","last_became_active":"2024-09-21T22:06:05.101507+0000","last_became_peered":"2024-09-21T22:06:05.101507+0000","last_unstale":"2024-09-21T22:06:39.818270+0000","last_undegraded":"2024-09-21T22:06:39.818270+0000","last_fullsized":"2024-09-21T22:06:39.818270+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:43:06.377355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001125968,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":212,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-21T22:06:39.781559+0000","last_change":"2024-09-21T22:06:24.228963+0000","last_active":"2024-09-21T22:06:39.781559+0000","last_peered":"2024-09-21T22:06:39.781559+0000","last_clean":"2024-09-21T22:06:39.781559+0000","last_became_active":"2024-09-21T22:06:24.227898+0000","last_became_peered":"2024-09-21T22:06:24.227898+0000","last_unstale":"2024-09-21T22:06:39.781559+0000","last_undegraded":"2024-09-21T22:06:39.781559+0000","last_fullsized":"2024-09-21T22:06:39.781559+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":27,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:00.990331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:00.990331+0000","last_clean_scrub_stamp":"2024-09-21T22:06:00.990331+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T09:06:54.254990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":1771104,"data_compressed":20551,"data_compressed_allocated":110592,"data_compressed_original":1736704,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640269,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27376,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344464,"statfs":{"total":96636764160,"available":96608731136,"internally_reserved":0,"allocated":356352,"data_stored":124605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640268,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27372,"kb_used_data":344,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344468,"statfs":{"total":96636764160,"available":96608735232,"internally_reserved":0,"allocated":352256,"data_stored":124586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640271,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27428,"kb_used_data":400,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344412,"statfs":{"total":96636764160,"available":96608677888,"internally_reserved":0,"allocated":409600,"data_stored":714954,"data_compressed":6561,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640270,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31532,"kb_used_data":344,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340308,"statfs":{"total":96636764160,"available":96604475392,"internally_reserved":0,"allocated":352256,"data_stored":124586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640270,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31580,"kb_used_data":388,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340260,"statfs":{"total":96636764160,"available":96604426240,"internally_reserved":0,"allocated":397312,"data_stored":714973,"data_compressed":7429,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":0,"seq":47244640271,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6561,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":7429,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6561,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T22:06:42.560 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-09-21T22:06:42.560 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T22:06:42.560 INFO:tasks.thrashosds.thrasher:Adding osd 0 2024-09-21T22:06:42.560 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 0 2024-09-21T22:06:42.594 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:06:42.588+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:06:42.615 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:42.615 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:42.629 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T22:06:42.715 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:06:42.712+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:06:42.785 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T22:06:42.799 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:42.799 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:42.814 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T22:06:42.830 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:06:42.829+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:06:42.835 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-21T22:06:42.897 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:06:42.893+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:06:42.921 INFO:tasks.ceph.mon.a.smithi039.stderr:2024-09-21T22:06:42.917+0000 7f9967196640 -1 mon.a@0(leader).osd e36 definitely_dead 0 2024-09-21T22:06:42.929 INFO:teuthology.orchestra.run.smithi039.stdout:osd.0: {} 2024-09-21T22:06:42.929 INFO:teuthology.orchestra.run.smithi039.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T22:06:42.946 INFO:teuthology.orchestra.run.smithi039.stdout:osd.1: {} 2024-09-21T22:06:42.946 INFO:teuthology.orchestra.run.smithi039.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T22:06:42.964 INFO:teuthology.orchestra.run.smithi039.stdout:osd.2: {} 2024-09-21T22:06:42.964 INFO:teuthology.orchestra.run.smithi039.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T22:06:42.975 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:42.975 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:42.984 INFO:teuthology.orchestra.run.smithi039.stdout:osd.3: {} 2024-09-21T22:06:42.984 INFO:teuthology.orchestra.run.smithi039.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T22:06:42.990 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T22:06:43.003 INFO:teuthology.orchestra.run.smithi039.stdout:osd.4: {} 2024-09-21T22:06:43.003 INFO:teuthology.orchestra.run.smithi039.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T22:06:43.021 INFO:teuthology.orchestra.run.smithi039.stdout:osd.5: {} 2024-09-21T22:06:43.021 INFO:teuthology.orchestra.run.smithi039.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T22:06:43.035 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:06:43.032+0000 7fc8d39f7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25294) UID: 0 2024-09-21T22:06:43.099 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:06:43.097+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:06:43.118 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:06:43.118 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-21T22:06:43.118 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-21T22:06:43.119 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-21T22:06:43.119 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:43.119 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4224.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-21T22:06:43.119 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:05.105895+0000", 2024-09-21T22:06:43.119 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 38.011506609999998, 2024-09-21T22:06:43.119 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0034226500000000002, 2024-09-21T22:06:43.119 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:43.119 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:43.119 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:43.119 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:43.119 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4224", 2024-09-21T22:06:43.119 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.39:0/795178185", 2024-09-21T22:06:43.120 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1 2024-09-21T22:06:43.120 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:43.120 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:43.120 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:43.120 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:43.120 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.105895+0000", 2024-09-21T22:06:43.120 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:43.120 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:43.120 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:43.120 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:43.120 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.105895+0000", 2024-09-21T22:06:43.120 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:43.120 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:43.121 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:43.121 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:43.121 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.105905+0000", 2024-09-21T22:06:43.121 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8069999999999994e-06 2024-09-21T22:06:43.121 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:43.121 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:43.121 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:43.121 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.105919+0000", 2024-09-21T22:06:43.121 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3976e-05 2024-09-21T22:06:43.121 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:43.121 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:43.121 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:43.122 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.105955+0000", 2024-09-21T22:06:43.122 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6776999999999997e-05 2024-09-21T22:06:43.122 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:43.122 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:43.122 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:43.122 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.105983+0000", 2024-09-21T22:06:43.122 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7535999999999999e-05 2024-09-21T22:06:43.122 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:43.122 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:43.122 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:43.122 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.106162+0000", 2024-09-21T22:06:43.122 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000178839 2024-09-21T22:06:43.123 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:43.123 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:43.123 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:43.123 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.106437+0000", 2024-09-21T22:06:43.123 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027542099999999998 2024-09-21T22:06:43.123 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:43.123 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:43.123 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 2", 2024-09-21T22:06:43.123 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.106726+0000", 2024-09-21T22:06:43.123 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00028856800000000001 2024-09-21T22:06:43.123 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:43.123 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:43.124 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:43.124 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.108194+0000", 2024-09-21T22:06:43.124 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001468095 2024-09-21T22:06:43.124 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:43.124 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:43.124 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:43.124 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.109130+0000", 2024-09-21T22:06:43.124 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00093653599999999997 2024-09-21T22:06:43.124 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:43.124 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:43.124 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:43.124 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.109216+0000", 2024-09-21T22:06:43.125 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5191999999999993e-05 2024-09-21T22:06:43.125 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:43.125 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:43.125 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:43.125 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.109317+0000", 2024-09-21T22:06:43.125 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000101903 2024-09-21T22:06:43.125 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:43.125 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:43.125 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:43.125 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:43.125 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:43.125 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4224.0:1 2.3 e17/16 ondisk, result = 0)", 2024-09-21T22:06:43.126 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:05.108920+0000", 2024-09-21T22:06:43.126 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 38.008481103000001, 2024-09-21T22:06:43.126 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00041366599999999998, 2024-09-21T22:06:43.126 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:43.126 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:43.126 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:06:43.126 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:43.126 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:43.126 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:43.126 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.108920+0000", 2024-09-21T22:06:43.126 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:43.127 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:43.127 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:43.127 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:43.127 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.108920+0000", 2024-09-21T22:06:43.127 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:43.127 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:43.127 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:43.127 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:43.127 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.108922+0000", 2024-09-21T22:06:43.127 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.703e-06 2024-09-21T22:06:43.127 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:43.127 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:43.128 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:43.128 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.108938+0000", 2024-09-21T22:06:43.128 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6350000000000001e-05 2024-09-21T22:06:43.128 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:43.128 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:43.128 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:43.128 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.108974+0000", 2024-09-21T22:06:43.128 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5612999999999999e-05 2024-09-21T22:06:43.128 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:43.128 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:43.128 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:43.128 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.108989+0000", 2024-09-21T22:06:43.129 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4576000000000001e-05 2024-09-21T22:06:43.129 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:43.129 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:43.129 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:43.129 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.109108+0000", 2024-09-21T22:06:43.129 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011901899999999999 2024-09-21T22:06:43.129 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:43.129 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:43.129 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:43.129 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.109120+0000", 2024-09-21T22:06:43.129 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.293e-05 2024-09-21T22:06:43.129 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:43.129 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:43.130 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:43.130 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.109334+0000", 2024-09-21T22:06:43.130 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000213475 2024-09-21T22:06:43.130 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:43.130 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:43.130 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:43.130 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:43.130 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:43.130 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4224.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e17)", 2024-09-21T22:06:43.130 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:05.115572+0000", 2024-09-21T22:06:43.130 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 38.001829194999999, 2024-09-21T22:06:43.130 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00041313, 2024-09-21T22:06:43.131 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:43.131 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:43.131 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-21T22:06:43.131 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:43.131 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4224", 2024-09-21T22:06:43.131 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.39:0/795178185", 2024-09-21T22:06:43.131 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 2 2024-09-21T22:06:43.131 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:43.131 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:43.131 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:43.131 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:43.131 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.115572+0000", 2024-09-21T22:06:43.132 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:43.132 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:43.132 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:43.132 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:43.132 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.115572+0000", 2024-09-21T22:06:43.132 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:43.132 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:43.132 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:43.132 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:43.132 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.115574+0000", 2024-09-21T22:06:43.132 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0530000000000001e-06 2024-09-21T22:06:43.132 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:43.133 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:43.133 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:43.133 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.115585+0000", 2024-09-21T22:06:43.133 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0947e-05 2024-09-21T22:06:43.133 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:43.133 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:43.133 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:43.133 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.115615+0000", 2024-09-21T22:06:43.133 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9408999999999999e-05 2024-09-21T22:06:43.133 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:43.133 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:43.133 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:43.133 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.115637+0000", 2024-09-21T22:06:43.134 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2073e-05 2024-09-21T22:06:43.134 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:43.134 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:43.134 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:43.134 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.115768+0000", 2024-09-21T22:06:43.134 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013153899999999999 2024-09-21T22:06:43.134 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:43.134 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:43.134 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:43.134 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.115985+0000", 2024-09-21T22:06:43.134 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000217109 2024-09-21T22:06:43.134 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:43.135 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:43.135 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:43.135 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:43.135 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:43.135 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4224.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e17)", 2024-09-21T22:06:43.135 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:05.116456+0000", 2024-09-21T22:06:43.135 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 38.000945649000002, 2024-09-21T22:06:43.135 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00030938500000000002, 2024-09-21T22:06:43.135 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:43.135 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:43.135 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "reached pg", 2024-09-21T22:06:43.135 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:43.136 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4224", 2024-09-21T22:06:43.136 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.39:0/795178185", 2024-09-21T22:06:43.136 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3 2024-09-21T22:06:43.136 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:43.136 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:43.136 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:43.136 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:43.136 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.116456+0000", 2024-09-21T22:06:43.136 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:43.136 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:43.136 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:43.136 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:43.137 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.116456+0000", 2024-09-21T22:06:43.137 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:43.137 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:43.137 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:43.137 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:43.137 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.116458+0000", 2024-09-21T22:06:43.137 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.77e-06 2024-09-21T22:06:43.137 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:43.137 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:43.137 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:43.137 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.116466+0000", 2024-09-21T22:06:43.137 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1799999999999996e-06 2024-09-21T22:06:43.137 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:43.138 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:43.138 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:43.138 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.116492+0000", 2024-09-21T22:06:43.138 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6523999999999999e-05 2024-09-21T22:06:43.138 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:43.138 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:43.138 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:43.138 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.116513+0000", 2024-09-21T22:06:43.138 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0400000000000001e-05 2024-09-21T22:06:43.138 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:43.138 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:43.138 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:43.139 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.116616+0000", 2024-09-21T22:06:43.139 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000103862 2024-09-21T22:06:43.139 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:43.139 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:43.139 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:43.139 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:05.116765+0000", 2024-09-21T22:06:43.139 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014864899999999999 2024-09-21T22:06:43.139 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:43.139 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:43.139 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:43.139 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:43.139 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:43.140 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4224.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-21T22:06:43.140 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:06.109464+0000", 2024-09-21T22:06:43.140 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 37.007936970999999, 2024-09-21T22:06:43.140 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0070431790000000001, 2024-09-21T22:06:43.140 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:43.140 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:43.140 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:43.140 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:43.140 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4224", 2024-09-21T22:06:43.140 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.39:0/795178185", 2024-09-21T22:06:43.140 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 5 2024-09-21T22:06:43.140 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:43.140 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:43.141 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:43.141 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:43.141 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.109464+0000", 2024-09-21T22:06:43.141 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:43.141 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:43.141 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:43.141 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:43.141 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.109464+0000", 2024-09-21T22:06:43.141 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:43.141 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:43.141 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:43.141 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:43.142 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.109467+0000", 2024-09-21T22:06:43.142 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.066e-06 2024-09-21T22:06:43.142 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:43.142 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:43.142 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:43.142 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.109485+0000", 2024-09-21T22:06:43.142 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7464000000000001e-05 2024-09-21T22:06:43.142 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:43.142 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:43.142 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:43.142 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.109518+0000", 2024-09-21T22:06:43.142 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3343000000000002e-05 2024-09-21T22:06:43.142 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:43.143 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:43.143 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:43.143 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.109537+0000", 2024-09-21T22:06:43.143 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8743e-05 2024-09-21T22:06:43.143 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:43.143 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:43.143 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:43.143 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.109708+0000", 2024-09-21T22:06:43.143 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017073899999999999 2024-09-21T22:06:43.143 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:43.143 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:43.144 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op must wait for map", 2024-09-21T22:06:43.144 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.109725+0000", 2024-09-21T22:06:43.144 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7013000000000001e-05 2024-09-21T22:06:43.144 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:43.144 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:43.144 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:43.144 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.113449+0000", 2024-09-21T22:06:43.144 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0037244579999999999 2024-09-21T22:06:43.144 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:43.144 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:43.144 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:43.144 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.113659+0000", 2024-09-21T22:06:43.144 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020968199999999999 2024-09-21T22:06:43.145 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:43.145 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:43.145 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-09-21T22:06:43.145 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.113872+0000", 2024-09-21T22:06:43.145 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000213062 2024-09-21T22:06:43.145 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:43.145 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:43.145 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:43.145 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.115375+0000", 2024-09-21T22:06:43.145 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015028979999999999 2024-09-21T22:06:43.145 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:43.145 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:43.145 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:43.146 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.116356+0000", 2024-09-21T22:06:43.146 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00098098200000000003 2024-09-21T22:06:43.146 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:43.146 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:43.146 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:43.146 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.116442+0000", 2024-09-21T22:06:43.146 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6533000000000007e-05 2024-09-21T22:06:43.146 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:43.146 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:43.146 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:43.146 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.116508+0000", 2024-09-21T22:06:43.147 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5196000000000004e-05 2024-09-21T22:06:43.147 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:43.147 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:43.147 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:43.147 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:43.147 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:43.147 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4224.0:5 2.2 e17/16 ondisk, result = 0)", 2024-09-21T22:06:43.147 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:06.116133+0000", 2024-09-21T22:06:43.147 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 37.001268486000001, 2024-09-21T22:06:43.147 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00038649700000000002, 2024-09-21T22:06:43.147 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:43.147 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:43.147 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:06:43.148 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:43.148 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:43.148 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:43.148 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.116133+0000", 2024-09-21T22:06:43.148 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:43.148 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:43.148 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:43.148 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:43.148 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.116133+0000", 2024-09-21T22:06:43.148 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:43.148 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:43.148 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:43.149 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:43.149 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.116135+0000", 2024-09-21T22:06:43.149 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8870000000000001e-06 2024-09-21T22:06:43.149 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:43.149 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:43.149 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:43.149 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.116146+0000", 2024-09-21T22:06:43.149 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0953e-05 2024-09-21T22:06:43.149 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:43.149 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:43.149 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:43.150 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.116175+0000", 2024-09-21T22:06:43.150 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9383e-05 2024-09-21T22:06:43.150 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:43.150 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:43.150 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:43.150 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.116190+0000", 2024-09-21T22:06:43.150 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4613000000000001e-05 2024-09-21T22:06:43.150 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:43.150 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:43.150 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:43.150 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.116328+0000", 2024-09-21T22:06:43.150 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013848300000000001 2024-09-21T22:06:43.151 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:43.151 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:43.151 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:43.151 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.116345+0000", 2024-09-21T22:06:43.151 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6883000000000001e-05 2024-09-21T22:06:43.151 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:43.151 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:43.151 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:43.151 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:06.116519+0000", 2024-09-21T22:06:43.151 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017429500000000001 2024-09-21T22:06:43.151 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:43.151 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:43.152 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:43.152 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:43.152 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:43.152 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4224.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19)", 2024-09-21T22:06:43.152 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:07.124322+0000", 2024-09-21T22:06:43.152 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 35.993079180999999, 2024-09-21T22:06:43.152 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0067709010000000002, 2024-09-21T22:06:43.152 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:43.152 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:43.152 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:43.152 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:43.152 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4224", 2024-09-21T22:06:43.153 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.39:0/795178185", 2024-09-21T22:06:43.153 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 7 2024-09-21T22:06:43.153 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:43.153 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:43.153 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:43.153 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:43.153 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.124322+0000", 2024-09-21T22:06:43.153 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:43.153 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:43.153 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:43.153 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:43.153 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.124322+0000", 2024-09-21T22:06:43.153 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:43.154 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:43.154 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:43.154 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:43.154 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.124325+0000", 2024-09-21T22:06:43.154 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1489999999999998e-06 2024-09-21T22:06:43.154 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:43.154 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:43.154 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:43.154 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.124342+0000", 2024-09-21T22:06:43.154 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.666e-05 2024-09-21T22:06:43.154 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:43.154 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:43.155 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:43.155 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.124379+0000", 2024-09-21T22:06:43.155 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6539999999999999e-05 2024-09-21T22:06:43.155 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:43.155 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:43.155 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:43.155 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.124406+0000", 2024-09-21T22:06:43.155 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7886e-05 2024-09-21T22:06:43.155 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:43.155 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:43.155 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:43.155 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.124535+0000", 2024-09-21T22:06:43.156 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000128749 2024-09-21T22:06:43.156 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:43.156 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:43.156 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op must wait for map", 2024-09-21T22:06:43.156 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.124547+0000", 2024-09-21T22:06:43.156 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.136e-05 2024-09-21T22:06:43.156 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:43.156 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:43.156 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:43.156 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.127774+0000", 2024-09-21T22:06:43.156 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0032275519999999999 2024-09-21T22:06:43.156 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:43.156 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:43.157 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:43.157 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.128014+0000", 2024-09-21T22:06:43.157 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023956800000000001 2024-09-21T22:06:43.157 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:43.157 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:43.157 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-09-21T22:06:43.157 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.128462+0000", 2024-09-21T22:06:43.157 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00044797700000000001 2024-09-21T22:06:43.157 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:43.157 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:43.157 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:43.157 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.130185+0000", 2024-09-21T22:06:43.158 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00172313 2024-09-21T22:06:43.158 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:43.158 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:43.158 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:43.158 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.130937+0000", 2024-09-21T22:06:43.158 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00075244799999999996 2024-09-21T22:06:43.158 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:43.158 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:43.158 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:43.158 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.131025+0000", 2024-09-21T22:06:43.158 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7998999999999996e-05 2024-09-21T22:06:43.158 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:43.159 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:43.159 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:43.159 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.131093+0000", 2024-09-21T22:06:43.159 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7883e-05 2024-09-21T22:06:43.159 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:43.159 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:43.159 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:43.159 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:43.159 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:43.159 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4224.0:7 2.2 e18/16 ondisk, result = 0)", 2024-09-21T22:06:43.159 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:07.130777+0000", 2024-09-21T22:06:43.159 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 35.986624135, 2024-09-21T22:06:43.159 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00032786799999999999, 2024-09-21T22:06:43.160 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:43.160 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:43.160 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:06:43.160 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:43.160 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:43.160 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:43.160 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.130777+0000", 2024-09-21T22:06:43.160 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:43.160 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:43.160 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:43.160 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:43.160 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.130777+0000", 2024-09-21T22:06:43.160 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:43.161 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:43.161 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:43.161 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:43.161 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.130779+0000", 2024-09-21T22:06:43.161 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7e-06 2024-09-21T22:06:43.161 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:43.161 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:43.161 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:43.161 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.130788+0000", 2024-09-21T22:06:43.161 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6670000000000002e-06 2024-09-21T22:06:43.161 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:43.161 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:43.162 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:43.162 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.130814+0000", 2024-09-21T22:06:43.162 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6590000000000001e-05 2024-09-21T22:06:43.162 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:43.162 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:43.162 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:43.162 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.130828+0000", 2024-09-21T22:06:43.162 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.415e-05 2024-09-21T22:06:43.162 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:43.162 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:43.162 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:43.162 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.130915+0000", 2024-09-21T22:06:43.162 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6759000000000001e-05 2024-09-21T22:06:43.163 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:43.163 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:43.163 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:43.163 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.130928+0000", 2024-09-21T22:06:43.163 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.328e-05 2024-09-21T22:06:43.163 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:43.163 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:43.163 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:43.163 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:07.131105+0000", 2024-09-21T22:06:43.163 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017672200000000001 2024-09-21T22:06:43.163 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:43.163 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:43.164 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:43.164 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:43.164 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:43.164 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:06:43.165 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T22:06:43.214 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:06:43.213+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:06:43.295 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:06:43.295 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:06:43.295 INFO:teuthology.orchestra.run.smithi039.stdout: "num_ops": 0 2024-09-21T22:06:43.295 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:06:43.301 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:06:43.300+0000 7fc8d39f7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25294) UID: 0 2024-09-21T22:06:43.309 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T22:06:43.402 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:06:43.396+0000 7fc8d39f7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25294) UID: 0 2024-09-21T22:06:43.477 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:06:43.478 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:06:43.478 INFO:teuthology.orchestra.run.smithi039.stdout: "complaint_time": 30, 2024-09-21T22:06:43.478 INFO:teuthology.orchestra.run.smithi039.stdout: "num_blocked_ops": 0 2024-09-21T22:06:43.478 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:06:43.492 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T22:06:43.503 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:06:43.500+0000 7fc8d39f7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25294) UID: 0 2024-09-21T22:06:43.604 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:06:43.601+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:06:43.623 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:06:43.623 INFO:teuthology.orchestra.run.smithi039.stdout: "size": 20, 2024-09-21T22:06:43.624 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 600, 2024-09-21T22:06:43.624 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [ 2024-09-21T22:06:43.624 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.624 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:20 1.0 e14/13 1:30306672:devicehealth::main.db.0000000000000000:head v 14'10, mlcod=14'8)", 2024-09-21T22:06:43.624 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.137639+0000", 2024-09-21T22:06:43.624 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 41.483750295, 2024-09-21T22:06:43.624 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0039902100000000001, 2024-09-21T22:06:43.624 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:43.624 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:43.624 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:43.624 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:43.624 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.624 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:43.624 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.137639+0000", 2024-09-21T22:06:43.625 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:43.625 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.625 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.625 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:43.625 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.137639+0000", 2024-09-21T22:06:43.625 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:43.625 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.625 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.625 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:43.625 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.137641+0000", 2024-09-21T22:06:43.625 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.686e-06 2024-09-21T22:06:43.625 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.625 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.625 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:43.626 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.137934+0000", 2024-09-21T22:06:43.626 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00029319099999999999 2024-09-21T22:06:43.626 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.626 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.626 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:43.626 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.137963+0000", 2024-09-21T22:06:43.626 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9502999999999999e-05 2024-09-21T22:06:43.626 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.626 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.626 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:43.626 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.137989+0000", 2024-09-21T22:06:43.626 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5573000000000001e-05 2024-09-21T22:06:43.626 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.626 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.627 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:43.627 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.138086+0000", 2024-09-21T22:06:43.627 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.6968999999999999e-05 2024-09-21T22:06:43.627 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.627 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.627 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:43.627 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.138120+0000", 2024-09-21T22:06:43.627 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.4560000000000001e-05 2024-09-21T22:06:43.627 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.627 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.627 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:43.627 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.141539+0000", 2024-09-21T22:06:43.627 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0034183289999999999 2024-09-21T22:06:43.627 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.628 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.628 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:43.628 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.141629+0000", 2024-09-21T22:06:43.628 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.0399e-05 2024-09-21T22:06:43.628 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:43.628 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:43.628 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:43.628 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.628 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.628 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:23 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'13, mlcod=14'11)", 2024-09-21T22:06:43.628 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.144914+0000", 2024-09-21T22:06:43.628 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 41.476474977000002, 2024-09-21T22:06:43.629 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0040223230000000004, 2024-09-21T22:06:43.629 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:43.629 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:43.629 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:43.629 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:43.629 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.629 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:43.629 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.144914+0000", 2024-09-21T22:06:43.629 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:43.629 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.629 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.629 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:43.629 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.144914+0000", 2024-09-21T22:06:43.629 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:43.630 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.630 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.630 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:43.630 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.144916+0000", 2024-09-21T22:06:43.630 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6470000000000001e-06 2024-09-21T22:06:43.630 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.630 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.630 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:43.630 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.144929+0000", 2024-09-21T22:06:43.630 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2903e-05 2024-09-21T22:06:43.630 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.630 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.630 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:43.630 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.144958+0000", 2024-09-21T22:06:43.631 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9280000000000001e-05 2024-09-21T22:06:43.631 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.631 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.631 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:43.631 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.144971+0000", 2024-09-21T22:06:43.631 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3407000000000001e-05 2024-09-21T22:06:43.631 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.631 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.631 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:43.631 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.145106+0000", 2024-09-21T22:06:43.631 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013440500000000001 2024-09-21T22:06:43.631 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.631 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.631 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:43.632 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.145170+0000", 2024-09-21T22:06:43.632 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.4597000000000005e-05 2024-09-21T22:06:43.632 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.632 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.632 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:43.632 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.148854+0000", 2024-09-21T22:06:43.632 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0036833249999999999 2024-09-21T22:06:43.632 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.632 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.632 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:43.632 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.148937+0000", 2024-09-21T22:06:43.632 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.2758999999999998e-05 2024-09-21T22:06:43.632 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:43.632 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:43.633 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:43.633 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.633 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.633 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:31 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'17, mlcod=14'14)", 2024-09-21T22:06:43.633 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.163958+0000", 2024-09-21T22:06:43.633 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 41.457430709999997, 2024-09-21T22:06:43.633 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0033891149999999998, 2024-09-21T22:06:43.633 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:43.633 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:43.633 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:43.633 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:43.633 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.633 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:43.633 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.163958+0000", 2024-09-21T22:06:43.634 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:43.634 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.634 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.634 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:43.634 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.163958+0000", 2024-09-21T22:06:43.634 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:43.634 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.634 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.634 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:43.634 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.163960+0000", 2024-09-21T22:06:43.634 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.64e-06 2024-09-21T22:06:43.634 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.634 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.635 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:43.635 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.163971+0000", 2024-09-21T22:06:43.635 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0797e-05 2024-09-21T22:06:43.635 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.635 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.635 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:43.635 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.163995+0000", 2024-09-21T22:06:43.635 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3876e-05 2024-09-21T22:06:43.635 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.635 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.635 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:43.635 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.164008+0000", 2024-09-21T22:06:43.635 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3674e-05 2024-09-21T22:06:43.636 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.636 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.636 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:43.636 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.164203+0000", 2024-09-21T22:06:43.636 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00019457799999999999 2024-09-21T22:06:43.636 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.636 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.636 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:43.636 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.164265+0000", 2024-09-21T22:06:43.636 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.2415999999999993e-05 2024-09-21T22:06:43.636 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.636 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.636 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:43.636 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.167209+0000", 2024-09-21T22:06:43.636 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0029436380000000002 2024-09-21T22:06:43.637 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.637 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.637 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:43.637 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.167348+0000", 2024-09-21T22:06:43.637 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000138496 2024-09-21T22:06:43.637 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:43.637 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:43.637 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:43.637 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.637 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.637 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:32 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'18, mlcod=14'14)", 2024-09-21T22:06:43.637 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.165319+0000", 2024-09-21T22:06:43.637 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 41.456069859999999, 2024-09-21T22:06:43.637 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0033722280000000001, 2024-09-21T22:06:43.638 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:43.638 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:43.638 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:43.638 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:43.638 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.638 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:43.638 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.165319+0000", 2024-09-21T22:06:43.638 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:43.638 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.638 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.638 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:43.638 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.165319+0000", 2024-09-21T22:06:43.638 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:43.639 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.639 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.639 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:43.639 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.165321+0000", 2024-09-21T22:06:43.639 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5999999999999999e-06 2024-09-21T22:06:43.639 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.639 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.639 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:43.639 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.165377+0000", 2024-09-21T22:06:43.639 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.5877000000000001e-05 2024-09-21T22:06:43.639 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.639 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.639 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:43.639 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.165407+0000", 2024-09-21T22:06:43.640 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.0263000000000001e-05 2024-09-21T22:06:43.640 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.640 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.640 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:43.640 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.165441+0000", 2024-09-21T22:06:43.640 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.3966e-05 2024-09-21T22:06:43.640 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.640 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.640 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:43.640 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.165574+0000", 2024-09-21T22:06:43.640 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013327599999999999 2024-09-21T22:06:43.640 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.640 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.641 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:43.641 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.165635+0000", 2024-09-21T22:06:43.641 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.0949000000000002e-05 2024-09-21T22:06:43.641 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.641 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.641 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:43.641 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.168619+0000", 2024-09-21T22:06:43.641 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0029836979999999999 2024-09-21T22:06:43.641 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.641 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.641 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:43.641 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.168692+0000", 2024-09-21T22:06:43.641 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.2599000000000001e-05 2024-09-21T22:06:43.642 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:43.642 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:43.642 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:43.642 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.642 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.642 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:38 1.0 e14/13 1:30306672:devicehealth::main.db.0000000000000000:head v 14'24, mlcod=14'22)", 2024-09-21T22:06:43.642 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.180244+0000", 2024-09-21T22:06:43.642 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 41.441145214999999, 2024-09-21T22:06:43.642 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0041712219999999996, 2024-09-21T22:06:43.642 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:43.642 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:43.642 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:43.642 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:43.643 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.643 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:43.643 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.180244+0000", 2024-09-21T22:06:43.643 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:43.643 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.643 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.643 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:43.643 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.180244+0000", 2024-09-21T22:06:43.643 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:43.643 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.643 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.643 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:43.643 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.180246+0000", 2024-09-21T22:06:43.643 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6959999999999999e-06 2024-09-21T22:06:43.644 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.644 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.644 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:43.644 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.180508+0000", 2024-09-21T22:06:43.644 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00026275199999999999 2024-09-21T22:06:43.644 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.644 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.644 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:43.644 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.180538+0000", 2024-09-21T22:06:43.644 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9142999999999999e-05 2024-09-21T22:06:43.644 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.644 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.644 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:43.644 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.180552+0000", 2024-09-21T22:06:43.644 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4466e-05 2024-09-21T22:06:43.645 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.645 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.645 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:43.645 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.180677+0000", 2024-09-21T22:06:43.645 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000125413 2024-09-21T22:06:43.645 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.645 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.645 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:43.645 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.180739+0000", 2024-09-21T22:06:43.645 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.1122000000000002e-05 2024-09-21T22:06:43.645 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.645 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.645 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:43.646 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.184291+0000", 2024-09-21T22:06:43.646 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0035525510000000001 2024-09-21T22:06:43.646 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.646 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.646 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:43.646 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.184415+0000", 2024-09-21T22:06:43.646 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000124079 2024-09-21T22:06:43.646 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:43.646 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:43.646 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:43.646 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.646 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.646 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:54 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'32, mlcod=14'29)", 2024-09-21T22:06:43.647 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.214326+0000", 2024-09-21T22:06:43.647 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 41.407062977999999, 2024-09-21T22:06:43.647 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0035226569999999998, 2024-09-21T22:06:43.647 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:43.647 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:43.647 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:43.647 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:43.647 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.647 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:43.647 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.214326+0000", 2024-09-21T22:06:43.647 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:43.647 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.647 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.647 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:43.648 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.214326+0000", 2024-09-21T22:06:43.648 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:43.648 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.648 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.648 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:43.648 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.214328+0000", 2024-09-21T22:06:43.648 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6270000000000001e-06 2024-09-21T22:06:43.648 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.648 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.648 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:43.648 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.214340+0000", 2024-09-21T22:06:43.648 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2286000000000001e-05 2024-09-21T22:06:43.648 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.649 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.649 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:43.649 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.214369+0000", 2024-09-21T22:06:43.649 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9297000000000001e-05 2024-09-21T22:06:43.649 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.649 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.649 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:43.649 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.214383+0000", 2024-09-21T22:06:43.649 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3773e-05 2024-09-21T22:06:43.649 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.649 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.649 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:43.649 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.214513+0000", 2024-09-21T22:06:43.649 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000130056 2024-09-21T22:06:43.650 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.650 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.650 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:43.650 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.214568+0000", 2024-09-21T22:06:43.650 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.4873000000000001e-05 2024-09-21T22:06:43.650 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.650 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.650 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:43.650 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.217739+0000", 2024-09-21T22:06:43.650 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0031712720000000002 2024-09-21T22:06:43.650 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.650 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.650 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:43.650 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.217849+0000", 2024-09-21T22:06:43.651 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000109473 2024-09-21T22:06:43.651 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:43.651 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:43.651 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:43.651 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.651 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.651 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:57 1.0 e14/13 1:30306672:devicehealth::main.db.0000000000000000:head v 14'35, mlcod=14'33)", 2024-09-21T22:06:43.651 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.221549+0000", 2024-09-21T22:06:43.651 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 41.399840478999998, 2024-09-21T22:06:43.651 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0041720860000000002, 2024-09-21T22:06:43.651 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:43.651 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:43.651 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:43.652 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:43.652 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.652 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:43.652 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.221549+0000", 2024-09-21T22:06:43.652 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:43.652 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.652 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.652 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:43.652 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.221549+0000", 2024-09-21T22:06:43.652 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:43.652 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.652 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.652 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:43.653 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.221550+0000", 2024-09-21T22:06:43.653 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.717e-06 2024-09-21T22:06:43.653 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.653 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.653 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:43.653 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.221673+0000", 2024-09-21T22:06:43.653 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000122612 2024-09-21T22:06:43.653 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.653 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.653 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:43.653 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.221703+0000", 2024-09-21T22:06:43.653 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9539999999999998e-05 2024-09-21T22:06:43.653 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.653 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.654 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:43.654 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.221716+0000", 2024-09-21T22:06:43.654 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3149999999999999e-05 2024-09-21T22:06:43.654 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.654 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.654 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:43.654 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.221848+0000", 2024-09-21T22:06:43.654 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013201900000000001 2024-09-21T22:06:43.654 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.654 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.654 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:43.654 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.221908+0000", 2024-09-21T22:06:43.654 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.0452999999999999e-05 2024-09-21T22:06:43.654 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.655 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.655 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:43.655 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.225573+0000", 2024-09-21T22:06:43.655 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.003664676 2024-09-21T22:06:43.655 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.655 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.655 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:43.655 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.225721+0000", 2024-09-21T22:06:43.655 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00014791899999999999 2024-09-21T22:06:43.655 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:43.655 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:43.655 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:43.655 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.655 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.656 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:68 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'42, mlcod=14'39)", 2024-09-21T22:06:43.656 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.247413+0000", 2024-09-21T22:06:43.656 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 41.373976130000003, 2024-09-21T22:06:43.656 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0039572339999999996, 2024-09-21T22:06:43.656 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:43.656 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:43.656 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:43.656 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:43.656 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.656 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:43.656 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.247413+0000", 2024-09-21T22:06:43.656 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:43.656 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.656 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.657 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:43.657 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.247413+0000", 2024-09-21T22:06:43.657 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:43.657 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.657 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.657 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:43.657 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.247415+0000", 2024-09-21T22:06:43.657 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.883e-06 2024-09-21T22:06:43.657 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.657 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.657 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:43.657 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.247427+0000", 2024-09-21T22:06:43.657 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2557e-05 2024-09-21T22:06:43.657 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.658 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.658 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:43.658 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.247457+0000", 2024-09-21T22:06:43.658 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9966000000000001e-05 2024-09-21T22:06:43.658 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.658 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.658 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:43.658 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.247471+0000", 2024-09-21T22:06:43.658 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3747000000000001e-05 2024-09-21T22:06:43.658 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.658 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.658 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:43.658 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.247603+0000", 2024-09-21T22:06:43.659 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013171600000000001 2024-09-21T22:06:43.659 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.659 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.659 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:43.659 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.247656+0000", 2024-09-21T22:06:43.659 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.3529999999999997e-05 2024-09-21T22:06:43.659 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.659 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.659 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:43.659 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.251225+0000", 2024-09-21T22:06:43.659 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00356865 2024-09-21T22:06:43.659 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.659 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.659 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:43.659 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.251370+0000", 2024-09-21T22:06:43.660 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00014518500000000001 2024-09-21T22:06:43.660 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:43.660 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:43.660 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:43.660 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.660 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.660 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:69 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'43, mlcod=14'39)", 2024-09-21T22:06:43.660 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.248839+0000", 2024-09-21T22:06:43.660 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 41.372550199999999, 2024-09-21T22:06:43.660 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0034954470000000001, 2024-09-21T22:06:43.660 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:43.660 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:43.660 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:43.660 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:43.661 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.661 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:43.661 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.248839+0000", 2024-09-21T22:06:43.661 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:43.661 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.661 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.661 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:43.661 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.248839+0000", 2024-09-21T22:06:43.661 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:43.661 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.661 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.661 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:43.661 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.248841+0000", 2024-09-21T22:06:43.661 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6759999999999999e-06 2024-09-21T22:06:43.662 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.662 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.662 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:43.662 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.249058+0000", 2024-09-21T22:06:43.662 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000217455 2024-09-21T22:06:43.662 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.662 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.662 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:43.662 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.249085+0000", 2024-09-21T22:06:43.662 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.7002999999999999e-05 2024-09-21T22:06:43.662 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.662 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.662 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:43.662 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.249099+0000", 2024-09-21T22:06:43.663 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.402e-05 2024-09-21T22:06:43.663 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.663 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.663 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:43.663 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.249241+0000", 2024-09-21T22:06:43.663 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00014194899999999999 2024-09-21T22:06:43.663 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.663 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.663 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:43.663 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.249298+0000", 2024-09-21T22:06:43.663 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.6687000000000001e-05 2024-09-21T22:06:43.663 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.663 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.663 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:43.663 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.252266+0000", 2024-09-21T22:06:43.664 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.002968348 2024-09-21T22:06:43.664 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.664 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.664 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:43.664 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.252334+0000", 2024-09-21T22:06:43.664 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.8308999999999999e-05 2024-09-21T22:06:43.664 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:43.664 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:43.664 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:43.664 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.664 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.664 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:71 1.0 e14/13 1:30306672:devicehealth::main.db.0000000000000000:head v 14'45, mlcod=14'43)", 2024-09-21T22:06:43.664 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.256068+0000", 2024-09-21T22:06:43.664 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 41.365320728, 2024-09-21T22:06:43.664 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0041712579999999997, 2024-09-21T22:06:43.665 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:43.665 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:43.665 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:43.665 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:43.665 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.665 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:43.665 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.256068+0000", 2024-09-21T22:06:43.665 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:43.665 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.665 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.665 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:43.665 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.256068+0000", 2024-09-21T22:06:43.665 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:43.665 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.666 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.666 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:43.666 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.256070+0000", 2024-09-21T22:06:43.666 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.61e-06 2024-09-21T22:06:43.666 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.666 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.666 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:43.666 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.256129+0000", 2024-09-21T22:06:43.666 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.8702999999999997e-05 2024-09-21T22:06:43.666 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.666 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.666 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:43.666 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.256158+0000", 2024-09-21T22:06:43.666 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9533e-05 2024-09-21T22:06:43.667 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.667 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.667 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:43.667 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.256173+0000", 2024-09-21T22:06:43.667 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4267000000000001e-05 2024-09-21T22:06:43.667 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.667 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.667 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:43.667 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.256304+0000", 2024-09-21T22:06:43.667 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000131352 2024-09-21T22:06:43.667 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.667 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.667 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:43.668 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.256357+0000", 2024-09-21T22:06:43.668 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.3492999999999998e-05 2024-09-21T22:06:43.668 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.668 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.668 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:43.668 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.260147+0000", 2024-09-21T22:06:43.668 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0037900410000000001 2024-09-21T22:06:43.668 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.668 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.668 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:43.668 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.260240+0000", 2024-09-21T22:06:43.668 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.2258999999999999e-05 2024-09-21T22:06:43.668 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:43.668 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:43.669 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:43.669 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.669 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.669 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:72 1.0 e14/13 1:30306672:devicehealth::main.db.0000000000000000:head v 14'46, mlcod=14'43)", 2024-09-21T22:06:43.669 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.258358+0000", 2024-09-21T22:06:43.669 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 41.363030735000002, 2024-09-21T22:06:43.669 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0041492919999999997, 2024-09-21T22:06:43.669 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:43.669 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:43.669 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:43.669 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:43.669 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.669 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:43.669 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.258358+0000", 2024-09-21T22:06:43.670 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:43.670 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.670 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.670 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:43.670 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.258358+0000", 2024-09-21T22:06:43.670 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:43.670 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.670 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.670 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:43.670 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.258360+0000", 2024-09-21T22:06:43.670 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.623e-06 2024-09-21T22:06:43.670 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.670 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.671 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:43.671 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.258415+0000", 2024-09-21T22:06:43.671 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.4756000000000002e-05 2024-09-21T22:06:43.671 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.671 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.671 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:43.671 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.258444+0000", 2024-09-21T22:06:43.671 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9164000000000001e-05 2024-09-21T22:06:43.671 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.671 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.671 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:43.671 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.258459+0000", 2024-09-21T22:06:43.671 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4996e-05 2024-09-21T22:06:43.672 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.672 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.672 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:43.672 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.258637+0000", 2024-09-21T22:06:43.672 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00017796500000000001 2024-09-21T22:06:43.672 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.672 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.672 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:43.672 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.258695+0000", 2024-09-21T22:06:43.672 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.8207e-05 2024-09-21T22:06:43.672 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.672 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.672 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:43.672 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.262421+0000", 2024-09-21T22:06:43.673 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0037261550000000001 2024-09-21T22:06:43.673 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.673 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.673 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:43.673 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.262508+0000", 2024-09-21T22:06:43.673 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.6426000000000002e-05 2024-09-21T22:06:43.673 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:43.673 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:43.673 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:43.673 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.673 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.673 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:87 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'57, mlcod=14'52)", 2024-09-21T22:06:43.674 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.290115+0000", 2024-09-21T22:06:43.674 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 41.331274290000003, 2024-09-21T22:06:43.674 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0030478200000000001, 2024-09-21T22:06:43.674 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:43.674 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:43.674 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:43.674 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:43.674 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.674 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:43.674 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.290115+0000", 2024-09-21T22:06:43.674 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:43.674 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.674 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.675 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:43.675 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.290115+0000", 2024-09-21T22:06:43.675 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:43.675 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.675 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.675 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:43.675 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.290116+0000", 2024-09-21T22:06:43.675 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3829999999999999e-06 2024-09-21T22:06:43.675 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.675 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.675 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:43.675 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.290191+0000", 2024-09-21T22:06:43.675 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.4562999999999997e-05 2024-09-21T22:06:43.675 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.676 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.676 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:43.676 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.290216+0000", 2024-09-21T22:06:43.676 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.565e-05 2024-09-21T22:06:43.676 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.676 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.676 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:43.676 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.290236+0000", 2024-09-21T22:06:43.676 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9326000000000001e-05 2024-09-21T22:06:43.676 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.676 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.676 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:43.676 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.290340+0000", 2024-09-21T22:06:43.676 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0001047 2024-09-21T22:06:43.677 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.677 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.677 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:43.677 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.290377+0000", 2024-09-21T22:06:43.677 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.6356000000000002e-05 2024-09-21T22:06:43.677 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.677 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.677 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:43.677 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.293096+0000", 2024-09-21T22:06:43.677 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.002718996 2024-09-21T22:06:43.677 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.677 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.677 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:43.677 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.293163+0000", 2024-09-21T22:06:43.677 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.6846000000000003e-05 2024-09-21T22:06:43.678 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:43.678 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:43.678 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:43.678 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.678 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.678 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:114 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'76, mlcod=14'73)", 2024-09-21T22:06:43.678 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.490030+0000", 2024-09-21T22:06:43.678 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 41.131358851000002, 2024-09-21T22:06:43.678 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.004227873, 2024-09-21T22:06:43.678 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:43.678 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:43.678 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:43.678 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:43.678 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.679 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:43.679 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.490030+0000", 2024-09-21T22:06:43.679 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:43.679 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.679 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.679 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:43.679 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.490030+0000", 2024-09-21T22:06:43.679 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:43.679 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.679 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.679 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:43.679 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.490031+0000", 2024-09-21T22:06:43.679 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.3300000000000001e-07 2024-09-21T22:06:43.679 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.679 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.680 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:43.680 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.490042+0000", 2024-09-21T22:06:43.680 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.1095e-05 2024-09-21T22:06:43.680 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.680 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.680 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:43.680 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.490053+0000", 2024-09-21T22:06:43.680 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0501e-05 2024-09-21T22:06:43.680 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.680 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.680 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:43.680 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.490060+0000", 2024-09-21T22:06:43.680 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.9829999999999999e-06 2024-09-21T22:06:43.680 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.681 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.681 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:43.681 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.490107+0000", 2024-09-21T22:06:43.681 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.7278999999999999e-05 2024-09-21T22:06:43.681 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.681 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.681 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:43.681 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.490124+0000", 2024-09-21T22:06:43.681 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7190999999999999e-05 2024-09-21T22:06:43.681 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.681 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.681 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:43.681 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.494229+0000", 2024-09-21T22:06:43.681 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0041050440000000004 2024-09-21T22:06:43.682 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.682 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.682 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:43.682 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.494258+0000", 2024-09-21T22:06:43.682 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9047000000000002e-05 2024-09-21T22:06:43.682 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:43.682 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:43.682 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:43.682 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.682 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.682 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:115 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'77, mlcod=14'73)", 2024-09-21T22:06:43.682 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.491380+0000", 2024-09-21T22:06:43.682 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 41.130009229000002, 2024-09-21T22:06:43.682 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0031550889999999998, 2024-09-21T22:06:43.683 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:43.683 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:43.683 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:43.683 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:43.683 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.683 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:43.683 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.491380+0000", 2024-09-21T22:06:43.683 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:43.683 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.683 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.683 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:43.683 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.491380+0000", 2024-09-21T22:06:43.683 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:43.683 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.684 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.684 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:43.684 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.491381+0000", 2024-09-21T22:06:43.684 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.1399999999999996e-07 2024-09-21T22:06:43.684 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.684 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.684 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:43.684 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.491435+0000", 2024-09-21T22:06:43.684 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.3797e-05 2024-09-21T22:06:43.684 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.684 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.684 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:43.684 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.491447+0000", 2024-09-21T22:06:43.684 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2642e-05 2024-09-21T22:06:43.684 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.685 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.685 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:43.685 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.491453+0000", 2024-09-21T22:06:43.685 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.4820000000000002e-06 2024-09-21T22:06:43.685 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.685 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.685 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:43.685 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.491502+0000", 2024-09-21T22:06:43.685 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.9038e-05 2024-09-21T22:06:43.685 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.685 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.685 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:43.685 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.491514+0000", 2024-09-21T22:06:43.685 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2483999999999999e-05 2024-09-21T22:06:43.686 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.686 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.686 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:43.686 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.494486+0000", 2024-09-21T22:06:43.686 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.002971338 2024-09-21T22:06:43.686 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.686 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.686 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:43.686 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.494535+0000", 2024-09-21T22:06:43.686 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.9493999999999997e-05 2024-09-21T22:06:43.686 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:43.686 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:43.686 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:43.687 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.687 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.687 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:118 1.0 e14/13 1:30306672:devicehealth::main.db.0000000000000000:head v 14'80, mlcod=14'78)", 2024-09-21T22:06:43.687 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.499045+0000", 2024-09-21T22:06:43.687 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 41.122344226999999, 2024-09-21T22:06:43.687 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0029889589999999998, 2024-09-21T22:06:43.687 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:43.687 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:43.687 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:43.687 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:43.687 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.687 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:43.687 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.499045+0000", 2024-09-21T22:06:43.687 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:43.688 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.688 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.688 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:43.688 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.499045+0000", 2024-09-21T22:06:43.688 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:43.688 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.688 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.688 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:43.688 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.499046+0000", 2024-09-21T22:06:43.688 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.0299999999999998e-07 2024-09-21T22:06:43.688 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.688 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.688 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:43.688 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.499110+0000", 2024-09-21T22:06:43.689 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.4688000000000001e-05 2024-09-21T22:06:43.689 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.689 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.689 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:43.689 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.499125+0000", 2024-09-21T22:06:43.689 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4387e-05 2024-09-21T22:06:43.689 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.689 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.689 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:43.689 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.499131+0000", 2024-09-21T22:06:43.689 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.2129999999999996e-06 2024-09-21T22:06:43.689 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.689 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.689 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:43.690 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.499255+0000", 2024-09-21T22:06:43.690 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00012424199999999999 2024-09-21T22:06:43.690 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.690 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.690 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:43.690 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.499312+0000", 2024-09-21T22:06:43.690 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.6709999999999997e-05 2024-09-21T22:06:43.690 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.690 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.690 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:43.690 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.502005+0000", 2024-09-21T22:06:43.690 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0026927169999999999 2024-09-21T22:06:43.690 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.690 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.691 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:43.691 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.502034+0000", 2024-09-21T22:06:43.691 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9198999999999999e-05 2024-09-21T22:06:43.691 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:43.691 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:43.691 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:43.691 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.691 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.691 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:136 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'94, mlcod=14'89)", 2024-09-21T22:06:43.691 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.903810+0000", 2024-09-21T22:06:43.691 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 40.717579256, 2024-09-21T22:06:43.691 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0062746829999999996, 2024-09-21T22:06:43.692 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:43.692 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:43.692 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:43.692 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:43.692 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.692 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:43.692 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.903810+0000", 2024-09-21T22:06:43.692 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:43.692 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.692 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.692 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:43.692 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.903810+0000", 2024-09-21T22:06:43.692 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:43.692 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.692 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.693 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:43.693 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.903810+0000", 2024-09-21T22:06:43.693 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.03e-07 2024-09-21T22:06:43.693 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.693 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.693 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:43.693 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.903814+0000", 2024-09-21T22:06:43.693 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.676e-06 2024-09-21T22:06:43.693 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.693 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.693 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:43.693 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.903822+0000", 2024-09-21T22:06:43.693 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.2879999999999993e-06 2024-09-21T22:06:43.693 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.694 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.694 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:43.694 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.903829+0000", 2024-09-21T22:06:43.694 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.2920000000000001e-06 2024-09-21T22:06:43.694 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.694 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.694 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:43.694 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.903879+0000", 2024-09-21T22:06:43.694 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.0253999999999997e-05 2024-09-21T22:06:43.694 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.694 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.694 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:43.694 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.903897+0000", 2024-09-21T22:06:43.695 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8468999999999999e-05 2024-09-21T22:06:43.695 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.695 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.695 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:43.695 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.910044+0000", 2024-09-21T22:06:43.695 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0061468950000000003 2024-09-21T22:06:43.695 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.695 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.695 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:43.695 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.910085+0000", 2024-09-21T22:06:43.695 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.0405999999999999e-05 2024-09-21T22:06:43.695 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:43.695 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:43.695 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:43.696 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.696 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.696 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:137 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'95, mlcod=14'90)", 2024-09-21T22:06:43.696 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.904439+0000", 2024-09-21T22:06:43.696 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 40.71694986, 2024-09-21T22:06:43.696 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0072625019999999997, 2024-09-21T22:06:43.696 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:43.696 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:43.696 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:43.696 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:43.696 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.696 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:43.696 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.904439+0000", 2024-09-21T22:06:43.696 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:43.697 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.697 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.697 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:43.697 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.904439+0000", 2024-09-21T22:06:43.697 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:43.697 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.697 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.697 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:43.697 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.904440+0000", 2024-09-21T22:06:43.697 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8400000000000005e-07 2024-09-21T22:06:43.697 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.697 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.697 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:43.697 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.904498+0000", 2024-09-21T22:06:43.697 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.8320999999999999e-05 2024-09-21T22:06:43.698 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.698 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.698 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:43.698 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.904507+0000", 2024-09-21T22:06:43.698 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.7790000000000001e-06 2024-09-21T22:06:43.698 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.698 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.698 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:43.698 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.904511+0000", 2024-09-21T22:06:43.698 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.9700000000000001e-06 2024-09-21T22:06:43.698 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.698 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.698 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:43.698 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.904551+0000", 2024-09-21T22:06:43.699 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.9910000000000002e-05 2024-09-21T22:06:43.699 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.699 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.699 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:43.699 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.904567+0000", 2024-09-21T22:06:43.699 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5903999999999999e-05 2024-09-21T22:06:43.699 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.699 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.699 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:43.699 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.911666+0000", 2024-09-21T22:06:43.699 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0070996929999999998 2024-09-21T22:06:43.699 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.699 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.699 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:43.700 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.911702+0000", 2024-09-21T22:06:43.700 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.5441000000000002e-05 2024-09-21T22:06:43.700 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:43.700 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:43.700 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:43.700 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.700 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.700 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:138 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'96, mlcod=14'90)", 2024-09-21T22:06:43.700 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.905402+0000", 2024-09-21T22:06:43.700 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 40.715986882999999, 2024-09-21T22:06:43.700 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0063366890000000004, 2024-09-21T22:06:43.700 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:43.700 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:43.700 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:43.700 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:43.701 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.701 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:43.701 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905402+0000", 2024-09-21T22:06:43.701 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:43.701 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.701 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.701 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:43.701 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905402+0000", 2024-09-21T22:06:43.701 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:43.701 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.701 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.701 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:43.701 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905403+0000", 2024-09-21T22:06:43.702 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.9200000000000002e-07 2024-09-21T22:06:43.702 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.702 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.702 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:43.702 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905406+0000", 2024-09-21T22:06:43.702 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.326e-06 2024-09-21T22:06:43.702 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.702 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.702 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:43.702 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905413+0000", 2024-09-21T22:06:43.702 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.3320000000000003e-06 2024-09-21T22:06:43.702 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.702 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.702 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:43.703 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905417+0000", 2024-09-21T22:06:43.703 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.7139999999999999e-06 2024-09-21T22:06:43.703 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.703 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.703 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:43.703 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.910115+0000", 2024-09-21T22:06:43.703 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0046983939999999998 2024-09-21T22:06:43.703 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.703 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.703 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:43.703 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.910140+0000", 2024-09-21T22:06:43.703 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4689999999999999e-05 2024-09-21T22:06:43.703 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.703 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.704 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:43.704 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.911706+0000", 2024-09-21T22:06:43.704 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0015658479999999999 2024-09-21T22:06:43.704 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.704 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.704 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:43.704 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.911739+0000", 2024-09-21T22:06:43.704 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.2993000000000001e-05 2024-09-21T22:06:43.704 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:43.704 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:43.704 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:43.704 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.704 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.704 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:139 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'97, mlcod=14'90)", 2024-09-21T22:06:43.705 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.905983+0000", 2024-09-21T22:06:43.705 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 40.715405965000002, 2024-09-21T22:06:43.705 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0057906030000000001, 2024-09-21T22:06:43.705 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:43.705 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:43.705 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:43.705 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:43.705 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.705 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:43.705 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905983+0000", 2024-09-21T22:06:43.705 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:43.705 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.705 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.705 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:43.706 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905983+0000", 2024-09-21T22:06:43.706 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:43.706 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.706 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.706 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:43.706 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905984+0000", 2024-09-21T22:06:43.706 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.15e-07 2024-09-21T22:06:43.706 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.706 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.706 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:43.706 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905986+0000", 2024-09-21T22:06:43.706 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.852e-06 2024-09-21T22:06:43.706 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.706 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.707 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:43.707 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905993+0000", 2024-09-21T22:06:43.707 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.652e-06 2024-09-21T22:06:43.707 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.707 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.707 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:43.707 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905996+0000", 2024-09-21T22:06:43.707 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.3519999999999998e-06 2024-09-21T22:06:43.707 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.707 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.707 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:43.707 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.910519+0000", 2024-09-21T22:06:43.707 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0045230820000000003 2024-09-21T22:06:43.707 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.707 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.708 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:43.708 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.910548+0000", 2024-09-21T22:06:43.708 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8399999999999999e-05 2024-09-21T22:06:43.708 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.708 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.708 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:43.708 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.911742+0000", 2024-09-21T22:06:43.708 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011939909999999999 2024-09-21T22:06:43.708 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.708 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.708 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:43.708 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.911774+0000", 2024-09-21T22:06:43.708 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.1958999999999997e-05 2024-09-21T22:06:43.708 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:43.709 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:43.709 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:43.709 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.709 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.709 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:140 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'98, mlcod=14'90)", 2024-09-21T22:06:43.709 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.906441+0000", 2024-09-21T22:06:43.709 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 40.714947836, 2024-09-21T22:06:43.709 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0057052989999999996, 2024-09-21T22:06:43.709 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:43.709 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:43.709 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:43.709 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:43.709 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.709 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:43.710 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.906441+0000", 2024-09-21T22:06:43.710 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:43.710 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.710 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.710 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:43.710 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.906441+0000", 2024-09-21T22:06:43.710 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:43.710 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.710 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.710 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:43.710 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.906442+0000", 2024-09-21T22:06:43.710 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.7e-07 2024-09-21T22:06:43.710 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.710 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.711 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:43.711 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.906511+0000", 2024-09-21T22:06:43.711 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.8857999999999997e-05 2024-09-21T22:06:43.711 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.711 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.711 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:43.711 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.906518+0000", 2024-09-21T22:06:43.711 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.1690000000000003e-06 2024-09-21T22:06:43.711 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.711 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.711 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:43.711 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.906521+0000", 2024-09-21T22:06:43.712 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.4829999999999999e-06 2024-09-21T22:06:43.712 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.712 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.712 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:43.712 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.910939+0000", 2024-09-21T22:06:43.712 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0044176809999999997 2024-09-21T22:06:43.712 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.712 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.712 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:43.712 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.910959+0000", 2024-09-21T22:06:43.712 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9898000000000002e-05 2024-09-21T22:06:43.712 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.712 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.713 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:43.713 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.912113+0000", 2024-09-21T22:06:43.713 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001153811 2024-09-21T22:06:43.713 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:43.713 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:43.713 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:43.713 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.912147+0000", 2024-09-21T22:06:43.713 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.4029000000000002e-05 2024-09-21T22:06:43.713 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:43.713 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:43.713 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:43.713 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:43.713 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:43.713 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:06:43.715 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T22:06:43.718 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:06:43.712+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:06:43.816 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:06:43.812+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:06:43.851 INFO:teuthology.orchestra.run.smithi039.stderr:noscrub is unset 2024-09-21T22:06:43.856 INFO:teuthology.orchestra.run.smithi039.stderr:marked in osd.0. 2024-09-21T22:06:43.879 INFO:tasks.thrashosds.thrasher:Added osd 0 2024-09-21T22:06:43.895 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:06:43.896 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:06:43.896 INFO:teuthology.orchestra.run.smithi039.stdout: "num_ops": 0 2024-09-21T22:06:43.896 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:06:43.909 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T22:06:43.916 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:06:43.912+0000 7fc8d39f7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25294) UID: 0 2024-09-21T22:06:44.017 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:06:44.013+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:06:44.067 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:06:44.067 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:06:44.067 INFO:teuthology.orchestra.run.smithi039.stdout: "complaint_time": 30, 2024-09-21T22:06:44.067 INFO:teuthology.orchestra.run.smithi039.stdout: "num_blocked_ops": 0 2024-09-21T22:06:44.067 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:06:44.082 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T22:06:44.118 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:06:44.112+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:06:44.219 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:06:44.217+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:06:44.250 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:06:44.250 INFO:teuthology.orchestra.run.smithi039.stdout: "size": 20, 2024-09-21T22:06:44.250 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 600, 2024-09-21T22:06:44.250 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [ 2024-09-21T22:06:44.250 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:44.250 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4224.0:5 2.2 e18/16 2:5cea7035:::rbd_info:head v 18'1, mlcod=0'0)", 2024-09-21T22:06:44.250 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:06.114955+0000", 2024-09-21T22:06:44.251 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 38.135685203999998, 2024-09-21T22:06:44.251 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0017716170000000001, 2024-09-21T22:06:44.251 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:44.251 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:44.251 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:44.251 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:44.251 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:44.251 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:44.251 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:06.114955+0000", 2024-09-21T22:06:44.251 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:44.251 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:44.252 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:44.252 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:44.252 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:06.114955+0000", 2024-09-21T22:06:44.252 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:44.252 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:44.252 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:44.252 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:44.252 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:06.114956+0000", 2024-09-21T22:06:44.252 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.9299999999999997e-07 2024-09-21T22:06:44.252 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:44.252 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:44.252 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:44.252 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:06.114964+0000", 2024-09-21T22:06:44.252 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.9840000000000001e-06 2024-09-21T22:06:44.253 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:44.253 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:44.253 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:44.253 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:06.115001+0000", 2024-09-21T22:06:44.253 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.7719999999999998e-05 2024-09-21T22:06:44.253 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:44.253 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:44.253 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:44.253 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:06.115019+0000", 2024-09-21T22:06:44.253 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7943000000000001e-05 2024-09-21T22:06:44.253 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:44.253 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:44.253 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:44.253 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:06.115168+0000", 2024-09-21T22:06:44.253 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000148589 2024-09-21T22:06:44.254 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:44.254 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:44.254 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:44.254 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:06.115225+0000", 2024-09-21T22:06:44.254 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.6975999999999998e-05 2024-09-21T22:06:44.254 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:44.254 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:44.254 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:44.254 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:06.116585+0000", 2024-09-21T22:06:44.254 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00136038 2024-09-21T22:06:44.254 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:44.254 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:44.254 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:44.255 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:06.116726+0000", 2024-09-21T22:06:44.255 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00014133200000000001 2024-09-21T22:06:44.255 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:44.255 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:44.255 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:44.255 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:44.255 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:44.255 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4224.0:7 2.2 e19/16 2:5cea7035:::rbd_info:head v 19'2, mlcod=0'0)", 2024-09-21T22:06:44.255 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:07.129567+0000", 2024-09-21T22:06:44.255 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 37.121073195000001, 2024-09-21T22:06:44.255 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0018460239999999999, 2024-09-21T22:06:44.255 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:44.255 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:44.256 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:44.256 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:44.256 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:44.256 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:44.256 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:07.129567+0000", 2024-09-21T22:06:44.256 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:44.256 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:44.256 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:44.256 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:44.256 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:07.129567+0000", 2024-09-21T22:06:44.256 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:44.256 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:44.256 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:44.256 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:44.257 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:07.129567+0000", 2024-09-21T22:06:44.257 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.9499999999999998e-07 2024-09-21T22:06:44.257 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:44.257 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:44.257 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:44.257 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:07.129580+0000", 2024-09-21T22:06:44.257 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2335e-05 2024-09-21T22:06:44.257 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:44.257 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:44.257 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:44.257 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:07.129606+0000", 2024-09-21T22:06:44.257 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.6528000000000001e-05 2024-09-21T22:06:44.257 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:44.258 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:44.258 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:44.258 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:07.129628+0000", 2024-09-21T22:06:44.258 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1549999999999999e-05 2024-09-21T22:06:44.258 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:44.258 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:44.258 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:44.258 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:07.129744+0000", 2024-09-21T22:06:44.258 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000116551 2024-09-21T22:06:44.258 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:44.258 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:44.258 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:44.258 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:07.129780+0000", 2024-09-21T22:06:44.258 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.5447000000000001e-05 2024-09-21T22:06:44.259 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:44.259 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:44.259 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:44.259 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:07.131348+0000", 2024-09-21T22:06:44.259 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001568631 2024-09-21T22:06:44.259 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:44.259 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:44.259 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:44.259 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:07.131413+0000", 2024-09-21T22:06:44.259 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.4586999999999997e-05 2024-09-21T22:06:44.259 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:44.259 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:44.259 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:44.259 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:44.260 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:44.260 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:06:44.264 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T22:06:44.355 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:06:44.353+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:06:44.421 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:06:44.416+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:06:44.434 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:06:44.434 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:06:44.434 INFO:teuthology.orchestra.run.smithi039.stdout: "num_ops": 0 2024-09-21T22:06:44.434 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:06:44.449 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T22:06:44.534 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:06:44.533+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:06:44.616 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:06:44.616 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:06:44.616 INFO:teuthology.orchestra.run.smithi039.stdout: "complaint_time": 30, 2024-09-21T22:06:44.616 INFO:teuthology.orchestra.run.smithi039.stdout: "num_blocked_ops": 0 2024-09-21T22:06:44.616 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:06:44.624 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:06:44.621+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:06:44.630 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T22:06:44.724 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:06:44.720+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:06:44.792 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:06:44.793 INFO:teuthology.orchestra.run.smithi039.stdout: "size": 20, 2024-09-21T22:06:44.793 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 600, 2024-09-21T22:06:44.793 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [ 2024-09-21T22:06:44.793 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:44.793 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4224.0:1 2.3 e17/16 2:c4c92e5a:::rbd_trash:head v 17'1, mlcod=0'0)", 2024-09-21T22:06:44.793 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:05.107810+0000", 2024-09-21T22:06:44.793 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 39.685345390000002, 2024-09-21T22:06:44.793 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0017679200000000001, 2024-09-21T22:06:44.793 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:44.793 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:44.793 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:44.793 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:44.794 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:44.794 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:44.794 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.107810+0000", 2024-09-21T22:06:44.794 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:44.794 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:44.794 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:44.794 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:44.794 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.107810+0000", 2024-09-21T22:06:44.794 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:44.794 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:44.794 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:44.794 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:44.794 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.107811+0000", 2024-09-21T22:06:44.795 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.6700000000000003e-07 2024-09-21T22:06:44.795 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:44.795 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:44.795 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:44.795 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.107822+0000", 2024-09-21T22:06:44.795 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0443e-05 2024-09-21T22:06:44.795 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:44.795 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:44.795 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:44.795 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.107853+0000", 2024-09-21T22:06:44.795 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.1046000000000001e-05 2024-09-21T22:06:44.795 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:44.795 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:44.795 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:44.796 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.107876+0000", 2024-09-21T22:06:44.796 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3547e-05 2024-09-21T22:06:44.796 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:44.796 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:44.796 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:44.796 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.108002+0000", 2024-09-21T22:06:44.796 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000125599 2024-09-21T22:06:44.796 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:44.796 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:44.796 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:44.796 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.108069+0000", 2024-09-21T22:06:44.796 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.7162999999999994e-05 2024-09-21T22:06:44.796 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:44.796 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:44.796 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:44.797 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.109385+0000", 2024-09-21T22:06:44.797 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0013162899999999999 2024-09-21T22:06:44.797 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:44.797 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:44.797 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:44.797 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.109578+0000", 2024-09-21T22:06:44.797 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000193065 2024-09-21T22:06:44.797 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:44.797 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:44.797 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:44.797 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:44.797 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:44.797 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:06:44.807 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T22:06:44.855 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:06:44.852+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:06:44.926 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:06:44.920+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:06:44.946 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:06:44.946 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:06:44.946 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-09-21T22:06:44.946 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:06:44.961 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T22:06:45.027 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:06:45.024+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:06:45.037 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T22:06:45.127 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:06:45.124+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:06:45.131 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:06:45.131 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:06:45.131 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-21T22:06:45.131 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-21T22:06:45.131 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:06:45.146 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T22:06:45.193 INFO:teuthology.orchestra.run.smithi039.stdout:osd.0: {} 2024-09-21T22:06:45.194 INFO:teuthology.orchestra.run.smithi039.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T22:06:45.210 INFO:teuthology.orchestra.run.smithi039.stdout:osd.1: {} 2024-09-21T22:06:45.210 INFO:teuthology.orchestra.run.smithi039.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T22:06:45.227 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:06:45.224+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:06:45.229 INFO:teuthology.orchestra.run.smithi039.stdout:osd.2: {} 2024-09-21T22:06:45.229 INFO:teuthology.orchestra.run.smithi039.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T22:06:45.247 INFO:teuthology.orchestra.run.smithi039.stdout:osd.3: {} 2024-09-21T22:06:45.247 INFO:teuthology.orchestra.run.smithi039.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T22:06:45.268 INFO:teuthology.orchestra.run.smithi039.stdout:osd.4: {} 2024-09-21T22:06:45.268 INFO:teuthology.orchestra.run.smithi039.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T22:06:45.290 INFO:teuthology.orchestra.run.smithi039.stdout:osd.5: {} 2024-09-21T22:06:45.290 INFO:teuthology.orchestra.run.smithi039.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T22:06:45.313 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:45.314 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:45.329 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T22:06:45.330 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:06:45.324+0000 7fc8d39f7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25294) UID: 0 2024-09-21T22:06:45.430 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:06:45.428+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:06:45.494 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:45.494 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:45.508 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T22:06:45.531 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:06:45.528+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:06:45.631 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:06:45.628+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:06:45.634 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:45.634 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:45.649 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T22:06:45.733 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:06:45.728+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:06:45.777 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:45.777 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:45.791 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T22:06:45.834 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:06:45.828+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:06:45.877 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-21T22:06:45.935 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:06:45.932+0000 7fc8d39f7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25294) UID: 0 2024-09-21T22:06:45.962 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:45.962 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:45.978 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T22:06:46.035 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:06:46.032+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:06:46.136 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:06:46.132+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:06:46.153 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:46.153 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:46.168 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T22:06:46.237 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:06:46.232+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:06:46.295 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:46.296 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:46.310 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T22:06:46.351 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:06:46.348+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:06:46.440 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:06:46.436+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:06:46.442 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:46.442 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:46.457 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T22:06:46.541 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:06:46.536+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:06:46.583 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:46.583 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:46.599 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T22:06:46.642 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:06:46.640+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:06:46.718 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:46.718 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:46.733 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T22:06:46.775 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:06:46.772+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:06:46.845 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:06:46.840+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:06:46.899 INFO:teuthology.orchestra.run.smithi039.stderr:nodeep-scrub is unset 2024-09-21T22:06:46.904 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:46.904 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:46.922 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T22:06:46.946 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:06:46.940+0000 7fc8d39f7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25294) UID: 0 2024-09-21T22:06:47.047 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:06:47.044+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:06:47.083 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:47.083 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:47.098 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T22:06:47.148 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:06:47.144+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:06:47.217 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:47.217 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:47.231 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T22:06:47.249 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:06:47.244+0000 7fc8d39f7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25294) UID: 0 2024-09-21T22:06:47.309 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T22:06:47.349 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:06:47.344+0000 7fc8d39f7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25294) UID: 0 2024-09-21T22:06:47.353 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:47.353 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:47.368 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T22:06:47.458 INFO:teuthology.orchestra.run.smithi039.stdout:osd.0: {} 2024-09-21T22:06:47.459 INFO:teuthology.orchestra.run.smithi039.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T22:06:47.459 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:06:47.456+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:06:47.475 INFO:teuthology.orchestra.run.smithi039.stdout:osd.1: {} 2024-09-21T22:06:47.475 INFO:teuthology.orchestra.run.smithi039.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T22:06:47.492 INFO:teuthology.orchestra.run.smithi039.stdout:osd.2: {} 2024-09-21T22:06:47.492 INFO:teuthology.orchestra.run.smithi039.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T22:06:47.529 INFO:teuthology.orchestra.run.smithi039.stdout:osd.3: {} 2024-09-21T22:06:47.529 INFO:teuthology.orchestra.run.smithi039.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T22:06:47.551 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:06:47.548+0000 7fc8d39f7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25294) UID: 0 2024-09-21T22:06:47.563 INFO:teuthology.orchestra.run.smithi039.stdout:osd.4: {} 2024-09-21T22:06:47.564 INFO:teuthology.orchestra.run.smithi039.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T22:06:47.568 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:06:47.568 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:06:47.568 INFO:teuthology.orchestra.run.smithi039.stdout: "complaint_time": 30, 2024-09-21T22:06:47.568 INFO:teuthology.orchestra.run.smithi039.stdout: "num_blocked_ops": 0 2024-09-21T22:06:47.568 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:06:47.583 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T22:06:47.584 INFO:teuthology.orchestra.run.smithi039.stdout:osd.5: {} 2024-09-21T22:06:47.585 INFO:teuthology.orchestra.run.smithi039.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T22:06:47.651 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:06:47.648+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:06:47.719 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:06:47.720 INFO:teuthology.orchestra.run.smithi039.stdout: "size": 20, 2024-09-21T22:06:47.720 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 600, 2024-09-21T22:06:47.720 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [ 2024-09-21T22:06:47.720 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:47.720 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4224.0:1 2.3 e17/16 2:c4c92e5a:::rbd_trash:head v 17'1, mlcod=0'0)", 2024-09-21T22:06:47.720 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:05.107810+0000", 2024-09-21T22:06:47.720 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 42.612124885999997, 2024-09-21T22:06:47.720 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0017679200000000001, 2024-09-21T22:06:47.720 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:47.720 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:47.720 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:47.720 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:47.720 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:47.721 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:47.721 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.107810+0000", 2024-09-21T22:06:47.721 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:47.721 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:47.721 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:47.721 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:47.721 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.107810+0000", 2024-09-21T22:06:47.721 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:47.721 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:47.721 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:47.721 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:47.721 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.107811+0000", 2024-09-21T22:06:47.721 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.6700000000000003e-07 2024-09-21T22:06:47.721 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:47.721 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:47.722 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:47.722 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.107822+0000", 2024-09-21T22:06:47.722 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0443e-05 2024-09-21T22:06:47.722 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:47.722 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:47.722 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:47.722 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.107853+0000", 2024-09-21T22:06:47.722 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.1046000000000001e-05 2024-09-21T22:06:47.722 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:47.722 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:47.722 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:47.722 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.107876+0000", 2024-09-21T22:06:47.723 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3547e-05 2024-09-21T22:06:47.723 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:47.723 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:47.723 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:47.723 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.108002+0000", 2024-09-21T22:06:47.723 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000125599 2024-09-21T22:06:47.723 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:47.723 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:47.723 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:47.723 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.108069+0000", 2024-09-21T22:06:47.723 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.7162999999999994e-05 2024-09-21T22:06:47.723 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:47.723 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:47.723 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:47.723 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.109385+0000", 2024-09-21T22:06:47.723 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0013162899999999999 2024-09-21T22:06:47.724 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:47.724 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:47.724 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:47.724 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.109578+0000", 2024-09-21T22:06:47.724 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000193065 2024-09-21T22:06:47.724 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:47.724 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:47.724 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:47.724 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:47.724 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:47.724 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:06:47.735 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T22:06:47.779 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:06:47.776+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:06:47.854 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:06:47.848+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:06:47.871 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:06:47.871 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:06:47.871 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-09-21T22:06:47.871 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:06:47.886 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T22:06:47.954 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:06:47.948+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:06:48.013 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:06:48.014 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:06:48.014 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-21T22:06:48.014 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-21T22:06:48.014 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:06:48.028 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T22:06:48.071 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:06:48.068+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:06:48.157 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:06:48.156+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:06:48.201 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:06:48.201 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-21T22:06:48.201 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-21T22:06:48.201 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-21T22:06:48.201 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.201 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:06:48.202 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.036355+0000", 2024-09-21T22:06:48.202 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 46.160360054999998, 2024-09-21T22:06:48.202 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.062441934999999997, 2024-09-21T22:06:48.202 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:48.202 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:48.202 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:48.202 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:48.202 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:06:48.202 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:06:48.202 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1 2024-09-21T22:06:48.202 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.202 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:48.202 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.202 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:48.203 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036355+0000", 2024-09-21T22:06:48.203 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:48.203 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.203 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.203 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:48.203 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036355+0000", 2024-09-21T22:06:48.203 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:48.203 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.203 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.203 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:48.204 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036357+0000", 2024-09-21T22:06:48.204 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.446e-06 2024-09-21T22:06:48.204 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.204 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.204 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:48.204 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036373+0000", 2024-09-21T22:06:48.204 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5447e-05 2024-09-21T22:06:48.204 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.204 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.204 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:48.204 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036399+0000", 2024-09-21T22:06:48.204 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6423000000000001e-05 2024-09-21T22:06:48.204 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.204 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.204 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:48.205 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036420+0000", 2024-09-21T22:06:48.205 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1129999999999999e-05 2024-09-21T22:06:48.205 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.205 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.205 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:48.205 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036557+0000", 2024-09-21T22:06:48.205 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013686899999999999 2024-09-21T22:06:48.205 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.205 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.205 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-21T22:06:48.205 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036585+0000", 2024-09-21T22:06:48.205 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8362999999999999e-05 2024-09-21T22:06:48.205 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.205 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.206 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:48.206 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.095068+0000", 2024-09-21T22:06:48.206 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058482972000000001 2024-09-21T22:06:48.206 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.206 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.206 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:48.206 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.095516+0000", 2024-09-21T22:06:48.206 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00044802999999999998 2024-09-21T22:06:48.206 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.206 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.206 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:06:48.207 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.095869+0000", 2024-09-21T22:06:48.207 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000352104 2024-09-21T22:06:48.207 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.207 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.207 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:48.207 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.097055+0000", 2024-09-21T22:06:48.207 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011865599999999999 2024-09-21T22:06:48.207 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.207 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.207 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:48.207 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.098637+0000", 2024-09-21T22:06:48.207 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001581458 2024-09-21T22:06:48.207 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.207 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.207 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:48.208 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.098728+0000", 2024-09-21T22:06:48.208 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1043000000000002e-05 2024-09-21T22:06:48.208 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.208 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.208 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:48.208 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.098797+0000", 2024-09-21T22:06:48.208 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9090000000000004e-05 2024-09-21T22:06:48.208 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:48.208 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:48.208 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:48.208 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.208 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.208 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:06:48.209 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.161242+0000", 2024-09-21T22:06:48.209 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 46.035473142999997, 2024-09-21T22:06:48.209 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014477078000000001, 2024-09-21T22:06:48.209 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:48.209 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:48.209 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:48.209 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:48.209 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:06:48.209 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:06:48.209 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 35 2024-09-21T22:06:48.209 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.209 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:48.209 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.209 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:48.210 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161242+0000", 2024-09-21T22:06:48.210 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:48.210 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.210 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.210 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:48.210 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161242+0000", 2024-09-21T22:06:48.210 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:48.210 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.210 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.210 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:48.210 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161243+0000", 2024-09-21T22:06:48.210 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4729999999999999e-06 2024-09-21T22:06:48.210 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.210 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.210 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:48.211 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161385+0000", 2024-09-21T22:06:48.211 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014226899999999999 2024-09-21T22:06:48.211 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.211 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.211 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:48.211 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161417+0000", 2024-09-21T22:06:48.211 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1406999999999999e-05 2024-09-21T22:06:48.211 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.211 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.211 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:48.211 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161434+0000", 2024-09-21T22:06:48.211 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7623000000000001e-05 2024-09-21T22:06:48.211 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.211 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.212 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:48.212 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.169728+0000", 2024-09-21T22:06:48.212 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0082933090000000004 2024-09-21T22:06:48.212 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.212 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.212 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:48.212 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.169861+0000", 2024-09-21T22:06:48.212 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013367300000000001 2024-09-21T22:06:48.212 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.212 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.212 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:06:48.212 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.170186+0000", 2024-09-21T22:06:48.212 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00032466400000000001 2024-09-21T22:06:48.212 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.213 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.213 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:48.213 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.174759+0000", 2024-09-21T22:06:48.213 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0045727880000000004 2024-09-21T22:06:48.213 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.213 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.213 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:48.213 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.175497+0000", 2024-09-21T22:06:48.213 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00073803400000000004 2024-09-21T22:06:48.213 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.213 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.213 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:48.213 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.175607+0000", 2024-09-21T22:06:48.213 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010987199999999999 2024-09-21T22:06:48.213 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.213 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.214 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:48.214 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.175719+0000", 2024-09-21T22:06:48.214 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011196600000000001 2024-09-21T22:06:48.214 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:48.214 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:48.214 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:48.214 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.214 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.214 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:06:48.214 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.161483+0000", 2024-09-21T22:06:48.214 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 46.035231631999999, 2024-09-21T22:06:48.214 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016155732999999999, 2024-09-21T22:06:48.214 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:48.214 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:48.215 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:48.215 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:48.215 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:06:48.215 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:06:48.215 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 36 2024-09-21T22:06:48.215 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.215 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:48.215 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.215 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:48.215 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161483+0000", 2024-09-21T22:06:48.215 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:48.215 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.215 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.215 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:48.216 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161483+0000", 2024-09-21T22:06:48.216 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:48.216 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.216 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.216 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:48.216 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161485+0000", 2024-09-21T22:06:48.216 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5999999999999999e-06 2024-09-21T22:06:48.216 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.216 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.216 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:48.216 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161498+0000", 2024-09-21T22:06:48.216 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3166999999999999e-05 2024-09-21T22:06:48.216 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.217 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.217 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:48.217 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161526+0000", 2024-09-21T22:06:48.217 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7786000000000001e-05 2024-09-21T22:06:48.217 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.217 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.217 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:48.217 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161545+0000", 2024-09-21T22:06:48.217 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8899999999999999e-05 2024-09-21T22:06:48.217 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.217 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.217 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:48.217 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.172615+0000", 2024-09-21T22:06:48.217 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011070465 2024-09-21T22:06:48.218 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.218 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.218 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:48.218 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.172759+0000", 2024-09-21T22:06:48.218 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014416900000000001 2024-09-21T22:06:48.218 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.218 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.218 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:06:48.218 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.172967+0000", 2024-09-21T22:06:48.218 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020758500000000001 2024-09-21T22:06:48.218 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.218 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.218 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:48.218 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.174775+0000", 2024-09-21T22:06:48.218 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0018080990000000001 2024-09-21T22:06:48.219 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.219 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.219 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:48.219 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.177447+0000", 2024-09-21T22:06:48.219 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026717429999999999 2024-09-21T22:06:48.219 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.219 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.219 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:48.219 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.177525+0000", 2024-09-21T22:06:48.219 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7846e-05 2024-09-21T22:06:48.219 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.219 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.219 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:48.219 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.177639+0000", 2024-09-21T22:06:48.220 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000114373 2024-09-21T22:06:48.220 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:48.220 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:48.220 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:48.220 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.220 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.220 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:06:48.220 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.161577+0000", 2024-09-21T22:06:48.220 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 46.035138099000001, 2024-09-21T22:06:48.220 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016652991999999998, 2024-09-21T22:06:48.220 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:48.220 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:48.220 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:48.220 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:48.221 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:06:48.221 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:06:48.221 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 37 2024-09-21T22:06:48.221 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.221 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:48.221 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.221 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:48.221 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161577+0000", 2024-09-21T22:06:48.221 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:48.221 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.221 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.221 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:48.221 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161577+0000", 2024-09-21T22:06:48.221 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:48.222 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.222 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.222 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:48.222 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161578+0000", 2024-09-21T22:06:48.222 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5269999999999999e-06 2024-09-21T22:06:48.222 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.222 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.222 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:48.222 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161591+0000", 2024-09-21T22:06:48.222 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.296e-05 2024-09-21T22:06:48.222 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.222 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.222 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:48.222 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161629+0000", 2024-09-21T22:06:48.223 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7799000000000002e-05 2024-09-21T22:06:48.223 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.223 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.223 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:48.223 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161644+0000", 2024-09-21T22:06:48.223 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5333e-05 2024-09-21T22:06:48.223 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.223 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.224 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:48.224 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.173636+0000", 2024-09-21T22:06:48.224 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011992078 2024-09-21T22:06:48.224 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.224 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.224 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:48.224 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.173862+0000", 2024-09-21T22:06:48.224 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022556799999999999 2024-09-21T22:06:48.224 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.224 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.224 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:06:48.224 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.174193+0000", 2024-09-21T22:06:48.225 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000331088 2024-09-21T22:06:48.225 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.225 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.225 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:48.225 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.175740+0000", 2024-09-21T22:06:48.225 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015467910000000001 2024-09-21T22:06:48.225 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.225 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.225 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:48.225 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.178073+0000", 2024-09-21T22:06:48.225 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002333356 2024-09-21T22:06:48.225 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.225 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.225 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:48.226 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.178162+0000", 2024-09-21T22:06:48.226 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8306000000000005e-05 2024-09-21T22:06:48.226 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.226 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.226 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:48.226 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.178230+0000", 2024-09-21T22:06:48.226 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8186000000000001e-05 2024-09-21T22:06:48.226 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:48.226 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:48.226 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:48.227 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.227 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.227 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:06:48.227 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.280540+0000", 2024-09-21T22:06:48.227 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 45.916174527000003, 2024-09-21T22:06:48.227 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013246212, 2024-09-21T22:06:48.227 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:48.227 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:48.227 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:48.227 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:48.227 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:06:48.227 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:06:48.227 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 88 2024-09-21T22:06:48.227 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.228 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:48.228 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.228 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:48.228 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.280540+0000", 2024-09-21T22:06:48.228 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:48.228 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.228 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.228 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:48.228 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.280540+0000", 2024-09-21T22:06:48.228 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:48.228 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.228 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.228 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:48.228 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.280541+0000", 2024-09-21T22:06:48.228 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.54e-07 2024-09-21T22:06:48.229 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.229 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.229 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:48.229 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.280549+0000", 2024-09-21T22:06:48.229 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3329999999999997e-06 2024-09-21T22:06:48.229 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.229 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.229 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:48.229 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.280560+0000", 2024-09-21T22:06:48.229 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1003e-05 2024-09-21T22:06:48.229 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.229 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.229 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:48.229 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.280569+0000", 2024-09-21T22:06:48.230 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2430000000000005e-06 2024-09-21T22:06:48.230 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.230 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.230 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:48.230 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.291095+0000", 2024-09-21T22:06:48.230 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010526259 2024-09-21T22:06:48.230 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.230 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.230 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:48.230 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.291231+0000", 2024-09-21T22:06:48.230 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013561899999999999 2024-09-21T22:06:48.230 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.230 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.230 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:06:48.231 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.291457+0000", 2024-09-21T22:06:48.231 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022686200000000001 2024-09-21T22:06:48.231 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.231 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.231 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:48.231 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.293117+0000", 2024-09-21T22:06:48.231 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001659474 2024-09-21T22:06:48.231 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.231 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.231 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:48.231 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.293524+0000", 2024-09-21T22:06:48.231 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00040701300000000001 2024-09-21T22:06:48.231 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.231 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.231 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:48.232 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.293673+0000", 2024-09-21T22:06:48.232 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000148782 2024-09-21T22:06:48.232 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.232 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.232 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:48.232 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.293786+0000", 2024-09-21T22:06:48.232 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011377 2024-09-21T22:06:48.232 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:48.232 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:48.232 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:48.232 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.232 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.232 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:06:48.232 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.466485+0000", 2024-09-21T22:06:48.232 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 45.730229796000003, 2024-09-21T22:06:48.233 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014443981, 2024-09-21T22:06:48.233 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:48.233 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:48.233 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:48.233 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:48.233 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:06:48.233 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:06:48.233 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 107 2024-09-21T22:06:48.233 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.233 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:48.233 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.233 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:48.233 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.466485+0000", 2024-09-21T22:06:48.233 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:48.234 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.234 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.234 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:48.234 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.466485+0000", 2024-09-21T22:06:48.234 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:48.234 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.234 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.234 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:48.234 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.466486+0000", 2024-09-21T22:06:48.234 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6299999999999993e-07 2024-09-21T22:06:48.234 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.234 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.234 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:48.234 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.466493+0000", 2024-09-21T22:06:48.235 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9639999999999997e-06 2024-09-21T22:06:48.235 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.235 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.235 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:48.235 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.466507+0000", 2024-09-21T22:06:48.235 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4216000000000001e-05 2024-09-21T22:06:48.235 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.235 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.235 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:48.235 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.466515+0000", 2024-09-21T22:06:48.235 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.157e-06 2024-09-21T22:06:48.235 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.235 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.236 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:48.236 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.472033+0000", 2024-09-21T22:06:48.236 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0055175839999999999 2024-09-21T22:06:48.236 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.236 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.236 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:48.236 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.472174+0000", 2024-09-21T22:06:48.236 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000141612 2024-09-21T22:06:48.236 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.236 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.236 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:06:48.237 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.472381+0000", 2024-09-21T22:06:48.237 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000206961 2024-09-21T22:06:48.237 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.237 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.237 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:48.237 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.473924+0000", 2024-09-21T22:06:48.237 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015427119999999999 2024-09-21T22:06:48.237 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.237 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.237 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:48.237 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.480666+0000", 2024-09-21T22:06:48.237 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0067415410000000002 2024-09-21T22:06:48.237 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.237 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.238 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:48.238 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.480811+0000", 2024-09-21T22:06:48.238 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000145422 2024-09-21T22:06:48.238 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.238 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.238 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:48.238 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.480929+0000", 2024-09-21T22:06:48.238 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000117849 2024-09-21T22:06:48.238 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:48.238 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:48.239 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:48.239 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.239 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.239 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:06:48.239 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.467057+0000", 2024-09-21T22:06:48.239 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 45.729657537000001, 2024-09-21T22:06:48.239 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018350195, 2024-09-21T22:06:48.239 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:48.239 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:48.239 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:48.239 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:48.239 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:06:48.239 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:06:48.239 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 108 2024-09-21T22:06:48.240 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.240 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:48.240 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.240 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:48.240 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467057+0000", 2024-09-21T22:06:48.240 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:48.240 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.240 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.240 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:48.240 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467057+0000", 2024-09-21T22:06:48.240 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:48.240 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.240 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.240 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:48.240 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467059+0000", 2024-09-21T22:06:48.241 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.826e-06 2024-09-21T22:06:48.241 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.241 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.241 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:48.241 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467071+0000", 2024-09-21T22:06:48.241 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2224e-05 2024-09-21T22:06:48.241 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.241 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.241 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:48.241 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467098+0000", 2024-09-21T22:06:48.241 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6953000000000001e-05 2024-09-21T22:06:48.241 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.241 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.241 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:48.242 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467113+0000", 2024-09-21T22:06:48.242 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5119999999999999e-05 2024-09-21T22:06:48.242 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.242 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.242 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:48.242 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.474060+0000", 2024-09-21T22:06:48.242 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0069462919999999997 2024-09-21T22:06:48.242 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.242 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.242 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:48.242 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.474283+0000", 2024-09-21T22:06:48.242 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000223315 2024-09-21T22:06:48.243 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.243 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.243 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:06:48.243 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.474660+0000", 2024-09-21T22:06:48.243 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00037729400000000002 2024-09-21T22:06:48.243 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.243 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.243 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:48.243 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.480622+0000", 2024-09-21T22:06:48.243 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0059613369999999997 2024-09-21T22:06:48.243 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.244 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.244 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:48.244 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.485145+0000", 2024-09-21T22:06:48.244 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0045231960000000002 2024-09-21T22:06:48.244 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.244 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.244 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:48.244 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.485288+0000", 2024-09-21T22:06:48.244 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000143232 2024-09-21T22:06:48.244 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.244 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.244 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:48.244 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.485407+0000", 2024-09-21T22:06:48.245 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000119406 2024-09-21T22:06:48.245 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:48.245 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:48.245 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:48.245 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.245 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.245 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:06:48.245 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.467144+0000", 2024-09-21T22:06:48.245 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 45.729570318, 2024-09-21T22:06:48.245 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018907146999999999, 2024-09-21T22:06:48.245 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:48.245 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:48.245 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:48.245 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:48.246 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:06:48.246 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:06:48.246 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 109 2024-09-21T22:06:48.246 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.246 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:48.246 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.246 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:48.246 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467144+0000", 2024-09-21T22:06:48.246 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:48.246 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.246 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.246 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:48.246 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467144+0000", 2024-09-21T22:06:48.247 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:48.247 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.247 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.247 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:48.247 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467145+0000", 2024-09-21T22:06:48.247 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9999999999999996e-07 2024-09-21T22:06:48.247 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.247 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.247 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:48.247 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467331+0000", 2024-09-21T22:06:48.247 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018518899999999999 2024-09-21T22:06:48.247 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.247 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.248 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:48.248 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467350+0000", 2024-09-21T22:06:48.248 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8966000000000001e-05 2024-09-21T22:06:48.248 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.248 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.248 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:48.248 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467361+0000", 2024-09-21T22:06:48.248 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1233000000000001e-05 2024-09-21T22:06:48.248 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.248 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.248 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:48.248 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.475613+0000", 2024-09-21T22:06:48.248 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0082518629999999999 2024-09-21T22:06:48.249 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.249 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.249 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:48.249 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.475859+0000", 2024-09-21T22:06:48.249 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024633800000000003 2024-09-21T22:06:48.249 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.249 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.249 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:06:48.249 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.476223+0000", 2024-09-21T22:06:48.249 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00036408700000000003 2024-09-21T22:06:48.249 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.249 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.249 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:48.249 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.481046+0000", 2024-09-21T22:06:48.250 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.004823356 2024-09-21T22:06:48.250 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.250 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.250 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:48.250 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.485722+0000", 2024-09-21T22:06:48.250 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0046753640000000004 2024-09-21T22:06:48.250 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.250 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.250 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:48.250 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.485908+0000", 2024-09-21T22:06:48.250 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000185859 2024-09-21T22:06:48.250 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.250 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.251 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:48.251 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.486052+0000", 2024-09-21T22:06:48.251 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014399200000000001 2024-09-21T22:06:48.251 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:48.251 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:48.251 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:48.251 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.251 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.251 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:06:48.251 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.467384+0000", 2024-09-21T22:06:48.251 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 45.729330490000002, 2024-09-21T22:06:48.251 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.01894144, 2024-09-21T22:06:48.252 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:48.252 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:48.252 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:48.252 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:48.252 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:06:48.252 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:06:48.252 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 110 2024-09-21T22:06:48.252 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.252 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:48.252 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.252 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:48.252 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467384+0000", 2024-09-21T22:06:48.252 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:48.253 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.253 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.253 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:48.253 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467384+0000", 2024-09-21T22:06:48.253 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:48.253 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.253 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.253 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:48.253 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467385+0000", 2024-09-21T22:06:48.253 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0470000000000001e-06 2024-09-21T22:06:48.253 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.253 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.253 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:48.253 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467393+0000", 2024-09-21T22:06:48.254 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3869999999999997e-06 2024-09-21T22:06:48.254 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.254 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.254 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:48.254 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467413+0000", 2024-09-21T22:06:48.254 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9953e-05 2024-09-21T22:06:48.254 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.254 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.254 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:48.254 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467421+0000", 2024-09-21T22:06:48.254 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2400000000000007e-06 2024-09-21T22:06:48.254 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.254 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.254 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:48.254 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.479542+0000", 2024-09-21T22:06:48.255 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012120832 2024-09-21T22:06:48.255 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.255 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.255 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:48.255 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.479705+0000", 2024-09-21T22:06:48.255 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016362899999999999 2024-09-21T22:06:48.255 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.255 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.255 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:06:48.255 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.479914+0000", 2024-09-21T22:06:48.255 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020814499999999999 2024-09-21T22:06:48.255 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.255 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.255 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:48.256 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.485536+0000", 2024-09-21T22:06:48.256 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0056222800000000003 2024-09-21T22:06:48.256 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.256 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.256 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:48.256 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.486077+0000", 2024-09-21T22:06:48.256 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00054079900000000001 2024-09-21T22:06:48.256 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.256 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.256 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:48.256 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.486213+0000", 2024-09-21T22:06:48.256 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000136682 2024-09-21T22:06:48.256 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.256 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.256 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:48.257 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.486326+0000", 2024-09-21T22:06:48.257 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000112446 2024-09-21T22:06:48.257 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:48.257 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:48.257 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:48.257 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.257 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.257 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:06:48.257 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.467436+0000", 2024-09-21T22:06:48.257 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 45.729278989999997, 2024-09-21T22:06:48.257 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019152168000000001, 2024-09-21T22:06:48.257 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:48.257 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:48.258 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:48.258 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:48.258 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:06:48.258 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:06:48.258 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 111 2024-09-21T22:06:48.258 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.258 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:48.258 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.258 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:48.258 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467436+0000", 2024-09-21T22:06:48.258 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:48.258 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.259 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.259 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:48.259 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467436+0000", 2024-09-21T22:06:48.259 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:48.259 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.259 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.259 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:48.259 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467437+0000", 2024-09-21T22:06:48.259 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0699999999999996e-07 2024-09-21T22:06:48.259 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.259 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.259 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:48.259 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467445+0000", 2024-09-21T22:06:48.260 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.789e-06 2024-09-21T22:06:48.260 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.260 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.260 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:48.260 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467467+0000", 2024-09-21T22:06:48.260 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1253000000000001e-05 2024-09-21T22:06:48.260 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.260 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.260 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:48.260 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467475+0000", 2024-09-21T22:06:48.260 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0039999999999999e-06 2024-09-21T22:06:48.260 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.260 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.260 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:48.260 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.481164+0000", 2024-09-21T22:06:48.261 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013689114 2024-09-21T22:06:48.261 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.261 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.261 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:48.261 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.481394+0000", 2024-09-21T22:06:48.261 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000229931 2024-09-21T22:06:48.261 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.261 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.261 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:06:48.261 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.481779+0000", 2024-09-21T22:06:48.261 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00038545400000000003 2024-09-21T22:06:48.261 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.261 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.261 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:48.261 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.485675+0000", 2024-09-21T22:06:48.262 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0038958370000000001 2024-09-21T22:06:48.262 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.262 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.262 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:48.262 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.486346+0000", 2024-09-21T22:06:48.262 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00067119100000000002 2024-09-21T22:06:48.262 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.262 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.262 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:48.262 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.486478+0000", 2024-09-21T22:06:48.262 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013182600000000001 2024-09-21T22:06:48.262 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.262 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.262 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:48.263 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.486588+0000", 2024-09-21T22:06:48.263 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000110062 2024-09-21T22:06:48.263 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:48.263 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:48.263 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:48.263 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.263 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.263 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:06:48.263 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.467488+0000", 2024-09-21T22:06:48.263 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 45.729226363999999, 2024-09-21T22:06:48.263 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021038304000000001, 2024-09-21T22:06:48.263 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:48.263 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:48.264 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:48.264 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:48.264 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:06:48.264 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:06:48.264 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 112 2024-09-21T22:06:48.264 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.264 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:48.264 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.264 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:48.264 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467488+0000", 2024-09-21T22:06:48.264 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:48.264 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.264 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.265 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:48.265 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467488+0000", 2024-09-21T22:06:48.265 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:48.265 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.265 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.265 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:48.265 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467489+0000", 2024-09-21T22:06:48.265 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7999999999999995e-07 2024-09-21T22:06:48.265 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.265 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.265 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:48.265 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467687+0000", 2024-09-21T22:06:48.265 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019776200000000001 2024-09-21T22:06:48.265 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.265 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.266 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:48.266 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467704+0000", 2024-09-21T22:06:48.266 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7193e-05 2024-09-21T22:06:48.266 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.266 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.266 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:48.266 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467719+0000", 2024-09-21T22:06:48.266 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4703e-05 2024-09-21T22:06:48.267 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.267 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.267 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:48.267 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.482745+0000", 2024-09-21T22:06:48.267 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015025801 2024-09-21T22:06:48.267 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.267 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.267 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:48.267 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.483015+0000", 2024-09-21T22:06:48.267 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026999400000000002 2024-09-21T22:06:48.267 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.267 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.267 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:06:48.268 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.483440+0000", 2024-09-21T22:06:48.268 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000425697 2024-09-21T22:06:48.268 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.268 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.268 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:48.268 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.486701+0000", 2024-09-21T22:06:48.268 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0032611950000000002 2024-09-21T22:06:48.268 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.268 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.268 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:48.268 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.488262+0000", 2024-09-21T22:06:48.268 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015610439999999999 2024-09-21T22:06:48.268 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.268 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.269 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:48.269 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.488406+0000", 2024-09-21T22:06:48.269 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000143232 2024-09-21T22:06:48.269 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.269 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.269 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:48.269 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.488527+0000", 2024-09-21T22:06:48.269 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012100299999999999 2024-09-21T22:06:48.269 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:48.269 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:48.269 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:48.269 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.269 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.269 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:06:48.270 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.467786+0000", 2024-09-21T22:06:48.270 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 45.728928938999999, 2024-09-21T22:06:48.270 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026339741999999999, 2024-09-21T22:06:48.270 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:48.270 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:48.270 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:48.270 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:48.270 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:06:48.270 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:06:48.270 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 113 2024-09-21T22:06:48.270 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.270 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:48.270 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.270 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:48.271 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467786+0000", 2024-09-21T22:06:48.271 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:48.271 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.271 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.271 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:48.271 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467786+0000", 2024-09-21T22:06:48.271 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:48.271 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.271 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.271 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:48.271 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467787+0000", 2024-09-21T22:06:48.271 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.446e-06 2024-09-21T22:06:48.271 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.271 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.272 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:48.272 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467802+0000", 2024-09-21T22:06:48.272 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4769999999999999e-05 2024-09-21T22:06:48.272 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.272 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.272 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:48.272 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467826+0000", 2024-09-21T22:06:48.272 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3430000000000001e-05 2024-09-21T22:06:48.272 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.272 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.272 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:48.272 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467843+0000", 2024-09-21T22:06:48.272 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7065999999999999e-05 2024-09-21T22:06:48.273 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.273 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.273 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:48.273 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.486820+0000", 2024-09-21T22:06:48.273 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018977233999999999 2024-09-21T22:06:48.273 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.273 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.273 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:48.273 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.487065+0000", 2024-09-21T22:06:48.273 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024558799999999998 2024-09-21T22:06:48.273 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.273 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.273 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:06:48.274 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.487442+0000", 2024-09-21T22:06:48.274 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000376707 2024-09-21T22:06:48.274 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.274 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.274 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:48.274 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.492256+0000", 2024-09-21T22:06:48.274 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0048141620000000003 2024-09-21T22:06:48.274 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.274 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.274 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:48.274 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.493865+0000", 2024-09-21T22:06:48.274 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016090150000000001 2024-09-21T22:06:48.274 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.274 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.275 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:48.275 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.494007+0000", 2024-09-21T22:06:48.275 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000141359 2024-09-21T22:06:48.275 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.275 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.275 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:48.275 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.494126+0000", 2024-09-21T22:06:48.275 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000118965 2024-09-21T22:06:48.275 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:48.275 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:48.275 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:48.275 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.275 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.276 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:06:48.276 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.467878+0000", 2024-09-21T22:06:48.276 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 45.728836450000003, 2024-09-21T22:06:48.276 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.027783958000000001, 2024-09-21T22:06:48.276 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:48.276 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:48.276 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:48.276 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:48.276 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:06:48.276 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:06:48.276 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 114 2024-09-21T22:06:48.276 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.276 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:48.276 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.277 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:48.277 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467878+0000", 2024-09-21T22:06:48.277 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:48.277 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.277 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.277 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:48.277 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467878+0000", 2024-09-21T22:06:48.277 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:48.277 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.277 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.277 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:48.277 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467880+0000", 2024-09-21T22:06:48.277 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.223e-06 2024-09-21T22:06:48.277 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.277 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.278 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:48.278 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467891+0000", 2024-09-21T22:06:48.278 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1527e-05 2024-09-21T22:06:48.278 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.278 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.278 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:48.278 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467913+0000", 2024-09-21T22:06:48.278 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.143e-05 2024-09-21T22:06:48.278 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.278 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.278 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:48.278 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467930+0000", 2024-09-21T22:06:48.278 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7620000000000001e-05 2024-09-21T22:06:48.279 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.279 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.279 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:48.279 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.488690+0000", 2024-09-21T22:06:48.279 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020759588999999998 2024-09-21T22:06:48.279 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.279 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.279 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:48.279 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.488882+0000", 2024-09-21T22:06:48.279 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019236200000000001 2024-09-21T22:06:48.279 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.279 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.279 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:06:48.280 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.489093+0000", 2024-09-21T22:06:48.280 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000211318 2024-09-21T22:06:48.280 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.280 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.280 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:48.280 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.494152+0000", 2024-09-21T22:06:48.280 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0050582769999999999 2024-09-21T22:06:48.280 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.280 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.280 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:48.280 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.495373+0000", 2024-09-21T22:06:48.280 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001221747 2024-09-21T22:06:48.280 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.280 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.281 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:48.281 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.495543+0000", 2024-09-21T22:06:48.281 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000169259 2024-09-21T22:06:48.281 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.281 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.281 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:48.281 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.495662+0000", 2024-09-21T22:06:48.281 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011960600000000001 2024-09-21T22:06:48.281 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:48.281 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:48.281 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:48.281 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.281 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.281 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:06:48.282 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.468118+0000", 2024-09-21T22:06:48.282 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 45.728596709000001, 2024-09-21T22:06:48.282 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.027938877000000001, 2024-09-21T22:06:48.282 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:48.282 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:48.282 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:48.282 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:48.282 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:06:48.282 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:06:48.282 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 115 2024-09-21T22:06:48.282 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.282 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:48.282 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.282 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:48.283 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468118+0000", 2024-09-21T22:06:48.283 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:48.283 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.283 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.283 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:48.283 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468118+0000", 2024-09-21T22:06:48.283 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:48.283 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.283 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.283 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:48.283 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468120+0000", 2024-09-21T22:06:48.283 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7069999999999999e-06 2024-09-21T22:06:48.283 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.284 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.284 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:48.284 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468381+0000", 2024-09-21T22:06:48.284 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026135399999999999 2024-09-21T22:06:48.284 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.284 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.284 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:48.284 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468409+0000", 2024-09-21T22:06:48.284 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.781e-05 2024-09-21T22:06:48.284 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.284 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.284 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:48.284 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468433+0000", 2024-09-21T22:06:48.284 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4343999999999999e-05 2024-09-21T22:06:48.285 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.285 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.285 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:48.285 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.489785+0000", 2024-09-21T22:06:48.285 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021351221 2024-09-21T22:06:48.285 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.285 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.285 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:48.285 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.490018+0000", 2024-09-21T22:06:48.285 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023317800000000001 2024-09-21T22:06:48.285 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.285 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.285 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:06:48.285 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.490412+0000", 2024-09-21T22:06:48.285 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00039417400000000001 2024-09-21T22:06:48.286 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.286 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.286 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:48.286 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.495694+0000", 2024-09-21T22:06:48.286 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.005282335 2024-09-21T22:06:48.286 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.286 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.286 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:48.286 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.495852+0000", 2024-09-21T22:06:48.286 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015777200000000001 2024-09-21T22:06:48.286 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.286 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.286 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:48.286 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.495938+0000", 2024-09-21T22:06:48.287 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6493e-05 2024-09-21T22:06:48.287 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.287 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.287 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:48.287 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.496057+0000", 2024-09-21T22:06:48.287 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000118489 2024-09-21T22:06:48.287 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:48.287 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:48.287 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:48.287 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.287 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.287 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:06:48.287 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.468474+0000", 2024-09-21T22:06:48.287 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 45.728240948, 2024-09-21T22:06:48.288 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.028015895999999998, 2024-09-21T22:06:48.288 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:48.288 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:48.288 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:48.288 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:48.288 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:06:48.288 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:06:48.288 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 116 2024-09-21T22:06:48.288 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.288 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:48.288 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.288 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:48.288 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468474+0000", 2024-09-21T22:06:48.288 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:48.289 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.289 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.289 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:48.289 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468474+0000", 2024-09-21T22:06:48.289 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:48.289 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.289 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.289 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:48.289 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468476+0000", 2024-09-21T22:06:48.289 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7060000000000001e-06 2024-09-21T22:06:48.289 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.289 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.289 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:48.289 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468488+0000", 2024-09-21T22:06:48.290 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.234e-05 2024-09-21T22:06:48.290 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.290 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.290 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:48.290 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468517+0000", 2024-09-21T22:06:48.290 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.898e-05 2024-09-21T22:06:48.290 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.290 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.290 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:48.290 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468536+0000", 2024-09-21T22:06:48.290 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8899999999999999e-05 2024-09-21T22:06:48.290 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.290 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.290 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:48.291 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.492376+0000", 2024-09-21T22:06:48.291 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023840659 2024-09-21T22:06:48.291 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.291 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.291 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:48.291 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.492621+0000", 2024-09-21T22:06:48.291 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000244794 2024-09-21T22:06:48.291 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.291 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.291 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:06:48.291 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.493008+0000", 2024-09-21T22:06:48.291 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00038629099999999998 2024-09-21T22:06:48.291 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.292 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.292 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:48.292 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.496192+0000", 2024-09-21T22:06:48.292 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0031847220000000001 2024-09-21T22:06:48.292 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.292 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.292 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:48.292 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.496233+0000", 2024-09-21T22:06:48.292 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0410000000000001e-05 2024-09-21T22:06:48.292 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.292 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.292 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:48.293 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.496374+0000", 2024-09-21T22:06:48.293 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140872 2024-09-21T22:06:48.293 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.293 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.293 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:48.293 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.496490+0000", 2024-09-21T22:06:48.293 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000116222 2024-09-21T22:06:48.293 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:48.293 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:48.293 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:48.293 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.293 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.293 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:06:48.293 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.468568+0000", 2024-09-21T22:06:48.294 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 45.728146829000003, 2024-09-21T22:06:48.294 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.028181578999999998, 2024-09-21T22:06:48.294 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:48.294 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:48.294 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:48.294 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:48.294 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:06:48.294 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:06:48.294 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 117 2024-09-21T22:06:48.294 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.294 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:48.294 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.294 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:48.295 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468568+0000", 2024-09-21T22:06:48.295 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:48.295 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.295 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.295 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:48.295 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468568+0000", 2024-09-21T22:06:48.295 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:48.295 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.295 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.295 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:48.295 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468569+0000", 2024-09-21T22:06:48.295 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4139999999999999e-06 2024-09-21T22:06:48.295 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.295 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.296 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:48.296 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468583+0000", 2024-09-21T22:06:48.296 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3409e-05 2024-09-21T22:06:48.296 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.296 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.296 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:48.296 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468608+0000", 2024-09-21T22:06:48.296 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5559999999999999e-05 2024-09-21T22:06:48.296 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.296 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.296 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:48.296 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468626+0000", 2024-09-21T22:06:48.296 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7863000000000002e-05 2024-09-21T22:06:48.297 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.297 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.297 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:48.297 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.494260+0000", 2024-09-21T22:06:48.297 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025633744999999999 2024-09-21T22:06:48.297 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.297 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.297 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:48.297 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.494519+0000", 2024-09-21T22:06:48.297 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025924800000000001 2024-09-21T22:06:48.297 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.297 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.297 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:06:48.298 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.494747+0000", 2024-09-21T22:06:48.298 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022792799999999999 2024-09-21T22:06:48.298 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.298 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.298 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:48.298 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.496519+0000", 2024-09-21T22:06:48.298 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017720030000000001 2024-09-21T22:06:48.298 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.298 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.298 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:48.298 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.496607+0000", 2024-09-21T22:06:48.298 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7873000000000004e-05 2024-09-21T22:06:48.298 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.298 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.299 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:48.299 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.496691+0000", 2024-09-21T22:06:48.299 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4009000000000002e-05 2024-09-21T22:06:48.299 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.299 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.299 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:48.299 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.496750+0000", 2024-09-21T22:06:48.299 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8526999999999997e-05 2024-09-21T22:06:48.299 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:48.299 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:48.299 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:48.299 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.299 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.299 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:06:48.299 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.898244+0000", 2024-09-21T22:06:48.300 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 45.298470526000003, 2024-09-21T22:06:48.300 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014749464, 2024-09-21T22:06:48.300 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:48.300 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:48.300 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:48.300 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:48.300 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:06:48.300 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:06:48.300 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 140 2024-09-21T22:06:48.300 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.300 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:48.300 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.300 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:48.300 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898244+0000", 2024-09-21T22:06:48.301 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:48.301 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.301 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.301 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:48.301 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898244+0000", 2024-09-21T22:06:48.301 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:48.301 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.301 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.301 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:48.301 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898245+0000", 2024-09-21T22:06:48.301 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3000000000000002e-07 2024-09-21T22:06:48.301 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.301 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.301 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:48.302 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898308+0000", 2024-09-21T22:06:48.302 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3468999999999998e-05 2024-09-21T22:06:48.302 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.302 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.302 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:48.302 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898315+0000", 2024-09-21T22:06:48.302 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.725e-06 2024-09-21T22:06:48.302 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.302 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.302 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:48.302 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898321+0000", 2024-09-21T22:06:48.302 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2090000000000002e-06 2024-09-21T22:06:48.303 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.303 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.303 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:48.303 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.905452+0000", 2024-09-21T22:06:48.303 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0071305989999999996 2024-09-21T22:06:48.303 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.303 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.303 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:48.303 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.905504+0000", 2024-09-21T22:06:48.303 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2638e-05 2024-09-21T22:06:48.303 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.303 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.303 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:06:48.303 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.905590+0000", 2024-09-21T22:06:48.304 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5333000000000004e-05 2024-09-21T22:06:48.304 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.304 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.304 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:48.304 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.911542+0000", 2024-09-21T22:06:48.304 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0059519819999999998 2024-09-21T22:06:48.304 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.304 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.304 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:48.304 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.912908+0000", 2024-09-21T22:06:48.304 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0013666640000000001 2024-09-21T22:06:48.304 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.304 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.305 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:48.305 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.912957+0000", 2024-09-21T22:06:48.305 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9203000000000003e-05 2024-09-21T22:06:48.305 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.305 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.305 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:48.305 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.912994+0000", 2024-09-21T22:06:48.305 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6312000000000001e-05 2024-09-21T22:06:48.305 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:48.305 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:48.305 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:48.305 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.305 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.305 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:06:48.306 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.898342+0000", 2024-09-21T22:06:48.306 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 45.298372387999997, 2024-09-21T22:06:48.306 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014787918000000001, 2024-09-21T22:06:48.306 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:48.306 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:48.306 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:48.306 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:48.306 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:06:48.306 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:06:48.306 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 141 2024-09-21T22:06:48.306 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.306 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:48.306 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.307 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:48.307 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898342+0000", 2024-09-21T22:06:48.307 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:48.307 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.307 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.307 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:48.307 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898342+0000", 2024-09-21T22:06:48.307 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:48.307 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.307 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.307 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:48.307 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898343+0000", 2024-09-21T22:06:48.307 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3000000000000001e-07 2024-09-21T22:06:48.307 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.308 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.308 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:48.308 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898346+0000", 2024-09-21T22:06:48.308 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6009999999999999e-06 2024-09-21T22:06:48.308 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.308 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.308 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:48.308 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898352+0000", 2024-09-21T22:06:48.308 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8089999999999998e-06 2024-09-21T22:06:48.308 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.308 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.308 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:48.308 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898358+0000", 2024-09-21T22:06:48.309 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4349999999999996e-06 2024-09-21T22:06:48.309 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.309 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.309 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:48.309 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.909387+0000", 2024-09-21T22:06:48.309 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011029105000000001 2024-09-21T22:06:48.309 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.309 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.309 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:48.309 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.909462+0000", 2024-09-21T22:06:48.309 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5192999999999996e-05 2024-09-21T22:06:48.309 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.309 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.310 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:06:48.310 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.909551+0000", 2024-09-21T22:06:48.310 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9046000000000001e-05 2024-09-21T22:06:48.310 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.310 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.310 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:48.310 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.911852+0000", 2024-09-21T22:06:48.310 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0023005579999999999 2024-09-21T22:06:48.310 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.310 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.310 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:48.310 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.913002+0000", 2024-09-21T22:06:48.310 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011502929999999999 2024-09-21T22:06:48.311 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.311 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.311 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:48.311 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.913060+0000", 2024-09-21T22:06:48.311 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8427000000000002e-05 2024-09-21T22:06:48.311 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.311 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.311 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:48.311 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.913130+0000", 2024-09-21T22:06:48.311 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0020999999999999e-05 2024-09-21T22:06:48.311 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:48.311 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:48.311 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:48.312 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.312 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.312 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:158 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T22:06:48.312 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:03.511201+0000", 2024-09-21T22:06:48.312 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 44.685513905000001, 2024-09-21T22:06:48.312 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013353538999999999, 2024-09-21T22:06:48.312 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:48.312 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:48.312 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:48.312 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:48.312 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:06:48.312 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:06:48.312 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 158 2024-09-21T22:06:48.312 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.313 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:48.313 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.313 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:48.313 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511201+0000", 2024-09-21T22:06:48.313 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:48.313 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.313 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.313 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:48.313 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511201+0000", 2024-09-21T22:06:48.313 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:48.313 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.313 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.313 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:48.313 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511202+0000", 2024-09-21T22:06:48.314 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9299999999999997e-07 2024-09-21T22:06:48.314 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.314 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.314 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:48.314 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511315+0000", 2024-09-21T22:06:48.314 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000113909 2024-09-21T22:06:48.314 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.314 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.314 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:48.314 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511333+0000", 2024-09-21T22:06:48.314 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7062999999999999e-05 2024-09-21T22:06:48.314 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.314 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.315 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:48.315 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511344+0000", 2024-09-21T22:06:48.315 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1177e-05 2024-09-21T22:06:48.315 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.315 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.315 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:48.315 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.517415+0000", 2024-09-21T22:06:48.315 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0060709830000000003 2024-09-21T22:06:48.315 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.315 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.315 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:48.315 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.517539+0000", 2024-09-21T22:06:48.315 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000124432 2024-09-21T22:06:48.315 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.316 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.316 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:06:48.316 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.517732+0000", 2024-09-21T22:06:48.316 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019262200000000001 2024-09-21T22:06:48.316 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.316 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.316 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:48.316 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.520709+0000", 2024-09-21T22:06:48.316 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0029767470000000001 2024-09-21T22:06:48.316 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.316 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.316 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:48.316 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.524389+0000", 2024-09-21T22:06:48.316 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003680975 2024-09-21T22:06:48.317 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.317 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.317 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:48.317 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.524486+0000", 2024-09-21T22:06:48.317 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6575000000000003e-05 2024-09-21T22:06:48.317 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.317 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.317 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:48.317 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.524554+0000", 2024-09-21T22:06:48.317 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8362999999999996e-05 2024-09-21T22:06:48.317 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:48.317 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:48.317 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:48.317 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.318 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.318 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T22:06:48.318 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:03.511368+0000", 2024-09-21T22:06:48.318 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 44.685347002999997, 2024-09-21T22:06:48.318 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013312133, 2024-09-21T22:06:48.318 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:48.318 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:48.318 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:48.318 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:48.318 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:06:48.318 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:06:48.318 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 159 2024-09-21T22:06:48.318 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.318 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:48.319 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.319 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:48.319 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511368+0000", 2024-09-21T22:06:48.319 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:48.319 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.319 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.319 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:48.319 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511368+0000", 2024-09-21T22:06:48.319 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:48.319 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.319 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.319 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:48.319 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511369+0000", 2024-09-21T22:06:48.320 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9599999999999998e-07 2024-09-21T22:06:48.320 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.320 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.320 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:48.320 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511382+0000", 2024-09-21T22:06:48.320 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3487e-05 2024-09-21T22:06:48.320 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.320 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.320 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:48.320 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511394+0000", 2024-09-21T22:06:48.320 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1337e-05 2024-09-21T22:06:48.320 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.320 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.320 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:48.320 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511402+0000", 2024-09-21T22:06:48.321 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0059999999999992e-06 2024-09-21T22:06:48.321 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.321 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.321 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:48.321 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.519376+0000", 2024-09-21T22:06:48.321 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0079746520000000005 2024-09-21T22:06:48.321 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.321 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.321 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:48.321 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.519498+0000", 2024-09-21T22:06:48.321 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012212900000000001 2024-09-21T22:06:48.321 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.321 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.322 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:06:48.322 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.519728+0000", 2024-09-21T22:06:48.322 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022985799999999999 2024-09-21T22:06:48.322 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.322 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.322 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:48.322 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.521344+0000", 2024-09-21T22:06:48.322 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016163169999999999 2024-09-21T22:06:48.322 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.322 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.322 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:48.322 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.524568+0000", 2024-09-21T22:06:48.322 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0032234220000000001 2024-09-21T22:06:48.322 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.322 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.323 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:48.323 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.524633+0000", 2024-09-21T22:06:48.323 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5126e-05 2024-09-21T22:06:48.323 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:48.323 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:48.323 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:48.323 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.524680+0000", 2024-09-21T22:06:48.323 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6903000000000001e-05 2024-09-21T22:06:48.323 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:48.323 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:48.323 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:48.323 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:48.323 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:48.323 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:06:48.325 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T22:06:48.325 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:06:48.324+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:06:48.426 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:06:48.420+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:06:48.457 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:06:48.458 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:06:48.458 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-09-21T22:06:48.458 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:06:48.473 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T22:06:48.559 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:06:48.556+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:06:48.628 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:06:48.624+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:06:48.642 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:06:48.643 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:06:48.643 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-21T22:06:48.643 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-21T22:06:48.643 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:06:48.657 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T22:06:48.728 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:06:48.728+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:06:48.818 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:06:48.818 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-21T22:06:48.818 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-21T22:06:48.818 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [] 2024-09-21T22:06:48.818 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:06:48.829 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:06:48.824+0000 7fc8d39f7640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25294) UID: 0 2024-09-21T22:06:48.832 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T22:06:48.880 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 0] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-09-21T22:06:48.880 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T22:06:48.880 INFO:tasks.thrashosds.thrasher:Killing osd 5, live_osds are [0, 1, 2, 3, 4, 5] 2024-09-21T22:06:48.881 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-09-21T22:06:48.881 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T22:06:48.925 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-21T22:06:48.930 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:06:48.924+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:06:48.957 INFO:tasks.ceph.osd.5:Stopped 2024-09-21T22:06:48.957 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.5 2024-09-21T22:06:48.957 INFO:tasks.thrashosds.thrasher:remote for osd 5 is ubuntu@smithi064.front.sepia.ceph.com 2024-09-21T22:06:48.958 DEBUG:teuthology.orchestra.run.smithi064:> type ceph-objectstore-tool 2024-09-21T22:06:48.988 INFO:teuthology.orchestra.run.smithi064.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-09-21T22:06:48.990 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-21T22:06:48.991 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T22:06:48.992 INFO:teuthology.orchestra.run.smithi064.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-21T22:06:48.992 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal --op list-pgs 2024-09-21T22:06:49.031 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:06:49.028+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:06:49.132 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:06:49.128+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:06:49.150 INFO:teuthology.orchestra.run.smithi064.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-09-21T22:06:49.153 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-21T22:06:49.154 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T22:06:49.232 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:06:49.228+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:06:49.263 INFO:teuthology.orchestra.run.smithi064.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-09-21T22:06:49.265 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-21T22:06:49.266 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T22:06:49.333 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:06:49.328+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:06:49.388 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:06:49.388 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:06:49.388 INFO:teuthology.orchestra.run.smithi039.stdout: "num_ops": 0 2024-09-21T22:06:49.388 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:06:49.402 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T22:06:49.434 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:06:49.428+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:06:49.535 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:06:49.532+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:06:49.578 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:06:49.578 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:06:49.578 INFO:teuthology.orchestra.run.smithi039.stdout: "complaint_time": 30, 2024-09-21T22:06:49.578 INFO:teuthology.orchestra.run.smithi039.stdout: "num_blocked_ops": 0 2024-09-21T22:06:49.578 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:06:49.593 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T22:06:49.603 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T22:06:49.635 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:06:49.632+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:06:49.716 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:06:49.716 INFO:teuthology.orchestra.run.smithi039.stdout: "size": 20, 2024-09-21T22:06:49.716 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 600, 2024-09-21T22:06:49.716 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [ 2024-09-21T22:06:49.717 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.717 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:20 1.0 e14/13 1:30306672:devicehealth::main.db.0000000000000000:head v 14'10, mlcod=14'8)", 2024-09-21T22:06:49.717 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.137639+0000", 2024-09-21T22:06:49.717 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 47.576062043999997, 2024-09-21T22:06:49.717 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0039902100000000001, 2024-09-21T22:06:49.717 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:49.717 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:49.717 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:49.717 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:49.717 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.717 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:49.718 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.137639+0000", 2024-09-21T22:06:49.718 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:49.718 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.718 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.718 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:49.718 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.137639+0000", 2024-09-21T22:06:49.718 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:49.718 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.718 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.718 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:49.718 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.137641+0000", 2024-09-21T22:06:49.719 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.686e-06 2024-09-21T22:06:49.719 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.719 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.719 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:49.719 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.137934+0000", 2024-09-21T22:06:49.719 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00029319099999999999 2024-09-21T22:06:49.719 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.719 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.719 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:49.719 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.137963+0000", 2024-09-21T22:06:49.719 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9502999999999999e-05 2024-09-21T22:06:49.719 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.719 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.720 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:49.720 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.137989+0000", 2024-09-21T22:06:49.720 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5573000000000001e-05 2024-09-21T22:06:49.720 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.720 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.720 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:49.720 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.138086+0000", 2024-09-21T22:06:49.720 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.6968999999999999e-05 2024-09-21T22:06:49.720 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.720 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.720 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:49.720 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.138120+0000", 2024-09-21T22:06:49.720 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.4560000000000001e-05 2024-09-21T22:06:49.720 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.721 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.721 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:49.721 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.141539+0000", 2024-09-21T22:06:49.721 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0034183289999999999 2024-09-21T22:06:49.721 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.721 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.721 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:49.721 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.141629+0000", 2024-09-21T22:06:49.721 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.0399e-05 2024-09-21T22:06:49.721 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:49.721 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:49.721 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:49.721 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.721 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.722 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:23 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'13, mlcod=14'11)", 2024-09-21T22:06:49.722 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.144914+0000", 2024-09-21T22:06:49.722 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 47.568786725999999, 2024-09-21T22:06:49.722 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0040223230000000004, 2024-09-21T22:06:49.722 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:49.722 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:49.722 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:49.722 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:49.722 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.722 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:49.722 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.144914+0000", 2024-09-21T22:06:49.722 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:49.723 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.723 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.723 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:49.723 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.144914+0000", 2024-09-21T22:06:49.723 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:49.723 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.723 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.723 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:49.723 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.144916+0000", 2024-09-21T22:06:49.723 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6470000000000001e-06 2024-09-21T22:06:49.723 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.723 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.724 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:49.724 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.144929+0000", 2024-09-21T22:06:49.724 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2903e-05 2024-09-21T22:06:49.724 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.724 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.724 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:49.724 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.144958+0000", 2024-09-21T22:06:49.724 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9280000000000001e-05 2024-09-21T22:06:49.724 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.724 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.724 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:49.724 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.144971+0000", 2024-09-21T22:06:49.725 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3407000000000001e-05 2024-09-21T22:06:49.725 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.725 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.725 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:49.725 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.145106+0000", 2024-09-21T22:06:49.725 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013440500000000001 2024-09-21T22:06:49.725 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.725 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.725 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:49.725 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.145170+0000", 2024-09-21T22:06:49.725 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.4597000000000005e-05 2024-09-21T22:06:49.725 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.726 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.726 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:49.726 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.148854+0000", 2024-09-21T22:06:49.726 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0036833249999999999 2024-09-21T22:06:49.726 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.726 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.726 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:49.726 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.148937+0000", 2024-09-21T22:06:49.726 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.2758999999999998e-05 2024-09-21T22:06:49.726 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:49.726 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:49.726 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:49.727 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.727 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.727 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:31 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'17, mlcod=14'14)", 2024-09-21T22:06:49.727 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.163958+0000", 2024-09-21T22:06:49.727 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 47.549742459000001, 2024-09-21T22:06:49.727 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0033891149999999998, 2024-09-21T22:06:49.727 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:49.727 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:49.727 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:49.727 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:49.727 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.727 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:49.728 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.163958+0000", 2024-09-21T22:06:49.728 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:49.728 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.728 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.728 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:49.728 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.163958+0000", 2024-09-21T22:06:49.728 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:49.728 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.728 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.728 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:49.728 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.163960+0000", 2024-09-21T22:06:49.729 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.64e-06 2024-09-21T22:06:49.729 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.729 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.729 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:49.729 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.163971+0000", 2024-09-21T22:06:49.729 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0797e-05 2024-09-21T22:06:49.729 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.729 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.729 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:49.729 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.163995+0000", 2024-09-21T22:06:49.729 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3876e-05 2024-09-21T22:06:49.729 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.730 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.730 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:49.730 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.164008+0000", 2024-09-21T22:06:49.730 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3674e-05 2024-09-21T22:06:49.730 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.730 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.730 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:49.730 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.164203+0000", 2024-09-21T22:06:49.730 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00019457799999999999 2024-09-21T22:06:49.730 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.730 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.731 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:49.731 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.164265+0000", 2024-09-21T22:06:49.731 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.2415999999999993e-05 2024-09-21T22:06:49.731 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.731 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.731 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:49.731 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.167209+0000", 2024-09-21T22:06:49.731 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0029436380000000002 2024-09-21T22:06:49.731 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.731 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.731 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:49.731 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.167348+0000", 2024-09-21T22:06:49.731 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000138496 2024-09-21T22:06:49.732 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:49.732 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:49.732 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:49.732 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.732 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.732 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:32 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'18, mlcod=14'14)", 2024-09-21T22:06:49.732 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.165319+0000", 2024-09-21T22:06:49.732 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 47.548381609000003, 2024-09-21T22:06:49.732 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0033722280000000001, 2024-09-21T22:06:49.732 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:49.732 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:49.733 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:49.733 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:49.733 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.733 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:49.733 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.165319+0000", 2024-09-21T22:06:49.733 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:49.733 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.733 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.733 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:49.733 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.165319+0000", 2024-09-21T22:06:49.733 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:49.733 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.733 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.734 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:49.734 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.165321+0000", 2024-09-21T22:06:49.734 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5999999999999999e-06 2024-09-21T22:06:49.734 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.734 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.734 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:49.734 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.165377+0000", 2024-09-21T22:06:49.734 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.5877000000000001e-05 2024-09-21T22:06:49.734 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.734 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.734 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:49.734 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.165407+0000", 2024-09-21T22:06:49.734 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.0263000000000001e-05 2024-09-21T22:06:49.734 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.735 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.735 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:49.735 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.165441+0000", 2024-09-21T22:06:49.735 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.3966e-05 2024-09-21T22:06:49.735 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.735 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.735 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:49.735 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.165574+0000", 2024-09-21T22:06:49.735 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013327599999999999 2024-09-21T22:06:49.735 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.735 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.735 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:49.735 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.165635+0000", 2024-09-21T22:06:49.735 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.0949000000000002e-05 2024-09-21T22:06:49.736 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.736 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.736 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:49.736 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.168619+0000", 2024-09-21T22:06:49.736 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0029836979999999999 2024-09-21T22:06:49.736 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.736 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.736 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:49.736 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.168692+0000", 2024-09-21T22:06:49.736 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.2599000000000001e-05 2024-09-21T22:06:49.736 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:49.736 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:49.736 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:49.736 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.736 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.737 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:38 1.0 e14/13 1:30306672:devicehealth::main.db.0000000000000000:head v 14'24, mlcod=14'22)", 2024-09-21T22:06:49.737 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.180244+0000", 2024-09-21T22:06:49.737 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 47.533456964000003, 2024-09-21T22:06:49.737 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0041712219999999996, 2024-09-21T22:06:49.737 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:49.737 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:49.737 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:49.737 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:49.737 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.737 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:49.737 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.180244+0000", 2024-09-21T22:06:49.737 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:49.737 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.737 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.738 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:49.738 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.180244+0000", 2024-09-21T22:06:49.738 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:49.738 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.738 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.738 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:49.738 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.180246+0000", 2024-09-21T22:06:49.738 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6959999999999999e-06 2024-09-21T22:06:49.738 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.738 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.738 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:49.738 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.180508+0000", 2024-09-21T22:06:49.738 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00026275199999999999 2024-09-21T22:06:49.738 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.739 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.739 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:49.739 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.180538+0000", 2024-09-21T22:06:49.739 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9142999999999999e-05 2024-09-21T22:06:49.739 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.739 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.739 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:49.739 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.180552+0000", 2024-09-21T22:06:49.739 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4466e-05 2024-09-21T22:06:49.739 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.739 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.739 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:49.739 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.180677+0000", 2024-09-21T22:06:49.739 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000125413 2024-09-21T22:06:49.740 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.740 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.740 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:49.740 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.180739+0000", 2024-09-21T22:06:49.740 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.1122000000000002e-05 2024-09-21T22:06:49.740 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.740 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.740 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:49.740 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.184291+0000", 2024-09-21T22:06:49.740 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0035525510000000001 2024-09-21T22:06:49.740 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.740 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.740 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:49.740 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.184415+0000", 2024-09-21T22:06:49.741 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000124079 2024-09-21T22:06:49.741 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:49.741 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:49.741 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:49.741 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.741 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.741 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:54 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'32, mlcod=14'29)", 2024-09-21T22:06:49.741 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.214326+0000", 2024-09-21T22:06:49.741 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 47.499374727000003, 2024-09-21T22:06:49.741 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0035226569999999998, 2024-09-21T22:06:49.741 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:49.741 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:49.741 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:49.741 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:49.742 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.742 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:49.742 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.214326+0000", 2024-09-21T22:06:49.742 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:49.742 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.742 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.742 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:49.742 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.214326+0000", 2024-09-21T22:06:49.742 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:49.742 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.742 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.742 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:49.742 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.214328+0000", 2024-09-21T22:06:49.743 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6270000000000001e-06 2024-09-21T22:06:49.743 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.743 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.743 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:49.743 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.214340+0000", 2024-09-21T22:06:49.743 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2286000000000001e-05 2024-09-21T22:06:49.743 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.743 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.743 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:49.743 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.214369+0000", 2024-09-21T22:06:49.743 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9297000000000001e-05 2024-09-21T22:06:49.743 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.744 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.744 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:49.744 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.214383+0000", 2024-09-21T22:06:49.744 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3773e-05 2024-09-21T22:06:49.744 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.744 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.744 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:49.744 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.214513+0000", 2024-09-21T22:06:49.744 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000130056 2024-09-21T22:06:49.744 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.744 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.744 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:49.745 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.214568+0000", 2024-09-21T22:06:49.745 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.4873000000000001e-05 2024-09-21T22:06:49.745 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.745 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.745 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:49.745 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.217739+0000", 2024-09-21T22:06:49.745 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0031712720000000002 2024-09-21T22:06:49.745 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.745 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.745 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:49.745 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.217849+0000", 2024-09-21T22:06:49.745 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000109473 2024-09-21T22:06:49.746 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:49.746 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:49.746 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:49.746 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.746 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.746 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:57 1.0 e14/13 1:30306672:devicehealth::main.db.0000000000000000:head v 14'35, mlcod=14'33)", 2024-09-21T22:06:49.746 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.221549+0000", 2024-09-21T22:06:49.746 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 47.492152228000002, 2024-09-21T22:06:49.746 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0041720860000000002, 2024-09-21T22:06:49.746 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:49.746 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:49.746 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:49.747 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:49.747 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.747 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:49.747 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.221549+0000", 2024-09-21T22:06:49.747 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:49.747 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.747 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.747 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:49.747 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.221549+0000", 2024-09-21T22:06:49.747 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:49.747 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.748 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.748 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:49.748 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.221550+0000", 2024-09-21T22:06:49.748 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.717e-06 2024-09-21T22:06:49.748 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.748 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.748 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:49.748 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.221673+0000", 2024-09-21T22:06:49.748 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000122612 2024-09-21T22:06:49.748 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.748 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.748 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:49.749 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.221703+0000", 2024-09-21T22:06:49.749 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9539999999999998e-05 2024-09-21T22:06:49.749 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.749 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.749 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:49.749 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.221716+0000", 2024-09-21T22:06:49.749 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3149999999999999e-05 2024-09-21T22:06:49.749 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.749 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.749 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:49.750 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.221848+0000", 2024-09-21T22:06:49.750 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013201900000000001 2024-09-21T22:06:49.750 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.750 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.750 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:49.750 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.221908+0000", 2024-09-21T22:06:49.750 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.0452999999999999e-05 2024-09-21T22:06:49.750 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.750 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.750 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:49.750 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.225573+0000", 2024-09-21T22:06:49.751 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.003664676 2024-09-21T22:06:49.751 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.751 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.751 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:49.751 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.225721+0000", 2024-09-21T22:06:49.751 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00014791899999999999 2024-09-21T22:06:49.751 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:49.751 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:49.751 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:49.751 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.751 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.752 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:68 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'42, mlcod=14'39)", 2024-09-21T22:06:49.752 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.247413+0000", 2024-09-21T22:06:49.752 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 47.466287878999999, 2024-09-21T22:06:49.752 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0039572339999999996, 2024-09-21T22:06:49.752 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:49.752 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:49.752 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:49.752 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:49.752 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.752 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:49.752 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.247413+0000", 2024-09-21T22:06:49.753 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:49.753 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.753 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.753 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:49.753 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.247413+0000", 2024-09-21T22:06:49.753 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:49.753 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.753 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.753 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:49.753 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.247415+0000", 2024-09-21T22:06:49.753 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.883e-06 2024-09-21T22:06:49.753 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.754 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.754 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:49.754 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.247427+0000", 2024-09-21T22:06:49.754 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2557e-05 2024-09-21T22:06:49.754 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.754 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.754 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:49.754 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.247457+0000", 2024-09-21T22:06:49.754 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9966000000000001e-05 2024-09-21T22:06:49.754 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.754 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.755 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:49.755 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.247471+0000", 2024-09-21T22:06:49.755 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3747000000000001e-05 2024-09-21T22:06:49.755 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.755 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.755 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:49.755 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.247603+0000", 2024-09-21T22:06:49.755 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013171600000000001 2024-09-21T22:06:49.755 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.755 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.755 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:49.756 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.247656+0000", 2024-09-21T22:06:49.756 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.3529999999999997e-05 2024-09-21T22:06:49.756 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.756 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.756 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:49.756 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.251225+0000", 2024-09-21T22:06:49.756 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00356865 2024-09-21T22:06:49.756 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.756 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.756 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:49.756 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.251370+0000", 2024-09-21T22:06:49.756 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00014518500000000001 2024-09-21T22:06:49.757 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:49.757 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:49.757 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:49.757 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.757 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.757 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:69 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'43, mlcod=14'39)", 2024-09-21T22:06:49.757 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.248839+0000", 2024-09-21T22:06:49.757 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 47.464861949000003, 2024-09-21T22:06:49.757 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0034954470000000001, 2024-09-21T22:06:49.757 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:49.758 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:49.758 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:49.758 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:49.758 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.758 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:49.758 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.248839+0000", 2024-09-21T22:06:49.758 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:49.758 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.758 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.758 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:49.758 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.248839+0000", 2024-09-21T22:06:49.759 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:49.759 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.759 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.759 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:49.759 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.248841+0000", 2024-09-21T22:06:49.759 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6759999999999999e-06 2024-09-21T22:06:49.759 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.759 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.759 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:49.759 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.249058+0000", 2024-09-21T22:06:49.760 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000217455 2024-09-21T22:06:49.760 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.760 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.760 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:49.760 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.249085+0000", 2024-09-21T22:06:49.760 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.7002999999999999e-05 2024-09-21T22:06:49.760 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.760 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.760 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:49.760 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.249099+0000", 2024-09-21T22:06:49.760 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.402e-05 2024-09-21T22:06:49.760 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.761 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.761 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:49.761 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.249241+0000", 2024-09-21T22:06:49.761 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00014194899999999999 2024-09-21T22:06:49.761 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.761 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.761 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:49.761 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.249298+0000", 2024-09-21T22:06:49.761 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.6687000000000001e-05 2024-09-21T22:06:49.761 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.761 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.761 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:49.761 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.252266+0000", 2024-09-21T22:06:49.762 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.002968348 2024-09-21T22:06:49.762 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.762 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.762 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:49.762 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.252334+0000", 2024-09-21T22:06:49.762 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.8308999999999999e-05 2024-09-21T22:06:49.762 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:49.762 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:49.762 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:49.762 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.762 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.762 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:71 1.0 e14/13 1:30306672:devicehealth::main.db.0000000000000000:head v 14'45, mlcod=14'43)", 2024-09-21T22:06:49.762 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.256068+0000", 2024-09-21T22:06:49.762 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 47.457632476999997, 2024-09-21T22:06:49.763 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0041712579999999997, 2024-09-21T22:06:49.763 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:49.763 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:49.763 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:49.763 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:49.763 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.763 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:49.763 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.256068+0000", 2024-09-21T22:06:49.763 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:49.763 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.763 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.763 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:49.763 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.256068+0000", 2024-09-21T22:06:49.763 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:49.764 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.764 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.764 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:49.764 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.256070+0000", 2024-09-21T22:06:49.764 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.61e-06 2024-09-21T22:06:49.764 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.764 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.764 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:49.764 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.256129+0000", 2024-09-21T22:06:49.764 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.8702999999999997e-05 2024-09-21T22:06:49.764 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.764 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.764 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:49.764 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.256158+0000", 2024-09-21T22:06:49.765 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9533e-05 2024-09-21T22:06:49.765 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.765 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.765 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:49.765 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.256173+0000", 2024-09-21T22:06:49.765 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4267000000000001e-05 2024-09-21T22:06:49.765 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.765 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.765 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:49.765 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.256304+0000", 2024-09-21T22:06:49.765 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000131352 2024-09-21T22:06:49.765 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.765 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.765 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:49.766 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.256357+0000", 2024-09-21T22:06:49.766 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.3492999999999998e-05 2024-09-21T22:06:49.766 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.766 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.766 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:49.766 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.260147+0000", 2024-09-21T22:06:49.766 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0037900410000000001 2024-09-21T22:06:49.766 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.766 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.766 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:49.766 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.260240+0000", 2024-09-21T22:06:49.766 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.2258999999999999e-05 2024-09-21T22:06:49.767 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:49.767 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:49.767 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:49.767 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.767 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.767 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:72 1.0 e14/13 1:30306672:devicehealth::main.db.0000000000000000:head v 14'46, mlcod=14'43)", 2024-09-21T22:06:49.767 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.258358+0000", 2024-09-21T22:06:49.767 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 47.455342483999999, 2024-09-21T22:06:49.767 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0041492919999999997, 2024-09-21T22:06:49.767 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:49.767 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:49.767 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:49.767 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:49.767 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.768 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:49.768 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.258358+0000", 2024-09-21T22:06:49.768 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:49.768 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.768 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.768 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:49.768 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.258358+0000", 2024-09-21T22:06:49.768 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:49.768 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.768 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.768 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:49.768 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.258360+0000", 2024-09-21T22:06:49.768 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.623e-06 2024-09-21T22:06:49.768 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.769 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.769 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:49.769 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.258415+0000", 2024-09-21T22:06:49.769 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.4756000000000002e-05 2024-09-21T22:06:49.769 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.769 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.769 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:49.769 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.258444+0000", 2024-09-21T22:06:49.769 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9164000000000001e-05 2024-09-21T22:06:49.769 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.769 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.769 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:49.769 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.258459+0000", 2024-09-21T22:06:49.769 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4996e-05 2024-09-21T22:06:49.769 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.770 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.770 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:49.770 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.258637+0000", 2024-09-21T22:06:49.770 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00017796500000000001 2024-09-21T22:06:49.770 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.770 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.770 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:49.770 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.258695+0000", 2024-09-21T22:06:49.770 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.8207e-05 2024-09-21T22:06:49.770 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.770 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.770 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:49.770 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.262421+0000", 2024-09-21T22:06:49.771 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0037261550000000001 2024-09-21T22:06:49.771 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.771 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.771 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:49.771 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.262508+0000", 2024-09-21T22:06:49.771 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.6426000000000002e-05 2024-09-21T22:06:49.771 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:49.771 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:49.771 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:49.771 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.771 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.771 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:87 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'57, mlcod=14'52)", 2024-09-21T22:06:49.771 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.290115+0000", 2024-09-21T22:06:49.772 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 47.423586039, 2024-09-21T22:06:49.772 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0030478200000000001, 2024-09-21T22:06:49.772 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:49.772 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:49.772 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:49.772 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:49.772 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.772 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:49.772 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.290115+0000", 2024-09-21T22:06:49.772 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:49.772 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.773 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.773 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:49.773 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.290115+0000", 2024-09-21T22:06:49.773 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:49.773 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.773 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.773 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:49.773 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.290116+0000", 2024-09-21T22:06:49.773 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3829999999999999e-06 2024-09-21T22:06:49.773 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.773 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.774 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:49.774 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.290191+0000", 2024-09-21T22:06:49.774 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.4562999999999997e-05 2024-09-21T22:06:49.774 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.774 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.774 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:49.774 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.290216+0000", 2024-09-21T22:06:49.774 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.565e-05 2024-09-21T22:06:49.774 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.774 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.774 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:49.775 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.290236+0000", 2024-09-21T22:06:49.775 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9326000000000001e-05 2024-09-21T22:06:49.775 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.775 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.775 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:49.775 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.290340+0000", 2024-09-21T22:06:49.775 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0001047 2024-09-21T22:06:49.775 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.775 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.775 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:49.775 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.290377+0000", 2024-09-21T22:06:49.775 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.6356000000000002e-05 2024-09-21T22:06:49.776 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.776 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.776 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:49.776 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.293096+0000", 2024-09-21T22:06:49.776 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.002718996 2024-09-21T22:06:49.776 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.776 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.776 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:49.776 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.293163+0000", 2024-09-21T22:06:49.777 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.6846000000000003e-05 2024-09-21T22:06:49.777 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:49.777 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:49.777 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:49.777 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.777 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.777 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:114 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'76, mlcod=14'73)", 2024-09-21T22:06:49.777 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.490030+0000", 2024-09-21T22:06:49.777 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 47.223670599999998, 2024-09-21T22:06:49.777 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.004227873, 2024-09-21T22:06:49.777 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:49.778 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:49.778 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:49.778 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:49.778 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.778 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:49.778 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.490030+0000", 2024-09-21T22:06:49.778 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:49.778 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.778 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.778 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:49.778 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.490030+0000", 2024-09-21T22:06:49.779 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:49.779 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.779 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.779 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:49.779 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.490031+0000", 2024-09-21T22:06:49.779 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.3300000000000001e-07 2024-09-21T22:06:49.779 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.779 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.779 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:49.779 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.490042+0000", 2024-09-21T22:06:49.779 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.1095e-05 2024-09-21T22:06:49.779 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.780 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.780 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:49.780 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.490053+0000", 2024-09-21T22:06:49.780 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0501e-05 2024-09-21T22:06:49.780 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.780 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.780 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:49.780 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.490060+0000", 2024-09-21T22:06:49.780 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.9829999999999999e-06 2024-09-21T22:06:49.780 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.780 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.781 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:49.781 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.490107+0000", 2024-09-21T22:06:49.781 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.7278999999999999e-05 2024-09-21T22:06:49.781 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.781 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.781 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:49.781 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.490124+0000", 2024-09-21T22:06:49.781 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7190999999999999e-05 2024-09-21T22:06:49.781 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.781 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.781 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:49.781 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.494229+0000", 2024-09-21T22:06:49.781 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0041050440000000004 2024-09-21T22:06:49.781 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.781 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.782 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:49.782 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.494258+0000", 2024-09-21T22:06:49.782 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9047000000000002e-05 2024-09-21T22:06:49.782 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:49.782 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:49.782 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:49.782 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.782 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.782 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:115 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'77, mlcod=14'73)", 2024-09-21T22:06:49.782 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.491380+0000", 2024-09-21T22:06:49.782 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 47.222320977999999, 2024-09-21T22:06:49.782 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0031550889999999998, 2024-09-21T22:06:49.783 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:49.783 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:49.783 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:49.783 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:49.783 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.783 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:49.783 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.491380+0000", 2024-09-21T22:06:49.783 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:49.783 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.783 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.783 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:49.784 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.491380+0000", 2024-09-21T22:06:49.784 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:49.784 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.784 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.784 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:49.784 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.491381+0000", 2024-09-21T22:06:49.784 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.1399999999999996e-07 2024-09-21T22:06:49.784 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.784 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.784 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:49.784 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.491435+0000", 2024-09-21T22:06:49.784 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.3797e-05 2024-09-21T22:06:49.785 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.785 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.785 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:49.785 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.491447+0000", 2024-09-21T22:06:49.785 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2642e-05 2024-09-21T22:06:49.785 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.785 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.785 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:49.785 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.491453+0000", 2024-09-21T22:06:49.785 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.4820000000000002e-06 2024-09-21T22:06:49.785 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.785 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.786 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:49.786 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.491502+0000", 2024-09-21T22:06:49.786 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.9038e-05 2024-09-21T22:06:49.786 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.786 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.786 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:49.786 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.491514+0000", 2024-09-21T22:06:49.786 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2483999999999999e-05 2024-09-21T22:06:49.786 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.786 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.787 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:49.787 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.494486+0000", 2024-09-21T22:06:49.787 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.002971338 2024-09-21T22:06:49.787 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.787 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.787 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:49.787 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.494535+0000", 2024-09-21T22:06:49.787 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.9493999999999997e-05 2024-09-21T22:06:49.787 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:49.787 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:49.787 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:49.788 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.788 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.788 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:118 1.0 e14/13 1:30306672:devicehealth::main.db.0000000000000000:head v 14'80, mlcod=14'78)", 2024-09-21T22:06:49.788 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.499045+0000", 2024-09-21T22:06:49.788 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 47.214655976000003, 2024-09-21T22:06:49.788 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0029889589999999998, 2024-09-21T22:06:49.788 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:49.788 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:49.788 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:49.788 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:49.788 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.788 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:49.789 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.499045+0000", 2024-09-21T22:06:49.789 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:49.789 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.789 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.789 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:49.789 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.499045+0000", 2024-09-21T22:06:49.789 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:49.789 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.789 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.789 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:49.789 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.499046+0000", 2024-09-21T22:06:49.789 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.0299999999999998e-07 2024-09-21T22:06:49.790 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.790 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.790 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:49.790 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.499110+0000", 2024-09-21T22:06:49.790 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.4688000000000001e-05 2024-09-21T22:06:49.790 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.790 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.790 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:49.790 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.499125+0000", 2024-09-21T22:06:49.790 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4387e-05 2024-09-21T22:06:49.790 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.790 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.791 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:49.791 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.499131+0000", 2024-09-21T22:06:49.791 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.2129999999999996e-06 2024-09-21T22:06:49.791 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.791 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.791 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:49.791 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.499255+0000", 2024-09-21T22:06:49.791 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00012424199999999999 2024-09-21T22:06:49.791 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.791 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.791 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:49.791 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.499312+0000", 2024-09-21T22:06:49.791 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.6709999999999997e-05 2024-09-21T22:06:49.792 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.792 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.792 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:49.792 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.502005+0000", 2024-09-21T22:06:49.792 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0026927169999999999 2024-09-21T22:06:49.792 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.792 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.792 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:49.792 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.502034+0000", 2024-09-21T22:06:49.792 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9198999999999999e-05 2024-09-21T22:06:49.792 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:49.792 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:49.792 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:49.792 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.792 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.793 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:136 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'94, mlcod=14'89)", 2024-09-21T22:06:49.793 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.903810+0000", 2024-09-21T22:06:49.793 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 46.809891004999997, 2024-09-21T22:06:49.793 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0062746829999999996, 2024-09-21T22:06:49.793 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:49.793 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:49.793 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:49.793 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:49.793 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.793 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:49.793 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.903810+0000", 2024-09-21T22:06:49.793 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:49.793 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.793 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.793 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:49.794 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.903810+0000", 2024-09-21T22:06:49.794 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:49.794 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.794 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.794 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:49.794 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.903810+0000", 2024-09-21T22:06:49.794 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.03e-07 2024-09-21T22:06:49.794 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.794 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.794 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:49.794 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.903814+0000", 2024-09-21T22:06:49.794 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.676e-06 2024-09-21T22:06:49.794 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.795 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.795 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:49.795 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.903822+0000", 2024-09-21T22:06:49.795 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.2879999999999993e-06 2024-09-21T22:06:49.795 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.795 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.795 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:49.795 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.903829+0000", 2024-09-21T22:06:49.795 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.2920000000000001e-06 2024-09-21T22:06:49.795 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.795 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.795 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:49.795 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.903879+0000", 2024-09-21T22:06:49.795 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.0253999999999997e-05 2024-09-21T22:06:49.796 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.796 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.796 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:49.796 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.903897+0000", 2024-09-21T22:06:49.796 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8468999999999999e-05 2024-09-21T22:06:49.796 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.796 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.796 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:49.796 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.910044+0000", 2024-09-21T22:06:49.796 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0061468950000000003 2024-09-21T22:06:49.796 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.796 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.796 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:49.796 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.910085+0000", 2024-09-21T22:06:49.797 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.0405999999999999e-05 2024-09-21T22:06:49.797 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:49.797 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:49.797 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:49.797 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.797 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.797 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:137 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'95, mlcod=14'90)", 2024-09-21T22:06:49.797 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.904439+0000", 2024-09-21T22:06:49.797 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 46.809261609000004, 2024-09-21T22:06:49.797 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0072625019999999997, 2024-09-21T22:06:49.797 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:49.797 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:49.797 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:49.797 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:49.798 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.798 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:49.798 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.904439+0000", 2024-09-21T22:06:49.798 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:49.798 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.798 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.798 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:49.798 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.904439+0000", 2024-09-21T22:06:49.798 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:49.798 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.798 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.798 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:49.798 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.904440+0000", 2024-09-21T22:06:49.799 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8400000000000005e-07 2024-09-21T22:06:49.799 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.799 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.799 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:49.799 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.904498+0000", 2024-09-21T22:06:49.799 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.8320999999999999e-05 2024-09-21T22:06:49.799 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.799 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.799 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:49.799 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.904507+0000", 2024-09-21T22:06:49.799 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.7790000000000001e-06 2024-09-21T22:06:49.799 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.799 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.800 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:49.800 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.904511+0000", 2024-09-21T22:06:49.800 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.9700000000000001e-06 2024-09-21T22:06:49.800 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.800 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.800 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:49.800 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.904551+0000", 2024-09-21T22:06:49.800 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.9910000000000002e-05 2024-09-21T22:06:49.800 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.800 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.800 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:49.800 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.904567+0000", 2024-09-21T22:06:49.800 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5903999999999999e-05 2024-09-21T22:06:49.801 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.801 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.801 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:49.801 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.911666+0000", 2024-09-21T22:06:49.801 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0070996929999999998 2024-09-21T22:06:49.801 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.801 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.801 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:49.801 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.911702+0000", 2024-09-21T22:06:49.801 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.5441000000000002e-05 2024-09-21T22:06:49.801 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:49.801 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:49.801 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:49.801 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.802 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.802 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:138 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'96, mlcod=14'90)", 2024-09-21T22:06:49.802 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.905402+0000", 2024-09-21T22:06:49.802 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 46.808298632000003, 2024-09-21T22:06:49.802 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0063366890000000004, 2024-09-21T22:06:49.802 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:49.802 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:49.802 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:49.802 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:49.802 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.802 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:49.802 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905402+0000", 2024-09-21T22:06:49.803 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:49.803 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.803 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.803 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:49.803 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905402+0000", 2024-09-21T22:06:49.803 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:49.803 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.803 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.803 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:49.803 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905403+0000", 2024-09-21T22:06:49.803 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.9200000000000002e-07 2024-09-21T22:06:49.803 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.803 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.804 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:49.804 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905406+0000", 2024-09-21T22:06:49.804 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.326e-06 2024-09-21T22:06:49.804 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.804 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.804 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:49.804 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905413+0000", 2024-09-21T22:06:49.804 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.3320000000000003e-06 2024-09-21T22:06:49.804 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.804 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.804 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:49.804 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905417+0000", 2024-09-21T22:06:49.805 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.7139999999999999e-06 2024-09-21T22:06:49.805 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.805 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.805 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:49.805 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.910115+0000", 2024-09-21T22:06:49.805 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0046983939999999998 2024-09-21T22:06:49.805 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.805 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.805 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:49.805 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.910140+0000", 2024-09-21T22:06:49.805 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4689999999999999e-05 2024-09-21T22:06:49.805 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.806 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.806 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:49.806 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.911706+0000", 2024-09-21T22:06:49.806 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0015658479999999999 2024-09-21T22:06:49.806 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.806 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.806 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:49.806 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.911739+0000", 2024-09-21T22:06:49.806 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.2993000000000001e-05 2024-09-21T22:06:49.806 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:49.806 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:49.807 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:49.807 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.807 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.807 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:139 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'97, mlcod=14'90)", 2024-09-21T22:06:49.807 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.905983+0000", 2024-09-21T22:06:49.807 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 46.807717713999999, 2024-09-21T22:06:49.807 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0057906030000000001, 2024-09-21T22:06:49.807 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:49.807 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:49.807 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:49.807 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:49.807 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.807 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:49.808 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905983+0000", 2024-09-21T22:06:49.808 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:49.808 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.808 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.808 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:49.808 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905983+0000", 2024-09-21T22:06:49.808 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:49.808 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.808 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.808 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:49.808 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905984+0000", 2024-09-21T22:06:49.808 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.15e-07 2024-09-21T22:06:49.809 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.809 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.809 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:49.809 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905986+0000", 2024-09-21T22:06:49.809 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.852e-06 2024-09-21T22:06:49.809 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.809 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.809 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:49.809 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905993+0000", 2024-09-21T22:06:49.809 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.652e-06 2024-09-21T22:06:49.809 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.809 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.810 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:49.810 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905996+0000", 2024-09-21T22:06:49.810 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.3519999999999998e-06 2024-09-21T22:06:49.810 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.810 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.810 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:49.810 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.910519+0000", 2024-09-21T22:06:49.810 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0045230820000000003 2024-09-21T22:06:49.810 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.810 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.810 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:49.810 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.910548+0000", 2024-09-21T22:06:49.810 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8399999999999999e-05 2024-09-21T22:06:49.811 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.811 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.811 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:49.811 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.911742+0000", 2024-09-21T22:06:49.811 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011939909999999999 2024-09-21T22:06:49.811 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.811 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.811 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:49.811 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.911774+0000", 2024-09-21T22:06:49.811 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.1958999999999997e-05 2024-09-21T22:06:49.812 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:49.812 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:49.812 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:49.812 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.812 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.812 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:140 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'98, mlcod=14'90)", 2024-09-21T22:06:49.812 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.906441+0000", 2024-09-21T22:06:49.812 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 46.807259584999997, 2024-09-21T22:06:49.812 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0057052989999999996, 2024-09-21T22:06:49.812 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:49.812 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:49.812 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:49.813 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:49.813 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.813 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:49.813 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.906441+0000", 2024-09-21T22:06:49.813 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:49.813 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.813 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.813 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:49.813 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.906441+0000", 2024-09-21T22:06:49.813 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:49.813 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.813 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.814 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:49.814 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.906442+0000", 2024-09-21T22:06:49.814 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.7e-07 2024-09-21T22:06:49.814 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.814 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.814 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:49.814 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.906511+0000", 2024-09-21T22:06:49.814 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.8857999999999997e-05 2024-09-21T22:06:49.814 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.814 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.814 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:49.815 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.906518+0000", 2024-09-21T22:06:49.815 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.1690000000000003e-06 2024-09-21T22:06:49.815 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.815 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.815 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:49.815 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.906521+0000", 2024-09-21T22:06:49.815 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.4829999999999999e-06 2024-09-21T22:06:49.815 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.815 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.815 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:49.815 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.910939+0000", 2024-09-21T22:06:49.816 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0044176809999999997 2024-09-21T22:06:49.816 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.816 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.816 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:49.816 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.910959+0000", 2024-09-21T22:06:49.816 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9898000000000002e-05 2024-09-21T22:06:49.816 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.816 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.816 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:49.816 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.912113+0000", 2024-09-21T22:06:49.816 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001153811 2024-09-21T22:06:49.816 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:49.817 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:49.817 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:49.817 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.912147+0000", 2024-09-21T22:06:49.817 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.4029000000000002e-05 2024-09-21T22:06:49.817 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:49.817 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:49.817 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:49.817 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:49.817 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:49.817 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:06:49.819 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T22:06:49.820 INFO:teuthology.orchestra.run.smithi039.stdout:osd.0: {} 2024-09-21T22:06:49.820 INFO:teuthology.orchestra.run.smithi039.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T22:06:49.820 INFO:teuthology.orchestra.run.smithi039.stdout:osd.1: {} 2024-09-21T22:06:49.820 INFO:teuthology.orchestra.run.smithi039.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T22:06:49.821 INFO:teuthology.orchestra.run.smithi039.stdout:osd.2: {} 2024-09-21T22:06:49.821 INFO:teuthology.orchestra.run.smithi039.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T22:06:49.821 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:06:49.816+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:06:49.831 INFO:teuthology.orchestra.run.smithi039.stdout:osd.3: {} 2024-09-21T22:06:49.831 INFO:teuthology.orchestra.run.smithi039.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T22:06:49.851 INFO:teuthology.orchestra.run.smithi039.stdout:osd.4: {} 2024-09-21T22:06:49.851 INFO:teuthology.orchestra.run.smithi039.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T22:06:49.920 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:06:49.916+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:06:49.948 INFO:teuthology.orchestra.run.smithi039.stderr:noscrub is set 2024-09-21T22:06:49.953 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:49.953 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:49.971 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T22:06:50.059 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:06:50.056+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:06:50.121 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:06:50.116+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:06:50.140 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:50.140 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:50.154 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T22:06:50.223 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:06:50.220+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:06:50.323 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:06:50.320+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:06:50.324 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:50.324 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:50.339 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T22:06:50.431 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:06:50.428+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:06:50.513 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:50.513 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:50.525 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:06:50.520+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:06:50.529 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T22:06:50.626 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:06:50.620+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:06:50.636 INFO:teuthology.orchestra.run.smithi064.stdout:2.3 2024-09-21T22:06:50.636 INFO:teuthology.orchestra.run.smithi064.stdout:2.2 2024-09-21T22:06:50.689 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:50.689 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:50.703 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T22:06:50.727 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:06:50.724+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:06:50.828 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:06:50.824+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:06:50.873 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:50.873 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:50.887 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T22:06:50.931 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:06:50.928+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:06:51.024 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:51.024 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:51.031 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:06:51.028+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:06:51.039 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T22:06:51.132 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:06:51.128+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:06:51.165 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:51.165 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:51.180 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T22:06:51.181 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal --op export --pgid 2.3 --file /var/log/ceph/exp.2.3.5 2024-09-21T22:06:51.232 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:06:51.232+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:06:51.333 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:06:51.328+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:06:51.358 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:51.358 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:51.373 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T22:06:51.459 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:06:51.456+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:06:51.534 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:06:51.528+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:06:51.541 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:51.541 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:51.555 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T22:06:51.647 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:06:51.644+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:06:51.714 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:51.714 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:51.728 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T22:06:51.736 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:06:51.736+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:06:51.838 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:06:51.832+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:06:51.900 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:51.900 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:51.918 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T22:06:51.963 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:06:51.960+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:06:51.970 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-21T22:06:52.040 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:06:52.036+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:06:52.054 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:52.054 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:52.069 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T22:06:52.155 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:06:52.152+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:06:52.227 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:52.227 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:52.242 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T22:06:52.242 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:06:52.236+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:06:52.343 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:06:52.340+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:06:52.403 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:52.532 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:52.534 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T22:06:52.534 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:06:52.528+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:06:52.635 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:06:52.632+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:06:52.666 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:52.666 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:52.681 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T22:06:52.767 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:06:52.764+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:06:52.788 INFO:teuthology.orchestra.run.smithi064.stderr:Exporting 2.3 info 2.3( v 17'1 (0'0,17'1] local-lis/les=16/17 n=1 ec=16/16 lis/c=16/16 les/c/f=17/17/0 sis=16) 2024-09-21T22:06:52.795 INFO:teuthology.orchestra.run.smithi064.stderr:Read #2:c4c92e5a:::rbd_trash:head# 2024-09-21T22:06:52.795 INFO:teuthology.orchestra.run.smithi064.stderr:Export successful 2024-09-21T22:06:52.837 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:06:52.832+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:06:52.851 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:52.851 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:52.866 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T22:06:52.938 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:06:52.932+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:06:52.963 INFO:teuthology.orchestra.run.smithi039.stderr:nodeep-scrub is set 2024-09-21T22:06:53.039 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:06:53.036+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:06:53.043 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:53.043 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:53.057 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T22:06:53.140 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:06:53.136+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:06:53.234 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:53.234 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:53.240 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:06:53.236+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:06:53.249 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T22:06:53.340 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal --force --op remove --pgid 2.3 2024-09-21T22:06:53.341 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:06:53.336+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:06:53.411 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:53.411 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:53.426 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T22:06:53.471 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:06:53.468+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:06:53.543 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:06:53.540+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:06:53.590 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:53.591 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:53.606 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T22:06:53.647 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:06:53.644+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:06:53.742 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:53.742 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:53.746 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:06:53.740+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:06:53.757 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T22:06:53.847 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:06:53.844+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:06:53.918 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:53.918 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:53.933 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T22:06:53.948 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:06:53.944+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:06:54.048 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:06:54.044+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:06:54.108 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:54.109 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:54.125 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T22:06:54.149 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:06:54.144+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:06:54.250 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:06:54.244+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:06:54.253 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:54.253 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:54.267 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T22:06:54.351 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:06:54.348+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:06:54.440 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:54.440 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:54.452 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:06:54.452+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:06:54.454 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T22:06:54.553 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:06:54.548+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:06:54.581 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:54.581 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:54.597 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T22:06:54.654 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:06:54.652+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:06:54.732 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:54.732 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:54.746 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T22:06:54.763 INFO:teuthology.orchestra.run.smithi039.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-21T22:06:54.764 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:06:54.760+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:06:54.767 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T22:06:54.767 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T22:06:54.856 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:06:54.852+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:06:54.866 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:54.866 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:54.881 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T22:06:54.923 INFO:teuthology.orchestra.run.smithi064.stdout: marking collection for removal 2024-09-21T22:06:54.924 INFO:teuthology.orchestra.run.smithi064.stdout:setting '_remove' omap key 2024-09-21T22:06:54.924 INFO:teuthology.orchestra.run.smithi064.stdout:finish_remove_pgs 2.3_head removing 2.3 2024-09-21T22:06:54.933 INFO:teuthology.orchestra.run.smithi064.stdout:Remove successful 2024-09-21T22:06:54.957 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:06:54.952+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:06:54.987 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-21T22:06:55.008 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:55.008 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:55.024 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T22:06:55.071 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:06:55.068+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:06:55.159 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:06:55.156+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:06:55.190 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:55.190 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:55.207 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T22:06:55.259 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:06:55.256+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:06:55.330 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:55.330 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:55.345 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T22:06:55.360 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:06:55.356+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:06:55.461 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:06:55.456+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:06:55.471 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.2.3.5 2024-09-21T22:06:55.513 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:55.514 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:55.528 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T22:06:55.571 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:06:55.568+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:06:55.663 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:06:55.660+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:06:55.698 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:55.698 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:55.713 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T22:06:55.799 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:06:55.796+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:06:55.865 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:06:55.860+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:06:55.883 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:55.883 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:55.899 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T22:06:55.966 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:06:55.964+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:06:55.982 INFO:teuthology.orchestra.run.smithi039.stderr:noscrub is unset 2024-09-21T22:06:56.067 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:06:56.064+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:06:56.073 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:56.073 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:56.088 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T22:06:56.168 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:06:56.164+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:06:56.227 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:56.227 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:56.243 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T22:06:56.268 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:06:56.264+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:06:56.369 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:06:56.368+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:06:56.373 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:56.374 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:56.387 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T22:06:56.470 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:06:56.468+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:06:56.512 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:56.512 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:56.526 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T22:06:56.571 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:06:56.568+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:06:56.652 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:56.652 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:56.666 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T22:06:56.707 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:06:56.704+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:06:56.768 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T22:06:56.772 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:06:56.768+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:06:56.834 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:56.834 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:56.847 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T22:06:56.873 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:06:56.868+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:06:56.927 INFO:teuthology.orchestra.run.smithi039.stdout:osd.0: {} 2024-09-21T22:06:56.928 INFO:teuthology.orchestra.run.smithi039.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T22:06:56.944 INFO:teuthology.orchestra.run.smithi039.stdout:osd.1: {} 2024-09-21T22:06:56.945 INFO:teuthology.orchestra.run.smithi039.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T22:06:56.961 INFO:teuthology.orchestra.run.smithi039.stdout:osd.2: {} 2024-09-21T22:06:56.961 INFO:teuthology.orchestra.run.smithi039.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T22:06:56.973 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:06:56.968+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:06:56.978 INFO:teuthology.orchestra.run.smithi039.stdout:osd.3: {} 2024-09-21T22:06:56.978 INFO:teuthology.orchestra.run.smithi039.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T22:06:57.000 INFO:teuthology.orchestra.run.smithi039.stdout:osd.4: {} 2024-09-21T22:06:57.000 INFO:teuthology.orchestra.run.smithi039.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T22:06:57.001 INFO:teuthology.orchestra.run.smithi039.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-21T22:06:57.005 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T22:06:57.005 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T22:06:57.015 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:06:57.016 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-21T22:06:57.016 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-21T22:06:57.016 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [] 2024-09-21T22:06:57.016 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:06:57.030 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T22:06:57.074 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:06:57.072+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:06:57.102 INFO:teuthology.orchestra.run.smithi064.stdout:get_pg_num_history pg_num_history pg_num_history(e41 pg_nums {1={13=1},2={16=8}} deleted_pools ) 2024-09-21T22:06:57.102 INFO:teuthology.orchestra.run.smithi064.stdout:Importing pgid 2.3 2024-09-21T22:06:57.102 INFO:teuthology.orchestra.run.smithi064.stdout:Write #2:c4c92e5a:::rbd_trash:head# 2024-09-21T22:06:57.102 INFO:teuthology.orchestra.run.smithi064.stdout:snapset 0=[]:{} 2024-09-21T22:06:57.110 INFO:teuthology.orchestra.run.smithi064.stdout:write_pg epoch 20 info 2.3( v 17'1 (0'0,17'1] local-lis/les=16/17 n=1 ec=16/16 lis/c=16/16 les/c/f=17/17/0 sis=16) 2024-09-21T22:06:57.110 INFO:teuthology.orchestra.run.smithi064.stdout:Import successful 2024-09-21T22:06:57.174 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:06:57.174 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:06:57.174 INFO:teuthology.orchestra.run.smithi039.stdout: "num_ops": 0 2024-09-21T22:06:57.174 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:06:57.175 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:06:57.172+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:06:57.189 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T22:06:57.279 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:06:57.276+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:06:57.358 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:06:57.358 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:06:57.358 INFO:teuthology.orchestra.run.smithi039.stdout: "complaint_time": 30, 2024-09-21T22:06:57.358 INFO:teuthology.orchestra.run.smithi039.stdout: "num_blocked_ops": 0 2024-09-21T22:06:57.358 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:06:57.373 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T22:06:57.378 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:06:57.372+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:06:57.478 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:06:57.472+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:06:57.542 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:06:57.542 INFO:teuthology.orchestra.run.smithi039.stdout: "size": 20, 2024-09-21T22:06:57.542 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 600, 2024-09-21T22:06:57.542 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [ 2024-09-21T22:06:57.542 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.542 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:20 1.0 e14/13 1:30306672:devicehealth::main.db.0000000000000000:head v 14'10, mlcod=14'8)", 2024-09-21T22:06:57.543 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.137639+0000", 2024-09-21T22:06:57.543 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 55.402676141999997, 2024-09-21T22:06:57.543 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0039902100000000001, 2024-09-21T22:06:57.543 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:57.543 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:57.543 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:57.543 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:57.543 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.543 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:57.543 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.137639+0000", 2024-09-21T22:06:57.543 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:57.543 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.543 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.544 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:57.544 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.137639+0000", 2024-09-21T22:06:57.544 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:57.544 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.544 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.544 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:57.544 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.137641+0000", 2024-09-21T22:06:57.544 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.686e-06 2024-09-21T22:06:57.545 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.545 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.545 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:57.545 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.137934+0000", 2024-09-21T22:06:57.545 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00029319099999999999 2024-09-21T22:06:57.545 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.545 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.545 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:57.545 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.137963+0000", 2024-09-21T22:06:57.545 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9502999999999999e-05 2024-09-21T22:06:57.545 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.545 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.545 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:57.546 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.137989+0000", 2024-09-21T22:06:57.546 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5573000000000001e-05 2024-09-21T22:06:57.546 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.546 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.546 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:57.546 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.138086+0000", 2024-09-21T22:06:57.546 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.6968999999999999e-05 2024-09-21T22:06:57.546 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.546 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.546 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:57.546 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.138120+0000", 2024-09-21T22:06:57.546 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.4560000000000001e-05 2024-09-21T22:06:57.546 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.547 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.547 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:57.547 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.141539+0000", 2024-09-21T22:06:57.547 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0034183289999999999 2024-09-21T22:06:57.547 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.547 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.547 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:57.547 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.141629+0000", 2024-09-21T22:06:57.547 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.0399e-05 2024-09-21T22:06:57.547 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:57.547 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:57.548 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:57.548 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.548 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.548 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:23 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'13, mlcod=14'11)", 2024-09-21T22:06:57.548 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.144914+0000", 2024-09-21T22:06:57.548 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 55.395400823999999, 2024-09-21T22:06:57.548 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0040223230000000004, 2024-09-21T22:06:57.548 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:57.548 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:57.548 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:57.548 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:57.548 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.548 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:57.548 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.144914+0000", 2024-09-21T22:06:57.549 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:57.549 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.549 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.549 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:57.549 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.144914+0000", 2024-09-21T22:06:57.549 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:57.549 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.549 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.549 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:57.550 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.144916+0000", 2024-09-21T22:06:57.550 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6470000000000001e-06 2024-09-21T22:06:57.550 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.550 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.550 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:57.550 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.144929+0000", 2024-09-21T22:06:57.550 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2903e-05 2024-09-21T22:06:57.550 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.550 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.550 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:57.551 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.144958+0000", 2024-09-21T22:06:57.551 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9280000000000001e-05 2024-09-21T22:06:57.551 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.551 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.551 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:57.551 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.144971+0000", 2024-09-21T22:06:57.551 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3407000000000001e-05 2024-09-21T22:06:57.551 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.551 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.551 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:57.552 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.145106+0000", 2024-09-21T22:06:57.552 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013440500000000001 2024-09-21T22:06:57.552 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.552 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.552 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:57.552 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.145170+0000", 2024-09-21T22:06:57.552 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.4597000000000005e-05 2024-09-21T22:06:57.552 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.552 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.552 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:57.552 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.148854+0000", 2024-09-21T22:06:57.552 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0036833249999999999 2024-09-21T22:06:57.553 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.553 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.553 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:57.553 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.148937+0000", 2024-09-21T22:06:57.553 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.2758999999999998e-05 2024-09-21T22:06:57.553 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:57.553 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:57.553 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:57.553 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.554 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.554 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:31 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'17, mlcod=14'14)", 2024-09-21T22:06:57.554 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.163958+0000", 2024-09-21T22:06:57.554 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 55.376356557000001, 2024-09-21T22:06:57.554 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0033891149999999998, 2024-09-21T22:06:57.554 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:57.555 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:57.555 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:57.555 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:57.555 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.555 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:57.555 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.163958+0000", 2024-09-21T22:06:57.555 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:57.555 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.555 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.556 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:57.556 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.163958+0000", 2024-09-21T22:06:57.556 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:57.556 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.556 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.556 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:57.556 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.163960+0000", 2024-09-21T22:06:57.556 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.64e-06 2024-09-21T22:06:57.556 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.556 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.556 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:57.556 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.163971+0000", 2024-09-21T22:06:57.556 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0797e-05 2024-09-21T22:06:57.557 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.557 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.557 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:57.557 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.163995+0000", 2024-09-21T22:06:57.557 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3876e-05 2024-09-21T22:06:57.557 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.557 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.557 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:57.557 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.164008+0000", 2024-09-21T22:06:57.557 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3674e-05 2024-09-21T22:06:57.557 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.557 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.557 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:57.558 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.164203+0000", 2024-09-21T22:06:57.558 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00019457799999999999 2024-09-21T22:06:57.558 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.558 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.558 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:57.558 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.164265+0000", 2024-09-21T22:06:57.558 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.2415999999999993e-05 2024-09-21T22:06:57.558 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.558 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.558 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:57.558 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.167209+0000", 2024-09-21T22:06:57.558 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0029436380000000002 2024-09-21T22:06:57.559 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.559 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.559 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:57.559 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.167348+0000", 2024-09-21T22:06:57.559 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000138496 2024-09-21T22:06:57.559 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:57.559 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:57.559 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:57.559 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.559 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.559 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:32 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'18, mlcod=14'14)", 2024-09-21T22:06:57.559 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.165319+0000", 2024-09-21T22:06:57.559 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 55.374995706999997, 2024-09-21T22:06:57.560 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0033722280000000001, 2024-09-21T22:06:57.560 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:57.560 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:57.560 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:57.560 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:57.560 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.560 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:57.560 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.165319+0000", 2024-09-21T22:06:57.560 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:57.560 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.560 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.560 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:57.561 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.165319+0000", 2024-09-21T22:06:57.561 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:57.561 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.561 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.561 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:57.561 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.165321+0000", 2024-09-21T22:06:57.561 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5999999999999999e-06 2024-09-21T22:06:57.561 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.561 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.561 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:57.561 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.165377+0000", 2024-09-21T22:06:57.561 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.5877000000000001e-05 2024-09-21T22:06:57.561 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.562 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.562 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:57.562 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.165407+0000", 2024-09-21T22:06:57.562 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.0263000000000001e-05 2024-09-21T22:06:57.562 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.562 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.562 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:57.562 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.165441+0000", 2024-09-21T22:06:57.563 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.3966e-05 2024-09-21T22:06:57.563 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.563 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.563 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:57.563 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.165574+0000", 2024-09-21T22:06:57.563 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013327599999999999 2024-09-21T22:06:57.563 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.563 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.563 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:57.563 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.165635+0000", 2024-09-21T22:06:57.563 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.0949000000000002e-05 2024-09-21T22:06:57.563 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.564 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.564 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:57.564 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.168619+0000", 2024-09-21T22:06:57.564 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0029836979999999999 2024-09-21T22:06:57.564 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.564 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.564 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:57.564 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.168692+0000", 2024-09-21T22:06:57.564 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.2599000000000001e-05 2024-09-21T22:06:57.564 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:57.564 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:57.564 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:57.565 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.565 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.565 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:38 1.0 e14/13 1:30306672:devicehealth::main.db.0000000000000000:head v 14'24, mlcod=14'22)", 2024-09-21T22:06:57.565 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.180244+0000", 2024-09-21T22:06:57.565 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 55.360071062000003, 2024-09-21T22:06:57.565 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0041712219999999996, 2024-09-21T22:06:57.565 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:57.565 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:57.565 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:57.565 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:57.565 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.565 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:57.565 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.180244+0000", 2024-09-21T22:06:57.566 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:57.566 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.566 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.566 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:57.566 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.180244+0000", 2024-09-21T22:06:57.566 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:57.566 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.566 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.566 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:57.566 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.180246+0000", 2024-09-21T22:06:57.566 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6959999999999999e-06 2024-09-21T22:06:57.566 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.567 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.567 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:57.567 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.180508+0000", 2024-09-21T22:06:57.567 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00026275199999999999 2024-09-21T22:06:57.567 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.567 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.567 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:57.567 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.180538+0000", 2024-09-21T22:06:57.567 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9142999999999999e-05 2024-09-21T22:06:57.567 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.567 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.567 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:57.568 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.180552+0000", 2024-09-21T22:06:57.568 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4466e-05 2024-09-21T22:06:57.568 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.568 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.568 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:57.568 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.180677+0000", 2024-09-21T22:06:57.568 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000125413 2024-09-21T22:06:57.568 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.568 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.568 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:57.568 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.180739+0000", 2024-09-21T22:06:57.568 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.1122000000000002e-05 2024-09-21T22:06:57.568 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.569 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.569 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:57.569 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.184291+0000", 2024-09-21T22:06:57.569 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0035525510000000001 2024-09-21T22:06:57.569 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.569 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.569 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:57.569 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.184415+0000", 2024-09-21T22:06:57.569 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000124079 2024-09-21T22:06:57.569 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:57.569 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:57.569 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:57.570 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.570 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.570 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:54 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'32, mlcod=14'29)", 2024-09-21T22:06:57.570 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.214326+0000", 2024-09-21T22:06:57.570 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 55.325988825000003, 2024-09-21T22:06:57.570 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0035226569999999998, 2024-09-21T22:06:57.570 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:57.570 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:57.570 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:57.570 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:57.570 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.570 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:57.571 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.214326+0000", 2024-09-21T22:06:57.571 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:57.571 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.571 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.571 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:57.571 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.214326+0000", 2024-09-21T22:06:57.571 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:57.571 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.571 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.571 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:57.571 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.214328+0000", 2024-09-21T22:06:57.571 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6270000000000001e-06 2024-09-21T22:06:57.572 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.572 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.572 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:57.572 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.214340+0000", 2024-09-21T22:06:57.572 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2286000000000001e-05 2024-09-21T22:06:57.572 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.572 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.572 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:57.572 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.214369+0000", 2024-09-21T22:06:57.572 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9297000000000001e-05 2024-09-21T22:06:57.572 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.572 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.573 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:57.573 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.214383+0000", 2024-09-21T22:06:57.573 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3773e-05 2024-09-21T22:06:57.573 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.573 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.573 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:57.573 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.214513+0000", 2024-09-21T22:06:57.573 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000130056 2024-09-21T22:06:57.573 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.573 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.573 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:57.573 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.214568+0000", 2024-09-21T22:06:57.573 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.4873000000000001e-05 2024-09-21T22:06:57.574 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.574 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.574 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:57.574 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.217739+0000", 2024-09-21T22:06:57.574 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0031712720000000002 2024-09-21T22:06:57.574 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.574 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.574 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:57.574 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.217849+0000", 2024-09-21T22:06:57.574 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000109473 2024-09-21T22:06:57.574 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:57.574 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:57.575 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:57.575 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.575 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.575 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:57 1.0 e14/13 1:30306672:devicehealth::main.db.0000000000000000:head v 14'35, mlcod=14'33)", 2024-09-21T22:06:57.575 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.221549+0000", 2024-09-21T22:06:57.575 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 55.318766326000002, 2024-09-21T22:06:57.575 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0041720860000000002, 2024-09-21T22:06:57.575 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:57.575 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:57.575 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:57.575 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:57.575 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.575 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:57.576 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.221549+0000", 2024-09-21T22:06:57.576 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:57.576 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.576 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.576 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:57.576 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.221549+0000", 2024-09-21T22:06:57.576 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:57.576 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.576 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.576 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:57.576 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.221550+0000", 2024-09-21T22:06:57.576 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.717e-06 2024-09-21T22:06:57.577 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.577 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.577 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:57.577 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.221673+0000", 2024-09-21T22:06:57.577 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000122612 2024-09-21T22:06:57.577 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.577 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.577 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:57.577 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.221703+0000", 2024-09-21T22:06:57.577 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9539999999999998e-05 2024-09-21T22:06:57.577 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.577 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.578 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:57.578 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.221716+0000", 2024-09-21T22:06:57.578 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3149999999999999e-05 2024-09-21T22:06:57.578 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.578 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.578 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:57.578 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.221848+0000", 2024-09-21T22:06:57.578 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013201900000000001 2024-09-21T22:06:57.578 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.578 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.578 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:57.578 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.221908+0000", 2024-09-21T22:06:57.579 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.0452999999999999e-05 2024-09-21T22:06:57.579 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.579 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.579 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:57.579 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.225573+0000", 2024-09-21T22:06:57.579 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.003664676 2024-09-21T22:06:57.579 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.579 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.579 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:57.579 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.225721+0000", 2024-09-21T22:06:57.579 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00014791899999999999 2024-09-21T22:06:57.580 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:57.580 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:57.580 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:57.580 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.580 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.580 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:68 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'42, mlcod=14'39)", 2024-09-21T22:06:57.580 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.247413+0000", 2024-09-21T22:06:57.580 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 55.292901977, 2024-09-21T22:06:57.580 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0039572339999999996, 2024-09-21T22:06:57.580 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:57.580 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:57.580 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:57.581 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:57.581 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.581 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:57.581 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.247413+0000", 2024-09-21T22:06:57.581 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:57.581 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.581 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.581 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:57.581 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.247413+0000", 2024-09-21T22:06:57.581 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:57.581 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.581 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.581 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:57.582 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.247415+0000", 2024-09-21T22:06:57.582 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.883e-06 2024-09-21T22:06:57.582 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.582 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.582 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:57.582 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.247427+0000", 2024-09-21T22:06:57.582 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2557e-05 2024-09-21T22:06:57.582 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.582 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.582 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:57.582 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.247457+0000", 2024-09-21T22:06:57.582 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9966000000000001e-05 2024-09-21T22:06:57.583 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.583 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.583 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:57.583 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.247471+0000", 2024-09-21T22:06:57.583 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3747000000000001e-05 2024-09-21T22:06:57.583 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.583 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.583 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:57.583 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.247603+0000", 2024-09-21T22:06:57.583 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013171600000000001 2024-09-21T22:06:57.583 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.583 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.584 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:57.584 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.247656+0000", 2024-09-21T22:06:57.584 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.3529999999999997e-05 2024-09-21T22:06:57.584 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.584 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.584 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:57.584 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.251225+0000", 2024-09-21T22:06:57.584 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00356865 2024-09-21T22:06:57.584 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.584 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.584 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:57.584 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.251370+0000", 2024-09-21T22:06:57.585 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00014518500000000001 2024-09-21T22:06:57.585 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:57.585 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:57.585 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:57.585 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.585 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.585 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:69 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'43, mlcod=14'39)", 2024-09-21T22:06:57.585 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.248839+0000", 2024-09-21T22:06:57.585 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 55.291476047000003, 2024-09-21T22:06:57.585 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0034954470000000001, 2024-09-21T22:06:57.585 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:57.586 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:57.586 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:57.586 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:57.586 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.586 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:57.586 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.248839+0000", 2024-09-21T22:06:57.586 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:57.586 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.586 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.586 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:57.586 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.248839+0000", 2024-09-21T22:06:57.586 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:57.587 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.587 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.587 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:57.587 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.248841+0000", 2024-09-21T22:06:57.587 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6759999999999999e-06 2024-09-21T22:06:57.587 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.587 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.587 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:57.587 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.249058+0000", 2024-09-21T22:06:57.587 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000217455 2024-09-21T22:06:57.587 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.587 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.587 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:57.588 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.249085+0000", 2024-09-21T22:06:57.588 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.7002999999999999e-05 2024-09-21T22:06:57.588 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.588 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.588 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:57.588 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.249099+0000", 2024-09-21T22:06:57.588 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.402e-05 2024-09-21T22:06:57.588 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.588 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.588 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:57.588 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.249241+0000", 2024-09-21T22:06:57.588 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00014194899999999999 2024-09-21T22:06:57.589 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.589 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.589 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:57.589 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.249298+0000", 2024-09-21T22:06:57.589 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.6687000000000001e-05 2024-09-21T22:06:57.589 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.589 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.589 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:57.589 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.252266+0000", 2024-09-21T22:06:57.589 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.002968348 2024-09-21T22:06:57.589 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.589 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.590 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:57.590 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.252334+0000", 2024-09-21T22:06:57.590 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.8308999999999999e-05 2024-09-21T22:06:57.590 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:57.590 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:57.590 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:57.590 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.590 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.590 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:71 1.0 e14/13 1:30306672:devicehealth::main.db.0000000000000000:head v 14'45, mlcod=14'43)", 2024-09-21T22:06:57.590 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.256068+0000", 2024-09-21T22:06:57.590 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 55.284246574999997, 2024-09-21T22:06:57.590 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0041712579999999997, 2024-09-21T22:06:57.591 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:57.591 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:57.591 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:57.591 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:57.591 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.591 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:57.591 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.256068+0000", 2024-09-21T22:06:57.591 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:57.591 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.591 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.591 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:57.591 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.256068+0000", 2024-09-21T22:06:57.592 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:57.592 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.592 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.592 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:57.592 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.256070+0000", 2024-09-21T22:06:57.592 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.61e-06 2024-09-21T22:06:57.592 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.592 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.592 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:57.592 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.256129+0000", 2024-09-21T22:06:57.592 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.8702999999999997e-05 2024-09-21T22:06:57.592 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.592 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.593 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:57.593 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.256158+0000", 2024-09-21T22:06:57.593 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9533e-05 2024-09-21T22:06:57.593 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.593 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.593 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:57.593 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.256173+0000", 2024-09-21T22:06:57.593 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4267000000000001e-05 2024-09-21T22:06:57.593 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.593 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.593 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:57.593 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.256304+0000", 2024-09-21T22:06:57.594 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000131352 2024-09-21T22:06:57.594 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.594 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.594 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:57.594 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.256357+0000", 2024-09-21T22:06:57.594 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.3492999999999998e-05 2024-09-21T22:06:57.594 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.594 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.594 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:57.594 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.260147+0000", 2024-09-21T22:06:57.594 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0037900410000000001 2024-09-21T22:06:57.594 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.595 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.595 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:57.595 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.260240+0000", 2024-09-21T22:06:57.595 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.2258999999999999e-05 2024-09-21T22:06:57.595 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:57.595 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:57.595 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:57.595 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.595 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.595 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:72 1.0 e14/13 1:30306672:devicehealth::main.db.0000000000000000:head v 14'46, mlcod=14'43)", 2024-09-21T22:06:57.595 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.258358+0000", 2024-09-21T22:06:57.595 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 55.281956581999999, 2024-09-21T22:06:57.595 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0041492919999999997, 2024-09-21T22:06:57.596 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:57.596 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:57.596 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:57.596 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:57.596 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.596 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:57.596 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.258358+0000", 2024-09-21T22:06:57.596 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:57.596 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.596 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.596 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:57.596 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.258358+0000", 2024-09-21T22:06:57.596 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:57.597 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.597 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.597 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:57.597 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.258360+0000", 2024-09-21T22:06:57.597 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.623e-06 2024-09-21T22:06:57.597 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.597 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.597 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:57.597 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.258415+0000", 2024-09-21T22:06:57.597 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.4756000000000002e-05 2024-09-21T22:06:57.597 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.597 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.598 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:57.598 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.258444+0000", 2024-09-21T22:06:57.598 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9164000000000001e-05 2024-09-21T22:06:57.598 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.598 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.598 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:57.598 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.258459+0000", 2024-09-21T22:06:57.598 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4996e-05 2024-09-21T22:06:57.598 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.598 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.598 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:57.598 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.258637+0000", 2024-09-21T22:06:57.599 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00017796500000000001 2024-09-21T22:06:57.599 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.599 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.599 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:57.599 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.258695+0000", 2024-09-21T22:06:57.599 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.8207e-05 2024-09-21T22:06:57.599 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.599 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.599 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:57.599 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.262421+0000", 2024-09-21T22:06:57.599 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0037261550000000001 2024-09-21T22:06:57.599 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.599 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.600 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:57.600 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.262508+0000", 2024-09-21T22:06:57.600 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.6426000000000002e-05 2024-09-21T22:06:57.600 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:57.600 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:57.600 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:57.600 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.600 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.600 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:87 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'57, mlcod=14'52)", 2024-09-21T22:06:57.600 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.290115+0000", 2024-09-21T22:06:57.600 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 55.250200137, 2024-09-21T22:06:57.600 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0030478200000000001, 2024-09-21T22:06:57.600 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:57.601 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:57.601 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:57.601 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:57.601 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.601 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:57.601 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.290115+0000", 2024-09-21T22:06:57.601 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:57.601 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.601 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.601 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:57.601 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.290115+0000", 2024-09-21T22:06:57.601 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:57.602 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.602 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.602 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:57.602 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.290116+0000", 2024-09-21T22:06:57.602 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3829999999999999e-06 2024-09-21T22:06:57.602 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.602 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.602 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:57.602 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.290191+0000", 2024-09-21T22:06:57.602 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.4562999999999997e-05 2024-09-21T22:06:57.602 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.602 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.603 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:57.603 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.290216+0000", 2024-09-21T22:06:57.603 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.565e-05 2024-09-21T22:06:57.603 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.603 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.603 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:57.603 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.290236+0000", 2024-09-21T22:06:57.603 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9326000000000001e-05 2024-09-21T22:06:57.603 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.603 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.603 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:57.603 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.290340+0000", 2024-09-21T22:06:57.603 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0001047 2024-09-21T22:06:57.604 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.604 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.604 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:57.604 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.290377+0000", 2024-09-21T22:06:57.604 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.6356000000000002e-05 2024-09-21T22:06:57.604 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.604 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.604 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:57.604 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.293096+0000", 2024-09-21T22:06:57.604 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.002718996 2024-09-21T22:06:57.604 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.604 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.604 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:57.605 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.293163+0000", 2024-09-21T22:06:57.605 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.6846000000000003e-05 2024-09-21T22:06:57.605 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:57.605 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:57.605 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:57.605 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.605 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.605 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:114 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'76, mlcod=14'73)", 2024-09-21T22:06:57.605 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.490030+0000", 2024-09-21T22:06:57.605 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 55.050284697999999, 2024-09-21T22:06:57.605 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.004227873, 2024-09-21T22:06:57.605 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:57.605 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:57.606 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:57.606 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:57.606 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.606 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:57.606 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.490030+0000", 2024-09-21T22:06:57.606 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:57.606 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.606 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.606 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:57.606 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.490030+0000", 2024-09-21T22:06:57.606 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:57.606 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.607 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.607 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:57.607 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.490031+0000", 2024-09-21T22:06:57.607 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.3300000000000001e-07 2024-09-21T22:06:57.607 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.607 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.607 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:57.607 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.490042+0000", 2024-09-21T22:06:57.607 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.1095e-05 2024-09-21T22:06:57.607 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.607 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.607 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:57.607 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.490053+0000", 2024-09-21T22:06:57.608 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0501e-05 2024-09-21T22:06:57.608 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.608 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.608 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:57.608 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.490060+0000", 2024-09-21T22:06:57.608 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.9829999999999999e-06 2024-09-21T22:06:57.608 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.608 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.608 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:57.608 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.490107+0000", 2024-09-21T22:06:57.608 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.7278999999999999e-05 2024-09-21T22:06:57.608 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.609 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.609 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:57.609 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.490124+0000", 2024-09-21T22:06:57.609 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7190999999999999e-05 2024-09-21T22:06:57.609 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.609 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.609 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:57.609 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.494229+0000", 2024-09-21T22:06:57.609 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0041050440000000004 2024-09-21T22:06:57.609 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.609 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.609 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:57.609 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.494258+0000", 2024-09-21T22:06:57.610 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9047000000000002e-05 2024-09-21T22:06:57.610 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:57.610 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:57.610 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:57.610 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.610 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.610 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:115 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'77, mlcod=14'73)", 2024-09-21T22:06:57.610 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.491380+0000", 2024-09-21T22:06:57.610 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 55.048935075999999, 2024-09-21T22:06:57.610 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0031550889999999998, 2024-09-21T22:06:57.610 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:57.610 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:57.611 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:57.611 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:57.611 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.611 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:57.611 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.491380+0000", 2024-09-21T22:06:57.611 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:57.611 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.611 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.611 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:57.611 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.491380+0000", 2024-09-21T22:06:57.611 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:57.611 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.612 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.612 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:57.612 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.491381+0000", 2024-09-21T22:06:57.612 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.1399999999999996e-07 2024-09-21T22:06:57.612 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.612 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.612 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:57.612 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.491435+0000", 2024-09-21T22:06:57.612 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.3797e-05 2024-09-21T22:06:57.612 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.612 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.612 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:57.613 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.491447+0000", 2024-09-21T22:06:57.613 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2642e-05 2024-09-21T22:06:57.613 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.613 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.613 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:57.613 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.491453+0000", 2024-09-21T22:06:57.613 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.4820000000000002e-06 2024-09-21T22:06:57.613 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.613 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.613 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:57.613 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.491502+0000", 2024-09-21T22:06:57.613 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.9038e-05 2024-09-21T22:06:57.614 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.614 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.614 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:57.614 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.491514+0000", 2024-09-21T22:06:57.614 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2483999999999999e-05 2024-09-21T22:06:57.614 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.614 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.614 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:57.614 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.494486+0000", 2024-09-21T22:06:57.614 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.002971338 2024-09-21T22:06:57.614 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.614 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.615 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:57.615 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.494535+0000", 2024-09-21T22:06:57.615 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.9493999999999997e-05 2024-09-21T22:06:57.615 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:57.615 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:57.615 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:57.615 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.615 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.615 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:118 1.0 e14/13 1:30306672:devicehealth::main.db.0000000000000000:head v 14'80, mlcod=14'78)", 2024-09-21T22:06:57.615 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.499045+0000", 2024-09-21T22:06:57.615 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 55.041270074000003, 2024-09-21T22:06:57.615 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0029889589999999998, 2024-09-21T22:06:57.616 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:57.616 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:57.616 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:57.616 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:57.616 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.616 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:57.616 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.499045+0000", 2024-09-21T22:06:57.616 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:57.616 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.616 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.616 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:57.616 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.499045+0000", 2024-09-21T22:06:57.616 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:57.617 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.617 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.617 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:57.617 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.499046+0000", 2024-09-21T22:06:57.617 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.0299999999999998e-07 2024-09-21T22:06:57.617 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.617 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.617 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:57.617 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.499110+0000", 2024-09-21T22:06:57.617 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.4688000000000001e-05 2024-09-21T22:06:57.617 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.617 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.618 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:57.618 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.499125+0000", 2024-09-21T22:06:57.618 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4387e-05 2024-09-21T22:06:57.618 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.618 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.618 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:57.618 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.499131+0000", 2024-09-21T22:06:57.618 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.2129999999999996e-06 2024-09-21T22:06:57.618 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.618 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.618 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:57.618 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.499255+0000", 2024-09-21T22:06:57.619 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00012424199999999999 2024-09-21T22:06:57.619 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.619 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.619 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:57.619 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.499312+0000", 2024-09-21T22:06:57.619 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.6709999999999997e-05 2024-09-21T22:06:57.619 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.619 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.619 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:57.619 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.502005+0000", 2024-09-21T22:06:57.619 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0026927169999999999 2024-09-21T22:06:57.619 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.620 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.620 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:57.620 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.502034+0000", 2024-09-21T22:06:57.620 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9198999999999999e-05 2024-09-21T22:06:57.620 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:57.620 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:57.620 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:57.620 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.620 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.620 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:136 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'94, mlcod=14'89)", 2024-09-21T22:06:57.620 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.903810+0000", 2024-09-21T22:06:57.620 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 54.636505102999998, 2024-09-21T22:06:57.621 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0062746829999999996, 2024-09-21T22:06:57.621 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:57.621 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:57.621 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:57.621 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:57.621 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.621 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:57.621 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.903810+0000", 2024-09-21T22:06:57.621 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:57.621 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.621 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.621 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:57.621 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.903810+0000", 2024-09-21T22:06:57.622 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:57.622 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.622 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.622 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:57.622 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.903810+0000", 2024-09-21T22:06:57.622 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.03e-07 2024-09-21T22:06:57.622 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.622 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.622 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:57.622 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.903814+0000", 2024-09-21T22:06:57.622 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.676e-06 2024-09-21T22:06:57.622 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.623 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.623 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:57.623 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.903822+0000", 2024-09-21T22:06:57.623 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.2879999999999993e-06 2024-09-21T22:06:57.623 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.623 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.623 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:57.623 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.903829+0000", 2024-09-21T22:06:57.623 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.2920000000000001e-06 2024-09-21T22:06:57.623 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.623 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.623 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:57.623 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.903879+0000", 2024-09-21T22:06:57.624 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.0253999999999997e-05 2024-09-21T22:06:57.624 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.624 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.624 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:57.624 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.903897+0000", 2024-09-21T22:06:57.624 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8468999999999999e-05 2024-09-21T22:06:57.624 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.624 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.624 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:57.624 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.910044+0000", 2024-09-21T22:06:57.624 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0061468950000000003 2024-09-21T22:06:57.624 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.625 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.625 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:57.625 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.910085+0000", 2024-09-21T22:06:57.625 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.0405999999999999e-05 2024-09-21T22:06:57.625 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:57.625 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:57.625 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:57.625 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.625 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.625 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:137 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'95, mlcod=14'90)", 2024-09-21T22:06:57.625 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.904439+0000", 2024-09-21T22:06:57.625 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 54.635875706999997, 2024-09-21T22:06:57.626 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0072625019999999997, 2024-09-21T22:06:57.626 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:57.626 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:57.626 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:57.626 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:57.626 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.626 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:57.626 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.904439+0000", 2024-09-21T22:06:57.626 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:57.626 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.626 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.627 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:57.627 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.904439+0000", 2024-09-21T22:06:57.627 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:57.627 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.627 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.627 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:57.627 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.904440+0000", 2024-09-21T22:06:57.627 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8400000000000005e-07 2024-09-21T22:06:57.627 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.627 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.627 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:57.627 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.904498+0000", 2024-09-21T22:06:57.627 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.8320999999999999e-05 2024-09-21T22:06:57.628 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.628 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.628 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:57.628 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.904507+0000", 2024-09-21T22:06:57.628 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.7790000000000001e-06 2024-09-21T22:06:57.628 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.628 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.628 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:57.628 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.904511+0000", 2024-09-21T22:06:57.628 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.9700000000000001e-06 2024-09-21T22:06:57.628 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.628 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.629 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:57.629 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.904551+0000", 2024-09-21T22:06:57.629 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.9910000000000002e-05 2024-09-21T22:06:57.629 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.629 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.629 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:57.629 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.904567+0000", 2024-09-21T22:06:57.629 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5903999999999999e-05 2024-09-21T22:06:57.629 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.629 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.629 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:57.629 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.911666+0000", 2024-09-21T22:06:57.630 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0070996929999999998 2024-09-21T22:06:57.630 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.630 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.630 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:57.630 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.911702+0000", 2024-09-21T22:06:57.630 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.5441000000000002e-05 2024-09-21T22:06:57.630 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:57.630 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:57.630 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:57.630 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.630 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.630 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:138 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'96, mlcod=14'90)", 2024-09-21T22:06:57.631 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.905402+0000", 2024-09-21T22:06:57.631 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 54.634912730000003, 2024-09-21T22:06:57.631 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0063366890000000004, 2024-09-21T22:06:57.631 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:57.631 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:57.631 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:57.631 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:57.631 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.631 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:57.631 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905402+0000", 2024-09-21T22:06:57.631 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:57.631 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.631 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.632 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:57.632 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905402+0000", 2024-09-21T22:06:57.632 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:57.632 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.632 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.632 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:57.632 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905403+0000", 2024-09-21T22:06:57.632 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.9200000000000002e-07 2024-09-21T22:06:57.632 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.632 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.632 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:57.632 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905406+0000", 2024-09-21T22:06:57.633 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.326e-06 2024-09-21T22:06:57.633 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.633 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.633 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:57.633 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905413+0000", 2024-09-21T22:06:57.633 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.3320000000000003e-06 2024-09-21T22:06:57.633 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.633 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.633 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:57.633 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905417+0000", 2024-09-21T22:06:57.633 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.7139999999999999e-06 2024-09-21T22:06:57.633 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.634 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.634 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:57.634 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.910115+0000", 2024-09-21T22:06:57.634 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0046983939999999998 2024-09-21T22:06:57.634 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.634 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.634 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:57.634 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.910140+0000", 2024-09-21T22:06:57.634 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4689999999999999e-05 2024-09-21T22:06:57.634 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.634 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.634 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:57.635 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.911706+0000", 2024-09-21T22:06:57.635 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0015658479999999999 2024-09-21T22:06:57.635 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.635 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.635 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:57.635 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.911739+0000", 2024-09-21T22:06:57.635 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.2993000000000001e-05 2024-09-21T22:06:57.635 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:57.635 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:57.635 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:57.635 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.635 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.635 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:139 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'97, mlcod=14'90)", 2024-09-21T22:06:57.636 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.905983+0000", 2024-09-21T22:06:57.636 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 54.634331811999999, 2024-09-21T22:06:57.636 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0057906030000000001, 2024-09-21T22:06:57.636 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:57.636 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:57.636 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:57.636 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:57.636 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.636 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:57.636 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905983+0000", 2024-09-21T22:06:57.636 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:57.636 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.637 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.637 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:57.637 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905983+0000", 2024-09-21T22:06:57.637 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:57.637 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.637 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.637 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:57.637 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905984+0000", 2024-09-21T22:06:57.637 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.15e-07 2024-09-21T22:06:57.637 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.637 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.637 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:57.637 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905986+0000", 2024-09-21T22:06:57.638 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.852e-06 2024-09-21T22:06:57.638 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.638 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.638 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:57.638 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905993+0000", 2024-09-21T22:06:57.638 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.652e-06 2024-09-21T22:06:57.638 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.638 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.638 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:57.638 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905996+0000", 2024-09-21T22:06:57.638 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.3519999999999998e-06 2024-09-21T22:06:57.638 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.639 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.639 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:57.639 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.910519+0000", 2024-09-21T22:06:57.639 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0045230820000000003 2024-09-21T22:06:57.639 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.639 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.639 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:57.639 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.910548+0000", 2024-09-21T22:06:57.639 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8399999999999999e-05 2024-09-21T22:06:57.639 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.639 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.639 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:57.640 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.911742+0000", 2024-09-21T22:06:57.640 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011939909999999999 2024-09-21T22:06:57.640 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.640 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.640 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:57.640 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.911774+0000", 2024-09-21T22:06:57.640 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.1958999999999997e-05 2024-09-21T22:06:57.640 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:57.640 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:57.640 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:57.640 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.640 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.641 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:140 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'98, mlcod=14'90)", 2024-09-21T22:06:57.641 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.906441+0000", 2024-09-21T22:06:57.641 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 54.633873682999997, 2024-09-21T22:06:57.641 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0057052989999999996, 2024-09-21T22:06:57.641 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:57.641 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:57.641 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:57.641 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:57.641 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.641 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:57.641 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.906441+0000", 2024-09-21T22:06:57.641 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:57.642 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.642 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.642 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:57.642 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.906441+0000", 2024-09-21T22:06:57.642 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:57.642 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.642 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.642 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:57.642 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.906442+0000", 2024-09-21T22:06:57.642 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.7e-07 2024-09-21T22:06:57.642 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.643 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.643 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:57.643 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.906511+0000", 2024-09-21T22:06:57.643 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.8857999999999997e-05 2024-09-21T22:06:57.643 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.643 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.643 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:57.643 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.906518+0000", 2024-09-21T22:06:57.643 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.1690000000000003e-06 2024-09-21T22:06:57.643 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.643 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.643 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:57.644 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.906521+0000", 2024-09-21T22:06:57.644 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.4829999999999999e-06 2024-09-21T22:06:57.644 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.644 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.644 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:57.644 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.910939+0000", 2024-09-21T22:06:57.644 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0044176809999999997 2024-09-21T22:06:57.644 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.644 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.644 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:57.644 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.910959+0000", 2024-09-21T22:06:57.644 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9898000000000002e-05 2024-09-21T22:06:57.645 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.645 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.645 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:57.645 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.912113+0000", 2024-09-21T22:06:57.645 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001153811 2024-09-21T22:06:57.645 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:57.645 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:57.645 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:57.645 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.912147+0000", 2024-09-21T22:06:57.645 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.4029000000000002e-05 2024-09-21T22:06:57.645 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:57.645 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:57.645 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:57.646 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:57.646 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:57.646 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:06:57.647 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T22:06:57.648 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:06:57.644+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:06:57.649 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -f /var/log/ceph/exp.2.3.5 2024-09-21T22:06:57.748 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:06:57.744+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:06:57.822 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:06:57.822 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:06:57.822 INFO:teuthology.orchestra.run.smithi039.stdout: "num_ops": 0 2024-09-21T22:06:57.822 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:06:57.836 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T22:06:57.883 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:06:57.880+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:06:57.950 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:06:57.944+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:06:58.004 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-21T22:06:58.012 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:06:58.013 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:06:58.013 INFO:teuthology.orchestra.run.smithi039.stdout: "complaint_time": 30, 2024-09-21T22:06:58.013 INFO:teuthology.orchestra.run.smithi039.stdout: "num_blocked_ops": 0 2024-09-21T22:06:58.013 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:06:58.028 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T22:06:58.051 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:06:58.048+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:06:58.152 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:06:58.148+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:06:58.194 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:06:58.194 INFO:teuthology.orchestra.run.smithi039.stdout: "size": 20, 2024-09-21T22:06:58.194 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 600, 2024-09-21T22:06:58.194 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [ 2024-09-21T22:06:58.194 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:58.194 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4224.0:5 2.2 e18/16 2:5cea7035:::rbd_info:head v 18'1, mlcod=0'0)", 2024-09-21T22:06:58.194 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:06.114955+0000", 2024-09-21T22:06:58.195 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 52.079062051000001, 2024-09-21T22:06:58.195 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0017716170000000001, 2024-09-21T22:06:58.195 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:58.195 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:58.195 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:58.195 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:58.195 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:58.195 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:58.195 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:06.114955+0000", 2024-09-21T22:06:58.195 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:58.195 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:58.195 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:58.195 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:58.195 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:06.114955+0000", 2024-09-21T22:06:58.196 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:58.196 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:58.196 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:58.196 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:58.196 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:06.114956+0000", 2024-09-21T22:06:58.196 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.9299999999999997e-07 2024-09-21T22:06:58.196 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:58.196 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:58.196 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:58.196 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:06.114964+0000", 2024-09-21T22:06:58.196 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.9840000000000001e-06 2024-09-21T22:06:58.196 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:58.196 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:58.196 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:58.197 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:06.115001+0000", 2024-09-21T22:06:58.197 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.7719999999999998e-05 2024-09-21T22:06:58.197 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:58.197 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:58.197 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:58.197 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:06.115019+0000", 2024-09-21T22:06:58.197 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7943000000000001e-05 2024-09-21T22:06:58.197 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:58.197 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:58.197 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:58.197 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:06.115168+0000", 2024-09-21T22:06:58.197 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000148589 2024-09-21T22:06:58.197 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:58.197 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:58.198 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:58.198 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:06.115225+0000", 2024-09-21T22:06:58.198 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.6975999999999998e-05 2024-09-21T22:06:58.198 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:58.198 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:58.198 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:58.198 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:06.116585+0000", 2024-09-21T22:06:58.198 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00136038 2024-09-21T22:06:58.198 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:58.198 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:58.198 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:58.198 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:06.116726+0000", 2024-09-21T22:06:58.198 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00014133200000000001 2024-09-21T22:06:58.198 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:58.199 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:58.199 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:58.199 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:58.199 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:58.199 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4224.0:7 2.2 e19/16 2:5cea7035:::rbd_info:head v 19'2, mlcod=0'0)", 2024-09-21T22:06:58.199 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:07.129567+0000", 2024-09-21T22:06:58.199 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 51.064450041999997, 2024-09-21T22:06:58.199 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0018460239999999999, 2024-09-21T22:06:58.199 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:58.199 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:58.199 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:58.199 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:58.199 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:58.199 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:58.199 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:07.129567+0000", 2024-09-21T22:06:58.200 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:58.200 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:58.200 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:58.200 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:58.200 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:07.129567+0000", 2024-09-21T22:06:58.200 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:58.200 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:58.200 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:58.200 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:58.200 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:07.129567+0000", 2024-09-21T22:06:58.200 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.9499999999999998e-07 2024-09-21T22:06:58.200 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:58.200 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:58.200 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:58.201 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:07.129580+0000", 2024-09-21T22:06:58.201 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2335e-05 2024-09-21T22:06:58.201 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:58.201 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:58.201 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:58.201 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:07.129606+0000", 2024-09-21T22:06:58.201 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.6528000000000001e-05 2024-09-21T22:06:58.201 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:58.201 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:58.201 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:58.201 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:07.129628+0000", 2024-09-21T22:06:58.201 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1549999999999999e-05 2024-09-21T22:06:58.201 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:58.201 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:58.202 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:58.202 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:07.129744+0000", 2024-09-21T22:06:58.202 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000116551 2024-09-21T22:06:58.202 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:58.202 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:58.202 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:58.202 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:07.129780+0000", 2024-09-21T22:06:58.202 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.5447000000000001e-05 2024-09-21T22:06:58.202 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:58.202 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:58.202 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:58.202 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:07.131348+0000", 2024-09-21T22:06:58.202 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001568631 2024-09-21T22:06:58.202 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:58.203 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:58.203 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:58.203 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:07.131413+0000", 2024-09-21T22:06:58.203 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.4586999999999997e-05 2024-09-21T22:06:58.203 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:58.203 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:58.203 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:58.203 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:58.203 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:58.203 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:06:58.209 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T22:06:58.253 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:06:58.248+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:06:58.353 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:06:58.348+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:06:58.385 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:06:58.385 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:06:58.385 INFO:teuthology.orchestra.run.smithi039.stdout: "num_ops": 0 2024-09-21T22:06:58.385 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:06:58.399 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T22:06:58.491 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:06:58.488+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:06:58.555 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:06:58.548+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:06:58.567 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:06:58.568 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:06:58.568 INFO:teuthology.orchestra.run.smithi039.stdout: "complaint_time": 30, 2024-09-21T22:06:58.568 INFO:teuthology.orchestra.run.smithi039.stdout: "num_blocked_ops": 0 2024-09-21T22:06:58.568 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:06:58.583 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T22:06:58.671 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:06:58.668+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:06:58.748 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:06:58.749 INFO:teuthology.orchestra.run.smithi039.stdout: "size": 20, 2024-09-21T22:06:58.749 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 600, 2024-09-21T22:06:58.749 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [ 2024-09-21T22:06:58.749 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:58.749 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4224.0:1 2.3 e17/16 2:c4c92e5a:::rbd_trash:head v 17'1, mlcod=0'0)", 2024-09-21T22:06:58.749 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:05.107810+0000", 2024-09-21T22:06:58.749 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 53.640607842000001, 2024-09-21T22:06:58.749 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0017679200000000001, 2024-09-21T22:06:58.749 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:06:58.749 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:06:58.749 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:58.749 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:06:58.749 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:58.750 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:06:58.750 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.107810+0000", 2024-09-21T22:06:58.750 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:58.750 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:58.750 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:58.750 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:06:58.750 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.107810+0000", 2024-09-21T22:06:58.750 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:06:58.750 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:58.750 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:58.750 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:06:58.750 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.107811+0000", 2024-09-21T22:06:58.750 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.6700000000000003e-07 2024-09-21T22:06:58.751 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:58.751 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:58.751 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:06:58.751 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.107822+0000", 2024-09-21T22:06:58.751 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0443e-05 2024-09-21T22:06:58.751 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:58.751 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:58.751 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:06:58.751 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.107853+0000", 2024-09-21T22:06:58.751 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.1046000000000001e-05 2024-09-21T22:06:58.751 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:58.751 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:58.751 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:06:58.751 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.107876+0000", 2024-09-21T22:06:58.752 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3547e-05 2024-09-21T22:06:58.752 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:58.752 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:58.752 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:06:58.752 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.108002+0000", 2024-09-21T22:06:58.752 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000125599 2024-09-21T22:06:58.752 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:58.752 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:58.752 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:06:58.752 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.108069+0000", 2024-09-21T22:06:58.752 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.7162999999999994e-05 2024-09-21T22:06:58.752 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:58.752 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:58.752 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:06:58.753 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.109385+0000", 2024-09-21T22:06:58.753 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0013162899999999999 2024-09-21T22:06:58.753 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:06:58.753 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:06:58.753 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:06:58.753 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.109578+0000", 2024-09-21T22:06:58.753 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000193065 2024-09-21T22:06:58.753 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:58.753 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:58.753 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:58.753 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:06:58.753 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:06:58.753 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:06:58.756 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:06:58.751+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:06:58.765 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T22:06:58.857 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:06:58.851+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:06:58.900 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:06:58.900 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:06:58.900 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-09-21T22:06:58.900 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:06:58.916 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T22:06:58.957 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:06:58.951+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:06:59.000 INFO:teuthology.orchestra.run.smithi039.stderr:nodeep-scrub is unset 2024-09-21T22:06:59.006 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T22:06:59.036 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:06:59.036 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:06:59.036 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-21T22:06:59.036 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-21T22:06:59.036 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:06:59.051 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T22:06:59.075 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:06:59.071+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:06:59.159 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:06:59.155+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:06:59.168 INFO:teuthology.orchestra.run.smithi039.stdout:osd.0: {} 2024-09-21T22:06:59.168 INFO:teuthology.orchestra.run.smithi039.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T22:06:59.176 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:06:59.176 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-21T22:06:59.176 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-21T22:06:59.176 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-21T22:06:59.177 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.177 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:06:59.177 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.036355+0000", 2024-09-21T22:06:59.177 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 57.136437458000003, 2024-09-21T22:06:59.177 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.062441934999999997, 2024-09-21T22:06:59.177 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:59.177 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:59.177 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:59.177 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:59.177 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:06:59.177 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:06:59.177 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1 2024-09-21T22:06:59.177 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.178 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:59.178 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.178 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:59.178 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036355+0000", 2024-09-21T22:06:59.178 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:59.178 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.178 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.178 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:59.178 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036355+0000", 2024-09-21T22:06:59.178 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:59.178 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.178 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.179 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:59.179 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036357+0000", 2024-09-21T22:06:59.179 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.446e-06 2024-09-21T22:06:59.179 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.179 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.179 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:59.179 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036373+0000", 2024-09-21T22:06:59.179 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5447e-05 2024-09-21T22:06:59.179 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.179 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.180 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:59.180 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036399+0000", 2024-09-21T22:06:59.180 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6423000000000001e-05 2024-09-21T22:06:59.180 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.180 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.180 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:59.180 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036420+0000", 2024-09-21T22:06:59.180 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1129999999999999e-05 2024-09-21T22:06:59.180 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.180 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.180 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:59.180 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036557+0000", 2024-09-21T22:06:59.180 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013686899999999999 2024-09-21T22:06:59.180 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.180 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.181 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-21T22:06:59.181 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036585+0000", 2024-09-21T22:06:59.181 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8362999999999999e-05 2024-09-21T22:06:59.181 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.181 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.181 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:59.181 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.095068+0000", 2024-09-21T22:06:59.181 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058482972000000001 2024-09-21T22:06:59.181 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.181 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.181 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:59.181 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.095516+0000", 2024-09-21T22:06:59.181 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00044802999999999998 2024-09-21T22:06:59.181 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.181 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.182 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:06:59.182 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.095869+0000", 2024-09-21T22:06:59.182 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000352104 2024-09-21T22:06:59.182 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.182 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.182 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:59.182 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.097055+0000", 2024-09-21T22:06:59.182 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011865599999999999 2024-09-21T22:06:59.182 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.182 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.182 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:59.182 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.098637+0000", 2024-09-21T22:06:59.182 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001581458 2024-09-21T22:06:59.183 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.183 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.183 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:59.183 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.098728+0000", 2024-09-21T22:06:59.183 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1043000000000002e-05 2024-09-21T22:06:59.183 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.183 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.183 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:59.183 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.098797+0000", 2024-09-21T22:06:59.183 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9090000000000004e-05 2024-09-21T22:06:59.183 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:59.183 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:59.183 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:59.183 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.183 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.184 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:06:59.184 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.161242+0000", 2024-09-21T22:06:59.184 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 57.011550546000002, 2024-09-21T22:06:59.184 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014477078000000001, 2024-09-21T22:06:59.184 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:59.184 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:59.184 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:59.185 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:59.185 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:06:59.185 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:06:59.185 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 35 2024-09-21T22:06:59.185 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.185 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:59.185 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.185 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:59.185 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161242+0000", 2024-09-21T22:06:59.185 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:59.185 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.185 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.185 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:59.185 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161242+0000", 2024-09-21T22:06:59.186 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:59.186 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.186 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.186 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:59.186 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161243+0000", 2024-09-21T22:06:59.186 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4729999999999999e-06 2024-09-21T22:06:59.186 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.186 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.186 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:59.186 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161385+0000", 2024-09-21T22:06:59.186 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014226899999999999 2024-09-21T22:06:59.187 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.187 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.187 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:59.187 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161417+0000", 2024-09-21T22:06:59.187 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1406999999999999e-05 2024-09-21T22:06:59.187 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.187 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.187 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:59.187 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161434+0000", 2024-09-21T22:06:59.187 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7623000000000001e-05 2024-09-21T22:06:59.187 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.187 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.187 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:59.187 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.169728+0000", 2024-09-21T22:06:59.188 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0082933090000000004 2024-09-21T22:06:59.188 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.188 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.188 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:59.188 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.169861+0000", 2024-09-21T22:06:59.188 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013367300000000001 2024-09-21T22:06:59.188 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.188 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.188 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:06:59.188 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.170186+0000", 2024-09-21T22:06:59.188 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00032466400000000001 2024-09-21T22:06:59.188 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.188 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.188 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:59.188 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.174759+0000", 2024-09-21T22:06:59.189 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0045727880000000004 2024-09-21T22:06:59.189 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.189 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.189 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:59.189 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.175497+0000", 2024-09-21T22:06:59.189 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00073803400000000004 2024-09-21T22:06:59.189 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.189 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.189 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:59.189 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.175607+0000", 2024-09-21T22:06:59.189 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010987199999999999 2024-09-21T22:06:59.189 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.189 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.189 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:59.190 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.175719+0000", 2024-09-21T22:06:59.190 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011196600000000001 2024-09-21T22:06:59.190 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:59.190 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:59.190 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:59.190 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.190 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.190 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:06:59.190 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.161483+0000", 2024-09-21T22:06:59.190 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 57.011309034999996, 2024-09-21T22:06:59.190 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016155732999999999, 2024-09-21T22:06:59.191 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:59.191 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:59.191 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:59.191 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:59.191 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:06:59.191 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:06:59.191 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 36 2024-09-21T22:06:59.191 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.191 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:59.191 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.191 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:59.191 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161483+0000", 2024-09-21T22:06:59.191 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:59.192 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.192 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.192 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:59.192 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161483+0000", 2024-09-21T22:06:59.192 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:59.192 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.192 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.192 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:59.192 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161485+0000", 2024-09-21T22:06:59.192 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5999999999999999e-06 2024-09-21T22:06:59.192 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.192 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.192 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:59.192 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161498+0000", 2024-09-21T22:06:59.192 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3166999999999999e-05 2024-09-21T22:06:59.193 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.193 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.193 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:59.193 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161526+0000", 2024-09-21T22:06:59.193 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7786000000000001e-05 2024-09-21T22:06:59.193 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.193 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.193 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:59.193 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161545+0000", 2024-09-21T22:06:59.193 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8899999999999999e-05 2024-09-21T22:06:59.193 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.193 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.193 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:59.193 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.172615+0000", 2024-09-21T22:06:59.193 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011070465 2024-09-21T22:06:59.194 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.194 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.194 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:59.194 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.172759+0000", 2024-09-21T22:06:59.194 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014416900000000001 2024-09-21T22:06:59.194 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.194 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.194 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:06:59.194 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.172967+0000", 2024-09-21T22:06:59.194 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020758500000000001 2024-09-21T22:06:59.194 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.195 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.195 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:59.195 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.174775+0000", 2024-09-21T22:06:59.195 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0018080990000000001 2024-09-21T22:06:59.195 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.195 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.195 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:59.195 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.177447+0000", 2024-09-21T22:06:59.195 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026717429999999999 2024-09-21T22:06:59.195 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.196 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.196 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:59.196 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.177525+0000", 2024-09-21T22:06:59.196 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7846e-05 2024-09-21T22:06:59.196 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.196 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.196 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:59.196 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.177639+0000", 2024-09-21T22:06:59.196 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000114373 2024-09-21T22:06:59.196 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:59.196 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:59.196 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:59.196 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.196 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.197 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:06:59.197 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.161577+0000", 2024-09-21T22:06:59.197 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 57.011215501999999, 2024-09-21T22:06:59.197 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016652991999999998, 2024-09-21T22:06:59.197 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:59.197 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:59.197 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:59.197 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:59.197 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:06:59.197 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:06:59.197 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 37 2024-09-21T22:06:59.197 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.197 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:59.197 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.197 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:59.198 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161577+0000", 2024-09-21T22:06:59.198 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:59.198 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.198 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.198 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:59.198 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161577+0000", 2024-09-21T22:06:59.198 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:59.198 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.198 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.198 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:59.198 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161578+0000", 2024-09-21T22:06:59.198 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5269999999999999e-06 2024-09-21T22:06:59.198 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.198 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.199 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:59.199 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161591+0000", 2024-09-21T22:06:59.199 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.296e-05 2024-09-21T22:06:59.199 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.199 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.199 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:59.199 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161629+0000", 2024-09-21T22:06:59.199 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7799000000000002e-05 2024-09-21T22:06:59.199 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.199 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.199 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:59.199 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161644+0000", 2024-09-21T22:06:59.199 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5333e-05 2024-09-21T22:06:59.199 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.200 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.200 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:59.200 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.173636+0000", 2024-09-21T22:06:59.200 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011992078 2024-09-21T22:06:59.200 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.200 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.200 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:59.200 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.173862+0000", 2024-09-21T22:06:59.200 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022556799999999999 2024-09-21T22:06:59.200 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.200 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.200 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:06:59.200 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.174193+0000", 2024-09-21T22:06:59.200 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000331088 2024-09-21T22:06:59.200 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.201 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.201 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:59.201 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.175740+0000", 2024-09-21T22:06:59.201 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015467910000000001 2024-09-21T22:06:59.201 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.201 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.201 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:59.201 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.178073+0000", 2024-09-21T22:06:59.201 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002333356 2024-09-21T22:06:59.201 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.201 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.201 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:59.201 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.178162+0000", 2024-09-21T22:06:59.201 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8306000000000005e-05 2024-09-21T22:06:59.202 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.202 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.202 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:59.202 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.178230+0000", 2024-09-21T22:06:59.202 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8186000000000001e-05 2024-09-21T22:06:59.202 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:59.202 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:59.202 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:59.202 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.202 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.202 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:06:59.202 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.280540+0000", 2024-09-21T22:06:59.202 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 56.89225193, 2024-09-21T22:06:59.202 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013246212, 2024-09-21T22:06:59.203 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:59.203 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:59.203 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:59.203 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:59.203 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:06:59.203 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:06:59.203 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 88 2024-09-21T22:06:59.203 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.203 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:59.203 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.203 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:59.203 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.280540+0000", 2024-09-21T22:06:59.203 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:59.203 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.203 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.204 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:59.204 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.280540+0000", 2024-09-21T22:06:59.204 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:59.204 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.204 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.204 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:59.204 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.280541+0000", 2024-09-21T22:06:59.204 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.54e-07 2024-09-21T22:06:59.204 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.204 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.204 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:59.204 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.280549+0000", 2024-09-21T22:06:59.204 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3329999999999997e-06 2024-09-21T22:06:59.204 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.204 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.205 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:59.205 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.280560+0000", 2024-09-21T22:06:59.205 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1003e-05 2024-09-21T22:06:59.205 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.205 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.205 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:59.205 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.280569+0000", 2024-09-21T22:06:59.205 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2430000000000005e-06 2024-09-21T22:06:59.205 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.205 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.205 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:59.205 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.291095+0000", 2024-09-21T22:06:59.205 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010526259 2024-09-21T22:06:59.205 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.205 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.206 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:59.206 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.291231+0000", 2024-09-21T22:06:59.206 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013561899999999999 2024-09-21T22:06:59.206 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.206 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.206 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:06:59.206 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.291457+0000", 2024-09-21T22:06:59.206 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022686200000000001 2024-09-21T22:06:59.206 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.206 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.206 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:59.206 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.293117+0000", 2024-09-21T22:06:59.206 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001659474 2024-09-21T22:06:59.207 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.207 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.207 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:59.207 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.293524+0000", 2024-09-21T22:06:59.207 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00040701300000000001 2024-09-21T22:06:59.207 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.207 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.207 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:59.207 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.293673+0000", 2024-09-21T22:06:59.207 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000148782 2024-09-21T22:06:59.207 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.207 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.207 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:59.207 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.293786+0000", 2024-09-21T22:06:59.207 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011377 2024-09-21T22:06:59.208 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:59.208 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:59.208 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:59.208 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.208 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.208 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:06:59.208 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.466485+0000", 2024-09-21T22:06:59.208 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 56.706307199000001, 2024-09-21T22:06:59.208 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014443981, 2024-09-21T22:06:59.208 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:59.208 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:59.208 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:59.208 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:59.208 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:06:59.209 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:06:59.209 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 107 2024-09-21T22:06:59.209 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.209 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:59.209 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.209 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:59.209 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.466485+0000", 2024-09-21T22:06:59.209 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:59.209 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.209 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.209 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:59.209 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.466485+0000", 2024-09-21T22:06:59.209 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:59.209 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.209 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.210 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:59.210 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.466486+0000", 2024-09-21T22:06:59.210 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6299999999999993e-07 2024-09-21T22:06:59.210 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.210 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.210 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:59.210 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.466493+0000", 2024-09-21T22:06:59.210 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9639999999999997e-06 2024-09-21T22:06:59.210 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.210 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.210 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:59.210 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.466507+0000", 2024-09-21T22:06:59.210 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4216000000000001e-05 2024-09-21T22:06:59.210 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.211 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.211 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:59.211 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.466515+0000", 2024-09-21T22:06:59.211 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.157e-06 2024-09-21T22:06:59.211 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.211 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.211 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:59.211 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.472033+0000", 2024-09-21T22:06:59.211 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0055175839999999999 2024-09-21T22:06:59.211 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.211 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.211 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:59.211 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.472174+0000", 2024-09-21T22:06:59.211 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000141612 2024-09-21T22:06:59.212 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.212 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.212 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:06:59.212 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.472381+0000", 2024-09-21T22:06:59.212 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000206961 2024-09-21T22:06:59.212 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.212 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.212 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:59.212 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.473924+0000", 2024-09-21T22:06:59.212 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015427119999999999 2024-09-21T22:06:59.212 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.212 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.212 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:59.212 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.480666+0000", 2024-09-21T22:06:59.212 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0067415410000000002 2024-09-21T22:06:59.213 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.213 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.213 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:59.213 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.480811+0000", 2024-09-21T22:06:59.213 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000145422 2024-09-21T22:06:59.213 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.213 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.213 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:59.213 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.480929+0000", 2024-09-21T22:06:59.213 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000117849 2024-09-21T22:06:59.213 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:59.213 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:59.213 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:59.213 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.214 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.214 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:06:59.214 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.467057+0000", 2024-09-21T22:06:59.214 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 56.705734939999999, 2024-09-21T22:06:59.214 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018350195, 2024-09-21T22:06:59.214 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:59.214 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:59.214 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:59.214 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:59.214 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:06:59.214 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:06:59.215 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 108 2024-09-21T22:06:59.215 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.215 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:59.215 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.215 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:59.215 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467057+0000", 2024-09-21T22:06:59.215 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:59.215 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.215 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.215 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:59.215 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467057+0000", 2024-09-21T22:06:59.215 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:59.215 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.215 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.216 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:59.216 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467059+0000", 2024-09-21T22:06:59.216 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.826e-06 2024-09-21T22:06:59.216 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.216 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.216 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:59.216 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467071+0000", 2024-09-21T22:06:59.216 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2224e-05 2024-09-21T22:06:59.216 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.216 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.216 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:59.216 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467098+0000", 2024-09-21T22:06:59.216 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6953000000000001e-05 2024-09-21T22:06:59.216 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.216 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.217 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:59.217 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467113+0000", 2024-09-21T22:06:59.217 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5119999999999999e-05 2024-09-21T22:06:59.217 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.217 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.217 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:59.217 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.474060+0000", 2024-09-21T22:06:59.217 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0069462919999999997 2024-09-21T22:06:59.217 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.217 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.217 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:59.217 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.474283+0000", 2024-09-21T22:06:59.217 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000223315 2024-09-21T22:06:59.217 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.218 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.218 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:06:59.218 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.474660+0000", 2024-09-21T22:06:59.218 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00037729400000000002 2024-09-21T22:06:59.218 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.218 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.218 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:59.218 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.480622+0000", 2024-09-21T22:06:59.218 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0059613369999999997 2024-09-21T22:06:59.218 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.218 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.218 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:59.218 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.485145+0000", 2024-09-21T22:06:59.218 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0045231960000000002 2024-09-21T22:06:59.219 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.219 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.219 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:59.219 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.485288+0000", 2024-09-21T22:06:59.219 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000143232 2024-09-21T22:06:59.219 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.219 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.219 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:59.219 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.485407+0000", 2024-09-21T22:06:59.219 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000119406 2024-09-21T22:06:59.219 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:59.219 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:59.219 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:59.220 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.220 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.220 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:06:59.220 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.467144+0000", 2024-09-21T22:06:59.220 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 56.705647720999998, 2024-09-21T22:06:59.220 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018907146999999999, 2024-09-21T22:06:59.220 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:59.220 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:59.220 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:59.220 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:59.220 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:06:59.220 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:06:59.220 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 109 2024-09-21T22:06:59.220 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.221 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:59.221 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.221 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:59.221 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467144+0000", 2024-09-21T22:06:59.221 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:59.221 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.221 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.221 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:59.221 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467144+0000", 2024-09-21T22:06:59.221 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:59.221 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.221 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.221 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:59.221 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467145+0000", 2024-09-21T22:06:59.222 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9999999999999996e-07 2024-09-21T22:06:59.222 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.222 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.222 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:59.222 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467331+0000", 2024-09-21T22:06:59.222 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018518899999999999 2024-09-21T22:06:59.222 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.222 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.222 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:59.222 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467350+0000", 2024-09-21T22:06:59.222 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8966000000000001e-05 2024-09-21T22:06:59.222 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.222 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.223 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:59.223 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467361+0000", 2024-09-21T22:06:59.223 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1233000000000001e-05 2024-09-21T22:06:59.223 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.223 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.223 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:59.223 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.475613+0000", 2024-09-21T22:06:59.223 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0082518629999999999 2024-09-21T22:06:59.223 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.223 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.223 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:59.223 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.475859+0000", 2024-09-21T22:06:59.223 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024633800000000003 2024-09-21T22:06:59.223 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.223 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.224 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:06:59.224 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.476223+0000", 2024-09-21T22:06:59.224 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00036408700000000003 2024-09-21T22:06:59.224 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.224 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.224 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:59.224 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.481046+0000", 2024-09-21T22:06:59.224 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.004823356 2024-09-21T22:06:59.224 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.224 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.224 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:59.224 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.485722+0000", 2024-09-21T22:06:59.224 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0046753640000000004 2024-09-21T22:06:59.224 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.224 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.225 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:59.225 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.485908+0000", 2024-09-21T22:06:59.225 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000185859 2024-09-21T22:06:59.225 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.225 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.225 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:59.225 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.486052+0000", 2024-09-21T22:06:59.225 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014399200000000001 2024-09-21T22:06:59.225 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:59.225 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:59.225 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:59.225 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.225 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.225 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:06:59.225 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.467384+0000", 2024-09-21T22:06:59.226 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 56.705407893, 2024-09-21T22:06:59.226 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.01894144, 2024-09-21T22:06:59.226 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:59.226 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:59.226 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:59.226 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:59.226 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:06:59.226 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:06:59.226 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 110 2024-09-21T22:06:59.226 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.226 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:59.226 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.226 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:59.226 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467384+0000", 2024-09-21T22:06:59.227 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:59.227 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.227 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.227 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:59.227 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467384+0000", 2024-09-21T22:06:59.227 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:59.227 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.227 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.227 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:59.227 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467385+0000", 2024-09-21T22:06:59.227 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0470000000000001e-06 2024-09-21T22:06:59.227 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.227 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.227 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:59.228 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467393+0000", 2024-09-21T22:06:59.228 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3869999999999997e-06 2024-09-21T22:06:59.228 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.228 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.228 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:59.228 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467413+0000", 2024-09-21T22:06:59.228 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9953e-05 2024-09-21T22:06:59.228 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.228 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.228 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:59.228 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467421+0000", 2024-09-21T22:06:59.228 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2400000000000007e-06 2024-09-21T22:06:59.228 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.228 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.228 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:59.229 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.479542+0000", 2024-09-21T22:06:59.229 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012120832 2024-09-21T22:06:59.229 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.229 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.229 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:59.229 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.479705+0000", 2024-09-21T22:06:59.229 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016362899999999999 2024-09-21T22:06:59.229 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.229 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.229 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:06:59.229 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.479914+0000", 2024-09-21T22:06:59.229 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020814499999999999 2024-09-21T22:06:59.229 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.229 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.229 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:59.229 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.485536+0000", 2024-09-21T22:06:59.230 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0056222800000000003 2024-09-21T22:06:59.230 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.230 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.230 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:59.230 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.486077+0000", 2024-09-21T22:06:59.230 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00054079900000000001 2024-09-21T22:06:59.230 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.230 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.230 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:59.230 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.486213+0000", 2024-09-21T22:06:59.230 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000136682 2024-09-21T22:06:59.230 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.230 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.230 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:59.231 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.486326+0000", 2024-09-21T22:06:59.231 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000112446 2024-09-21T22:06:59.231 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:59.231 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:59.231 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:59.231 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.231 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.231 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:06:59.231 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.467436+0000", 2024-09-21T22:06:59.231 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 56.705356393000002, 2024-09-21T22:06:59.231 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019152168000000001, 2024-09-21T22:06:59.231 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:59.231 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:59.231 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:59.232 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:59.232 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:06:59.232 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:06:59.232 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 111 2024-09-21T22:06:59.232 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.232 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:59.232 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.232 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:59.232 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467436+0000", 2024-09-21T22:06:59.232 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:59.233 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.233 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.233 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:59.233 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467436+0000", 2024-09-21T22:06:59.233 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:59.233 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.233 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.233 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:59.233 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467437+0000", 2024-09-21T22:06:59.233 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0699999999999996e-07 2024-09-21T22:06:59.233 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.233 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.233 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:59.234 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467445+0000", 2024-09-21T22:06:59.234 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.789e-06 2024-09-21T22:06:59.234 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.234 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.234 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:59.234 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467467+0000", 2024-09-21T22:06:59.234 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1253000000000001e-05 2024-09-21T22:06:59.234 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.234 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.234 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:59.234 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467475+0000", 2024-09-21T22:06:59.235 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0039999999999999e-06 2024-09-21T22:06:59.235 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.235 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.235 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:59.235 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.481164+0000", 2024-09-21T22:06:59.235 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013689114 2024-09-21T22:06:59.235 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.235 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.235 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:59.235 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.481394+0000", 2024-09-21T22:06:59.235 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000229931 2024-09-21T22:06:59.235 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.236 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.236 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:06:59.236 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.481779+0000", 2024-09-21T22:06:59.236 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00038545400000000003 2024-09-21T22:06:59.236 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.236 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.237 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:59.237 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.485675+0000", 2024-09-21T22:06:59.237 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0038958370000000001 2024-09-21T22:06:59.237 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.237 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.237 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:59.237 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.486346+0000", 2024-09-21T22:06:59.237 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00067119100000000002 2024-09-21T22:06:59.238 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.238 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.238 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:59.238 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.486478+0000", 2024-09-21T22:06:59.238 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013182600000000001 2024-09-21T22:06:59.238 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.238 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.238 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:59.239 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.486588+0000", 2024-09-21T22:06:59.239 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000110062 2024-09-21T22:06:59.239 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:59.239 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:59.239 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:59.239 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.239 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.239 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:06:59.239 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.467488+0000", 2024-09-21T22:06:59.240 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 56.705303766999997, 2024-09-21T22:06:59.240 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021038304000000001, 2024-09-21T22:06:59.240 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:59.240 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:59.240 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:59.240 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:59.240 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:06:59.241 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:06:59.241 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 112 2024-09-21T22:06:59.241 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.241 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:59.241 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.241 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:59.241 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467488+0000", 2024-09-21T22:06:59.242 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:59.242 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.242 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.242 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:59.242 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467488+0000", 2024-09-21T22:06:59.242 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:59.242 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.243 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.243 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:59.243 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467489+0000", 2024-09-21T22:06:59.243 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7999999999999995e-07 2024-09-21T22:06:59.243 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.243 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.243 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:59.243 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467687+0000", 2024-09-21T22:06:59.244 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019776200000000001 2024-09-21T22:06:59.244 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.244 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.244 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:59.244 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467704+0000", 2024-09-21T22:06:59.244 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7193e-05 2024-09-21T22:06:59.244 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.244 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.244 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:59.244 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467719+0000", 2024-09-21T22:06:59.244 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4703e-05 2024-09-21T22:06:59.244 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.244 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.244 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:59.244 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.482745+0000", 2024-09-21T22:06:59.245 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015025801 2024-09-21T22:06:59.245 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.245 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.245 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:59.245 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.483015+0000", 2024-09-21T22:06:59.245 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026999400000000002 2024-09-21T22:06:59.245 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.245 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.245 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:06:59.245 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.483440+0000", 2024-09-21T22:06:59.245 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000425697 2024-09-21T22:06:59.245 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.245 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.245 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:59.245 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.486701+0000", 2024-09-21T22:06:59.246 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0032611950000000002 2024-09-21T22:06:59.246 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.246 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.246 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:59.246 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.488262+0000", 2024-09-21T22:06:59.246 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015610439999999999 2024-09-21T22:06:59.246 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.246 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.246 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:59.246 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.488406+0000", 2024-09-21T22:06:59.246 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000143232 2024-09-21T22:06:59.246 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.246 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.246 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:59.247 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.488527+0000", 2024-09-21T22:06:59.247 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012100299999999999 2024-09-21T22:06:59.247 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:59.247 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:59.247 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:59.247 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.247 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.247 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:06:59.247 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.467786+0000", 2024-09-21T22:06:59.247 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 56.705006341999997, 2024-09-21T22:06:59.247 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026339741999999999, 2024-09-21T22:06:59.247 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:59.247 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:59.247 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:59.247 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:59.248 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:06:59.248 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:06:59.248 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 113 2024-09-21T22:06:59.248 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.248 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:59.248 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.248 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:59.248 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467786+0000", 2024-09-21T22:06:59.248 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:59.248 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.248 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.248 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:59.248 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467786+0000", 2024-09-21T22:06:59.248 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:59.248 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.249 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.249 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:59.249 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467787+0000", 2024-09-21T22:06:59.249 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.446e-06 2024-09-21T22:06:59.249 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.249 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.249 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:59.249 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467802+0000", 2024-09-21T22:06:59.249 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4769999999999999e-05 2024-09-21T22:06:59.249 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.249 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.249 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:59.249 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467826+0000", 2024-09-21T22:06:59.249 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3430000000000001e-05 2024-09-21T22:06:59.249 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.250 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.250 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:59.250 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467843+0000", 2024-09-21T22:06:59.250 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7065999999999999e-05 2024-09-21T22:06:59.250 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.250 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.250 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:59.250 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.486820+0000", 2024-09-21T22:06:59.250 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018977233999999999 2024-09-21T22:06:59.250 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.250 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.250 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:59.250 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.487065+0000", 2024-09-21T22:06:59.251 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024558799999999998 2024-09-21T22:06:59.251 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.251 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.251 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:06:59.251 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.487442+0000", 2024-09-21T22:06:59.251 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000376707 2024-09-21T22:06:59.251 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.251 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.251 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:59.251 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.492256+0000", 2024-09-21T22:06:59.251 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0048141620000000003 2024-09-21T22:06:59.251 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.251 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.251 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:59.252 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.493865+0000", 2024-09-21T22:06:59.252 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016090150000000001 2024-09-21T22:06:59.252 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.252 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.252 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:59.252 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.494007+0000", 2024-09-21T22:06:59.252 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000141359 2024-09-21T22:06:59.252 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.252 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.252 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:59.252 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.494126+0000", 2024-09-21T22:06:59.252 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000118965 2024-09-21T22:06:59.252 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:59.252 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:59.253 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:59.253 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.253 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.253 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:06:59.253 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.467878+0000", 2024-09-21T22:06:59.253 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 56.704913853000001, 2024-09-21T22:06:59.253 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.027783958000000001, 2024-09-21T22:06:59.253 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:59.253 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:59.253 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:59.253 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:59.253 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:06:59.253 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:06:59.253 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 114 2024-09-21T22:06:59.253 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.254 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:59.254 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.254 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:59.254 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467878+0000", 2024-09-21T22:06:59.254 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:59.254 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.254 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.254 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:59.254 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467878+0000", 2024-09-21T22:06:59.254 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:59.254 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.254 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.254 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:59.254 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467880+0000", 2024-09-21T22:06:59.255 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.223e-06 2024-09-21T22:06:59.255 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.255 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.255 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:59.255 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467891+0000", 2024-09-21T22:06:59.255 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1527e-05 2024-09-21T22:06:59.255 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.255 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.255 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:59.255 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467913+0000", 2024-09-21T22:06:59.255 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.143e-05 2024-09-21T22:06:59.255 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.255 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.255 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:59.255 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467930+0000", 2024-09-21T22:06:59.256 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7620000000000001e-05 2024-09-21T22:06:59.256 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.256 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.256 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:59.256 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.488690+0000", 2024-09-21T22:06:59.256 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020759588999999998 2024-09-21T22:06:59.256 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.256 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.256 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:59.256 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.488882+0000", 2024-09-21T22:06:59.256 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019236200000000001 2024-09-21T22:06:59.256 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.256 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.256 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:06:59.257 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.489093+0000", 2024-09-21T22:06:59.257 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000211318 2024-09-21T22:06:59.257 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.257 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.257 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:59.257 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.494152+0000", 2024-09-21T22:06:59.257 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0050582769999999999 2024-09-21T22:06:59.257 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.257 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.257 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:59.257 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.495373+0000", 2024-09-21T22:06:59.257 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001221747 2024-09-21T22:06:59.257 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.257 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.257 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:59.258 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.495543+0000", 2024-09-21T22:06:59.258 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000169259 2024-09-21T22:06:59.258 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.258 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.258 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:59.258 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.495662+0000", 2024-09-21T22:06:59.258 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011960600000000001 2024-09-21T22:06:59.258 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:59.258 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:59.258 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:59.258 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.258 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.258 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:06:59.259 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.468118+0000", 2024-09-21T22:06:59.259 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 56.704674111999999, 2024-09-21T22:06:59.259 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.027938877000000001, 2024-09-21T22:06:59.259 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:59.259 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:59.259 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:59.259 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:59.259 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:06:59.259 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:06:59.259 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 115 2024-09-21T22:06:59.259 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.259 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:59.259 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.259 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:59.259 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468118+0000", 2024-09-21T22:06:59.260 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:59.260 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.260 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.260 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:59.260 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468118+0000", 2024-09-21T22:06:59.260 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:59.260 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.260 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.260 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:59.260 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468120+0000", 2024-09-21T22:06:59.260 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7069999999999999e-06 2024-09-21T22:06:59.260 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.260 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.260 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:59.260 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468381+0000", 2024-09-21T22:06:59.261 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026135399999999999 2024-09-21T22:06:59.261 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.261 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.261 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:59.261 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468409+0000", 2024-09-21T22:06:59.261 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.781e-05 2024-09-21T22:06:59.261 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.261 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.261 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:59.261 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468433+0000", 2024-09-21T22:06:59.261 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4343999999999999e-05 2024-09-21T22:06:59.261 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.261 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.262 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:59.262 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.489785+0000", 2024-09-21T22:06:59.262 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021351221 2024-09-21T22:06:59.262 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.262 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.262 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:59.262 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.490018+0000", 2024-09-21T22:06:59.262 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023317800000000001 2024-09-21T22:06:59.262 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.262 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.262 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:06:59.262 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.490412+0000", 2024-09-21T22:06:59.262 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00039417400000000001 2024-09-21T22:06:59.262 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.263 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.263 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:59.263 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.495694+0000", 2024-09-21T22:06:59.263 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.005282335 2024-09-21T22:06:59.263 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.263 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.263 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:59.263 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.495852+0000", 2024-09-21T22:06:59.263 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015777200000000001 2024-09-21T22:06:59.263 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.263 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.263 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:59.263 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.495938+0000", 2024-09-21T22:06:59.263 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6493e-05 2024-09-21T22:06:59.264 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.264 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.264 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:59.264 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.496057+0000", 2024-09-21T22:06:59.264 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000118489 2024-09-21T22:06:59.264 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:59.264 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:59.264 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:59.264 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.264 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.264 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:06:59.264 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.468474+0000", 2024-09-21T22:06:59.264 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 56.704318350999998, 2024-09-21T22:06:59.264 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.028015895999999998, 2024-09-21T22:06:59.264 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:59.265 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:59.265 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:59.265 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:59.265 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:06:59.265 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:06:59.265 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 116 2024-09-21T22:06:59.265 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.265 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:59.265 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.265 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:59.265 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468474+0000", 2024-09-21T22:06:59.265 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:59.265 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.265 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.265 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:59.265 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468474+0000", 2024-09-21T22:06:59.266 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:59.266 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.266 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.266 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:59.266 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468476+0000", 2024-09-21T22:06:59.266 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7060000000000001e-06 2024-09-21T22:06:59.266 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.266 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.266 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:59.266 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468488+0000", 2024-09-21T22:06:59.266 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.234e-05 2024-09-21T22:06:59.266 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.266 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.266 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:59.267 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468517+0000", 2024-09-21T22:06:59.267 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.898e-05 2024-09-21T22:06:59.267 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.267 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.267 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:59.267 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468536+0000", 2024-09-21T22:06:59.267 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8899999999999999e-05 2024-09-21T22:06:59.267 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.267 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.267 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:59.267 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.492376+0000", 2024-09-21T22:06:59.267 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023840659 2024-09-21T22:06:59.267 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.267 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.268 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:59.268 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.492621+0000", 2024-09-21T22:06:59.268 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000244794 2024-09-21T22:06:59.268 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.268 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.268 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:06:59.268 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.493008+0000", 2024-09-21T22:06:59.268 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00038629099999999998 2024-09-21T22:06:59.268 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.268 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.268 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:59.268 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.496192+0000", 2024-09-21T22:06:59.268 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0031847220000000001 2024-09-21T22:06:59.268 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.269 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.269 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:59.269 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.496233+0000", 2024-09-21T22:06:59.269 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0410000000000001e-05 2024-09-21T22:06:59.269 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.269 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.269 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:59.269 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.496374+0000", 2024-09-21T22:06:59.269 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140872 2024-09-21T22:06:59.269 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.269 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.269 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:59.269 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.496490+0000", 2024-09-21T22:06:59.269 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000116222 2024-09-21T22:06:59.269 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:59.270 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:59.270 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:59.270 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.270 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.270 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:06:59.270 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.468568+0000", 2024-09-21T22:06:59.270 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 56.704224232000001, 2024-09-21T22:06:59.270 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.028181578999999998, 2024-09-21T22:06:59.270 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:59.270 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:59.270 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:59.270 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:59.270 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:06:59.270 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:06:59.271 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 117 2024-09-21T22:06:59.271 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.271 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:59.271 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.271 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:59.271 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468568+0000", 2024-09-21T22:06:59.271 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:59.271 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.271 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.271 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:59.271 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468568+0000", 2024-09-21T22:06:59.271 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:59.271 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.271 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.271 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:59.272 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468569+0000", 2024-09-21T22:06:59.272 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4139999999999999e-06 2024-09-21T22:06:59.272 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.272 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.272 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:59.272 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468583+0000", 2024-09-21T22:06:59.272 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3409e-05 2024-09-21T22:06:59.272 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.272 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.272 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:59.272 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468608+0000", 2024-09-21T22:06:59.272 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5559999999999999e-05 2024-09-21T22:06:59.272 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.272 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.273 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:59.273 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468626+0000", 2024-09-21T22:06:59.273 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7863000000000002e-05 2024-09-21T22:06:59.273 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.273 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.273 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:59.273 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.494260+0000", 2024-09-21T22:06:59.273 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025633744999999999 2024-09-21T22:06:59.273 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.273 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.273 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:59.273 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.494519+0000", 2024-09-21T22:06:59.273 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025924800000000001 2024-09-21T22:06:59.273 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.273 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.274 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:06:59.274 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.494747+0000", 2024-09-21T22:06:59.274 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022792799999999999 2024-09-21T22:06:59.274 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.274 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.274 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:59.274 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.496519+0000", 2024-09-21T22:06:59.274 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017720030000000001 2024-09-21T22:06:59.274 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.274 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.274 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:59.274 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.496607+0000", 2024-09-21T22:06:59.274 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7873000000000004e-05 2024-09-21T22:06:59.275 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.275 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.275 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:59.275 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.496691+0000", 2024-09-21T22:06:59.275 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4009000000000002e-05 2024-09-21T22:06:59.275 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.275 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.275 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:59.275 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.496750+0000", 2024-09-21T22:06:59.275 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8526999999999997e-05 2024-09-21T22:06:59.275 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:59.275 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:59.275 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:59.275 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.276 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.276 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:06:59.276 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.898244+0000", 2024-09-21T22:06:59.276 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 56.274547929000001, 2024-09-21T22:06:59.276 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014749464, 2024-09-21T22:06:59.276 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:59.276 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:59.276 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:59.276 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:59.276 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:06:59.276 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:06:59.276 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 140 2024-09-21T22:06:59.276 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.276 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:59.276 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.277 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:59.277 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898244+0000", 2024-09-21T22:06:59.277 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:59.277 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.277 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.277 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:59.277 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898244+0000", 2024-09-21T22:06:59.277 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:59.277 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.277 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.277 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:59.277 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898245+0000", 2024-09-21T22:06:59.277 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3000000000000002e-07 2024-09-21T22:06:59.277 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.277 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.278 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:59.278 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898308+0000", 2024-09-21T22:06:59.278 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3468999999999998e-05 2024-09-21T22:06:59.278 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.278 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.278 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:59.278 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898315+0000", 2024-09-21T22:06:59.278 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.725e-06 2024-09-21T22:06:59.278 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.278 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.278 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:59.278 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898321+0000", 2024-09-21T22:06:59.278 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2090000000000002e-06 2024-09-21T22:06:59.279 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.279 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.279 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:59.279 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.905452+0000", 2024-09-21T22:06:59.279 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0071305989999999996 2024-09-21T22:06:59.279 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.279 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.279 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:59.279 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.905504+0000", 2024-09-21T22:06:59.279 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2638e-05 2024-09-21T22:06:59.279 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.279 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.279 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:06:59.279 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.905590+0000", 2024-09-21T22:06:59.279 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5333000000000004e-05 2024-09-21T22:06:59.280 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.280 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.280 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:59.280 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.911542+0000", 2024-09-21T22:06:59.280 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0059519819999999998 2024-09-21T22:06:59.280 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.280 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.280 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:59.280 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.912908+0000", 2024-09-21T22:06:59.280 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0013666640000000001 2024-09-21T22:06:59.280 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.280 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.280 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:59.280 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.912957+0000", 2024-09-21T22:06:59.280 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9203000000000003e-05 2024-09-21T22:06:59.281 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.281 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.281 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:59.281 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.912994+0000", 2024-09-21T22:06:59.281 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6312000000000001e-05 2024-09-21T22:06:59.281 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:59.281 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:59.281 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:59.281 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.281 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.281 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:06:59.281 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.898342+0000", 2024-09-21T22:06:59.281 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 56.274449791000002, 2024-09-21T22:06:59.281 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014787918000000001, 2024-09-21T22:06:59.281 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:59.282 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:59.282 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:59.282 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:59.282 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:06:59.282 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:06:59.282 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 141 2024-09-21T22:06:59.282 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.282 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:59.282 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.282 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:59.282 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898342+0000", 2024-09-21T22:06:59.282 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:59.282 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.282 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.283 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:59.283 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898342+0000", 2024-09-21T22:06:59.283 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:59.283 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.283 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.283 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:59.283 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898343+0000", 2024-09-21T22:06:59.283 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3000000000000001e-07 2024-09-21T22:06:59.283 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.283 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.283 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:59.283 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898346+0000", 2024-09-21T22:06:59.283 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6009999999999999e-06 2024-09-21T22:06:59.283 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.283 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.284 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:59.284 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898352+0000", 2024-09-21T22:06:59.284 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8089999999999998e-06 2024-09-21T22:06:59.284 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.284 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.284 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:59.284 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898358+0000", 2024-09-21T22:06:59.284 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4349999999999996e-06 2024-09-21T22:06:59.284 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.284 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.284 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:59.284 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.909387+0000", 2024-09-21T22:06:59.284 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011029105000000001 2024-09-21T22:06:59.284 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.285 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.285 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:59.285 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.909462+0000", 2024-09-21T22:06:59.285 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5192999999999996e-05 2024-09-21T22:06:59.285 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.285 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.285 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:06:59.285 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.909551+0000", 2024-09-21T22:06:59.285 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9046000000000001e-05 2024-09-21T22:06:59.285 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.285 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.285 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:59.285 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.911852+0000", 2024-09-21T22:06:59.285 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0023005579999999999 2024-09-21T22:06:59.285 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.286 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.286 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:59.286 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.913002+0000", 2024-09-21T22:06:59.286 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011502929999999999 2024-09-21T22:06:59.286 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.286 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.286 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:59.286 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.913060+0000", 2024-09-21T22:06:59.286 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8427000000000002e-05 2024-09-21T22:06:59.286 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.286 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.286 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:59.287 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.913130+0000", 2024-09-21T22:06:59.287 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0020999999999999e-05 2024-09-21T22:06:59.287 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:59.287 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:59.287 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:59.287 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.287 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.287 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:158 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T22:06:59.287 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:03.511201+0000", 2024-09-21T22:06:59.287 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 55.661591307999998, 2024-09-21T22:06:59.287 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013353538999999999, 2024-09-21T22:06:59.287 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:59.287 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:59.287 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:59.288 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:59.288 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:06:59.288 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:06:59.288 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 158 2024-09-21T22:06:59.288 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.288 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:59.288 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.288 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:59.288 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511201+0000", 2024-09-21T22:06:59.288 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:59.288 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.288 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.288 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:59.288 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511201+0000", 2024-09-21T22:06:59.288 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:59.288 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.289 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.289 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:59.289 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511202+0000", 2024-09-21T22:06:59.289 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9299999999999997e-07 2024-09-21T22:06:59.289 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.289 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.289 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:59.289 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511315+0000", 2024-09-21T22:06:59.289 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000113909 2024-09-21T22:06:59.289 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.289 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.289 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:59.289 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511333+0000", 2024-09-21T22:06:59.289 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7062999999999999e-05 2024-09-21T22:06:59.290 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.290 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.290 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:59.290 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511344+0000", 2024-09-21T22:06:59.290 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1177e-05 2024-09-21T22:06:59.290 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.290 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.290 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:59.290 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.517415+0000", 2024-09-21T22:06:59.290 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0060709830000000003 2024-09-21T22:06:59.290 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.290 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.290 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:59.291 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.517539+0000", 2024-09-21T22:06:59.291 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000124432 2024-09-21T22:06:59.291 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.291 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.291 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:06:59.291 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.517732+0000", 2024-09-21T22:06:59.291 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019262200000000001 2024-09-21T22:06:59.291 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.291 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.291 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:59.291 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.520709+0000", 2024-09-21T22:06:59.291 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0029767470000000001 2024-09-21T22:06:59.291 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.295 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.295 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:59.295 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.524389+0000", 2024-09-21T22:06:59.295 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003680975 2024-09-21T22:06:59.295 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.295 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.295 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:59.295 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.524486+0000", 2024-09-21T22:06:59.296 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6575000000000003e-05 2024-09-21T22:06:59.296 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.296 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.296 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:59.296 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.524554+0000", 2024-09-21T22:06:59.296 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8362999999999996e-05 2024-09-21T22:06:59.296 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:59.296 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:59.296 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:59.296 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.296 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.296 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T22:06:59.296 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:03.511368+0000", 2024-09-21T22:06:59.296 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 55.661424406000002, 2024-09-21T22:06:59.297 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013312133, 2024-09-21T22:06:59.297 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:06:59.297 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:06:59.297 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:06:59.297 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:06:59.297 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:06:59.297 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:06:59.297 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 159 2024-09-21T22:06:59.297 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.297 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:06:59.297 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.297 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:06:59.297 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511368+0000", 2024-09-21T22:06:59.297 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:59.298 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.298 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.298 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:06:59.298 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511368+0000", 2024-09-21T22:06:59.298 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:06:59.298 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.298 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.298 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:06:59.298 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511369+0000", 2024-09-21T22:06:59.298 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9599999999999998e-07 2024-09-21T22:06:59.298 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.298 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.298 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:06:59.299 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511382+0000", 2024-09-21T22:06:59.299 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3487e-05 2024-09-21T22:06:59.299 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.299 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.299 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:06:59.299 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511394+0000", 2024-09-21T22:06:59.299 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1337e-05 2024-09-21T22:06:59.299 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.299 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.299 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:06:59.299 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511402+0000", 2024-09-21T22:06:59.299 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0059999999999992e-06 2024-09-21T22:06:59.299 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.299 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.300 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:06:59.300 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.519376+0000", 2024-09-21T22:06:59.300 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0079746520000000005 2024-09-21T22:06:59.300 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.300 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.300 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:06:59.300 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.519498+0000", 2024-09-21T22:06:59.300 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012212900000000001 2024-09-21T22:06:59.300 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.300 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.300 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:06:59.300 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.519728+0000", 2024-09-21T22:06:59.300 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022985799999999999 2024-09-21T22:06:59.300 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.301 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.301 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:06:59.301 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.521344+0000", 2024-09-21T22:06:59.301 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016163169999999999 2024-09-21T22:06:59.301 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.301 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.301 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:06:59.301 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.524568+0000", 2024-09-21T22:06:59.301 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0032234220000000001 2024-09-21T22:06:59.301 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.301 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.301 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:06:59.301 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.524633+0000", 2024-09-21T22:06:59.301 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5126e-05 2024-09-21T22:06:59.301 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:06:59.302 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:06:59.302 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:06:59.302 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.524680+0000", 2024-09-21T22:06:59.302 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6903000000000001e-05 2024-09-21T22:06:59.302 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:59.302 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:59.302 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:59.302 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:06:59.302 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:06:59.302 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:06:59.303 INFO:teuthology.orchestra.run.smithi039.stdout:osd.1: {} 2024-09-21T22:06:59.304 INFO:teuthology.orchestra.run.smithi039.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T22:06:59.304 INFO:teuthology.orchestra.run.smithi039.stdout:osd.2: {} 2024-09-21T22:06:59.304 INFO:teuthology.orchestra.run.smithi039.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T22:06:59.305 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T22:06:59.305 INFO:teuthology.orchestra.run.smithi039.stdout:osd.3: {} 2024-09-21T22:06:59.305 INFO:teuthology.orchestra.run.smithi039.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T22:06:59.306 INFO:teuthology.orchestra.run.smithi039.stdout:osd.4: {} 2024-09-21T22:06:59.306 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T22:06:59.307 INFO:teuthology.orchestra.run.smithi039.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T22:06:59.307 INFO:teuthology.orchestra.run.smithi039.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-21T22:06:59.307 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T22:06:59.307 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:06:59.299+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:06:59.404 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:06:59.403+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:06:59.438 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:59.438 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:59.452 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T22:06:59.505 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:06:59.500+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:06:59.578 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:59.578 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:59.594 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T22:06:59.635 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:06:59.632+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:06:59.707 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:06:59.704+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:06:59.759 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:59.760 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:59.774 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T22:06:59.815 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:06:59.812+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:06:59.909 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:06:59.903+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:06:59.911 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:06:59.911 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:06:59.926 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T22:07:00.010 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:00.004+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:00.096 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:00.096 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:00.111 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T22:07:00.111 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:00.103+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:00.212 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:00.207+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:00.284 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:00.284 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:00.299 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T22:07:00.343 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:00.339+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:00.413 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:00.407+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:00.463 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:00.463 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:00.477 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T22:07:00.519 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:00.515+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:00.615 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:00.612+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:00.646 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:00.646 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:00.660 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T22:07:00.716 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:00.711+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:00.779 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:00.780 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:00.794 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T22:07:00.835 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:00.831+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:00.917 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:00.911+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:00.961 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:00.962 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:00.976 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T22:07:01.017 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:01.012+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:01.030 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-21T22:07:01.118 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:01.116+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:01.150 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:01.150 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:01.165 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T22:07:01.219 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:01.211+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:01.289 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:01.289 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:01.303 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T22:07:01.308 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T22:07:01.347 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:01.344+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:01.421 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:01.415+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:01.439 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:01.439 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:01.453 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T22:07:01.490 INFO:teuthology.orchestra.run.smithi039.stdout:osd.0: {} 2024-09-21T22:07:01.490 INFO:teuthology.orchestra.run.smithi039.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T22:07:01.506 INFO:teuthology.orchestra.run.smithi039.stdout:osd.1: {} 2024-09-21T22:07:01.507 INFO:teuthology.orchestra.run.smithi039.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T22:07:01.523 INFO:teuthology.orchestra.run.smithi039.stdout:osd.2: {} 2024-09-21T22:07:01.523 INFO:teuthology.orchestra.run.smithi039.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T22:07:01.542 INFO:teuthology.orchestra.run.smithi039.stdout:osd.3: {} 2024-09-21T22:07:01.542 INFO:teuthology.orchestra.run.smithi039.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T22:07:01.543 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:01.540+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:01.560 INFO:teuthology.orchestra.run.smithi039.stdout:osd.4: {} 2024-09-21T22:07:01.560 INFO:teuthology.orchestra.run.smithi039.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T22:07:01.561 INFO:teuthology.orchestra.run.smithi039.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-21T22:07:01.564 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T22:07:01.564 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T22:07:01.623 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:01.615+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:01.626 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:07:01.626 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:07:01.626 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-21T22:07:01.626 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-21T22:07:01.627 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:07:01.640 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T22:07:01.724 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:01.723+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:01.815 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:07:01.816 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-21T22:07:01.816 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-21T22:07:01.816 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-21T22:07:01.816 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.816 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:07:01.816 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.036355+0000", 2024-09-21T22:07:01.816 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 59.775745409000002, 2024-09-21T22:07:01.816 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.062441934999999997, 2024-09-21T22:07:01.816 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:01.816 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:01.816 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:01.816 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:01.817 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:07:01.817 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:07:01.817 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1 2024-09-21T22:07:01.817 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.817 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:01.817 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.817 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:01.817 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036355+0000", 2024-09-21T22:07:01.817 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:01.817 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.817 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.817 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:01.817 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036355+0000", 2024-09-21T22:07:01.817 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:01.818 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.818 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.818 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:01.818 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036357+0000", 2024-09-21T22:07:01.818 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.446e-06 2024-09-21T22:07:01.818 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.818 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.818 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:01.818 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036373+0000", 2024-09-21T22:07:01.818 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5447e-05 2024-09-21T22:07:01.818 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.818 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.818 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:01.818 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036399+0000", 2024-09-21T22:07:01.819 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6423000000000001e-05 2024-09-21T22:07:01.819 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.819 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.819 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:01.819 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036420+0000", 2024-09-21T22:07:01.819 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1129999999999999e-05 2024-09-21T22:07:01.819 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.819 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.819 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:01.819 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036557+0000", 2024-09-21T22:07:01.819 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013686899999999999 2024-09-21T22:07:01.819 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.819 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.819 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-21T22:07:01.820 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036585+0000", 2024-09-21T22:07:01.820 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8362999999999999e-05 2024-09-21T22:07:01.820 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.820 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.820 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:01.820 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.095068+0000", 2024-09-21T22:07:01.820 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058482972000000001 2024-09-21T22:07:01.820 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.820 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.820 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:01.820 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.095516+0000", 2024-09-21T22:07:01.820 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00044802999999999998 2024-09-21T22:07:01.820 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.820 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.821 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:01.821 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.095869+0000", 2024-09-21T22:07:01.821 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000352104 2024-09-21T22:07:01.821 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.821 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.821 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:01.821 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.097055+0000", 2024-09-21T22:07:01.821 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011865599999999999 2024-09-21T22:07:01.821 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.821 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.821 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:01.821 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.098637+0000", 2024-09-21T22:07:01.821 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001581458 2024-09-21T22:07:01.821 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.822 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.822 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:01.822 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.098728+0000", 2024-09-21T22:07:01.822 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1043000000000002e-05 2024-09-21T22:07:01.822 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.822 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.822 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:01.822 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.098797+0000", 2024-09-21T22:07:01.823 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9090000000000004e-05 2024-09-21T22:07:01.823 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:01.823 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:01.823 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:01.823 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.823 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.823 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:07:01.823 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.161242+0000", 2024-09-21T22:07:01.823 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 59.650858497000002, 2024-09-21T22:07:01.823 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014477078000000001, 2024-09-21T22:07:01.823 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:01.823 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:01.824 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:01.824 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:01.824 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:07:01.824 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:07:01.824 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 35 2024-09-21T22:07:01.824 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.824 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:01.824 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.824 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:01.824 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161242+0000", 2024-09-21T22:07:01.824 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:01.824 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.824 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.825 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:01.825 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161242+0000", 2024-09-21T22:07:01.825 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:01.825 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.825 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.825 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:01.825 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161243+0000", 2024-09-21T22:07:01.825 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4729999999999999e-06 2024-09-21T22:07:01.825 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.826 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.826 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:01.826 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161385+0000", 2024-09-21T22:07:01.826 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014226899999999999 2024-09-21T22:07:01.826 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.826 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.826 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:01.826 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161417+0000", 2024-09-21T22:07:01.826 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1406999999999999e-05 2024-09-21T22:07:01.826 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.826 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.826 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:01.827 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161434+0000", 2024-09-21T22:07:01.827 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7623000000000001e-05 2024-09-21T22:07:01.827 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.827 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.827 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:01.827 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.169728+0000", 2024-09-21T22:07:01.827 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0082933090000000004 2024-09-21T22:07:01.827 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.827 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.827 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:01.827 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.169861+0000", 2024-09-21T22:07:01.827 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013367300000000001 2024-09-21T22:07:01.827 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.827 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.828 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:01.828 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.170186+0000", 2024-09-21T22:07:01.828 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00032466400000000001 2024-09-21T22:07:01.828 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.828 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.828 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:01.828 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.174759+0000", 2024-09-21T22:07:01.828 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0045727880000000004 2024-09-21T22:07:01.828 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.828 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.828 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:01.828 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.175497+0000", 2024-09-21T22:07:01.828 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00073803400000000004 2024-09-21T22:07:01.829 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.829 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.829 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:01.829 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.175607+0000", 2024-09-21T22:07:01.829 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010987199999999999 2024-09-21T22:07:01.829 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.829 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.829 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:01.829 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.175719+0000", 2024-09-21T22:07:01.829 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011196600000000001 2024-09-21T22:07:01.829 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:01.829 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:01.829 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:01.829 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.829 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.830 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:07:01.830 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.161483+0000", 2024-09-21T22:07:01.830 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 59.650616986000003, 2024-09-21T22:07:01.830 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016155732999999999, 2024-09-21T22:07:01.830 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:01.830 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:01.830 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:01.830 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:01.830 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:07:01.830 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:07:01.830 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 36 2024-09-21T22:07:01.830 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.831 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:01.831 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.831 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:01.831 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161483+0000", 2024-09-21T22:07:01.831 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:01.831 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.831 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.831 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:01.831 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161483+0000", 2024-09-21T22:07:01.831 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:01.831 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.831 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.831 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:01.831 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161485+0000", 2024-09-21T22:07:01.831 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5999999999999999e-06 2024-09-21T22:07:01.832 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.832 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.832 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:01.832 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161498+0000", 2024-09-21T22:07:01.832 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3166999999999999e-05 2024-09-21T22:07:01.832 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.832 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.832 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:01.832 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161526+0000", 2024-09-21T22:07:01.832 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7786000000000001e-05 2024-09-21T22:07:01.832 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.832 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.832 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:01.832 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161545+0000", 2024-09-21T22:07:01.832 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8899999999999999e-05 2024-09-21T22:07:01.833 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.833 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.833 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:01.833 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.172615+0000", 2024-09-21T22:07:01.833 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011070465 2024-09-21T22:07:01.833 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.833 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.833 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:01.833 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.172759+0000", 2024-09-21T22:07:01.833 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014416900000000001 2024-09-21T22:07:01.833 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.833 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.833 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:01.833 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.172967+0000", 2024-09-21T22:07:01.834 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020758500000000001 2024-09-21T22:07:01.834 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.834 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.834 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:01.834 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.174775+0000", 2024-09-21T22:07:01.834 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0018080990000000001 2024-09-21T22:07:01.834 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.834 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.834 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:01.834 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.177447+0000", 2024-09-21T22:07:01.834 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026717429999999999 2024-09-21T22:07:01.834 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.834 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.834 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:01.835 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.177525+0000", 2024-09-21T22:07:01.835 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7846e-05 2024-09-21T22:07:01.835 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.835 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.835 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:01.835 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.177639+0000", 2024-09-21T22:07:01.835 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000114373 2024-09-21T22:07:01.835 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:01.835 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:01.835 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:01.835 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.836 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.836 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:07:01.836 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.161577+0000", 2024-09-21T22:07:01.836 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 59.650523452999998, 2024-09-21T22:07:01.836 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016652991999999998, 2024-09-21T22:07:01.836 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:01.836 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:01.836 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:01.836 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:01.836 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:07:01.836 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:07:01.836 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 37 2024-09-21T22:07:01.836 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.837 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:01.837 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.837 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:01.837 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161577+0000", 2024-09-21T22:07:01.837 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:01.837 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.837 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.837 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:01.837 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161577+0000", 2024-09-21T22:07:01.837 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:01.837 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.837 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.837 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:01.837 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161578+0000", 2024-09-21T22:07:01.838 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5269999999999999e-06 2024-09-21T22:07:01.838 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.838 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.838 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:01.838 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161591+0000", 2024-09-21T22:07:01.838 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.296e-05 2024-09-21T22:07:01.838 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.838 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.838 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:01.838 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161629+0000", 2024-09-21T22:07:01.838 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7799000000000002e-05 2024-09-21T22:07:01.838 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.838 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.838 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:01.839 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161644+0000", 2024-09-21T22:07:01.839 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5333e-05 2024-09-21T22:07:01.839 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.839 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.839 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:01.839 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.173636+0000", 2024-09-21T22:07:01.839 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011992078 2024-09-21T22:07:01.839 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.839 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.839 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:01.839 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.173862+0000", 2024-09-21T22:07:01.839 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022556799999999999 2024-09-21T22:07:01.839 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.839 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.839 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:01.840 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.174193+0000", 2024-09-21T22:07:01.840 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000331088 2024-09-21T22:07:01.840 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.840 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.840 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:01.840 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.175740+0000", 2024-09-21T22:07:01.840 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015467910000000001 2024-09-21T22:07:01.840 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.840 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.840 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:01.840 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.178073+0000", 2024-09-21T22:07:01.840 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002333356 2024-09-21T22:07:01.840 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.840 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.840 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:01.841 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.178162+0000", 2024-09-21T22:07:01.841 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8306000000000005e-05 2024-09-21T22:07:01.841 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.841 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.841 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:01.841 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.178230+0000", 2024-09-21T22:07:01.841 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8186000000000001e-05 2024-09-21T22:07:01.841 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:01.841 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:01.841 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:01.841 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.841 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.841 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:07:01.841 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.280540+0000", 2024-09-21T22:07:01.841 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 59.531559881, 2024-09-21T22:07:01.842 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013246212, 2024-09-21T22:07:01.842 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:01.842 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:01.842 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:01.842 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:01.842 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:07:01.842 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:07:01.842 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 88 2024-09-21T22:07:01.842 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.842 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:01.842 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.842 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:01.842 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.280540+0000", 2024-09-21T22:07:01.842 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:01.843 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.843 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.843 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:01.843 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.280540+0000", 2024-09-21T22:07:01.843 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:01.843 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.843 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.843 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:01.843 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.280541+0000", 2024-09-21T22:07:01.843 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.54e-07 2024-09-21T22:07:01.843 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.843 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.843 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:01.843 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.280549+0000", 2024-09-21T22:07:01.843 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3329999999999997e-06 2024-09-21T22:07:01.843 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.844 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.844 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:01.844 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.280560+0000", 2024-09-21T22:07:01.844 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1003e-05 2024-09-21T22:07:01.844 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.844 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.844 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:01.844 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.280569+0000", 2024-09-21T22:07:01.844 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2430000000000005e-06 2024-09-21T22:07:01.844 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.844 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.844 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:01.845 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.291095+0000", 2024-09-21T22:07:01.845 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010526259 2024-09-21T22:07:01.845 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.845 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.845 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:01.845 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.291231+0000", 2024-09-21T22:07:01.845 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013561899999999999 2024-09-21T22:07:01.845 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.845 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.845 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:01.845 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.291457+0000", 2024-09-21T22:07:01.845 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022686200000000001 2024-09-21T22:07:01.845 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.845 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.846 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:01.846 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.293117+0000", 2024-09-21T22:07:01.846 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001659474 2024-09-21T22:07:01.846 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.846 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.846 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:01.846 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.293524+0000", 2024-09-21T22:07:01.846 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00040701300000000001 2024-09-21T22:07:01.846 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.846 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.847 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:01.847 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.293673+0000", 2024-09-21T22:07:01.847 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000148782 2024-09-21T22:07:01.847 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.847 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.847 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:01.847 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.293786+0000", 2024-09-21T22:07:01.847 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011377 2024-09-21T22:07:01.847 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:01.847 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:01.847 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:01.847 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.847 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.847 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:07:01.848 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.466485+0000", 2024-09-21T22:07:01.848 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 59.34561515, 2024-09-21T22:07:01.848 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014443981, 2024-09-21T22:07:01.848 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:01.848 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:01.848 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:01.848 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:01.848 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:07:01.848 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:07:01.848 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 107 2024-09-21T22:07:01.848 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.848 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:01.848 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.848 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:01.849 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.466485+0000", 2024-09-21T22:07:01.849 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:01.849 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.849 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.849 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:01.849 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.466485+0000", 2024-09-21T22:07:01.849 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:01.849 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.849 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.849 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:01.849 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.466486+0000", 2024-09-21T22:07:01.849 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6299999999999993e-07 2024-09-21T22:07:01.849 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.849 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.849 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:01.850 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.466493+0000", 2024-09-21T22:07:01.850 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9639999999999997e-06 2024-09-21T22:07:01.850 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.850 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.850 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:01.850 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.466507+0000", 2024-09-21T22:07:01.850 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4216000000000001e-05 2024-09-21T22:07:01.850 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.850 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.850 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:01.850 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.466515+0000", 2024-09-21T22:07:01.850 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.157e-06 2024-09-21T22:07:01.850 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.850 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.851 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:01.851 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.472033+0000", 2024-09-21T22:07:01.851 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0055175839999999999 2024-09-21T22:07:01.851 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.851 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.851 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:01.851 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.472174+0000", 2024-09-21T22:07:01.851 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000141612 2024-09-21T22:07:01.851 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.851 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.851 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:01.851 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.472381+0000", 2024-09-21T22:07:01.851 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000206961 2024-09-21T22:07:01.851 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.851 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.852 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:01.852 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.473924+0000", 2024-09-21T22:07:01.852 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015427119999999999 2024-09-21T22:07:01.852 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.852 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.852 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:01.852 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.480666+0000", 2024-09-21T22:07:01.852 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0067415410000000002 2024-09-21T22:07:01.852 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.852 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.852 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:01.852 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.480811+0000", 2024-09-21T22:07:01.852 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000145422 2024-09-21T22:07:01.852 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.853 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.853 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:01.853 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.480929+0000", 2024-09-21T22:07:01.853 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000117849 2024-09-21T22:07:01.853 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:01.853 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:01.853 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:01.853 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.853 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.853 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:07:01.853 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.467057+0000", 2024-09-21T22:07:01.853 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 59.345042890999999, 2024-09-21T22:07:01.853 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018350195, 2024-09-21T22:07:01.853 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:01.853 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:01.854 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:01.854 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:01.854 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:07:01.854 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:07:01.854 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 108 2024-09-21T22:07:01.854 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.854 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:01.854 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.854 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:01.854 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467057+0000", 2024-09-21T22:07:01.854 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:01.854 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.854 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.855 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:01.855 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467057+0000", 2024-09-21T22:07:01.855 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:01.855 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.855 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.855 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:01.855 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467059+0000", 2024-09-21T22:07:01.855 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.826e-06 2024-09-21T22:07:01.855 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.855 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.855 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:01.855 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467071+0000", 2024-09-21T22:07:01.855 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2224e-05 2024-09-21T22:07:01.855 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.855 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.856 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:01.856 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467098+0000", 2024-09-21T22:07:01.856 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6953000000000001e-05 2024-09-21T22:07:01.856 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.856 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.856 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:01.856 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467113+0000", 2024-09-21T22:07:01.856 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5119999999999999e-05 2024-09-21T22:07:01.856 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.856 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.856 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:01.856 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.474060+0000", 2024-09-21T22:07:01.857 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0069462919999999997 2024-09-21T22:07:01.857 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.857 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.857 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:01.857 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.474283+0000", 2024-09-21T22:07:01.857 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000223315 2024-09-21T22:07:01.857 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.857 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.857 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:01.857 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.474660+0000", 2024-09-21T22:07:01.857 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00037729400000000002 2024-09-21T22:07:01.857 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.857 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.858 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:01.858 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.480622+0000", 2024-09-21T22:07:01.858 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0059613369999999997 2024-09-21T22:07:01.858 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.858 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.858 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:01.858 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.485145+0000", 2024-09-21T22:07:01.858 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0045231960000000002 2024-09-21T22:07:01.858 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.858 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.858 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:01.858 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.485288+0000", 2024-09-21T22:07:01.858 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000143232 2024-09-21T22:07:01.859 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.859 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.859 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:01.859 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.485407+0000", 2024-09-21T22:07:01.859 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000119406 2024-09-21T22:07:01.859 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:01.859 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:01.859 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:01.859 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.859 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.859 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:07:01.859 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.467144+0000", 2024-09-21T22:07:01.859 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 59.344955671999998, 2024-09-21T22:07:01.859 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018907146999999999, 2024-09-21T22:07:01.860 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:01.860 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:01.860 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:01.860 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:01.860 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:07:01.860 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:07:01.860 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 109 2024-09-21T22:07:01.860 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.860 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:01.860 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.860 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:01.860 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467144+0000", 2024-09-21T22:07:01.860 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:01.860 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.860 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.860 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:01.861 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467144+0000", 2024-09-21T22:07:01.861 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:01.861 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.861 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.861 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:01.861 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467145+0000", 2024-09-21T22:07:01.861 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9999999999999996e-07 2024-09-21T22:07:01.861 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.861 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.861 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:01.861 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467331+0000", 2024-09-21T22:07:01.861 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018518899999999999 2024-09-21T22:07:01.861 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.861 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.861 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:01.862 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467350+0000", 2024-09-21T22:07:01.862 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8966000000000001e-05 2024-09-21T22:07:01.862 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.862 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.862 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:01.862 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467361+0000", 2024-09-21T22:07:01.862 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1233000000000001e-05 2024-09-21T22:07:01.862 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.862 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.862 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:01.862 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.475613+0000", 2024-09-21T22:07:01.862 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0082518629999999999 2024-09-21T22:07:01.862 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.863 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.863 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:01.863 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.475859+0000", 2024-09-21T22:07:01.863 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024633800000000003 2024-09-21T22:07:01.863 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.863 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.863 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:01.863 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.476223+0000", 2024-09-21T22:07:01.863 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00036408700000000003 2024-09-21T22:07:01.863 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.863 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.863 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:01.863 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.481046+0000", 2024-09-21T22:07:01.863 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.004823356 2024-09-21T22:07:01.863 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.864 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.864 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:01.864 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.485722+0000", 2024-09-21T22:07:01.864 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0046753640000000004 2024-09-21T22:07:01.864 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.864 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.864 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:01.864 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.485908+0000", 2024-09-21T22:07:01.864 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000185859 2024-09-21T22:07:01.864 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.864 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.864 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:01.864 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.486052+0000", 2024-09-21T22:07:01.864 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014399200000000001 2024-09-21T22:07:01.864 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:01.865 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:01.865 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:01.865 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.865 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.865 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:07:01.865 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.467384+0000", 2024-09-21T22:07:01.865 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 59.344715844, 2024-09-21T22:07:01.865 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.01894144, 2024-09-21T22:07:01.865 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:01.865 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:01.865 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:01.865 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:01.865 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:07:01.866 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:07:01.866 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 110 2024-09-21T22:07:01.866 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.866 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:01.866 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.866 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:01.866 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467384+0000", 2024-09-21T22:07:01.866 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:01.866 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.866 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.866 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:01.866 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467384+0000", 2024-09-21T22:07:01.866 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:01.866 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.866 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.867 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:01.867 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467385+0000", 2024-09-21T22:07:01.867 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0470000000000001e-06 2024-09-21T22:07:01.867 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.867 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.867 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:01.867 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467393+0000", 2024-09-21T22:07:01.867 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3869999999999997e-06 2024-09-21T22:07:01.867 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.867 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.868 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:01.868 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467413+0000", 2024-09-21T22:07:01.868 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9953e-05 2024-09-21T22:07:01.868 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.868 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.868 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:01.868 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467421+0000", 2024-09-21T22:07:01.868 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2400000000000007e-06 2024-09-21T22:07:01.868 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.868 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.868 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:01.868 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.479542+0000", 2024-09-21T22:07:01.868 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012120832 2024-09-21T22:07:01.868 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.868 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.869 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:01.869 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.479705+0000", 2024-09-21T22:07:01.869 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016362899999999999 2024-09-21T22:07:01.869 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.869 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.869 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:01.869 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.479914+0000", 2024-09-21T22:07:01.869 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020814499999999999 2024-09-21T22:07:01.869 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.869 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.869 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:01.869 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.485536+0000", 2024-09-21T22:07:01.869 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0056222800000000003 2024-09-21T22:07:01.869 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.869 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.870 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:01.870 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.486077+0000", 2024-09-21T22:07:01.870 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00054079900000000001 2024-09-21T22:07:01.870 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.870 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.870 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:01.870 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.486213+0000", 2024-09-21T22:07:01.870 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000136682 2024-09-21T22:07:01.870 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.870 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.870 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:01.870 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.486326+0000", 2024-09-21T22:07:01.870 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000112446 2024-09-21T22:07:01.870 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:01.871 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:01.871 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:01.871 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.871 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.871 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:07:01.871 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.467436+0000", 2024-09-21T22:07:01.871 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 59.344664344000002, 2024-09-21T22:07:01.871 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019152168000000001, 2024-09-21T22:07:01.871 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:01.871 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:01.871 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:01.872 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:01.872 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:07:01.872 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:07:01.872 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 111 2024-09-21T22:07:01.872 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.872 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:01.872 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.872 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:01.872 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467436+0000", 2024-09-21T22:07:01.872 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:01.872 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.872 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.872 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:01.872 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467436+0000", 2024-09-21T22:07:01.872 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:01.873 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.873 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.873 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:01.873 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467437+0000", 2024-09-21T22:07:01.873 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0699999999999996e-07 2024-09-21T22:07:01.873 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.873 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.873 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:01.873 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467445+0000", 2024-09-21T22:07:01.873 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.789e-06 2024-09-21T22:07:01.873 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.873 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.874 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:01.874 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467467+0000", 2024-09-21T22:07:01.874 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1253000000000001e-05 2024-09-21T22:07:01.874 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.874 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.874 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:01.874 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467475+0000", 2024-09-21T22:07:01.874 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0039999999999999e-06 2024-09-21T22:07:01.874 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.874 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.874 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:01.874 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.481164+0000", 2024-09-21T22:07:01.874 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013689114 2024-09-21T22:07:01.875 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.875 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.875 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:01.875 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.481394+0000", 2024-09-21T22:07:01.875 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000229931 2024-09-21T22:07:01.875 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.875 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.875 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:01.875 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.481779+0000", 2024-09-21T22:07:01.875 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00038545400000000003 2024-09-21T22:07:01.875 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.875 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.875 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:01.875 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.485675+0000", 2024-09-21T22:07:01.875 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0038958370000000001 2024-09-21T22:07:01.876 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.876 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.876 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:01.876 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.486346+0000", 2024-09-21T22:07:01.876 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00067119100000000002 2024-09-21T22:07:01.876 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.876 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.876 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:01.876 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.486478+0000", 2024-09-21T22:07:01.876 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013182600000000001 2024-09-21T22:07:01.876 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.876 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.876 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:01.876 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.486588+0000", 2024-09-21T22:07:01.877 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000110062 2024-09-21T22:07:01.877 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:01.877 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:01.877 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:01.877 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.877 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.877 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:07:01.877 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.467488+0000", 2024-09-21T22:07:01.877 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 59.344611718000003, 2024-09-21T22:07:01.877 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021038304000000001, 2024-09-21T22:07:01.878 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:01.878 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:01.878 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:01.878 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:01.878 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:07:01.878 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:07:01.878 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 112 2024-09-21T22:07:01.878 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.878 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:01.878 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.878 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:01.878 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467488+0000", 2024-09-21T22:07:01.879 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:01.879 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.879 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.879 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:01.879 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467488+0000", 2024-09-21T22:07:01.879 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:01.879 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.879 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.879 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:01.879 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467489+0000", 2024-09-21T22:07:01.879 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7999999999999995e-07 2024-09-21T22:07:01.879 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.879 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.879 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:01.880 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467687+0000", 2024-09-21T22:07:01.880 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019776200000000001 2024-09-21T22:07:01.880 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.880 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.880 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:01.880 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467704+0000", 2024-09-21T22:07:01.880 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7193e-05 2024-09-21T22:07:01.880 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.880 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.880 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:01.880 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467719+0000", 2024-09-21T22:07:01.880 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4703e-05 2024-09-21T22:07:01.880 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.880 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.880 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:01.880 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.482745+0000", 2024-09-21T22:07:01.881 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015025801 2024-09-21T22:07:01.881 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.881 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.881 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:01.881 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.483015+0000", 2024-09-21T22:07:01.881 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026999400000000002 2024-09-21T22:07:01.881 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.881 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.881 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:01.881 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.483440+0000", 2024-09-21T22:07:01.881 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000425697 2024-09-21T22:07:01.881 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.881 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.881 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:01.881 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.486701+0000", 2024-09-21T22:07:01.882 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0032611950000000002 2024-09-21T22:07:01.882 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.882 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.882 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:01.882 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.488262+0000", 2024-09-21T22:07:01.882 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015610439999999999 2024-09-21T22:07:01.882 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.882 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.882 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:01.882 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.488406+0000", 2024-09-21T22:07:01.882 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000143232 2024-09-21T22:07:01.882 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.882 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.882 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:01.883 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.488527+0000", 2024-09-21T22:07:01.883 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012100299999999999 2024-09-21T22:07:01.883 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:01.883 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:01.883 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:01.883 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.883 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.883 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:07:01.883 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.467786+0000", 2024-09-21T22:07:01.883 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 59.344314292999996, 2024-09-21T22:07:01.883 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026339741999999999, 2024-09-21T22:07:01.883 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:01.883 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:01.883 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:01.884 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:01.884 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:07:01.884 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:07:01.884 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 113 2024-09-21T22:07:01.884 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.884 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:01.884 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.884 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:01.884 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467786+0000", 2024-09-21T22:07:01.885 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:01.885 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.885 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.885 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:01.885 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467786+0000", 2024-09-21T22:07:01.885 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:01.885 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.885 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.885 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:01.885 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467787+0000", 2024-09-21T22:07:01.885 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.446e-06 2024-09-21T22:07:01.885 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.885 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.885 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:01.885 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467802+0000", 2024-09-21T22:07:01.886 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4769999999999999e-05 2024-09-21T22:07:01.886 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.886 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.886 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:01.886 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467826+0000", 2024-09-21T22:07:01.886 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3430000000000001e-05 2024-09-21T22:07:01.886 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.886 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.886 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:01.886 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467843+0000", 2024-09-21T22:07:01.886 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7065999999999999e-05 2024-09-21T22:07:01.886 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.886 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.887 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:01.887 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.486820+0000", 2024-09-21T22:07:01.887 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018977233999999999 2024-09-21T22:07:01.887 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.887 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.887 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:01.887 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.487065+0000", 2024-09-21T22:07:01.887 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024558799999999998 2024-09-21T22:07:01.887 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.887 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.887 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:01.887 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.487442+0000", 2024-09-21T22:07:01.887 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000376707 2024-09-21T22:07:01.887 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.888 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.888 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:01.888 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.492256+0000", 2024-09-21T22:07:01.888 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0048141620000000003 2024-09-21T22:07:01.888 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.888 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.888 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:01.888 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.493865+0000", 2024-09-21T22:07:01.888 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016090150000000001 2024-09-21T22:07:01.888 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.888 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.889 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:01.889 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.494007+0000", 2024-09-21T22:07:01.889 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000141359 2024-09-21T22:07:01.889 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.889 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.889 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:01.889 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.494126+0000", 2024-09-21T22:07:01.889 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000118965 2024-09-21T22:07:01.889 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:01.889 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:01.889 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:01.889 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.889 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.889 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:07:01.890 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.467878+0000", 2024-09-21T22:07:01.890 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 59.344221804, 2024-09-21T22:07:01.890 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.027783958000000001, 2024-09-21T22:07:01.890 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:01.890 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:01.890 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:01.890 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:01.890 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:07:01.890 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:07:01.890 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 114 2024-09-21T22:07:01.890 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.890 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:01.890 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.890 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:01.891 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467878+0000", 2024-09-21T22:07:01.891 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:01.891 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.891 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.891 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:01.891 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467878+0000", 2024-09-21T22:07:01.891 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:01.891 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.891 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.891 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:01.891 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467880+0000", 2024-09-21T22:07:01.891 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.223e-06 2024-09-21T22:07:01.891 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.891 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.891 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:01.892 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467891+0000", 2024-09-21T22:07:01.892 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1527e-05 2024-09-21T22:07:01.892 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.892 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.892 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:01.892 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467913+0000", 2024-09-21T22:07:01.892 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.143e-05 2024-09-21T22:07:01.892 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.892 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.892 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:01.892 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467930+0000", 2024-09-21T22:07:01.892 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7620000000000001e-05 2024-09-21T22:07:01.893 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.893 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.893 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:01.893 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.488690+0000", 2024-09-21T22:07:01.893 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020759588999999998 2024-09-21T22:07:01.893 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.893 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.893 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:01.893 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.488882+0000", 2024-09-21T22:07:01.893 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019236200000000001 2024-09-21T22:07:01.893 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.893 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.893 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:01.893 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.489093+0000", 2024-09-21T22:07:01.894 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000211318 2024-09-21T22:07:01.894 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.894 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.894 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:01.894 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.494152+0000", 2024-09-21T22:07:01.894 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0050582769999999999 2024-09-21T22:07:01.894 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.894 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.894 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:01.894 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.495373+0000", 2024-09-21T22:07:01.894 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001221747 2024-09-21T22:07:01.894 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.895 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.895 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:01.895 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.495543+0000", 2024-09-21T22:07:01.895 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000169259 2024-09-21T22:07:01.895 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.895 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.895 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:01.895 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.495662+0000", 2024-09-21T22:07:01.895 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011960600000000001 2024-09-21T22:07:01.895 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:01.895 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:01.895 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:01.896 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.896 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.896 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:07:01.896 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.468118+0000", 2024-09-21T22:07:01.896 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 59.343982062999999, 2024-09-21T22:07:01.896 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.027938877000000001, 2024-09-21T22:07:01.896 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:01.896 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:01.896 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:01.896 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:01.896 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:07:01.896 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:07:01.896 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 115 2024-09-21T22:07:01.896 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.896 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:01.897 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.897 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:01.897 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468118+0000", 2024-09-21T22:07:01.897 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:01.897 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.897 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.897 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:01.897 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468118+0000", 2024-09-21T22:07:01.897 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:01.897 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.897 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.897 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:01.897 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468120+0000", 2024-09-21T22:07:01.897 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7069999999999999e-06 2024-09-21T22:07:01.897 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.898 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.898 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:01.898 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468381+0000", 2024-09-21T22:07:01.898 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026135399999999999 2024-09-21T22:07:01.898 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.898 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.898 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:01.898 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468409+0000", 2024-09-21T22:07:01.898 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.781e-05 2024-09-21T22:07:01.898 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.898 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.898 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:01.899 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468433+0000", 2024-09-21T22:07:01.899 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4343999999999999e-05 2024-09-21T22:07:01.899 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.899 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.899 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:01.899 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.489785+0000", 2024-09-21T22:07:01.899 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021351221 2024-09-21T22:07:01.899 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.899 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.899 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:01.899 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.490018+0000", 2024-09-21T22:07:01.899 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023317800000000001 2024-09-21T22:07:01.899 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.900 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.900 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:01.900 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.490412+0000", 2024-09-21T22:07:01.900 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00039417400000000001 2024-09-21T22:07:01.900 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.900 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.900 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:01.900 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.495694+0000", 2024-09-21T22:07:01.900 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.005282335 2024-09-21T22:07:01.900 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.900 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.900 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:01.900 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.495852+0000", 2024-09-21T22:07:01.900 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015777200000000001 2024-09-21T22:07:01.901 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.901 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.901 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:01.901 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.495938+0000", 2024-09-21T22:07:01.901 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6493e-05 2024-09-21T22:07:01.901 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.901 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.901 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:01.901 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.496057+0000", 2024-09-21T22:07:01.901 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000118489 2024-09-21T22:07:01.901 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:01.901 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:01.901 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:01.901 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.901 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.902 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:07:01.902 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.468474+0000", 2024-09-21T22:07:01.902 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 59.343626301999997, 2024-09-21T22:07:01.902 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.028015895999999998, 2024-09-21T22:07:01.902 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:01.902 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:01.902 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:01.902 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:01.902 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:07:01.902 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:07:01.902 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 116 2024-09-21T22:07:01.902 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.902 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:01.902 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.903 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:01.903 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468474+0000", 2024-09-21T22:07:01.903 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:01.903 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.903 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.903 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:01.903 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468474+0000", 2024-09-21T22:07:01.903 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:01.903 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.903 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.903 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:01.903 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468476+0000", 2024-09-21T22:07:01.904 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7060000000000001e-06 2024-09-21T22:07:01.904 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.904 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.904 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:01.904 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468488+0000", 2024-09-21T22:07:01.904 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.234e-05 2024-09-21T22:07:01.904 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.904 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.904 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:01.904 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468517+0000", 2024-09-21T22:07:01.904 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.898e-05 2024-09-21T22:07:01.904 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.904 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.904 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:01.905 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468536+0000", 2024-09-21T22:07:01.905 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8899999999999999e-05 2024-09-21T22:07:01.905 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.905 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.905 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:01.905 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.492376+0000", 2024-09-21T22:07:01.905 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023840659 2024-09-21T22:07:01.905 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.905 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.905 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:01.905 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.492621+0000", 2024-09-21T22:07:01.906 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000244794 2024-09-21T22:07:01.906 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.906 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.906 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:01.906 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.493008+0000", 2024-09-21T22:07:01.906 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00038629099999999998 2024-09-21T22:07:01.906 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.906 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.906 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:01.906 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.496192+0000", 2024-09-21T22:07:01.906 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0031847220000000001 2024-09-21T22:07:01.906 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.906 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.906 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:01.906 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.496233+0000", 2024-09-21T22:07:01.907 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0410000000000001e-05 2024-09-21T22:07:01.907 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.907 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.907 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:01.907 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.496374+0000", 2024-09-21T22:07:01.907 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140872 2024-09-21T22:07:01.907 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.907 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.907 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:01.907 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.496490+0000", 2024-09-21T22:07:01.907 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000116222 2024-09-21T22:07:01.907 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:01.907 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:01.907 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:01.907 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.908 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.908 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:07:01.908 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.468568+0000", 2024-09-21T22:07:01.908 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 59.343532183000001, 2024-09-21T22:07:01.908 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.028181578999999998, 2024-09-21T22:07:01.908 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:01.908 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:01.908 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:01.908 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:01.908 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:07:01.908 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:07:01.908 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 117 2024-09-21T22:07:01.909 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.909 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:01.909 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.909 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:01.909 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468568+0000", 2024-09-21T22:07:01.909 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:01.909 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.909 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.909 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:01.909 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468568+0000", 2024-09-21T22:07:01.909 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:01.910 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.910 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.910 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:01.910 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468569+0000", 2024-09-21T22:07:01.910 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4139999999999999e-06 2024-09-21T22:07:01.910 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.910 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.910 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:01.910 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468583+0000", 2024-09-21T22:07:01.910 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3409e-05 2024-09-21T22:07:01.910 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.910 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.910 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:01.911 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468608+0000", 2024-09-21T22:07:01.911 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5559999999999999e-05 2024-09-21T22:07:01.911 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.911 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.911 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:01.911 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468626+0000", 2024-09-21T22:07:01.911 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7863000000000002e-05 2024-09-21T22:07:01.911 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.911 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.911 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:01.911 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.494260+0000", 2024-09-21T22:07:01.911 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025633744999999999 2024-09-21T22:07:01.911 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.911 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.911 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:01.912 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.494519+0000", 2024-09-21T22:07:01.912 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025924800000000001 2024-09-21T22:07:01.912 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.912 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.912 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:01.912 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.494747+0000", 2024-09-21T22:07:01.912 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022792799999999999 2024-09-21T22:07:01.912 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.912 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.912 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:01.912 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.496519+0000", 2024-09-21T22:07:01.912 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017720030000000001 2024-09-21T22:07:01.912 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.913 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.913 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:01.913 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.496607+0000", 2024-09-21T22:07:01.913 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7873000000000004e-05 2024-09-21T22:07:01.913 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.913 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.913 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:01.913 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.496691+0000", 2024-09-21T22:07:01.913 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4009000000000002e-05 2024-09-21T22:07:01.913 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.913 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.913 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:01.914 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.496750+0000", 2024-09-21T22:07:01.914 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8526999999999997e-05 2024-09-21T22:07:01.914 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:01.914 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:01.914 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:01.914 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.914 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.914 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:07:01.914 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.898244+0000", 2024-09-21T22:07:01.914 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 58.91385588, 2024-09-21T22:07:01.914 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014749464, 2024-09-21T22:07:01.914 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:01.914 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:01.915 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:01.915 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:01.915 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:07:01.915 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:07:01.915 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 140 2024-09-21T22:07:01.915 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.915 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:01.915 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.915 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:01.915 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898244+0000", 2024-09-21T22:07:01.915 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:01.915 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.915 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.916 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:01.916 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898244+0000", 2024-09-21T22:07:01.916 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:01.916 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.916 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.916 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:01.916 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898245+0000", 2024-09-21T22:07:01.916 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3000000000000002e-07 2024-09-21T22:07:01.916 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.916 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.916 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:01.916 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898308+0000", 2024-09-21T22:07:01.917 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3468999999999998e-05 2024-09-21T22:07:01.917 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.917 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.917 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:01.917 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898315+0000", 2024-09-21T22:07:01.917 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.725e-06 2024-09-21T22:07:01.917 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.917 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.917 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:01.917 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898321+0000", 2024-09-21T22:07:01.917 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2090000000000002e-06 2024-09-21T22:07:01.917 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.917 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.917 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:01.917 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.905452+0000", 2024-09-21T22:07:01.917 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0071305989999999996 2024-09-21T22:07:01.918 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.918 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.918 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:01.918 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.905504+0000", 2024-09-21T22:07:01.918 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2638e-05 2024-09-21T22:07:01.918 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.918 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.918 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:01.918 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.905590+0000", 2024-09-21T22:07:01.918 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5333000000000004e-05 2024-09-21T22:07:01.918 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.918 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.918 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:01.918 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.911542+0000", 2024-09-21T22:07:01.919 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0059519819999999998 2024-09-21T22:07:01.919 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.919 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.919 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:01.919 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.912908+0000", 2024-09-21T22:07:01.919 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0013666640000000001 2024-09-21T22:07:01.919 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.919 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.919 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:01.919 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.912957+0000", 2024-09-21T22:07:01.919 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9203000000000003e-05 2024-09-21T22:07:01.919 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.920 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.920 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:01.920 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.912994+0000", 2024-09-21T22:07:01.920 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6312000000000001e-05 2024-09-21T22:07:01.920 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:01.920 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:01.920 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:01.920 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.920 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.920 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:07:01.920 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.898342+0000", 2024-09-21T22:07:01.920 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 58.913757742000001, 2024-09-21T22:07:01.921 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014787918000000001, 2024-09-21T22:07:01.921 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:01.921 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:01.921 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:01.921 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:01.921 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:07:01.921 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:07:01.921 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 141 2024-09-21T22:07:01.921 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.921 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:01.921 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.921 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:01.921 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898342+0000", 2024-09-21T22:07:01.921 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:01.922 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.922 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.922 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:01.922 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898342+0000", 2024-09-21T22:07:01.922 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:01.922 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.922 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.922 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:01.922 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898343+0000", 2024-09-21T22:07:01.922 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3000000000000001e-07 2024-09-21T22:07:01.922 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.923 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.923 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:01.923 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898346+0000", 2024-09-21T22:07:01.923 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6009999999999999e-06 2024-09-21T22:07:01.923 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.923 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.923 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:01.923 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898352+0000", 2024-09-21T22:07:01.923 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8089999999999998e-06 2024-09-21T22:07:01.923 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.923 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.923 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:01.923 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898358+0000", 2024-09-21T22:07:01.924 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4349999999999996e-06 2024-09-21T22:07:01.924 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.924 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.924 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:01.924 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.909387+0000", 2024-09-21T22:07:01.924 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011029105000000001 2024-09-21T22:07:01.924 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.924 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.924 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:01.924 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.909462+0000", 2024-09-21T22:07:01.924 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5192999999999996e-05 2024-09-21T22:07:01.924 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.924 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.924 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:01.924 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.909551+0000", 2024-09-21T22:07:01.925 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9046000000000001e-05 2024-09-21T22:07:01.925 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.925 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.925 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:01.925 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.911852+0000", 2024-09-21T22:07:01.925 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0023005579999999999 2024-09-21T22:07:01.925 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.925 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.925 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:01.925 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.913002+0000", 2024-09-21T22:07:01.925 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011502929999999999 2024-09-21T22:07:01.925 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.925 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.925 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:01.925 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.913060+0000", 2024-09-21T22:07:01.925 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8427000000000002e-05 2024-09-21T22:07:01.926 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.926 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.926 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:01.926 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.913130+0000", 2024-09-21T22:07:01.926 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0020999999999999e-05 2024-09-21T22:07:01.926 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:01.926 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:01.926 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:01.926 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.926 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.927 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:158 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T22:07:01.927 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:03.511201+0000", 2024-09-21T22:07:01.927 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 58.300899258999998, 2024-09-21T22:07:01.927 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013353538999999999, 2024-09-21T22:07:01.927 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:01.927 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:01.927 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:01.927 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:01.927 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:07:01.927 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:07:01.927 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 158 2024-09-21T22:07:01.927 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.927 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:01.927 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.928 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:01.928 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511201+0000", 2024-09-21T22:07:01.928 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:01.928 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.928 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.928 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:01.928 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511201+0000", 2024-09-21T22:07:01.928 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:01.928 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.928 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.928 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:01.928 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511202+0000", 2024-09-21T22:07:01.928 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9299999999999997e-07 2024-09-21T22:07:01.928 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.928 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.929 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:01.929 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511315+0000", 2024-09-21T22:07:01.929 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000113909 2024-09-21T22:07:01.929 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.929 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.929 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:01.929 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511333+0000", 2024-09-21T22:07:01.929 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7062999999999999e-05 2024-09-21T22:07:01.929 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.929 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.929 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:01.929 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511344+0000", 2024-09-21T22:07:01.929 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1177e-05 2024-09-21T22:07:01.929 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.929 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.930 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:01.930 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.517415+0000", 2024-09-21T22:07:01.930 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0060709830000000003 2024-09-21T22:07:01.930 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.930 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.930 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:01.930 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.517539+0000", 2024-09-21T22:07:01.930 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000124432 2024-09-21T22:07:01.930 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.931 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.931 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:01.931 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.517732+0000", 2024-09-21T22:07:01.931 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019262200000000001 2024-09-21T22:07:01.931 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.931 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.931 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:01.931 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.520709+0000", 2024-09-21T22:07:01.931 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0029767470000000001 2024-09-21T22:07:01.931 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.931 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.931 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:01.931 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.524389+0000", 2024-09-21T22:07:01.931 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003680975 2024-09-21T22:07:01.932 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.932 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.932 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:01.932 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.524486+0000", 2024-09-21T22:07:01.932 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6575000000000003e-05 2024-09-21T22:07:01.932 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.932 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.932 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:01.932 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.524554+0000", 2024-09-21T22:07:01.933 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8362999999999996e-05 2024-09-21T22:07:01.933 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:01.933 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:01.933 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:01.933 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.933 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.933 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T22:07:01.933 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:03.511368+0000", 2024-09-21T22:07:01.933 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 58.300732357000001, 2024-09-21T22:07:01.933 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013312133, 2024-09-21T22:07:01.933 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:01.933 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:01.934 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:01.934 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:01.934 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:07:01.934 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:07:01.934 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 159 2024-09-21T22:07:01.934 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.934 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:01.934 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.934 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:01.934 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511368+0000", 2024-09-21T22:07:01.934 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:01.934 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.934 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.935 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:01.935 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511368+0000", 2024-09-21T22:07:01.935 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:01.935 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.935 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.935 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:01.935 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511369+0000", 2024-09-21T22:07:01.935 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9599999999999998e-07 2024-09-21T22:07:01.935 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.935 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.935 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:01.935 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511382+0000", 2024-09-21T22:07:01.935 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3487e-05 2024-09-21T22:07:01.935 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.935 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.936 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:01.936 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511394+0000", 2024-09-21T22:07:01.936 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1337e-05 2024-09-21T22:07:01.936 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.936 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.936 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:01.936 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511402+0000", 2024-09-21T22:07:01.936 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0059999999999992e-06 2024-09-21T22:07:01.936 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.936 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.936 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:01.936 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.519376+0000", 2024-09-21T22:07:01.936 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0079746520000000005 2024-09-21T22:07:01.937 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.937 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.937 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:01.937 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.519498+0000", 2024-09-21T22:07:01.937 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012212900000000001 2024-09-21T22:07:01.937 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.937 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.937 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:01.937 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.519728+0000", 2024-09-21T22:07:01.937 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022985799999999999 2024-09-21T22:07:01.937 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.937 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.937 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:01.938 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.521344+0000", 2024-09-21T22:07:01.938 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016163169999999999 2024-09-21T22:07:01.938 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.938 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.938 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:01.938 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.524568+0000", 2024-09-21T22:07:01.938 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0032234220000000001 2024-09-21T22:07:01.938 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.938 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.938 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:01.938 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.524633+0000", 2024-09-21T22:07:01.938 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5126e-05 2024-09-21T22:07:01.938 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:01.938 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:01.939 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:01.939 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.524680+0000", 2024-09-21T22:07:01.939 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6903000000000001e-05 2024-09-21T22:07:01.939 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:01.939 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:01.939 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:01.939 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:01.939 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:01.939 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:07:01.941 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T22:07:01.941 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:01.939+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:02.006 INFO:teuthology.orchestra.run.smithi039.stderr:noscrub is set 2024-09-21T22:07:02.071 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:02.067+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:02.072 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:07:02.072 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:07:02.072 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-09-21T22:07:02.073 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:07:02.087 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T22:07:02.141 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:02.136+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:02.211 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:07:02.211 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:07:02.211 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-21T22:07:02.211 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-21T22:07:02.211 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:07:02.226 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T22:07:02.267 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:02.264+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:02.344 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:02.340+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:02.393 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:07:02.394 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-21T22:07:02.394 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-21T22:07:02.394 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [] 2024-09-21T22:07:02.394 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:07:02.408 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T22:07:02.451 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:02.448+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:02.545 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:07:02.545 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:07:02.545 INFO:teuthology.orchestra.run.smithi039.stdout: "num_ops": 0 2024-09-21T22:07:02.545 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:07:02.546 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:02.540+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:02.560 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T22:07:02.646 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:02.644+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:02.683 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:07:02.683 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:07:02.683 INFO:teuthology.orchestra.run.smithi039.stdout: "complaint_time": 30, 2024-09-21T22:07:02.683 INFO:teuthology.orchestra.run.smithi039.stdout: "num_blocked_ops": 0 2024-09-21T22:07:02.684 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:07:02.698 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T22:07:02.705 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 0] out_osds: [] dead_osds: [5] live_osds: [0, 1, 2, 3, 4] 2024-09-21T22:07:02.705 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T22:07:02.705 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-21T22:07:02.706 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-09-21T22:07:02.747 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:02.744+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:02.828 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:07:02.828 INFO:teuthology.orchestra.run.smithi039.stdout: "size": 20, 2024-09-21T22:07:02.828 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 600, 2024-09-21T22:07:02.828 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [ 2024-09-21T22:07:02.828 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.828 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:20 1.0 e14/13 1:30306672:devicehealth::main.db.0000000000000000:head v 14'10, mlcod=14'8)", 2024-09-21T22:07:02.828 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.137639+0000", 2024-09-21T22:07:02.828 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 60.687364666999997, 2024-09-21T22:07:02.829 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0039902100000000001, 2024-09-21T22:07:02.829 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:02.829 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:02.829 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:02.829 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:02.829 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.829 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:02.829 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.137639+0000", 2024-09-21T22:07:02.829 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:02.829 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.829 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.829 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:02.829 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.137639+0000", 2024-09-21T22:07:02.829 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:02.829 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.830 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.830 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:02.830 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.137641+0000", 2024-09-21T22:07:02.830 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.686e-06 2024-09-21T22:07:02.830 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.830 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.830 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:02.830 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.137934+0000", 2024-09-21T22:07:02.830 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00029319099999999999 2024-09-21T22:07:02.830 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.830 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.830 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:02.830 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.137963+0000", 2024-09-21T22:07:02.830 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9502999999999999e-05 2024-09-21T22:07:02.831 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.831 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.831 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:02.831 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.137989+0000", 2024-09-21T22:07:02.831 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5573000000000001e-05 2024-09-21T22:07:02.831 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.831 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.831 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:02.831 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.138086+0000", 2024-09-21T22:07:02.831 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.6968999999999999e-05 2024-09-21T22:07:02.831 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.831 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.831 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:02.831 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.138120+0000", 2024-09-21T22:07:02.831 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.4560000000000001e-05 2024-09-21T22:07:02.832 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.832 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.832 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:02.832 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.141539+0000", 2024-09-21T22:07:02.832 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0034183289999999999 2024-09-21T22:07:02.832 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.832 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.832 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:02.832 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.141629+0000", 2024-09-21T22:07:02.832 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.0399e-05 2024-09-21T22:07:02.832 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:02.832 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:02.832 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:02.832 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.832 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.833 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:23 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'13, mlcod=14'11)", 2024-09-21T22:07:02.833 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.144914+0000", 2024-09-21T22:07:02.833 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 60.680089348999999, 2024-09-21T22:07:02.833 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0040223230000000004, 2024-09-21T22:07:02.833 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:02.833 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:02.833 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:02.833 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:02.833 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.833 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:02.833 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.144914+0000", 2024-09-21T22:07:02.833 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:02.833 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.833 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.834 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:02.834 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.144914+0000", 2024-09-21T22:07:02.834 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:02.834 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.834 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.834 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:02.834 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.144916+0000", 2024-09-21T22:07:02.834 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6470000000000001e-06 2024-09-21T22:07:02.834 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.834 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.834 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:02.834 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.144929+0000", 2024-09-21T22:07:02.834 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2903e-05 2024-09-21T22:07:02.834 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.834 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.835 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:02.835 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.144958+0000", 2024-09-21T22:07:02.835 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9280000000000001e-05 2024-09-21T22:07:02.835 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.835 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.835 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:02.835 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.144971+0000", 2024-09-21T22:07:02.835 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3407000000000001e-05 2024-09-21T22:07:02.835 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.835 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.835 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:02.835 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.145106+0000", 2024-09-21T22:07:02.835 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013440500000000001 2024-09-21T22:07:02.835 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.836 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.836 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:02.836 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.145170+0000", 2024-09-21T22:07:02.836 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.4597000000000005e-05 2024-09-21T22:07:02.836 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.836 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.836 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:02.836 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.148854+0000", 2024-09-21T22:07:02.836 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0036833249999999999 2024-09-21T22:07:02.836 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.836 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.836 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:02.836 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.148937+0000", 2024-09-21T22:07:02.836 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.2758999999999998e-05 2024-09-21T22:07:02.836 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:02.837 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:02.837 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:02.837 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.837 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.837 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:31 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'17, mlcod=14'14)", 2024-09-21T22:07:02.837 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.163958+0000", 2024-09-21T22:07:02.837 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 60.661045082000001, 2024-09-21T22:07:02.837 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0033891149999999998, 2024-09-21T22:07:02.837 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:02.837 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:02.837 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:02.837 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:02.837 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.837 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:02.837 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.163958+0000", 2024-09-21T22:07:02.838 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:02.838 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.838 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.838 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:02.838 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.163958+0000", 2024-09-21T22:07:02.838 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:02.838 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.838 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.838 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:02.838 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.163960+0000", 2024-09-21T22:07:02.838 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.64e-06 2024-09-21T22:07:02.838 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.838 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.838 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:02.838 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.163971+0000", 2024-09-21T22:07:02.839 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0797e-05 2024-09-21T22:07:02.839 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.839 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.839 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:02.839 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.163995+0000", 2024-09-21T22:07:02.839 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3876e-05 2024-09-21T22:07:02.839 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.839 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.839 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:02.839 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.164008+0000", 2024-09-21T22:07:02.839 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3674e-05 2024-09-21T22:07:02.839 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.839 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.839 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:02.839 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.164203+0000", 2024-09-21T22:07:02.840 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00019457799999999999 2024-09-21T22:07:02.840 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.840 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.840 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:02.840 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.164265+0000", 2024-09-21T22:07:02.840 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.2415999999999993e-05 2024-09-21T22:07:02.840 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.840 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.840 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:02.840 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.167209+0000", 2024-09-21T22:07:02.840 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0029436380000000002 2024-09-21T22:07:02.840 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.840 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.840 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:02.840 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.167348+0000", 2024-09-21T22:07:02.841 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000138496 2024-09-21T22:07:02.841 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:02.841 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:02.841 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:02.841 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.841 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.841 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:32 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'18, mlcod=14'14)", 2024-09-21T22:07:02.841 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.165319+0000", 2024-09-21T22:07:02.841 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 60.659684232000004, 2024-09-21T22:07:02.841 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0033722280000000001, 2024-09-21T22:07:02.841 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:02.841 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:02.841 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:02.841 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:02.842 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.842 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:02.842 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.165319+0000", 2024-09-21T22:07:02.842 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:02.842 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.842 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.842 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:02.842 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.165319+0000", 2024-09-21T22:07:02.842 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:02.842 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.842 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.842 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:02.842 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.165321+0000", 2024-09-21T22:07:02.842 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5999999999999999e-06 2024-09-21T22:07:02.842 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.843 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.843 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:02.843 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.165377+0000", 2024-09-21T22:07:02.843 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.5877000000000001e-05 2024-09-21T22:07:02.843 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.843 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.843 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:02.843 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.165407+0000", 2024-09-21T22:07:02.843 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.0263000000000001e-05 2024-09-21T22:07:02.843 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.843 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.843 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:02.843 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.165441+0000", 2024-09-21T22:07:02.843 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.3966e-05 2024-09-21T22:07:02.844 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.844 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.844 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:02.844 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.165574+0000", 2024-09-21T22:07:02.844 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013327599999999999 2024-09-21T22:07:02.844 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.844 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.844 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:02.844 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.165635+0000", 2024-09-21T22:07:02.844 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.0949000000000002e-05 2024-09-21T22:07:02.844 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.844 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.844 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:02.844 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.168619+0000", 2024-09-21T22:07:02.844 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0029836979999999999 2024-09-21T22:07:02.845 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.845 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.845 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:02.845 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.168692+0000", 2024-09-21T22:07:02.845 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.2599000000000001e-05 2024-09-21T22:07:02.845 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:02.845 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:02.845 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:02.845 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.845 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.845 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:38 1.0 e14/13 1:30306672:devicehealth::main.db.0000000000000000:head v 14'24, mlcod=14'22)", 2024-09-21T22:07:02.845 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.180244+0000", 2024-09-21T22:07:02.845 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 60.644759587000003, 2024-09-21T22:07:02.845 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0041712219999999996, 2024-09-21T22:07:02.845 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:02.846 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:02.846 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:02.846 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:02.846 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.846 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:02.846 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.180244+0000", 2024-09-21T22:07:02.846 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:02.846 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.846 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.846 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:02.846 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.180244+0000", 2024-09-21T22:07:02.846 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:02.846 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.846 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.847 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:02.847 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.180246+0000", 2024-09-21T22:07:02.847 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6959999999999999e-06 2024-09-21T22:07:02.847 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.847 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.847 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:02.847 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.180508+0000", 2024-09-21T22:07:02.847 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00026275199999999999 2024-09-21T22:07:02.847 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.847 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.847 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:02.847 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.180538+0000", 2024-09-21T22:07:02.847 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9142999999999999e-05 2024-09-21T22:07:02.847 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.847 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.848 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:02.848 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.180552+0000", 2024-09-21T22:07:02.848 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4466e-05 2024-09-21T22:07:02.848 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.848 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.848 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:02.848 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.180677+0000", 2024-09-21T22:07:02.848 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000125413 2024-09-21T22:07:02.848 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.848 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.848 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:02.848 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.180739+0000", 2024-09-21T22:07:02.848 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.1122000000000002e-05 2024-09-21T22:07:02.848 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.849 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.849 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:02.849 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.184291+0000", 2024-09-21T22:07:02.849 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0035525510000000001 2024-09-21T22:07:02.849 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.849 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.849 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:02.849 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.184415+0000", 2024-09-21T22:07:02.849 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000124079 2024-09-21T22:07:02.849 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:02.849 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:02.849 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:02.849 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.849 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.849 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:54 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'32, mlcod=14'29)", 2024-09-21T22:07:02.850 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.214326+0000", 2024-09-21T22:07:02.850 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 60.610677350000003, 2024-09-21T22:07:02.850 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0035226569999999998, 2024-09-21T22:07:02.850 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:02.850 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:02.850 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:02.850 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:02.850 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.850 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:02.850 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.214326+0000", 2024-09-21T22:07:02.850 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:02.850 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.850 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.850 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:02.851 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.214326+0000", 2024-09-21T22:07:02.851 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:02.851 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.851 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.851 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:02.851 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.214328+0000", 2024-09-21T22:07:02.851 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6270000000000001e-06 2024-09-21T22:07:02.851 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.851 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.851 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:02.851 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.214340+0000", 2024-09-21T22:07:02.851 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2286000000000001e-05 2024-09-21T22:07:02.851 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.851 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.851 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:02.852 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.214369+0000", 2024-09-21T22:07:02.852 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9297000000000001e-05 2024-09-21T22:07:02.852 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.852 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.852 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:02.852 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.214383+0000", 2024-09-21T22:07:02.852 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3773e-05 2024-09-21T22:07:02.852 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.852 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.852 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:02.852 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.214513+0000", 2024-09-21T22:07:02.852 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000130056 2024-09-21T22:07:02.852 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.852 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.852 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:02.853 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.214568+0000", 2024-09-21T22:07:02.853 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.4873000000000001e-05 2024-09-21T22:07:02.853 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.853 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.853 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:02.853 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.217739+0000", 2024-09-21T22:07:02.853 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0031712720000000002 2024-09-21T22:07:02.853 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.853 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.853 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:02.853 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.217849+0000", 2024-09-21T22:07:02.853 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000109473 2024-09-21T22:07:02.853 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:02.853 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:02.853 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:02.854 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.854 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.854 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:57 1.0 e14/13 1:30306672:devicehealth::main.db.0000000000000000:head v 14'35, mlcod=14'33)", 2024-09-21T22:07:02.854 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.221549+0000", 2024-09-21T22:07:02.854 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 60.603454851000002, 2024-09-21T22:07:02.854 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0041720860000000002, 2024-09-21T22:07:02.854 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:02.854 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:02.854 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:02.854 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:02.854 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.854 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:02.854 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.221549+0000", 2024-09-21T22:07:02.854 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:02.855 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.855 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.855 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:02.855 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.221549+0000", 2024-09-21T22:07:02.855 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:02.855 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.855 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.855 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:02.855 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.221550+0000", 2024-09-21T22:07:02.855 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.717e-06 2024-09-21T22:07:02.855 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.855 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.855 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:02.855 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.221673+0000", 2024-09-21T22:07:02.855 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000122612 2024-09-21T22:07:02.856 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.856 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.856 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:02.856 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.221703+0000", 2024-09-21T22:07:02.856 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9539999999999998e-05 2024-09-21T22:07:02.856 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.856 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.856 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:02.856 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.221716+0000", 2024-09-21T22:07:02.856 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3149999999999999e-05 2024-09-21T22:07:02.856 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.856 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.856 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:02.856 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.221848+0000", 2024-09-21T22:07:02.857 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013201900000000001 2024-09-21T22:07:02.857 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.857 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.857 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:02.857 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.221908+0000", 2024-09-21T22:07:02.857 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.0452999999999999e-05 2024-09-21T22:07:02.857 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.857 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.857 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:02.857 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.225573+0000", 2024-09-21T22:07:02.857 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.003664676 2024-09-21T22:07:02.857 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.857 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.857 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:02.857 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.225721+0000", 2024-09-21T22:07:02.858 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00014791899999999999 2024-09-21T22:07:02.858 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:02.858 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:02.858 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:02.858 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.858 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.858 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:68 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'42, mlcod=14'39)", 2024-09-21T22:07:02.858 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.247413+0000", 2024-09-21T22:07:02.858 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 60.577590502, 2024-09-21T22:07:02.858 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0039572339999999996, 2024-09-21T22:07:02.858 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:02.858 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:02.858 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:02.858 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:02.859 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.859 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:02.859 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.247413+0000", 2024-09-21T22:07:02.859 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:02.859 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.859 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.859 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:02.859 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.247413+0000", 2024-09-21T22:07:02.859 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:02.859 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.859 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.859 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:02.859 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.247415+0000", 2024-09-21T22:07:02.859 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.883e-06 2024-09-21T22:07:02.859 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.860 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.860 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:02.860 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.247427+0000", 2024-09-21T22:07:02.860 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2557e-05 2024-09-21T22:07:02.860 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.860 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.860 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:02.860 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.247457+0000", 2024-09-21T22:07:02.860 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9966000000000001e-05 2024-09-21T22:07:02.860 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.860 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.860 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:02.860 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.247471+0000", 2024-09-21T22:07:02.860 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3747000000000001e-05 2024-09-21T22:07:02.861 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.861 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.861 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:02.861 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.247603+0000", 2024-09-21T22:07:02.861 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013171600000000001 2024-09-21T22:07:02.861 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.861 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.861 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:02.861 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.247656+0000", 2024-09-21T22:07:02.861 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.3529999999999997e-05 2024-09-21T22:07:02.861 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.861 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.861 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:02.861 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.251225+0000", 2024-09-21T22:07:02.861 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00356865 2024-09-21T22:07:02.862 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.862 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.862 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:02.862 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.251370+0000", 2024-09-21T22:07:02.862 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00014518500000000001 2024-09-21T22:07:02.862 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:02.862 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:02.862 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:02.862 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.862 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.862 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:69 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'43, mlcod=14'39)", 2024-09-21T22:07:02.862 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.248839+0000", 2024-09-21T22:07:02.862 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 60.576164572000003, 2024-09-21T22:07:02.862 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0034954470000000001, 2024-09-21T22:07:02.862 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:02.863 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:02.863 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:02.863 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:02.863 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.863 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:02.863 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.248839+0000", 2024-09-21T22:07:02.863 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:02.863 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.863 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.863 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:02.863 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.248839+0000", 2024-09-21T22:07:02.863 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:02.863 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.863 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.864 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:02.864 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.248841+0000", 2024-09-21T22:07:02.864 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6759999999999999e-06 2024-09-21T22:07:02.864 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.864 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.864 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:02.864 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.249058+0000", 2024-09-21T22:07:02.864 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000217455 2024-09-21T22:07:02.864 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.864 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.864 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:02.864 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.249085+0000", 2024-09-21T22:07:02.864 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.7002999999999999e-05 2024-09-21T22:07:02.864 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.864 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.865 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:02.865 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.249099+0000", 2024-09-21T22:07:02.865 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.402e-05 2024-09-21T22:07:02.865 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.865 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.865 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:02.865 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.249241+0000", 2024-09-21T22:07:02.865 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00014194899999999999 2024-09-21T22:07:02.865 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.865 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.865 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:02.865 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.249298+0000", 2024-09-21T22:07:02.865 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.6687000000000001e-05 2024-09-21T22:07:02.865 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.865 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.866 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:02.866 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.252266+0000", 2024-09-21T22:07:02.866 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.002968348 2024-09-21T22:07:02.866 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.866 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.866 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:02.866 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.252334+0000", 2024-09-21T22:07:02.866 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.8308999999999999e-05 2024-09-21T22:07:02.866 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:02.866 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:02.866 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:02.866 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.866 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.866 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:71 1.0 e14/13 1:30306672:devicehealth::main.db.0000000000000000:head v 14'45, mlcod=14'43)", 2024-09-21T22:07:02.866 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.256068+0000", 2024-09-21T22:07:02.867 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 60.568935099999997, 2024-09-21T22:07:02.867 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0041712579999999997, 2024-09-21T22:07:02.867 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:02.867 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:02.867 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:02.867 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:02.867 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.867 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:02.867 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.256068+0000", 2024-09-21T22:07:02.867 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:02.867 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.867 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.867 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:02.867 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.256068+0000", 2024-09-21T22:07:02.867 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:02.868 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.868 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.868 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:02.868 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.256070+0000", 2024-09-21T22:07:02.868 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.61e-06 2024-09-21T22:07:02.868 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.868 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.868 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:02.868 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.256129+0000", 2024-09-21T22:07:02.868 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.8702999999999997e-05 2024-09-21T22:07:02.868 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.868 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.868 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:02.868 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.256158+0000", 2024-09-21T22:07:02.869 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9533e-05 2024-09-21T22:07:02.869 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.869 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.869 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:02.869 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.256173+0000", 2024-09-21T22:07:02.869 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4267000000000001e-05 2024-09-21T22:07:02.869 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.869 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.869 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:02.869 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.256304+0000", 2024-09-21T22:07:02.869 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000131352 2024-09-21T22:07:02.869 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.869 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.869 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:02.869 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.256357+0000", 2024-09-21T22:07:02.870 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.3492999999999998e-05 2024-09-21T22:07:02.870 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.870 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.870 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:02.870 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.260147+0000", 2024-09-21T22:07:02.870 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0037900410000000001 2024-09-21T22:07:02.870 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.870 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.870 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:02.870 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.260240+0000", 2024-09-21T22:07:02.870 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.2258999999999999e-05 2024-09-21T22:07:02.870 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:02.870 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:02.870 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:02.871 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.871 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.871 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:72 1.0 e14/13 1:30306672:devicehealth::main.db.0000000000000000:head v 14'46, mlcod=14'43)", 2024-09-21T22:07:02.871 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.258358+0000", 2024-09-21T22:07:02.871 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 60.566645106999999, 2024-09-21T22:07:02.871 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0041492919999999997, 2024-09-21T22:07:02.871 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:02.871 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:02.871 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:02.871 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:02.871 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.871 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:02.871 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.258358+0000", 2024-09-21T22:07:02.871 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:02.871 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.872 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.872 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:02.872 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.258358+0000", 2024-09-21T22:07:02.872 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:02.872 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.872 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.872 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:02.872 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.258360+0000", 2024-09-21T22:07:02.872 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.623e-06 2024-09-21T22:07:02.872 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.872 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.872 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:02.872 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.258415+0000", 2024-09-21T22:07:02.872 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.4756000000000002e-05 2024-09-21T22:07:02.872 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.873 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.873 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:02.873 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.258444+0000", 2024-09-21T22:07:02.873 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9164000000000001e-05 2024-09-21T22:07:02.873 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.873 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.873 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:02.873 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.258459+0000", 2024-09-21T22:07:02.873 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4996e-05 2024-09-21T22:07:02.873 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.873 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.873 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:02.873 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.258637+0000", 2024-09-21T22:07:02.873 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00017796500000000001 2024-09-21T22:07:02.874 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.874 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.874 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:02.874 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.258695+0000", 2024-09-21T22:07:02.874 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.8207e-05 2024-09-21T22:07:02.874 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.874 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.874 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:02.874 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.262421+0000", 2024-09-21T22:07:02.874 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0037261550000000001 2024-09-21T22:07:02.874 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.874 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.874 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:02.874 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.262508+0000", 2024-09-21T22:07:02.874 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.6426000000000002e-05 2024-09-21T22:07:02.875 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:02.875 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:02.875 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:02.875 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.875 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.875 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:87 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'57, mlcod=14'52)", 2024-09-21T22:07:02.875 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.290115+0000", 2024-09-21T22:07:02.875 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 60.534888662, 2024-09-21T22:07:02.875 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0030478200000000001, 2024-09-21T22:07:02.875 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:02.875 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:02.875 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:02.875 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:02.875 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.876 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:02.876 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.290115+0000", 2024-09-21T22:07:02.876 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:02.876 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.876 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.876 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:02.876 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.290115+0000", 2024-09-21T22:07:02.876 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:02.876 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.876 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.876 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:02.876 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.290116+0000", 2024-09-21T22:07:02.876 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3829999999999999e-06 2024-09-21T22:07:02.876 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.876 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.877 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:02.877 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.290191+0000", 2024-09-21T22:07:02.877 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.4562999999999997e-05 2024-09-21T22:07:02.877 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.877 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.877 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:02.877 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.290216+0000", 2024-09-21T22:07:02.877 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.565e-05 2024-09-21T22:07:02.877 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.877 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.877 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:02.877 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.290236+0000", 2024-09-21T22:07:02.877 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9326000000000001e-05 2024-09-21T22:07:02.877 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.877 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.878 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:02.878 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.290340+0000", 2024-09-21T22:07:02.878 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0001047 2024-09-21T22:07:02.878 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.878 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.878 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:02.878 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.290377+0000", 2024-09-21T22:07:02.878 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.6356000000000002e-05 2024-09-21T22:07:02.878 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.878 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.878 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:02.878 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.293096+0000", 2024-09-21T22:07:02.878 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.002718996 2024-09-21T22:07:02.878 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.879 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.879 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:02.879 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.293163+0000", 2024-09-21T22:07:02.879 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.6846000000000003e-05 2024-09-21T22:07:02.879 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:02.879 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:02.879 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:02.879 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.879 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.879 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:114 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'76, mlcod=14'73)", 2024-09-21T22:07:02.879 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.490030+0000", 2024-09-21T22:07:02.879 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 60.334973222999999, 2024-09-21T22:07:02.879 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.004227873, 2024-09-21T22:07:02.879 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:02.879 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:02.880 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:02.880 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:02.880 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.880 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:02.880 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.490030+0000", 2024-09-21T22:07:02.880 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:02.880 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.880 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.880 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:02.880 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.490030+0000", 2024-09-21T22:07:02.880 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:02.880 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.880 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.880 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:02.880 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.490031+0000", 2024-09-21T22:07:02.881 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.3300000000000001e-07 2024-09-21T22:07:02.881 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.881 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.881 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:02.881 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.490042+0000", 2024-09-21T22:07:02.881 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.1095e-05 2024-09-21T22:07:02.881 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.881 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.881 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:02.881 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.490053+0000", 2024-09-21T22:07:02.881 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0501e-05 2024-09-21T22:07:02.881 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.881 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.881 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:02.881 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.490060+0000", 2024-09-21T22:07:02.882 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.9829999999999999e-06 2024-09-21T22:07:02.882 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.882 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.882 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:02.882 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.490107+0000", 2024-09-21T22:07:02.882 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.7278999999999999e-05 2024-09-21T22:07:02.882 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.882 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.882 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:02.882 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.490124+0000", 2024-09-21T22:07:02.882 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7190999999999999e-05 2024-09-21T22:07:02.882 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.882 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.882 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:02.883 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.494229+0000", 2024-09-21T22:07:02.883 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0041050440000000004 2024-09-21T22:07:02.883 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.883 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.883 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:02.883 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.494258+0000", 2024-09-21T22:07:02.883 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9047000000000002e-05 2024-09-21T22:07:02.883 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:02.883 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:02.883 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:02.883 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.883 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.883 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:115 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'77, mlcod=14'73)", 2024-09-21T22:07:02.883 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.491380+0000", 2024-09-21T22:07:02.884 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 60.333623600999999, 2024-09-21T22:07:02.884 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0031550889999999998, 2024-09-21T22:07:02.884 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:02.884 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:02.884 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:02.884 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:02.884 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.884 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:02.884 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.491380+0000", 2024-09-21T22:07:02.884 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:02.884 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.884 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.884 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:02.884 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.491380+0000", 2024-09-21T22:07:02.884 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:02.885 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.885 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.885 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:02.885 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.491381+0000", 2024-09-21T22:07:02.885 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.1399999999999996e-07 2024-09-21T22:07:02.885 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.885 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.885 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:02.885 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.491435+0000", 2024-09-21T22:07:02.885 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.3797e-05 2024-09-21T22:07:02.885 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.885 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.885 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:02.885 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.491447+0000", 2024-09-21T22:07:02.885 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2642e-05 2024-09-21T22:07:02.886 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.886 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.886 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:02.886 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.491453+0000", 2024-09-21T22:07:02.886 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.4820000000000002e-06 2024-09-21T22:07:02.886 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.886 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.886 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:02.886 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.491502+0000", 2024-09-21T22:07:02.886 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.9038e-05 2024-09-21T22:07:02.886 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.886 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.886 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:02.886 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.491514+0000", 2024-09-21T22:07:02.887 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2483999999999999e-05 2024-09-21T22:07:02.887 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.887 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.887 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:02.887 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.494486+0000", 2024-09-21T22:07:02.887 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.002971338 2024-09-21T22:07:02.887 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.887 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.887 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:02.887 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.494535+0000", 2024-09-21T22:07:02.887 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.9493999999999997e-05 2024-09-21T22:07:02.887 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:02.887 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:02.887 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:02.888 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.888 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.888 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:118 1.0 e14/13 1:30306672:devicehealth::main.db.0000000000000000:head v 14'80, mlcod=14'78)", 2024-09-21T22:07:02.888 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.499045+0000", 2024-09-21T22:07:02.888 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 60.325958599000003, 2024-09-21T22:07:02.888 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0029889589999999998, 2024-09-21T22:07:02.888 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:02.888 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:02.888 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:02.888 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:02.888 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.888 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:02.888 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.499045+0000", 2024-09-21T22:07:02.888 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:02.888 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.889 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.889 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:02.889 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.499045+0000", 2024-09-21T22:07:02.889 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:02.889 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.889 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.889 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:02.889 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.499046+0000", 2024-09-21T22:07:02.889 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.0299999999999998e-07 2024-09-21T22:07:02.889 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.889 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.889 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:02.889 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.499110+0000", 2024-09-21T22:07:02.889 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.4688000000000001e-05 2024-09-21T22:07:02.890 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.890 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.890 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:02.890 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.499125+0000", 2024-09-21T22:07:02.890 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4387e-05 2024-09-21T22:07:02.890 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.890 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.890 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:02.890 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.499131+0000", 2024-09-21T22:07:02.890 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.2129999999999996e-06 2024-09-21T22:07:02.890 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.890 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.890 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:02.890 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.499255+0000", 2024-09-21T22:07:02.891 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00012424199999999999 2024-09-21T22:07:02.891 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.891 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.891 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:02.891 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.499312+0000", 2024-09-21T22:07:02.891 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.6709999999999997e-05 2024-09-21T22:07:02.891 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.891 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.891 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:02.891 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.502005+0000", 2024-09-21T22:07:02.891 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0026927169999999999 2024-09-21T22:07:02.891 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.891 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.891 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:02.891 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.502034+0000", 2024-09-21T22:07:02.892 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9198999999999999e-05 2024-09-21T22:07:02.892 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:02.892 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:02.892 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:02.892 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.892 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.892 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:136 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'94, mlcod=14'89)", 2024-09-21T22:07:02.892 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.903810+0000", 2024-09-21T22:07:02.892 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 59.921193627999997, 2024-09-21T22:07:02.892 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0062746829999999996, 2024-09-21T22:07:02.892 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:02.892 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:02.892 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:02.892 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:02.893 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.893 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:02.893 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.903810+0000", 2024-09-21T22:07:02.893 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:02.893 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.893 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.893 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:02.893 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.903810+0000", 2024-09-21T22:07:02.893 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:02.893 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.893 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.893 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:02.893 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.903810+0000", 2024-09-21T22:07:02.893 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.03e-07 2024-09-21T22:07:02.893 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.894 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.894 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:02.894 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.903814+0000", 2024-09-21T22:07:02.894 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.676e-06 2024-09-21T22:07:02.894 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.894 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.894 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:02.894 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.903822+0000", 2024-09-21T22:07:02.894 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.2879999999999993e-06 2024-09-21T22:07:02.894 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.894 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.894 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:02.894 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.903829+0000", 2024-09-21T22:07:02.894 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.2920000000000001e-06 2024-09-21T22:07:02.894 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.895 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.895 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:02.895 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.903879+0000", 2024-09-21T22:07:02.895 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.0253999999999997e-05 2024-09-21T22:07:02.895 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.895 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.895 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:02.895 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.903897+0000", 2024-09-21T22:07:02.895 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8468999999999999e-05 2024-09-21T22:07:02.895 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.895 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.895 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:02.895 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.910044+0000", 2024-09-21T22:07:02.895 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0061468950000000003 2024-09-21T22:07:02.896 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.896 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.896 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:02.896 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.910085+0000", 2024-09-21T22:07:02.896 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.0405999999999999e-05 2024-09-21T22:07:02.896 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:02.896 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:02.896 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:02.896 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.896 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.896 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:137 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'95, mlcod=14'90)", 2024-09-21T22:07:02.896 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.904439+0000", 2024-09-21T22:07:02.896 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 59.920564231999997, 2024-09-21T22:07:02.896 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0072625019999999997, 2024-09-21T22:07:02.896 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:02.897 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:02.897 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:02.897 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:02.897 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.897 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:02.897 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.904439+0000", 2024-09-21T22:07:02.897 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:02.897 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.897 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.897 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:02.897 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.904439+0000", 2024-09-21T22:07:02.897 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:02.897 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.897 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.897 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:02.898 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.904440+0000", 2024-09-21T22:07:02.898 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8400000000000005e-07 2024-09-21T22:07:02.898 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.898 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.898 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:02.898 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.904498+0000", 2024-09-21T22:07:02.898 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.8320999999999999e-05 2024-09-21T22:07:02.898 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.898 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.898 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:02.898 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.904507+0000", 2024-09-21T22:07:02.898 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.7790000000000001e-06 2024-09-21T22:07:02.898 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.898 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.899 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:02.899 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.904511+0000", 2024-09-21T22:07:02.899 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.9700000000000001e-06 2024-09-21T22:07:02.899 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.899 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.899 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:02.899 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.904551+0000", 2024-09-21T22:07:02.899 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.9910000000000002e-05 2024-09-21T22:07:02.899 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.899 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.899 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:02.899 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.904567+0000", 2024-09-21T22:07:02.899 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5903999999999999e-05 2024-09-21T22:07:02.899 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.899 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.900 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:02.900 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.911666+0000", 2024-09-21T22:07:02.900 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0070996929999999998 2024-09-21T22:07:02.900 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.900 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.900 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:02.900 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.911702+0000", 2024-09-21T22:07:02.900 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.5441000000000002e-05 2024-09-21T22:07:02.900 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:02.900 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:02.900 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:02.900 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.900 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.900 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:138 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'96, mlcod=14'90)", 2024-09-21T22:07:02.900 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.905402+0000", 2024-09-21T22:07:02.901 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 59.919601255000003, 2024-09-21T22:07:02.901 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0063366890000000004, 2024-09-21T22:07:02.901 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:02.901 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:02.901 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:02.901 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:02.901 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.901 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:02.901 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905402+0000", 2024-09-21T22:07:02.901 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:02.901 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.901 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.901 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:02.901 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905402+0000", 2024-09-21T22:07:02.901 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:02.902 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.902 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.902 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:02.902 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905403+0000", 2024-09-21T22:07:02.902 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.9200000000000002e-07 2024-09-21T22:07:02.902 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.902 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.902 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:02.902 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905406+0000", 2024-09-21T22:07:02.902 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.326e-06 2024-09-21T22:07:02.902 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.902 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.902 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:02.902 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905413+0000", 2024-09-21T22:07:02.903 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.3320000000000003e-06 2024-09-21T22:07:02.903 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.903 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.903 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:02.903 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905417+0000", 2024-09-21T22:07:02.903 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.7139999999999999e-06 2024-09-21T22:07:02.903 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.903 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.903 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:02.903 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.910115+0000", 2024-09-21T22:07:02.903 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0046983939999999998 2024-09-21T22:07:02.903 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.903 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.903 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:02.904 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.910140+0000", 2024-09-21T22:07:02.904 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4689999999999999e-05 2024-09-21T22:07:02.904 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.904 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.904 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:02.904 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.911706+0000", 2024-09-21T22:07:02.904 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0015658479999999999 2024-09-21T22:07:02.904 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.904 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.904 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:02.904 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.911739+0000", 2024-09-21T22:07:02.904 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.2993000000000001e-05 2024-09-21T22:07:02.904 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:02.904 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:02.904 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:02.905 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.905 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.905 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:139 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'97, mlcod=14'90)", 2024-09-21T22:07:02.905 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.905983+0000", 2024-09-21T22:07:02.905 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 59.919020336999999, 2024-09-21T22:07:02.905 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0057906030000000001, 2024-09-21T22:07:02.905 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:02.905 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:02.905 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:02.905 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:02.905 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.905 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:02.905 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905983+0000", 2024-09-21T22:07:02.906 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:02.906 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.906 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.906 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:02.906 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905983+0000", 2024-09-21T22:07:02.906 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:02.906 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.906 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.906 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:02.906 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905984+0000", 2024-09-21T22:07:02.906 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.15e-07 2024-09-21T22:07:02.906 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.906 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.906 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:02.906 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905986+0000", 2024-09-21T22:07:02.907 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.852e-06 2024-09-21T22:07:02.907 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.907 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.907 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:02.907 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905993+0000", 2024-09-21T22:07:02.907 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.652e-06 2024-09-21T22:07:02.907 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.907 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.907 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:02.907 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905996+0000", 2024-09-21T22:07:02.907 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.3519999999999998e-06 2024-09-21T22:07:02.907 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.907 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.907 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:02.908 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.910519+0000", 2024-09-21T22:07:02.908 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0045230820000000003 2024-09-21T22:07:02.908 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.908 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.908 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:02.908 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.910548+0000", 2024-09-21T22:07:02.908 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8399999999999999e-05 2024-09-21T22:07:02.908 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.908 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.908 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:02.908 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.911742+0000", 2024-09-21T22:07:02.908 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011939909999999999 2024-09-21T22:07:02.908 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.908 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.909 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:02.909 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.911774+0000", 2024-09-21T22:07:02.909 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.1958999999999997e-05 2024-09-21T22:07:02.909 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:02.909 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:02.909 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:02.909 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.909 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.909 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:140 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'98, mlcod=14'90)", 2024-09-21T22:07:02.909 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.906441+0000", 2024-09-21T22:07:02.909 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 59.918562207999997, 2024-09-21T22:07:02.909 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0057052989999999996, 2024-09-21T22:07:02.909 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:02.909 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:02.909 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:02.910 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:02.910 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.910 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:02.910 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.906441+0000", 2024-09-21T22:07:02.910 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:02.910 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.910 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.910 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:02.910 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.906441+0000", 2024-09-21T22:07:02.910 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:02.910 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.910 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.910 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:02.910 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.906442+0000", 2024-09-21T22:07:02.911 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.7e-07 2024-09-21T22:07:02.911 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.911 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.911 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:02.911 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.906511+0000", 2024-09-21T22:07:02.911 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.8857999999999997e-05 2024-09-21T22:07:02.911 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.911 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.911 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:02.911 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.906518+0000", 2024-09-21T22:07:02.911 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.1690000000000003e-06 2024-09-21T22:07:02.911 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.911 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.911 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:02.911 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.906521+0000", 2024-09-21T22:07:02.911 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.4829999999999999e-06 2024-09-21T22:07:02.912 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.912 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.912 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:02.912 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.910939+0000", 2024-09-21T22:07:02.912 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0044176809999999997 2024-09-21T22:07:02.912 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.912 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.912 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:02.912 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.910959+0000", 2024-09-21T22:07:02.912 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9898000000000002e-05 2024-09-21T22:07:02.912 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.912 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.912 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:02.912 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.912113+0000", 2024-09-21T22:07:02.912 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001153811 2024-09-21T22:07:02.913 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:02.913 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:02.913 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:02.913 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.912147+0000", 2024-09-21T22:07:02.913 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.4029000000000002e-05 2024-09-21T22:07:02.913 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:02.913 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:02.913 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:02.913 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:02.913 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:02.913 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:07:02.915 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T22:07:02.915 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:02.907+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:03.015 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:03.011+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:03.043 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T22:07:03.043 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:07:03.043 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 47, 2024-09-21T22:07:03.043 INFO:teuthology.orchestra.run.smithi039.stdout: "fsid": "00bb09a3-e410-4e18-a2b2-08eaf61978e6", 2024-09-21T22:07:03.043 INFO:teuthology.orchestra.run.smithi039.stdout: "created": "2024-09-21T22:05:45.303341+0000", 2024-09-21T22:07:03.043 INFO:teuthology.orchestra.run.smithi039.stdout: "modified": "2024-09-21T22:07:01.994280+0000", 2024-09-21T22:07:03.043 INFO:teuthology.orchestra.run.smithi039.stdout: "last_up_change": "2024-09-21T22:06:49.928873+0000", 2024-09-21T22:07:03.043 INFO:teuthology.orchestra.run.smithi039.stdout: "last_in_change": "2024-09-21T22:06:43.837891+0000", 2024-09-21T22:07:03.043 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-21T22:07:03.043 INFO:teuthology.orchestra.run.smithi039.stdout: "flags_num": 5801984, 2024-09-21T22:07:03.044 INFO:teuthology.orchestra.run.smithi039.stdout: "flags_set": [ 2024-09-21T22:07:03.044 INFO:teuthology.orchestra.run.smithi039.stdout: "noscrub", 2024-09-21T22:07:03.044 INFO:teuthology.orchestra.run.smithi039.stdout: "pglog_hardlimit", 2024-09-21T22:07:03.044 INFO:teuthology.orchestra.run.smithi039.stdout: "purged_snapdirs", 2024-09-21T22:07:03.044 INFO:teuthology.orchestra.run.smithi039.stdout: "recovery_deletes", 2024-09-21T22:07:03.044 INFO:teuthology.orchestra.run.smithi039.stdout: "sortbitwise" 2024-09-21T22:07:03.044 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T22:07:03.044 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_version": 4, 2024-09-21T22:07:03.044 INFO:teuthology.orchestra.run.smithi039.stdout: "full_ratio": 0.94999998807907104, 2024-09-21T22:07:03.044 INFO:teuthology.orchestra.run.smithi039.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-21T22:07:03.044 INFO:teuthology.orchestra.run.smithi039.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-21T22:07:03.044 INFO:teuthology.orchestra.run.smithi039.stdout: "cluster_snapshot": "", 2024-09-21T22:07:03.044 INFO:teuthology.orchestra.run.smithi039.stdout: "pool_max": 2, 2024-09-21T22:07:03.044 INFO:teuthology.orchestra.run.smithi039.stdout: "max_osd": 6, 2024-09-21T22:07:03.045 INFO:teuthology.orchestra.run.smithi039.stdout: "require_min_compat_client": "luminous", 2024-09-21T22:07:03.045 INFO:teuthology.orchestra.run.smithi039.stdout: "min_compat_client": "jewel", 2024-09-21T22:07:03.045 INFO:teuthology.orchestra.run.smithi039.stdout: "require_osd_release": "squid", 2024-09-21T22:07:03.045 INFO:teuthology.orchestra.run.smithi039.stdout: "allow_crimson": false, 2024-09-21T22:07:03.045 INFO:teuthology.orchestra.run.smithi039.stdout: "pools": [ 2024-09-21T22:07:03.045 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:03.045 INFO:teuthology.orchestra.run.smithi039.stdout: "pool": 1, 2024-09-21T22:07:03.045 INFO:teuthology.orchestra.run.smithi039.stdout: "pool_name": ".mgr", 2024-09-21T22:07:03.045 INFO:teuthology.orchestra.run.smithi039.stdout: "create_time": "2024-09-21T22:06:00.868591+0000", 2024-09-21T22:07:03.045 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T22:07:03.045 INFO:teuthology.orchestra.run.smithi039.stdout: "flags_names": "hashpspool", 2024-09-21T22:07:03.045 INFO:teuthology.orchestra.run.smithi039.stdout: "type": 1, 2024-09-21T22:07:03.045 INFO:teuthology.orchestra.run.smithi039.stdout: "size": 2, 2024-09-21T22:07:03.045 INFO:teuthology.orchestra.run.smithi039.stdout: "min_size": 1, 2024-09-21T22:07:03.045 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_rule": 0, 2024-09-21T22:07:03.046 INFO:teuthology.orchestra.run.smithi039.stdout: "peering_crush_bucket_count": 0, 2024-09-21T22:07:03.046 INFO:teuthology.orchestra.run.smithi039.stdout: "peering_crush_bucket_target": 0, 2024-09-21T22:07:03.046 INFO:teuthology.orchestra.run.smithi039.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T22:07:03.046 INFO:teuthology.orchestra.run.smithi039.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T22:07:03.046 INFO:teuthology.orchestra.run.smithi039.stdout: "is_stretch_pool": false, 2024-09-21T22:07:03.046 INFO:teuthology.orchestra.run.smithi039.stdout: "object_hash": 2, 2024-09-21T22:07:03.046 INFO:teuthology.orchestra.run.smithi039.stdout: "pg_autoscale_mode": "off", 2024-09-21T22:07:03.046 INFO:teuthology.orchestra.run.smithi039.stdout: "pg_num": 1, 2024-09-21T22:07:03.046 INFO:teuthology.orchestra.run.smithi039.stdout: "pg_placement_num": 1, 2024-09-21T22:07:03.046 INFO:teuthology.orchestra.run.smithi039.stdout: "pg_placement_num_target": 1, 2024-09-21T22:07:03.046 INFO:teuthology.orchestra.run.smithi039.stdout: "pg_num_target": 1, 2024-09-21T22:07:03.046 INFO:teuthology.orchestra.run.smithi039.stdout: "pg_num_pending": 1, 2024-09-21T22:07:03.046 INFO:teuthology.orchestra.run.smithi039.stdout: "last_pg_merge_meta": { 2024-09-21T22:07:03.047 INFO:teuthology.orchestra.run.smithi039.stdout: "source_pgid": "0.0", 2024-09-21T22:07:03.047 INFO:teuthology.orchestra.run.smithi039.stdout: "ready_epoch": 0, 2024-09-21T22:07:03.047 INFO:teuthology.orchestra.run.smithi039.stdout: "last_epoch_started": 0, 2024-09-21T22:07:03.047 INFO:teuthology.orchestra.run.smithi039.stdout: "last_epoch_clean": 0, 2024-09-21T22:07:03.047 INFO:teuthology.orchestra.run.smithi039.stdout: "source_version": "0'0", 2024-09-21T22:07:03.047 INFO:teuthology.orchestra.run.smithi039.stdout: "target_version": "0'0" 2024-09-21T22:07:03.047 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:03.047 INFO:teuthology.orchestra.run.smithi039.stdout: "last_change": "15", 2024-09-21T22:07:03.047 INFO:teuthology.orchestra.run.smithi039.stdout: "last_force_op_resend": "0", 2024-09-21T22:07:03.047 INFO:teuthology.orchestra.run.smithi039.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T22:07:03.047 INFO:teuthology.orchestra.run.smithi039.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T22:07:03.047 INFO:teuthology.orchestra.run.smithi039.stdout: "auid": 0, 2024-09-21T22:07:03.047 INFO:teuthology.orchestra.run.smithi039.stdout: "snap_mode": "selfmanaged", 2024-09-21T22:07:03.047 INFO:teuthology.orchestra.run.smithi039.stdout: "snap_seq": 0, 2024-09-21T22:07:03.048 INFO:teuthology.orchestra.run.smithi039.stdout: "snap_epoch": 0, 2024-09-21T22:07:03.048 INFO:teuthology.orchestra.run.smithi039.stdout: "pool_snaps": [], 2024-09-21T22:07:03.048 INFO:teuthology.orchestra.run.smithi039.stdout: "removed_snaps": "[]", 2024-09-21T22:07:03.048 INFO:teuthology.orchestra.run.smithi039.stdout: "quota_max_bytes": 0, 2024-09-21T22:07:03.048 INFO:teuthology.orchestra.run.smithi039.stdout: "quota_max_objects": 0, 2024-09-21T22:07:03.048 INFO:teuthology.orchestra.run.smithi039.stdout: "tiers": [], 2024-09-21T22:07:03.048 INFO:teuthology.orchestra.run.smithi039.stdout: "tier_of": -1, 2024-09-21T22:07:03.048 INFO:teuthology.orchestra.run.smithi039.stdout: "read_tier": -1, 2024-09-21T22:07:03.048 INFO:teuthology.orchestra.run.smithi039.stdout: "write_tier": -1, 2024-09-21T22:07:03.048 INFO:teuthology.orchestra.run.smithi039.stdout: "cache_mode": "none", 2024-09-21T22:07:03.048 INFO:teuthology.orchestra.run.smithi039.stdout: "target_max_bytes": 0, 2024-09-21T22:07:03.048 INFO:teuthology.orchestra.run.smithi039.stdout: "target_max_objects": 0, 2024-09-21T22:07:03.048 INFO:teuthology.orchestra.run.smithi039.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T22:07:03.048 INFO:teuthology.orchestra.run.smithi039.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T22:07:03.049 INFO:teuthology.orchestra.run.smithi039.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T22:07:03.049 INFO:teuthology.orchestra.run.smithi039.stdout: "cache_min_flush_age": 0, 2024-09-21T22:07:03.049 INFO:teuthology.orchestra.run.smithi039.stdout: "cache_min_evict_age": 0, 2024-09-21T22:07:03.049 INFO:teuthology.orchestra.run.smithi039.stdout: "erasure_code_profile": "", 2024-09-21T22:07:03.049 INFO:teuthology.orchestra.run.smithi039.stdout: "hit_set_params": { 2024-09-21T22:07:03.049 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "none" 2024-09-21T22:07:03.049 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:03.049 INFO:teuthology.orchestra.run.smithi039.stdout: "hit_set_period": 0, 2024-09-21T22:07:03.049 INFO:teuthology.orchestra.run.smithi039.stdout: "hit_set_count": 0, 2024-09-21T22:07:03.049 INFO:teuthology.orchestra.run.smithi039.stdout: "use_gmt_hitset": true, 2024-09-21T22:07:03.049 INFO:teuthology.orchestra.run.smithi039.stdout: "min_read_recency_for_promote": 0, 2024-09-21T22:07:03.049 INFO:teuthology.orchestra.run.smithi039.stdout: "min_write_recency_for_promote": 0, 2024-09-21T22:07:03.049 INFO:teuthology.orchestra.run.smithi039.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T22:07:03.049 INFO:teuthology.orchestra.run.smithi039.stdout: "hit_set_search_last_n": 0, 2024-09-21T22:07:03.049 INFO:teuthology.orchestra.run.smithi039.stdout: "grade_table": [], 2024-09-21T22:07:03.050 INFO:teuthology.orchestra.run.smithi039.stdout: "stripe_width": 0, 2024-09-21T22:07:03.050 INFO:teuthology.orchestra.run.smithi039.stdout: "expected_num_objects": 0, 2024-09-21T22:07:03.050 INFO:teuthology.orchestra.run.smithi039.stdout: "fast_read": false, 2024-09-21T22:07:03.050 INFO:teuthology.orchestra.run.smithi039.stdout: "options": { 2024-09-21T22:07:03.050 INFO:teuthology.orchestra.run.smithi039.stdout: "pg_num_max": 32, 2024-09-21T22:07:03.050 INFO:teuthology.orchestra.run.smithi039.stdout: "pg_num_min": 1 2024-09-21T22:07:03.050 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:03.050 INFO:teuthology.orchestra.run.smithi039.stdout: "application_metadata": { 2024-09-21T22:07:03.050 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": {} 2024-09-21T22:07:03.050 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:03.050 INFO:teuthology.orchestra.run.smithi039.stdout: "read_balance": { 2024-09-21T22:07:03.050 INFO:teuthology.orchestra.run.smithi039.stdout: "score_type": "Fair distribution", 2024-09-21T22:07:03.050 INFO:teuthology.orchestra.run.smithi039.stdout: "score_acting": 6.059999942779541, 2024-09-21T22:07:03.051 INFO:teuthology.orchestra.run.smithi039.stdout: "score_stable": 6.059999942779541, 2024-09-21T22:07:03.051 INFO:teuthology.orchestra.run.smithi039.stdout: "optimal_score": 0.33000001311302185, 2024-09-21T22:07:03.051 INFO:teuthology.orchestra.run.smithi039.stdout: "raw_score_acting": 2, 2024-09-21T22:07:03.051 INFO:teuthology.orchestra.run.smithi039.stdout: "raw_score_stable": 2, 2024-09-21T22:07:03.051 INFO:teuthology.orchestra.run.smithi039.stdout: "primary_affinity_weighted": 1, 2024-09-21T22:07:03.051 INFO:teuthology.orchestra.run.smithi039.stdout: "average_primary_affinity": 1, 2024-09-21T22:07:03.051 INFO:teuthology.orchestra.run.smithi039.stdout: "average_primary_affinity_weighted": 1 2024-09-21T22:07:03.051 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:03.051 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:03.051 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:03.051 INFO:teuthology.orchestra.run.smithi039.stdout: "pool": 2, 2024-09-21T22:07:03.051 INFO:teuthology.orchestra.run.smithi039.stdout: "pool_name": "rbd", 2024-09-21T22:07:03.051 INFO:teuthology.orchestra.run.smithi039.stdout: "create_time": "2024-09-21T22:06:03.535857+0000", 2024-09-21T22:07:03.051 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 8193, 2024-09-21T22:07:03.052 INFO:teuthology.orchestra.run.smithi039.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-21T22:07:03.052 INFO:teuthology.orchestra.run.smithi039.stdout: "type": 1, 2024-09-21T22:07:03.052 INFO:teuthology.orchestra.run.smithi039.stdout: "size": 2, 2024-09-21T22:07:03.052 INFO:teuthology.orchestra.run.smithi039.stdout: "min_size": 1, 2024-09-21T22:07:03.052 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_rule": 0, 2024-09-21T22:07:03.052 INFO:teuthology.orchestra.run.smithi039.stdout: "peering_crush_bucket_count": 0, 2024-09-21T22:07:03.052 INFO:teuthology.orchestra.run.smithi039.stdout: "peering_crush_bucket_target": 0, 2024-09-21T22:07:03.052 INFO:teuthology.orchestra.run.smithi039.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T22:07:03.052 INFO:teuthology.orchestra.run.smithi039.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T22:07:03.052 INFO:teuthology.orchestra.run.smithi039.stdout: "is_stretch_pool": false, 2024-09-21T22:07:03.052 INFO:teuthology.orchestra.run.smithi039.stdout: "object_hash": 2, 2024-09-21T22:07:03.052 INFO:teuthology.orchestra.run.smithi039.stdout: "pg_autoscale_mode": "off", 2024-09-21T22:07:03.052 INFO:teuthology.orchestra.run.smithi039.stdout: "pg_num": 8, 2024-09-21T22:07:03.052 INFO:teuthology.orchestra.run.smithi039.stdout: "pg_placement_num": 8, 2024-09-21T22:07:03.052 INFO:teuthology.orchestra.run.smithi039.stdout: "pg_placement_num_target": 8, 2024-09-21T22:07:03.053 INFO:teuthology.orchestra.run.smithi039.stdout: "pg_num_target": 8, 2024-09-21T22:07:03.053 INFO:teuthology.orchestra.run.smithi039.stdout: "pg_num_pending": 8, 2024-09-21T22:07:03.053 INFO:teuthology.orchestra.run.smithi039.stdout: "last_pg_merge_meta": { 2024-09-21T22:07:03.053 INFO:teuthology.orchestra.run.smithi039.stdout: "source_pgid": "0.0", 2024-09-21T22:07:03.053 INFO:teuthology.orchestra.run.smithi039.stdout: "ready_epoch": 0, 2024-09-21T22:07:03.053 INFO:teuthology.orchestra.run.smithi039.stdout: "last_epoch_started": 0, 2024-09-21T22:07:03.053 INFO:teuthology.orchestra.run.smithi039.stdout: "last_epoch_clean": 0, 2024-09-21T22:07:03.053 INFO:teuthology.orchestra.run.smithi039.stdout: "source_version": "0'0", 2024-09-21T22:07:03.053 INFO:teuthology.orchestra.run.smithi039.stdout: "target_version": "0'0" 2024-09-21T22:07:03.053 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:03.053 INFO:teuthology.orchestra.run.smithi039.stdout: "last_change": "19", 2024-09-21T22:07:03.053 INFO:teuthology.orchestra.run.smithi039.stdout: "last_force_op_resend": "0", 2024-09-21T22:07:03.053 INFO:teuthology.orchestra.run.smithi039.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T22:07:03.053 INFO:teuthology.orchestra.run.smithi039.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T22:07:03.054 INFO:teuthology.orchestra.run.smithi039.stdout: "auid": 0, 2024-09-21T22:07:03.054 INFO:teuthology.orchestra.run.smithi039.stdout: "snap_mode": "selfmanaged", 2024-09-21T22:07:03.054 INFO:teuthology.orchestra.run.smithi039.stdout: "snap_seq": 2, 2024-09-21T22:07:03.054 INFO:teuthology.orchestra.run.smithi039.stdout: "snap_epoch": 19, 2024-09-21T22:07:03.054 INFO:teuthology.orchestra.run.smithi039.stdout: "pool_snaps": [], 2024-09-21T22:07:03.054 INFO:teuthology.orchestra.run.smithi039.stdout: "removed_snaps": "[]", 2024-09-21T22:07:03.054 INFO:teuthology.orchestra.run.smithi039.stdout: "quota_max_bytes": 0, 2024-09-21T22:07:03.054 INFO:teuthology.orchestra.run.smithi039.stdout: "quota_max_objects": 0, 2024-09-21T22:07:03.054 INFO:teuthology.orchestra.run.smithi039.stdout: "tiers": [], 2024-09-21T22:07:03.054 INFO:teuthology.orchestra.run.smithi039.stdout: "tier_of": -1, 2024-09-21T22:07:03.054 INFO:teuthology.orchestra.run.smithi039.stdout: "read_tier": -1, 2024-09-21T22:07:03.054 INFO:teuthology.orchestra.run.smithi039.stdout: "write_tier": -1, 2024-09-21T22:07:03.054 INFO:teuthology.orchestra.run.smithi039.stdout: "cache_mode": "none", 2024-09-21T22:07:03.054 INFO:teuthology.orchestra.run.smithi039.stdout: "target_max_bytes": 0, 2024-09-21T22:07:03.055 INFO:teuthology.orchestra.run.smithi039.stdout: "target_max_objects": 0, 2024-09-21T22:07:03.055 INFO:teuthology.orchestra.run.smithi039.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T22:07:03.055 INFO:teuthology.orchestra.run.smithi039.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T22:07:03.055 INFO:teuthology.orchestra.run.smithi039.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T22:07:03.055 INFO:teuthology.orchestra.run.smithi039.stdout: "cache_min_flush_age": 0, 2024-09-21T22:07:03.055 INFO:teuthology.orchestra.run.smithi039.stdout: "cache_min_evict_age": 0, 2024-09-21T22:07:03.055 INFO:teuthology.orchestra.run.smithi039.stdout: "erasure_code_profile": "", 2024-09-21T22:07:03.055 INFO:teuthology.orchestra.run.smithi039.stdout: "hit_set_params": { 2024-09-21T22:07:03.055 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "none" 2024-09-21T22:07:03.055 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:03.055 INFO:teuthology.orchestra.run.smithi039.stdout: "hit_set_period": 0, 2024-09-21T22:07:03.055 INFO:teuthology.orchestra.run.smithi039.stdout: "hit_set_count": 0, 2024-09-21T22:07:03.055 INFO:teuthology.orchestra.run.smithi039.stdout: "use_gmt_hitset": true, 2024-09-21T22:07:03.056 INFO:teuthology.orchestra.run.smithi039.stdout: "min_read_recency_for_promote": 0, 2024-09-21T22:07:03.056 INFO:teuthology.orchestra.run.smithi039.stdout: "min_write_recency_for_promote": 0, 2024-09-21T22:07:03.056 INFO:teuthology.orchestra.run.smithi039.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T22:07:03.056 INFO:teuthology.orchestra.run.smithi039.stdout: "hit_set_search_last_n": 0, 2024-09-21T22:07:03.056 INFO:teuthology.orchestra.run.smithi039.stdout: "grade_table": [], 2024-09-21T22:07:03.056 INFO:teuthology.orchestra.run.smithi039.stdout: "stripe_width": 0, 2024-09-21T22:07:03.056 INFO:teuthology.orchestra.run.smithi039.stdout: "expected_num_objects": 0, 2024-09-21T22:07:03.056 INFO:teuthology.orchestra.run.smithi039.stdout: "fast_read": false, 2024-09-21T22:07:03.056 INFO:teuthology.orchestra.run.smithi039.stdout: "options": {}, 2024-09-21T22:07:03.056 INFO:teuthology.orchestra.run.smithi039.stdout: "application_metadata": { 2024-09-21T22:07:03.056 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd": {} 2024-09-21T22:07:03.056 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:03.056 INFO:teuthology.orchestra.run.smithi039.stdout: "read_balance": { 2024-09-21T22:07:03.056 INFO:teuthology.orchestra.run.smithi039.stdout: "score_type": "Fair distribution", 2024-09-21T22:07:03.057 INFO:teuthology.orchestra.run.smithi039.stdout: "score_acting": 2.2699999809265137, 2024-09-21T22:07:03.057 INFO:teuthology.orchestra.run.smithi039.stdout: "score_stable": 2.2699999809265137, 2024-09-21T22:07:03.057 INFO:teuthology.orchestra.run.smithi039.stdout: "optimal_score": 0.82999998331069946, 2024-09-21T22:07:03.057 INFO:teuthology.orchestra.run.smithi039.stdout: "raw_score_acting": 1.8799999952316284, 2024-09-21T22:07:03.057 INFO:teuthology.orchestra.run.smithi039.stdout: "raw_score_stable": 1.8799999952316284, 2024-09-21T22:07:03.057 INFO:teuthology.orchestra.run.smithi039.stdout: "primary_affinity_weighted": 1, 2024-09-21T22:07:03.057 INFO:teuthology.orchestra.run.smithi039.stdout: "average_primary_affinity": 1, 2024-09-21T22:07:03.057 INFO:teuthology.orchestra.run.smithi039.stdout: "average_primary_affinity_weighted": 1 2024-09-21T22:07:03.057 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:03.057 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:03.057 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T22:07:03.057 INFO:teuthology.orchestra.run.smithi039.stdout: "osds": [ 2024-09-21T22:07:03.057 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:03.057 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": 0, 2024-09-21T22:07:03.058 INFO:teuthology.orchestra.run.smithi039.stdout: "uuid": "8336c397-2018-4c75-ab31-2e5f6ad564b5", 2024-09-21T22:07:03.058 INFO:teuthology.orchestra.run.smithi039.stdout: "up": 1, 2024-09-21T22:07:03.058 INFO:teuthology.orchestra.run.smithi039.stdout: "in": 1, 2024-09-21T22:07:03.058 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 1, 2024-09-21T22:07:03.058 INFO:teuthology.orchestra.run.smithi039.stdout: "primary_affinity": 1, 2024-09-21T22:07:03.058 INFO:teuthology.orchestra.run.smithi039.stdout: "last_clean_begin": 0, 2024-09-21T22:07:03.058 INFO:teuthology.orchestra.run.smithi039.stdout: "last_clean_end": 0, 2024-09-21T22:07:03.058 INFO:teuthology.orchestra.run.smithi039.stdout: "up_from": 11, 2024-09-21T22:07:03.058 INFO:teuthology.orchestra.run.smithi039.stdout: "up_thru": 0, 2024-09-21T22:07:03.058 INFO:teuthology.orchestra.run.smithi039.stdout: "down_at": 0, 2024-09-21T22:07:03.058 INFO:teuthology.orchestra.run.smithi039.stdout: "lost_at": 0, 2024-09-21T22:07:03.058 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2024-09-21T22:07:03.059 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2024-09-21T22:07:03.059 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:03.059 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2024-09-21T22:07:03.059 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6800", 2024-09-21T22:07:03.059 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 3054061468 2024-09-21T22:07:03.059 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:03.059 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:03.059 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-09-21T22:07:03.059 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6801", 2024-09-21T22:07:03.059 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 3054061468 2024-09-21T22:07:03.059 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:03.059 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:03.059 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:03.059 INFO:teuthology.orchestra.run.smithi039.stdout: "cluster_addrs": { 2024-09-21T22:07:03.060 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2024-09-21T22:07:03.060 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:03.060 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2024-09-21T22:07:03.060 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6802", 2024-09-21T22:07:03.060 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 3054061468 2024-09-21T22:07:03.060 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:03.060 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:03.060 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-09-21T22:07:03.060 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6803", 2024-09-21T22:07:03.060 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 3054061468 2024-09-21T22:07:03.060 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:03.060 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:03.060 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:03.060 INFO:teuthology.orchestra.run.smithi039.stdout: "heartbeat_back_addrs": { 2024-09-21T22:07:03.061 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2024-09-21T22:07:03.061 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:03.061 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2024-09-21T22:07:03.061 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6808", 2024-09-21T22:07:03.061 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 3054061468 2024-09-21T22:07:03.061 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:03.061 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:03.061 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-09-21T22:07:03.061 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6810", 2024-09-21T22:07:03.061 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 3054061468 2024-09-21T22:07:03.061 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:03.061 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:03.061 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:03.062 INFO:teuthology.orchestra.run.smithi039.stdout: "heartbeat_front_addrs": { 2024-09-21T22:07:03.062 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2024-09-21T22:07:03.062 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:03.062 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2024-09-21T22:07:03.062 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6804", 2024-09-21T22:07:03.062 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 3054061468 2024-09-21T22:07:03.062 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:03.062 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:03.062 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-09-21T22:07:03.062 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6806", 2024-09-21T22:07:03.062 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 3054061468 2024-09-21T22:07:03.062 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:03.062 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:03.062 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:03.063 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6801/3054061468", 2024-09-21T22:07:03.063 INFO:teuthology.orchestra.run.smithi039.stdout: "cluster_addr": "172.21.15.39:6803/3054061468", 2024-09-21T22:07:03.063 INFO:teuthology.orchestra.run.smithi039.stdout: "heartbeat_back_addr": "172.21.15.39:6810/3054061468", 2024-09-21T22:07:03.063 INFO:teuthology.orchestra.run.smithi039.stdout: "heartbeat_front_addr": "172.21.15.39:6806/3054061468", 2024-09-21T22:07:03.063 INFO:teuthology.orchestra.run.smithi039.stdout: "state": [ 2024-09-21T22:07:03.063 INFO:teuthology.orchestra.run.smithi039.stdout: "exists", 2024-09-21T22:07:03.063 INFO:teuthology.orchestra.run.smithi039.stdout: "up" 2024-09-21T22:07:03.063 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:03.063 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:03.063 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:03.063 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": 1, 2024-09-21T22:07:03.063 INFO:teuthology.orchestra.run.smithi039.stdout: "uuid": "bca64af6-7427-4c5d-97a4-a183b1f3a7d9", 2024-09-21T22:07:03.063 INFO:teuthology.orchestra.run.smithi039.stdout: "up": 1, 2024-09-21T22:07:03.063 INFO:teuthology.orchestra.run.smithi039.stdout: "in": 1, 2024-09-21T22:07:03.064 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 1, 2024-09-21T22:07:03.064 INFO:teuthology.orchestra.run.smithi039.stdout: "primary_affinity": 1, 2024-09-21T22:07:03.064 INFO:teuthology.orchestra.run.smithi039.stdout: "last_clean_begin": 0, 2024-09-21T22:07:03.064 INFO:teuthology.orchestra.run.smithi039.stdout: "last_clean_end": 0, 2024-09-21T22:07:03.064 INFO:teuthology.orchestra.run.smithi039.stdout: "up_from": 11, 2024-09-21T22:07:03.064 INFO:teuthology.orchestra.run.smithi039.stdout: "up_thru": 42, 2024-09-21T22:07:03.064 INFO:teuthology.orchestra.run.smithi039.stdout: "down_at": 0, 2024-09-21T22:07:03.064 INFO:teuthology.orchestra.run.smithi039.stdout: "lost_at": 0, 2024-09-21T22:07:03.064 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2024-09-21T22:07:03.064 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2024-09-21T22:07:03.064 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:03.064 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2024-09-21T22:07:03.064 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6816", 2024-09-21T22:07:03.064 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 2876495139 2024-09-21T22:07:03.065 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:03.065 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:03.065 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-09-21T22:07:03.065 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6817", 2024-09-21T22:07:03.065 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 2876495139 2024-09-21T22:07:03.065 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:03.065 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:03.065 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:03.065 INFO:teuthology.orchestra.run.smithi039.stdout: "cluster_addrs": { 2024-09-21T22:07:03.065 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2024-09-21T22:07:03.065 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:03.065 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2024-09-21T22:07:03.065 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6818", 2024-09-21T22:07:03.066 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 2876495139 2024-09-21T22:07:03.066 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:03.066 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:03.066 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-09-21T22:07:03.066 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6819", 2024-09-21T22:07:03.066 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 2876495139 2024-09-21T22:07:03.066 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:03.066 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:03.066 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:03.066 INFO:teuthology.orchestra.run.smithi039.stdout: "heartbeat_back_addrs": { 2024-09-21T22:07:03.066 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2024-09-21T22:07:03.066 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:03.066 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2024-09-21T22:07:03.066 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6822", 2024-09-21T22:07:03.067 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 2876495139 2024-09-21T22:07:03.067 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:03.067 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:03.067 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-09-21T22:07:03.067 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6823", 2024-09-21T22:07:03.067 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 2876495139 2024-09-21T22:07:03.067 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:03.067 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:03.067 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:03.067 INFO:teuthology.orchestra.run.smithi039.stdout: "heartbeat_front_addrs": { 2024-09-21T22:07:03.067 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2024-09-21T22:07:03.067 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:03.067 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2024-09-21T22:07:03.067 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6820", 2024-09-21T22:07:03.068 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 2876495139 2024-09-21T22:07:03.068 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:03.068 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:03.068 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-09-21T22:07:03.068 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6821", 2024-09-21T22:07:03.068 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 2876495139 2024-09-21T22:07:03.068 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:03.068 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:03.068 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:03.068 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6817/2876495139", 2024-09-21T22:07:03.068 INFO:teuthology.orchestra.run.smithi039.stdout: "cluster_addr": "172.21.15.39:6819/2876495139", 2024-09-21T22:07:03.068 INFO:teuthology.orchestra.run.smithi039.stdout: "heartbeat_back_addr": "172.21.15.39:6823/2876495139", 2024-09-21T22:07:03.068 INFO:teuthology.orchestra.run.smithi039.stdout: "heartbeat_front_addr": "172.21.15.39:6821/2876495139", 2024-09-21T22:07:03.068 INFO:teuthology.orchestra.run.smithi039.stdout: "state": [ 2024-09-21T22:07:03.069 INFO:teuthology.orchestra.run.smithi039.stdout: "exists", 2024-09-21T22:07:03.069 INFO:teuthology.orchestra.run.smithi039.stdout: "up" 2024-09-21T22:07:03.069 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:03.069 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:03.069 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:03.069 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": 2, 2024-09-21T22:07:03.069 INFO:teuthology.orchestra.run.smithi039.stdout: "uuid": "5f2678ed-6a23-4a67-96a4-c0ca91254bd7", 2024-09-21T22:07:03.069 INFO:teuthology.orchestra.run.smithi039.stdout: "up": 1, 2024-09-21T22:07:03.069 INFO:teuthology.orchestra.run.smithi039.stdout: "in": 1, 2024-09-21T22:07:03.069 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 1, 2024-09-21T22:07:03.069 INFO:teuthology.orchestra.run.smithi039.stdout: "primary_affinity": 1, 2024-09-21T22:07:03.069 INFO:teuthology.orchestra.run.smithi039.stdout: "last_clean_begin": 0, 2024-09-21T22:07:03.069 INFO:teuthology.orchestra.run.smithi039.stdout: "last_clean_end": 0, 2024-09-21T22:07:03.069 INFO:teuthology.orchestra.run.smithi039.stdout: "up_from": 11, 2024-09-21T22:07:03.070 INFO:teuthology.orchestra.run.smithi039.stdout: "up_thru": 42, 2024-09-21T22:07:03.070 INFO:teuthology.orchestra.run.smithi039.stdout: "down_at": 0, 2024-09-21T22:07:03.070 INFO:teuthology.orchestra.run.smithi039.stdout: "lost_at": 0, 2024-09-21T22:07:03.070 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2024-09-21T22:07:03.070 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2024-09-21T22:07:03.070 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:03.070 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2024-09-21T22:07:03.070 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6805", 2024-09-21T22:07:03.070 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 1262038957 2024-09-21T22:07:03.070 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:03.070 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:03.070 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-09-21T22:07:03.070 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6807", 2024-09-21T22:07:03.070 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 1262038957 2024-09-21T22:07:03.071 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:03.071 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:03.071 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:03.071 INFO:teuthology.orchestra.run.smithi039.stdout: "cluster_addrs": { 2024-09-21T22:07:03.071 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2024-09-21T22:07:03.071 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:03.071 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2024-09-21T22:07:03.071 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6809", 2024-09-21T22:07:03.071 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 1262038957 2024-09-21T22:07:03.071 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:03.071 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:03.071 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-09-21T22:07:03.071 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6811", 2024-09-21T22:07:03.071 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 1262038957 2024-09-21T22:07:03.072 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:03.072 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:03.072 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:03.072 INFO:teuthology.orchestra.run.smithi039.stdout: "heartbeat_back_addrs": { 2024-09-21T22:07:03.072 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2024-09-21T22:07:03.072 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:03.072 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2024-09-21T22:07:03.072 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6814", 2024-09-21T22:07:03.072 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 1262038957 2024-09-21T22:07:03.072 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:03.072 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:03.072 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-09-21T22:07:03.072 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6815", 2024-09-21T22:07:03.072 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 1262038957 2024-09-21T22:07:03.072 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:03.073 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:03.073 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:03.073 INFO:teuthology.orchestra.run.smithi039.stdout: "heartbeat_front_addrs": { 2024-09-21T22:07:03.073 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2024-09-21T22:07:03.073 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:03.073 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2024-09-21T22:07:03.073 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6812", 2024-09-21T22:07:03.073 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 1262038957 2024-09-21T22:07:03.073 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:03.073 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:03.073 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-09-21T22:07:03.073 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6813", 2024-09-21T22:07:03.073 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 1262038957 2024-09-21T22:07:03.073 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:03.073 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:03.074 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:03.074 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6807/1262038957", 2024-09-21T22:07:03.074 INFO:teuthology.orchestra.run.smithi039.stdout: "cluster_addr": "172.21.15.39:6811/1262038957", 2024-09-21T22:07:03.074 INFO:teuthology.orchestra.run.smithi039.stdout: "heartbeat_back_addr": "172.21.15.39:6815/1262038957", 2024-09-21T22:07:03.074 INFO:teuthology.orchestra.run.smithi039.stdout: "heartbeat_front_addr": "172.21.15.39:6813/1262038957", 2024-09-21T22:07:03.074 INFO:teuthology.orchestra.run.smithi039.stdout: "state": [ 2024-09-21T22:07:03.074 INFO:teuthology.orchestra.run.smithi039.stdout: "exists", 2024-09-21T22:07:03.074 INFO:teuthology.orchestra.run.smithi039.stdout: "up" 2024-09-21T22:07:03.074 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:03.074 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:03.074 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:03.074 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": 3, 2024-09-21T22:07:03.074 INFO:teuthology.orchestra.run.smithi039.stdout: "uuid": "aba30555-ef9e-47c9-a38b-ee1a4b8092f5", 2024-09-21T22:07:03.074 INFO:teuthology.orchestra.run.smithi039.stdout: "up": 1, 2024-09-21T22:07:03.075 INFO:teuthology.orchestra.run.smithi039.stdout: "in": 1, 2024-09-21T22:07:03.075 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 1, 2024-09-21T22:07:03.075 INFO:teuthology.orchestra.run.smithi039.stdout: "primary_affinity": 1, 2024-09-21T22:07:03.075 INFO:teuthology.orchestra.run.smithi039.stdout: "last_clean_begin": 0, 2024-09-21T22:07:03.075 INFO:teuthology.orchestra.run.smithi039.stdout: "last_clean_end": 0, 2024-09-21T22:07:03.075 INFO:teuthology.orchestra.run.smithi039.stdout: "up_from": 11, 2024-09-21T22:07:03.075 INFO:teuthology.orchestra.run.smithi039.stdout: "up_thru": 40, 2024-09-21T22:07:03.075 INFO:teuthology.orchestra.run.smithi039.stdout: "down_at": 0, 2024-09-21T22:07:03.075 INFO:teuthology.orchestra.run.smithi039.stdout: "lost_at": 0, 2024-09-21T22:07:03.075 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2024-09-21T22:07:03.075 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2024-09-21T22:07:03.075 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:03.075 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2024-09-21T22:07:03.075 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.64:6800", 2024-09-21T22:07:03.075 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 1132198485 2024-09-21T22:07:03.076 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:03.076 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:03.076 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-09-21T22:07:03.076 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.64:6801", 2024-09-21T22:07:03.076 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 1132198485 2024-09-21T22:07:03.076 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:03.076 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:03.076 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:03.076 INFO:teuthology.orchestra.run.smithi039.stdout: "cluster_addrs": { 2024-09-21T22:07:03.076 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2024-09-21T22:07:03.076 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:03.076 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2024-09-21T22:07:03.076 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.64:6802", 2024-09-21T22:07:03.076 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 1132198485 2024-09-21T22:07:03.077 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:03.077 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:03.077 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-09-21T22:07:03.077 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.64:6803", 2024-09-21T22:07:03.077 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 1132198485 2024-09-21T22:07:03.077 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:03.077 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:03.077 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:03.077 INFO:teuthology.orchestra.run.smithi039.stdout: "heartbeat_back_addrs": { 2024-09-21T22:07:03.077 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2024-09-21T22:07:03.077 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:03.077 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2024-09-21T22:07:03.077 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.64:6806", 2024-09-21T22:07:03.077 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 1132198485 2024-09-21T22:07:03.077 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:03.078 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:03.078 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-09-21T22:07:03.078 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.64:6807", 2024-09-21T22:07:03.078 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 1132198485 2024-09-21T22:07:03.078 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:03.078 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:03.078 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:03.078 INFO:teuthology.orchestra.run.smithi039.stdout: "heartbeat_front_addrs": { 2024-09-21T22:07:03.078 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2024-09-21T22:07:03.078 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:03.078 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2024-09-21T22:07:03.078 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.64:6804", 2024-09-21T22:07:03.078 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 1132198485 2024-09-21T22:07:03.078 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:03.079 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:03.079 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-09-21T22:07:03.079 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.64:6805", 2024-09-21T22:07:03.079 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 1132198485 2024-09-21T22:07:03.079 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:03.079 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:03.079 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:03.079 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.64:6801/1132198485", 2024-09-21T22:07:03.079 INFO:teuthology.orchestra.run.smithi039.stdout: "cluster_addr": "172.21.15.64:6803/1132198485", 2024-09-21T22:07:03.079 INFO:teuthology.orchestra.run.smithi039.stdout: "heartbeat_back_addr": "172.21.15.64:6807/1132198485", 2024-09-21T22:07:03.079 INFO:teuthology.orchestra.run.smithi039.stdout: "heartbeat_front_addr": "172.21.15.64:6805/1132198485", 2024-09-21T22:07:03.079 INFO:teuthology.orchestra.run.smithi039.stdout: "state": [ 2024-09-21T22:07:03.079 INFO:teuthology.orchestra.run.smithi039.stdout: "exists", 2024-09-21T22:07:03.079 INFO:teuthology.orchestra.run.smithi039.stdout: "up" 2024-09-21T22:07:03.080 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:03.080 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:03.080 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:03.080 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": 4, 2024-09-21T22:07:03.080 INFO:teuthology.orchestra.run.smithi039.stdout: "uuid": "2a1288d7-618d-4f44-8346-930cf56d4e80", 2024-09-21T22:07:03.080 INFO:teuthology.orchestra.run.smithi039.stdout: "up": 1, 2024-09-21T22:07:03.080 INFO:teuthology.orchestra.run.smithi039.stdout: "in": 1, 2024-09-21T22:07:03.080 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 1, 2024-09-21T22:07:03.080 INFO:teuthology.orchestra.run.smithi039.stdout: "primary_affinity": 1, 2024-09-21T22:07:03.080 INFO:teuthology.orchestra.run.smithi039.stdout: "last_clean_begin": 0, 2024-09-21T22:07:03.080 INFO:teuthology.orchestra.run.smithi039.stdout: "last_clean_end": 0, 2024-09-21T22:07:03.080 INFO:teuthology.orchestra.run.smithi039.stdout: "up_from": 11, 2024-09-21T22:07:03.080 INFO:teuthology.orchestra.run.smithi039.stdout: "up_thru": 0, 2024-09-21T22:07:03.080 INFO:teuthology.orchestra.run.smithi039.stdout: "down_at": 0, 2024-09-21T22:07:03.081 INFO:teuthology.orchestra.run.smithi039.stdout: "lost_at": 0, 2024-09-21T22:07:03.081 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2024-09-21T22:07:03.081 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2024-09-21T22:07:03.081 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:03.081 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2024-09-21T22:07:03.081 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.64:6808", 2024-09-21T22:07:03.081 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 3500786459 2024-09-21T22:07:03.081 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:03.081 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:03.081 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-09-21T22:07:03.081 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.64:6809", 2024-09-21T22:07:03.081 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 3500786459 2024-09-21T22:07:03.081 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:03.081 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:03.082 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:03.082 INFO:teuthology.orchestra.run.smithi039.stdout: "cluster_addrs": { 2024-09-21T22:07:03.082 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2024-09-21T22:07:03.082 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:03.082 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2024-09-21T22:07:03.082 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.64:6810", 2024-09-21T22:07:03.082 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 3500786459 2024-09-21T22:07:03.082 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:03.082 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:03.082 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-09-21T22:07:03.082 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.64:6811", 2024-09-21T22:07:03.082 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 3500786459 2024-09-21T22:07:03.082 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:03.082 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:03.083 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:03.083 INFO:teuthology.orchestra.run.smithi039.stdout: "heartbeat_back_addrs": { 2024-09-21T22:07:03.083 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2024-09-21T22:07:03.083 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:03.083 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2024-09-21T22:07:03.083 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.64:6814", 2024-09-21T22:07:03.083 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 3500786459 2024-09-21T22:07:03.083 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:03.083 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:03.083 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-09-21T22:07:03.083 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.64:6815", 2024-09-21T22:07:03.083 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 3500786459 2024-09-21T22:07:03.083 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:03.083 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:03.084 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:03.084 INFO:teuthology.orchestra.run.smithi039.stdout: "heartbeat_front_addrs": { 2024-09-21T22:07:03.084 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2024-09-21T22:07:03.084 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:03.084 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2024-09-21T22:07:03.084 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.64:6812", 2024-09-21T22:07:03.084 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 3500786459 2024-09-21T22:07:03.084 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:03.084 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:03.084 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-09-21T22:07:03.084 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.64:6813", 2024-09-21T22:07:03.084 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 3500786459 2024-09-21T22:07:03.084 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:03.084 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:03.085 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:03.085 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.64:6809/3500786459", 2024-09-21T22:07:03.085 INFO:teuthology.orchestra.run.smithi039.stdout: "cluster_addr": "172.21.15.64:6811/3500786459", 2024-09-21T22:07:03.085 INFO:teuthology.orchestra.run.smithi039.stdout: "heartbeat_back_addr": "172.21.15.64:6815/3500786459", 2024-09-21T22:07:03.085 INFO:teuthology.orchestra.run.smithi039.stdout: "heartbeat_front_addr": "172.21.15.64:6813/3500786459", 2024-09-21T22:07:03.085 INFO:teuthology.orchestra.run.smithi039.stdout: "state": [ 2024-09-21T22:07:03.085 INFO:teuthology.orchestra.run.smithi039.stdout: "exists", 2024-09-21T22:07:03.085 INFO:teuthology.orchestra.run.smithi039.stdout: "up" 2024-09-21T22:07:03.085 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:03.085 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:03.085 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:03.085 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": 5, 2024-09-21T22:07:03.085 INFO:teuthology.orchestra.run.smithi039.stdout: "uuid": "3ce719a8-34e9-4890-bd40-d22201e7bad8", 2024-09-21T22:07:03.085 INFO:teuthology.orchestra.run.smithi039.stdout: "up": 0, 2024-09-21T22:07:03.086 INFO:teuthology.orchestra.run.smithi039.stdout: "in": 1, 2024-09-21T22:07:03.086 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 1, 2024-09-21T22:07:03.086 INFO:teuthology.orchestra.run.smithi039.stdout: "primary_affinity": 1, 2024-09-21T22:07:03.086 INFO:teuthology.orchestra.run.smithi039.stdout: "last_clean_begin": 0, 2024-09-21T22:07:03.086 INFO:teuthology.orchestra.run.smithi039.stdout: "last_clean_end": 0, 2024-09-21T22:07:03.086 INFO:teuthology.orchestra.run.smithi039.stdout: "up_from": 11, 2024-09-21T22:07:03.086 INFO:teuthology.orchestra.run.smithi039.stdout: "up_thru": 16, 2024-09-21T22:07:03.086 INFO:teuthology.orchestra.run.smithi039.stdout: "down_at": 42, 2024-09-21T22:07:03.086 INFO:teuthology.orchestra.run.smithi039.stdout: "lost_at": 0, 2024-09-21T22:07:03.086 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2024-09-21T22:07:03.086 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2024-09-21T22:07:03.086 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:03.086 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2024-09-21T22:07:03.086 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.64:6816", 2024-09-21T22:07:03.087 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 2604542492 2024-09-21T22:07:03.087 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:03.087 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:03.087 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-09-21T22:07:03.087 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.64:6817", 2024-09-21T22:07:03.087 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 2604542492 2024-09-21T22:07:03.087 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:03.087 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:03.087 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:03.087 INFO:teuthology.orchestra.run.smithi039.stdout: "cluster_addrs": { 2024-09-21T22:07:03.087 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2024-09-21T22:07:03.087 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:03.087 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2024-09-21T22:07:03.087 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.64:6818", 2024-09-21T22:07:03.088 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 2604542492 2024-09-21T22:07:03.088 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:03.088 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:03.088 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-09-21T22:07:03.088 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.64:6819", 2024-09-21T22:07:03.088 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 2604542492 2024-09-21T22:07:03.088 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:03.088 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:03.088 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:03.088 INFO:teuthology.orchestra.run.smithi039.stdout: "heartbeat_back_addrs": { 2024-09-21T22:07:03.088 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2024-09-21T22:07:03.088 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:03.088 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2024-09-21T22:07:03.088 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.64:6822", 2024-09-21T22:07:03.089 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 2604542492 2024-09-21T22:07:03.089 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:03.089 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:03.089 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-09-21T22:07:03.089 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.64:6823", 2024-09-21T22:07:03.089 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 2604542492 2024-09-21T22:07:03.089 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:03.089 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:03.089 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:03.089 INFO:teuthology.orchestra.run.smithi039.stdout: "heartbeat_front_addrs": { 2024-09-21T22:07:03.089 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2024-09-21T22:07:03.089 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:03.089 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2024-09-21T22:07:03.089 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.64:6820", 2024-09-21T22:07:03.090 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 2604542492 2024-09-21T22:07:03.090 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:03.090 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:03.090 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-09-21T22:07:03.090 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.64:6821", 2024-09-21T22:07:03.090 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 2604542492 2024-09-21T22:07:03.090 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:03.090 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:03.090 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:03.090 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.64:6817/2604542492", 2024-09-21T22:07:03.090 INFO:teuthology.orchestra.run.smithi039.stdout: "cluster_addr": "172.21.15.64:6819/2604542492", 2024-09-21T22:07:03.090 INFO:teuthology.orchestra.run.smithi039.stdout: "heartbeat_back_addr": "172.21.15.64:6823/2604542492", 2024-09-21T22:07:03.090 INFO:teuthology.orchestra.run.smithi039.stdout: "heartbeat_front_addr": "172.21.15.64:6821/2604542492", 2024-09-21T22:07:03.090 INFO:teuthology.orchestra.run.smithi039.stdout: "state": [ 2024-09-21T22:07:03.091 INFO:teuthology.orchestra.run.smithi039.stdout: "exists" 2024-09-21T22:07:03.091 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:03.091 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:03.091 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T22:07:03.091 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_xinfo": [ 2024-09-21T22:07:03.091 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:03.091 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": 0, 2024-09-21T22:07:03.091 INFO:teuthology.orchestra.run.smithi039.stdout: "down_stamp": "0.000000", 2024-09-21T22:07:03.091 INFO:teuthology.orchestra.run.smithi039.stdout: "laggy_probability": 0, 2024-09-21T22:07:03.091 INFO:teuthology.orchestra.run.smithi039.stdout: "laggy_interval": 0, 2024-09-21T22:07:03.091 INFO:teuthology.orchestra.run.smithi039.stdout: "features": 4540701547738038271, 2024-09-21T22:07:03.091 INFO:teuthology.orchestra.run.smithi039.stdout: "old_weight": 0, 2024-09-21T22:07:03.091 INFO:teuthology.orchestra.run.smithi039.stdout: "last_purged_snaps_scrub": "2024-09-21T22:05:54.224881+0000", 2024-09-21T22:07:03.091 INFO:teuthology.orchestra.run.smithi039.stdout: "dead_epoch": 0 2024-09-21T22:07:03.092 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:03.092 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:03.092 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": 1, 2024-09-21T22:07:03.092 INFO:teuthology.orchestra.run.smithi039.stdout: "down_stamp": "0.000000", 2024-09-21T22:07:03.092 INFO:teuthology.orchestra.run.smithi039.stdout: "laggy_probability": 0, 2024-09-21T22:07:03.092 INFO:teuthology.orchestra.run.smithi039.stdout: "laggy_interval": 0, 2024-09-21T22:07:03.092 INFO:teuthology.orchestra.run.smithi039.stdout: "features": 4540701547738038271, 2024-09-21T22:07:03.092 INFO:teuthology.orchestra.run.smithi039.stdout: "old_weight": 0, 2024-09-21T22:07:03.092 INFO:teuthology.orchestra.run.smithi039.stdout: "last_purged_snaps_scrub": "2024-09-21T22:05:54.668978+0000", 2024-09-21T22:07:03.092 INFO:teuthology.orchestra.run.smithi039.stdout: "dead_epoch": 0 2024-09-21T22:07:03.092 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:03.092 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:03.092 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": 2, 2024-09-21T22:07:03.092 INFO:teuthology.orchestra.run.smithi039.stdout: "down_stamp": "0.000000", 2024-09-21T22:07:03.093 INFO:teuthology.orchestra.run.smithi039.stdout: "laggy_probability": 0, 2024-09-21T22:07:03.093 INFO:teuthology.orchestra.run.smithi039.stdout: "laggy_interval": 0, 2024-09-21T22:07:03.093 INFO:teuthology.orchestra.run.smithi039.stdout: "features": 4540701547738038271, 2024-09-21T22:07:03.093 INFO:teuthology.orchestra.run.smithi039.stdout: "old_weight": 0, 2024-09-21T22:07:03.093 INFO:teuthology.orchestra.run.smithi039.stdout: "last_purged_snaps_scrub": "2024-09-21T22:05:54.315876+0000", 2024-09-21T22:07:03.093 INFO:teuthology.orchestra.run.smithi039.stdout: "dead_epoch": 0 2024-09-21T22:07:03.093 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:03.093 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:03.093 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": 3, 2024-09-21T22:07:03.093 INFO:teuthology.orchestra.run.smithi039.stdout: "down_stamp": "0.000000", 2024-09-21T22:07:03.093 INFO:teuthology.orchestra.run.smithi039.stdout: "laggy_probability": 0, 2024-09-21T22:07:03.093 INFO:teuthology.orchestra.run.smithi039.stdout: "laggy_interval": 0, 2024-09-21T22:07:03.093 INFO:teuthology.orchestra.run.smithi039.stdout: "features": 4540701547738038271, 2024-09-21T22:07:03.093 INFO:teuthology.orchestra.run.smithi039.stdout: "old_weight": 0, 2024-09-21T22:07:03.094 INFO:teuthology.orchestra.run.smithi039.stdout: "last_purged_snaps_scrub": "2024-09-21T22:05:54.314795+0000", 2024-09-21T22:07:03.094 INFO:teuthology.orchestra.run.smithi039.stdout: "dead_epoch": 0 2024-09-21T22:07:03.094 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:03.094 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:03.094 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": 4, 2024-09-21T22:07:03.094 INFO:teuthology.orchestra.run.smithi039.stdout: "down_stamp": "0.000000", 2024-09-21T22:07:03.094 INFO:teuthology.orchestra.run.smithi039.stdout: "laggy_probability": 0, 2024-09-21T22:07:03.094 INFO:teuthology.orchestra.run.smithi039.stdout: "laggy_interval": 0, 2024-09-21T22:07:03.094 INFO:teuthology.orchestra.run.smithi039.stdout: "features": 4540701547738038271, 2024-09-21T22:07:03.094 INFO:teuthology.orchestra.run.smithi039.stdout: "old_weight": 0, 2024-09-21T22:07:03.094 INFO:teuthology.orchestra.run.smithi039.stdout: "last_purged_snaps_scrub": "2024-09-21T22:05:54.322403+0000", 2024-09-21T22:07:03.094 INFO:teuthology.orchestra.run.smithi039.stdout: "dead_epoch": 0 2024-09-21T22:07:03.094 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:03.095 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:03.095 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": 5, 2024-09-21T22:07:03.095 INFO:teuthology.orchestra.run.smithi039.stdout: "down_stamp": "0.000000", 2024-09-21T22:07:03.095 INFO:teuthology.orchestra.run.smithi039.stdout: "laggy_probability": 0, 2024-09-21T22:07:03.095 INFO:teuthology.orchestra.run.smithi039.stdout: "laggy_interval": 0, 2024-09-21T22:07:03.095 INFO:teuthology.orchestra.run.smithi039.stdout: "features": 4540701547738038271, 2024-09-21T22:07:03.095 INFO:teuthology.orchestra.run.smithi039.stdout: "old_weight": 0, 2024-09-21T22:07:03.095 INFO:teuthology.orchestra.run.smithi039.stdout: "last_purged_snaps_scrub": "2024-09-21T22:05:54.750633+0000", 2024-09-21T22:07:03.095 INFO:teuthology.orchestra.run.smithi039.stdout: "dead_epoch": 42 2024-09-21T22:07:03.095 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:03.095 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T22:07:03.095 INFO:teuthology.orchestra.run.smithi039.stdout: "pg_upmap": [], 2024-09-21T22:07:03.095 INFO:teuthology.orchestra.run.smithi039.stdout: "pg_upmap_items": [], 2024-09-21T22:07:03.095 INFO:teuthology.orchestra.run.smithi039.stdout: "pg_upmap_primaries": [], 2024-09-21T22:07:03.096 INFO:teuthology.orchestra.run.smithi039.stdout: "pg_temp": [], 2024-09-21T22:07:03.096 INFO:teuthology.orchestra.run.smithi039.stdout: "primary_temp": [], 2024-09-21T22:07:03.096 INFO:teuthology.orchestra.run.smithi039.stdout: "blocklist": {}, 2024-09-21T22:07:03.096 INFO:teuthology.orchestra.run.smithi039.stdout: "range_blocklist": {}, 2024-09-21T22:07:03.096 INFO:teuthology.orchestra.run.smithi039.stdout: "erasure_code_profiles": { 2024-09-21T22:07:03.096 INFO:teuthology.orchestra.run.smithi039.stdout: "default": { 2024-09-21T22:07:03.096 INFO:teuthology.orchestra.run.smithi039.stdout: "crush-failure-domain": "osd", 2024-09-21T22:07:03.096 INFO:teuthology.orchestra.run.smithi039.stdout: "k": "2", 2024-09-21T22:07:03.096 INFO:teuthology.orchestra.run.smithi039.stdout: "m": "1", 2024-09-21T22:07:03.096 INFO:teuthology.orchestra.run.smithi039.stdout: "plugin": "jerasure", 2024-09-21T22:07:03.096 INFO:teuthology.orchestra.run.smithi039.stdout: "technique": "reed_sol_van" 2024-09-21T22:07:03.096 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:03.096 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:03.096 INFO:teuthology.orchestra.run.smithi039.stdout: "removed_snaps_queue": [], 2024-09-21T22:07:03.097 INFO:teuthology.orchestra.run.smithi039.stdout: "new_removed_snaps": [], 2024-09-21T22:07:03.097 INFO:teuthology.orchestra.run.smithi039.stdout: "new_purged_snaps": [], 2024-09-21T22:07:03.097 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_node_flags": {}, 2024-09-21T22:07:03.097 INFO:teuthology.orchestra.run.smithi039.stdout: "device_class_flags": {}, 2024-09-21T22:07:03.097 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": { 2024-09-21T22:07:03.097 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode_enabled": false, 2024-09-21T22:07:03.097 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_bucket_count": 0, 2024-09-21T22:07:03.097 INFO:teuthology.orchestra.run.smithi039.stdout: "degraded_stretch_mode": 0, 2024-09-21T22:07:03.097 INFO:teuthology.orchestra.run.smithi039.stdout: "recovering_stretch_mode": 0, 2024-09-21T22:07:03.097 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode_bucket": 0 2024-09-21T22:07:03.097 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:03.097 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:07:03.099 INFO:tasks.thrashosds.thrasher:j is {'epoch': 47, 'fsid': '00bb09a3-e410-4e18-a2b2-08eaf61978e6', 'created': '2024-09-21T22:05:45.303341+0000', 'modified': '2024-09-21T22:07:01.994280+0000', 'last_up_change': '2024-09-21T22:06:49.928873+0000', 'last_in_change': '2024-09-21T22:06:43.837891+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 4, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 2, 'max_osd': 6, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-21T22:06:00.868591+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6.059999942779541, 'score_stable': 6.059999942779541, 'optimal_score': 0.33000001311302185, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-21T22:06:03.535857+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 19, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.2699999809265137, 'score_stable': 2.2699999809265137, 'optimal_score': 0.8299999833106995, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '8336c397-2018-4c75-ab31-2e5f6ad564b5', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6800', 'nonce': 3054061468}, {'type': 'v1', 'addr': '172.21.15.39:6801', 'nonce': 3054061468}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6802', 'nonce': 3054061468}, {'type': 'v1', 'addr': '172.21.15.39:6803', 'nonce': 3054061468}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6808', 'nonce': 3054061468}, {'type': 'v1', 'addr': '172.21.15.39:6810', 'nonce': 3054061468}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6804', 'nonce': 3054061468}, {'type': 'v1', 'addr': '172.21.15.39:6806', 'nonce': 3054061468}]}, 'public_addr': '172.21.15.39:6801/3054061468', 'cluster_addr': '172.21.15.39:6803/3054061468', 'heartbeat_back_addr': '172.21.15.39:6810/3054061468', 'heartbeat_front_addr': '172.21.15.39:6806/3054061468', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'bca64af6-7427-4c5d-97a4-a183b1f3a7d9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 42, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6816', 'nonce': 2876495139}, {'type': 'v1', 'addr': '172.21.15.39:6817', 'nonce': 2876495139}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6818', 'nonce': 2876495139}, {'type': 'v1', 'addr': '172.21.15.39:6819', 'nonce': 2876495139}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6822', 'nonce': 2876495139}, {'type': 'v1', 'addr': '172.21.15.39:6823', 'nonce': 2876495139}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6820', 'nonce': 2876495139}, {'type': 'v1', 'addr': '172.21.15.39:6821', 'nonce': 2876495139}]}, 'public_addr': '172.21.15.39:6817/2876495139', 'cluster_addr': '172.21.15.39:6819/2876495139', 'heartbeat_back_addr': '172.21.15.39:6823/2876495139', 'heartbeat_front_addr': '172.21.15.39:6821/2876495139', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '5f2678ed-6a23-4a67-96a4-c0ca91254bd7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 42, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6805', 'nonce': 1262038957}, {'type': 'v1', 'addr': '172.21.15.39:6807', 'nonce': 1262038957}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6809', 'nonce': 1262038957}, {'type': 'v1', 'addr': '172.21.15.39:6811', 'nonce': 1262038957}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6814', 'nonce': 1262038957}, {'type': 'v1', 'addr': '172.21.15.39:6815', 'nonce': 1262038957}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6812', 'nonce': 1262038957}, {'type': 'v1', 'addr': '172.21.15.39:6813', 'nonce': 1262038957}]}, 'public_addr': '172.21.15.39:6807/1262038957', 'cluster_addr': '172.21.15.39:6811/1262038957', 'heartbeat_back_addr': '172.21.15.39:6815/1262038957', 'heartbeat_front_addr': '172.21.15.39:6813/1262038957', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'aba30555-ef9e-47c9-a38b-ee1a4b8092f5', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 40, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6800', 'nonce': 1132198485}, {'type': 'v1', 'addr': '172.21.15.64:6801', 'nonce': 1132198485}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6802', 'nonce': 1132198485}, {'type': 'v1', 'addr': '172.21.15.64:6803', 'nonce': 1132198485}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6806', 'nonce': 1132198485}, {'type': 'v1', 'addr': '172.21.15.64:6807', 'nonce': 1132198485}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6804', 'nonce': 1132198485}, {'type': 'v1', 'addr': '172.21.15.64:6805', 'nonce': 1132198485}]}, 'public_addr': '172.21.15.64:6801/1132198485', 'cluster_addr': '172.21.15.64:6803/1132198485', 'heartbeat_back_addr': '172.21.15.64:6807/1132198485', 'heartbeat_front_addr': '172.21.15.64:6805/1132198485', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '2a1288d7-618d-4f44-8346-930cf56d4e80', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6808', 'nonce': 3500786459}, {'type': 'v1', 'addr': '172.21.15.64:6809', 'nonce': 3500786459}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6810', 'nonce': 3500786459}, {'type': 'v1', 'addr': '172.21.15.64:6811', 'nonce': 3500786459}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6814', 'nonce': 3500786459}, {'type': 'v1', 'addr': '172.21.15.64:6815', 'nonce': 3500786459}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6812', 'nonce': 3500786459}, {'type': 'v1', 'addr': '172.21.15.64:6813', 'nonce': 3500786459}]}, 'public_addr': '172.21.15.64:6809/3500786459', 'cluster_addr': '172.21.15.64:6811/3500786459', 'heartbeat_back_addr': '172.21.15.64:6815/3500786459', 'heartbeat_front_addr': '172.21.15.64:6813/3500786459', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '3ce719a8-34e9-4890-bd40-d22201e7bad8', 'up': 0, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 16, 'down_at': 42, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6816', 'nonce': 2604542492}, {'type': 'v1', 'addr': '172.21.15.64:6817', 'nonce': 2604542492}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6818', 'nonce': 2604542492}, {'type': 'v1', 'addr': '172.21.15.64:6819', 'nonce': 2604542492}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6822', 'nonce': 2604542492}, {'type': 'v1', 'addr': '172.21.15.64:6823', 'nonce': 2604542492}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6820', 'nonce': 2604542492}, {'type': 'v1', 'addr': '172.21.15.64:6821', 'nonce': 2604542492}]}, 'public_addr': '172.21.15.64:6817/2604542492', 'cluster_addr': '172.21.15.64:6819/2604542492', 'heartbeat_back_addr': '172.21.15.64:6823/2604542492', 'heartbeat_front_addr': '172.21.15.64:6821/2604542492', 'state': ['exists']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T22:05:54.224881+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T22:05:54.668978+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T22:05:54.315876+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T22:05:54.314795+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T22:05:54.322403+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T22:05:54.750633+0000', 'dead_epoch': 42}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-21T22:07:03.099 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T22:07:03.100 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:07:03.100 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:07:03.100 INFO:teuthology.orchestra.run.smithi039.stdout: "num_ops": 0 2024-09-21T22:07:03.100 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:07:03.100 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T22:07:03.140 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:03.135+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:03.217 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:03.211+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:03.258 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:07:03.259 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:07:03.259 INFO:teuthology.orchestra.run.smithi039.stdout: "complaint_time": 30, 2024-09-21T22:07:03.259 INFO:teuthology.orchestra.run.smithi039.stdout: "num_blocked_ops": 0 2024-09-21T22:07:03.259 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:07:03.273 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T22:07:03.318 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:03.312+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:03.419 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:03.416+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:03.432 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:07:03.433 INFO:teuthology.orchestra.run.smithi039.stdout: "size": 20, 2024-09-21T22:07:03.433 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 600, 2024-09-21T22:07:03.433 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [ 2024-09-21T22:07:03.433 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:03.433 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4224.0:5 2.2 e18/16 2:5cea7035:::rbd_info:head v 18'1, mlcod=0'0)", 2024-09-21T22:07:03.433 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:06.114955+0000", 2024-09-21T22:07:03.433 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 57.317275574999996, 2024-09-21T22:07:03.433 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0017716170000000001, 2024-09-21T22:07:03.433 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:03.433 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:03.433 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:03.433 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:03.433 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:03.434 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:03.434 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:06.114955+0000", 2024-09-21T22:07:03.434 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:03.434 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:03.434 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:03.434 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:03.434 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:06.114955+0000", 2024-09-21T22:07:03.434 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:03.434 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:03.434 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:03.434 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:03.434 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:06.114956+0000", 2024-09-21T22:07:03.434 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.9299999999999997e-07 2024-09-21T22:07:03.435 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:03.435 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:03.435 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:03.435 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:06.114964+0000", 2024-09-21T22:07:03.435 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.9840000000000001e-06 2024-09-21T22:07:03.435 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:03.435 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:03.435 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:03.435 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:06.115001+0000", 2024-09-21T22:07:03.435 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.7719999999999998e-05 2024-09-21T22:07:03.435 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:03.435 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:03.435 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:03.435 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:06.115019+0000", 2024-09-21T22:07:03.435 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7943000000000001e-05 2024-09-21T22:07:03.436 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:03.436 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:03.436 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:03.436 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:06.115168+0000", 2024-09-21T22:07:03.436 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000148589 2024-09-21T22:07:03.436 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:03.436 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:03.436 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:03.436 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:06.115225+0000", 2024-09-21T22:07:03.436 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.6975999999999998e-05 2024-09-21T22:07:03.436 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:03.436 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:03.436 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:03.436 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:06.116585+0000", 2024-09-21T22:07:03.437 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00136038 2024-09-21T22:07:03.437 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:03.437 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:03.437 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:03.437 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:06.116726+0000", 2024-09-21T22:07:03.437 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00014133200000000001 2024-09-21T22:07:03.437 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:03.437 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:03.437 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:03.437 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:03.437 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:03.437 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4224.0:7 2.2 e19/16 2:5cea7035:::rbd_info:head v 19'2, mlcod=0'0)", 2024-09-21T22:07:03.437 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:07.129567+0000", 2024-09-21T22:07:03.437 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 56.302663566, 2024-09-21T22:07:03.437 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0018460239999999999, 2024-09-21T22:07:03.438 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:03.438 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:03.438 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:03.438 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:03.438 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:03.438 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:03.438 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:07.129567+0000", 2024-09-21T22:07:03.438 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:03.438 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:03.438 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:03.438 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:03.438 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:07.129567+0000", 2024-09-21T22:07:03.438 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:03.438 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:03.438 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:03.439 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:03.439 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:07.129567+0000", 2024-09-21T22:07:03.439 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.9499999999999998e-07 2024-09-21T22:07:03.439 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:03.439 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:03.439 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:03.439 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:07.129580+0000", 2024-09-21T22:07:03.439 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2335e-05 2024-09-21T22:07:03.439 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:03.439 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:03.439 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:03.439 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:07.129606+0000", 2024-09-21T22:07:03.439 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.6528000000000001e-05 2024-09-21T22:07:03.439 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:03.440 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:03.440 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:03.440 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:07.129628+0000", 2024-09-21T22:07:03.440 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1549999999999999e-05 2024-09-21T22:07:03.440 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:03.440 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:03.440 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:03.440 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:07.129744+0000", 2024-09-21T22:07:03.440 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000116551 2024-09-21T22:07:03.440 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:03.440 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:03.440 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:03.440 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:07.129780+0000", 2024-09-21T22:07:03.440 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.5447000000000001e-05 2024-09-21T22:07:03.441 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:03.441 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:03.441 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:03.441 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:07.131348+0000", 2024-09-21T22:07:03.441 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001568631 2024-09-21T22:07:03.441 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:03.441 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:03.441 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:03.441 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:07.131413+0000", 2024-09-21T22:07:03.441 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.4586999999999997e-05 2024-09-21T22:07:03.441 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:03.441 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:03.441 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:03.441 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:03.441 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:03.442 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:07:03.447 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T22:07:03.447 INFO:teuthology.orchestra.run.smithi039.stderr:dumped all 2024-09-21T22:07:03.449 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T22:07:03.466 INFO:teuthology.orchestra.run.smithi039.stdout:{"pg_ready":true,"pg_map":{"version":74,"stamp":"2024-09-21T22:07:02.802180+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":181784,"kb_used_data":2972,"kb_used_omap":9,"kb_used_meta":178678,"kb_avail":566049256,"statfs":{"total":579820584960,"available":579634438144,"internally_reserved":0,"allocated":3043328,"data_stored":2235002,"data_compressed":13990,"data_compressed_allocated":73728,"data_compressed_original":1163264,"omap_allocated":9568,"internal_metadata":182966944},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.820725"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":71,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-21T22:06:58.998740+0000","last_change":"2024-09-21T22:06:07.122328+0000","last_active":"2024-09-21T22:06:58.998740+0000","last_peered":"2024-09-21T22:06:58.998740+0000","last_clean":"2024-09-21T22:06:58.998740+0000","last_became_active":"2024-09-21T22:06:05.220996+0000","last_became_peered":"2024-09-21T22:06:05.220996+0000","last_unstale":"2024-09-21T22:06:58.998740+0000","last_undegraded":"2024-09-21T22:06:58.998740+0000","last_fullsized":"2024-09-21T22:06:58.998740+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T23:45:49.178013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001473755,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":73,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-21T22:06:58.999604+0000","last_change":"2024-09-21T22:06:07.133757+0000","last_active":"2024-09-21T22:06:58.999604+0000","last_peered":"2024-09-21T22:06:58.999604+0000","last_clean":"2024-09-21T22:06:58.999604+0000","last_became_active":"2024-09-21T22:06:05.109065+0000","last_became_peered":"2024-09-21T22:06:05.109065+0000","last_unstale":"2024-09-21T22:06:58.999604+0000","last_undegraded":"2024-09-21T22:06:58.999604+0000","last_fullsized":"2024-09-21T22:06:58.999604+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:31:12.654893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00088865700000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":81,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-21T22:06:58.998738+0000","last_change":"2024-09-21T22:06:44.875038+0000","last_active":"2024-09-21T22:06:58.998738+0000","last_peered":"2024-09-21T22:06:58.998738+0000","last_clean":"2024-09-21T22:06:58.998738+0000","last_became_active":"2024-09-21T22:06:44.874697+0000","last_became_peered":"2024-09-21T22:06:44.874697+0000","last_unstale":"2024-09-21T22:06:58.998738+0000","last_undegraded":"2024-09-21T22:06:58.998738+0000","last_fullsized":"2024-09-21T22:06:58.998738+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T07:09:15.569760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091949900000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":83,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-21T22:06:58.999495+0000","last_change":"2024-09-21T22:06:44.875768+0000","last_active":"2024-09-21T22:06:58.999495+0000","last_peered":"2024-09-21T22:06:58.999495+0000","last_clean":"2024-09-21T22:06:58.999495+0000","last_became_active":"2024-09-21T22:06:44.875201+0000","last_became_peered":"2024-09-21T22:06:44.875201+0000","last_unstale":"2024-09-21T22:06:58.999495+0000","last_undegraded":"2024-09-21T22:06:58.999495+0000","last_fullsized":"2024-09-21T22:06:58.999495+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T08:49:08.148866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00090834100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"19'2","reported_seq":32,"reported_epoch":46,"state":"active+undersized+degraded","last_fresh":"2024-09-21T22:06:58.999356+0000","last_change":"2024-09-21T22:06:50.952428+0000","last_active":"2024-09-21T22:06:58.999356+0000","last_peered":"2024-09-21T22:06:58.999356+0000","last_clean":"2024-09-21T22:06:07.127707+0000","last_became_active":"2024-09-21T22:06:50.952428+0000","last_became_peered":"2024-09-21T22:06:50.952428+0000","last_unstale":"2024-09-21T22:06:58.999356+0000","last_undegraded":"2024-09-21T22:06:50.951200+0000","last_fullsized":"2024-09-21T22:06:50.950965+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:25:30.313564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":73,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-21T22:06:59.000345+0000","last_change":"2024-09-21T22:06:07.123496+0000","last_active":"2024-09-21T22:06:59.000345+0000","last_peered":"2024-09-21T22:06:59.000345+0000","last_clean":"2024-09-21T22:06:59.000345+0000","last_became_active":"2024-09-21T22:06:05.104941+0000","last_became_peered":"2024-09-21T22:06:05.104941+0000","last_unstale":"2024-09-21T22:06:59.000345+0000","last_undegraded":"2024-09-21T22:06:59.000345+0000","last_fullsized":"2024-09-21T22:06:59.000345+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:46:29.901819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010052629999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":71,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-21T22:06:58.998776+0000","last_change":"2024-09-21T22:06:07.121573+0000","last_active":"2024-09-21T22:06:58.998776+0000","last_peered":"2024-09-21T22:06:58.998776+0000","last_clean":"2024-09-21T22:06:58.998776+0000","last_became_active":"2024-09-21T22:06:05.108187+0000","last_became_peered":"2024-09-21T22:06:05.108187+0000","last_unstale":"2024-09-21T22:06:58.998776+0000","last_undegraded":"2024-09-21T22:06:58.998776+0000","last_fullsized":"2024-09-21T22:06:58.998776+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T05:05:08.108341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079478100000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"17'1","reported_seq":22,"reported_epoch":46,"state":"active+undersized+degraded","last_fresh":"2024-09-21T22:06:59.000405+0000","last_change":"2024-09-21T22:06:50.951695+0000","last_active":"2024-09-21T22:06:59.000405+0000","last_peered":"2024-09-21T22:06:59.000405+0000","last_clean":"2024-09-21T22:06:05.103775+0000","last_became_active":"2024-09-21T22:06:50.951695+0000","last_became_peered":"2024-09-21T22:06:50.951695+0000","last_unstale":"2024-09-21T22:06:59.000405+0000","last_undegraded":"2024-09-21T22:06:50.950664+0000","last_fullsized":"2024-09-21T22:06:50.950592+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T05:13:45.467897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"15'117","reported_seq":248,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-21T22:06:58.998824+0000","last_change":"2024-09-21T22:06:47.926376+0000","last_active":"2024-09-21T22:06:58.998824+0000","last_peered":"2024-09-21T22:06:58.998824+0000","last_clean":"2024-09-21T22:06:58.998824+0000","last_became_active":"2024-09-21T22:06:47.925841+0000","last_became_peered":"2024-09-21T22:06:47.925841+0000","last_unstale":"2024-09-21T22:06:58.998824+0000","last_undegraded":"2024-09-21T22:06:58.998824+0000","last_fullsized":"2024-09-21T22:06:58.998824+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:00.990331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:00.990331+0000","last_clean_scrub_stamp":"2024-09-21T22:06:00.990331+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:50:17.318694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":14,"acting":14,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":1771104,"data_compressed":21419,"data_compressed_allocated":110592,"data_compressed_original":1753088,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640270,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27428,"kb_used_data":396,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344412,"statfs":{"total":96636764160,"available":96608677888,"internally_reserved":0,"allocated":405504,"data_stored":142934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640272,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27516,"kb_used_data":488,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344324,"statfs":{"total":96636764160,"available":96608587776,"internally_reserved":0,"allocated":499712,"data_stored":179535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640275,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31744,"kb_used_data":556,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340096,"statfs":{"total":96636764160,"available":96604258304,"internally_reserved":0,"allocated":569344,"data_stored":774449,"data_compressed":6561,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640274,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31688,"kb_used_data":500,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340152,"statfs":{"total":96636764160,"available":96604315648,"internally_reserved":0,"allocated":512000,"data_stored":184081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640274,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31692,"kb_used_data":504,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340148,"statfs":{"total":96636764160,"available":96604311552,"internally_reserved":0,"allocated":516096,"data_stored":184100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640275,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31716,"kb_used_data":528,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340124,"statfs":{"total":96636764160,"available":96604286976,"internally_reserved":0,"allocated":540672,"data_stored":769903,"data_compressed":7429,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":7429,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":7429,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6561,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T22:07:03.467 INFO:tasks.thrashosds.thrasher:Setting 1.0 to [3, 2] 2024-09-21T22:07:03.467 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '1.0', '3', '2'] 2024-09-21T22:07:03.467 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap 1.0 3 2 2024-09-21T22:07:03.559 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:03.555+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:03.565 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T22:07:03.569 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:07:03.569 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:07:03.569 INFO:teuthology.orchestra.run.smithi039.stdout: "num_ops": 0 2024-09-21T22:07:03.569 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:07:03.586 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T22:07:03.621 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:03.616+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:03.704 INFO:teuthology.orchestra.run.smithi039.stdout:osd.0: {} 2024-09-21T22:07:03.704 INFO:teuthology.orchestra.run.smithi039.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T22:07:03.721 INFO:teuthology.orchestra.run.smithi039.stdout:osd.1: {} 2024-09-21T22:07:03.721 INFO:teuthology.orchestra.run.smithi039.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T22:07:03.722 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:03.715+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:03.738 INFO:teuthology.orchestra.run.smithi039.stdout:osd.2: {} 2024-09-21T22:07:03.738 INFO:teuthology.orchestra.run.smithi039.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T22:07:03.745 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:03.745 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:03.759 INFO:teuthology.orchestra.run.smithi039.stdout:osd.3: {} 2024-09-21T22:07:03.759 INFO:teuthology.orchestra.run.smithi039.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T22:07:03.760 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T22:07:03.780 INFO:teuthology.orchestra.run.smithi039.stdout:osd.4: {} 2024-09-21T22:07:03.780 INFO:teuthology.orchestra.run.smithi039.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T22:07:03.781 INFO:teuthology.orchestra.run.smithi039.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-21T22:07:03.783 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T22:07:03.784 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T22:07:03.831 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:03.827+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:03.894 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:03.894 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:03.907 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T22:07:03.951 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:03.947+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:04.024 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:04.020+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:04.029 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-21T22:07:04.041 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:04.042 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:04.049 INFO:teuthology.orchestra.run.smithi039.stderr:set 1.0 pg_upmap mapping to [3,2] 2024-09-21T22:07:04.058 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T22:07:04.145 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:04.140+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:04.224 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:04.224 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:04.225 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:04.220+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:04.238 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T22:07:04.325 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:04.319+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:04.410 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:04.410 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:04.425 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T22:07:04.426 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:04.423+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:04.527 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:04.524+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:04.547 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:04.547 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:04.561 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T22:07:04.651 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:04.648+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:04.729 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:04.727+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:04.731 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:04.731 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:04.746 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T22:07:04.831 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:04.823+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:04.907 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:04.907 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:04.921 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T22:07:04.967 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:04.963+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:05.022 INFO:teuthology.orchestra.run.smithi039.stderr:nodeep-scrub is set 2024-09-21T22:07:05.032 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:05.027+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:05.062 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:05.062 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:05.076 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T22:07:05.168 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:05.163+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:05.236 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:05.231+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:05.245 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:05.246 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:05.260 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T22:07:05.347 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:05.343+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:05.423 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:05.423 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:05.439 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T22:07:05.439 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:05.431+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:05.539 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:05.536+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:05.598 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:05.598 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:05.612 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T22:07:05.655 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:05.651+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:05.741 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:05.741 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:05.741 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:05.735+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:05.755 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T22:07:05.785 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T22:07:05.843 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:05.839+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:05.922 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:05.922 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:05.942 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T22:07:05.950 INFO:teuthology.orchestra.run.smithi039.stdout:osd.0: {} 2024-09-21T22:07:05.950 INFO:teuthology.orchestra.run.smithi039.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T22:07:05.951 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:05.943+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:05.967 INFO:teuthology.orchestra.run.smithi039.stdout:osd.1: {} 2024-09-21T22:07:05.968 INFO:teuthology.orchestra.run.smithi039.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T22:07:05.985 INFO:teuthology.orchestra.run.smithi039.stdout:osd.2: {} 2024-09-21T22:07:05.985 INFO:teuthology.orchestra.run.smithi039.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T22:07:06.006 INFO:teuthology.orchestra.run.smithi039.stdout:osd.3: {} 2024-09-21T22:07:06.006 INFO:teuthology.orchestra.run.smithi039.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T22:07:06.027 INFO:teuthology.orchestra.run.smithi039.stdout:osd.4: {} 2024-09-21T22:07:06.027 INFO:teuthology.orchestra.run.smithi039.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T22:07:06.027 INFO:teuthology.orchestra.run.smithi039.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-21T22:07:06.031 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T22:07:06.032 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T22:07:06.071 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:07:06.071 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:07:06.071 INFO:teuthology.orchestra.run.smithi039.stdout: "num_ops": 0 2024-09-21T22:07:06.071 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:07:06.071 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:06.063+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:06.085 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T22:07:06.171 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:06.167+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:06.246 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:06.239+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:06.248 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:07:06.248 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:07:06.248 INFO:teuthology.orchestra.run.smithi039.stdout: "complaint_time": 30, 2024-09-21T22:07:06.249 INFO:teuthology.orchestra.run.smithi039.stdout: "num_blocked_ops": 0 2024-09-21T22:07:06.249 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:07:06.263 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T22:07:06.347 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:06.344+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:06.436 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:07:06.437 INFO:teuthology.orchestra.run.smithi039.stdout: "size": 20, 2024-09-21T22:07:06.437 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 600, 2024-09-21T22:07:06.437 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [ 2024-09-21T22:07:06.437 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:06.437 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4224.0:1 2.3 e17/16 2:c4c92e5a:::rbd_trash:head v 17'1, mlcod=0'0)", 2024-09-21T22:07:06.437 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:05.107810+0000", 2024-09-21T22:07:06.437 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 61.328160721000003, 2024-09-21T22:07:06.437 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0017679200000000001, 2024-09-21T22:07:06.437 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:06.437 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:06.437 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:06.438 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:06.438 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:06.438 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:06.438 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.107810+0000", 2024-09-21T22:07:06.438 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:06.438 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:06.438 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:06.438 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:06.438 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.107810+0000", 2024-09-21T22:07:06.438 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:06.438 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:06.438 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:06.438 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:06.438 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.107811+0000", 2024-09-21T22:07:06.439 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.6700000000000003e-07 2024-09-21T22:07:06.439 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:06.439 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:06.439 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:06.439 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.107822+0000", 2024-09-21T22:07:06.439 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0443e-05 2024-09-21T22:07:06.439 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:06.439 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:06.439 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:06.439 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.107853+0000", 2024-09-21T22:07:06.439 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.1046000000000001e-05 2024-09-21T22:07:06.439 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:06.439 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:06.439 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:06.439 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.107876+0000", 2024-09-21T22:07:06.439 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3547e-05 2024-09-21T22:07:06.440 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:06.440 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:06.440 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:06.440 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.108002+0000", 2024-09-21T22:07:06.440 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000125599 2024-09-21T22:07:06.440 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:06.440 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:06.440 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:06.440 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.108069+0000", 2024-09-21T22:07:06.440 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.7162999999999994e-05 2024-09-21T22:07:06.440 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:06.440 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:06.440 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:06.440 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.109385+0000", 2024-09-21T22:07:06.440 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0013162899999999999 2024-09-21T22:07:06.441 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:06.441 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:06.441 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:06.441 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.109578+0000", 2024-09-21T22:07:06.441 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000193065 2024-09-21T22:07:06.441 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:06.441 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:06.441 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:06.441 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:06.441 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:06.441 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(1.0 50/49 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 15'116, data_included: [0~262688], data_size: 262688, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@15'116, size: 262688, copy_subset: [(0, 262688)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 262688, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T22:07:06.441 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:06.036813+0000", 2024-09-21T22:07:06.441 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.39915814199999999, 2024-09-21T22:07:06.441 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0022591270000000001, 2024-09-21T22:07:06.441 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:06.442 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:06.442 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:06.442 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:06.442 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:06.442 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:06.442 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:06.036813+0000", 2024-09-21T22:07:06.442 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:06.442 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:06.442 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:06.442 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:06.442 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:06.036813+0000", 2024-09-21T22:07:06.442 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:06.442 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:06.443 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:06.443 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:06.443 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:06.036814+0000", 2024-09-21T22:07:06.443 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.0400000000000005e-07 2024-09-21T22:07:06.443 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:06.443 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:06.443 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:06.443 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:06.037279+0000", 2024-09-21T22:07:06.443 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00046499600000000002 2024-09-21T22:07:06.443 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:06.443 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:06.443 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:06.443 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:06.037307+0000", 2024-09-21T22:07:06.443 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8359000000000001e-05 2024-09-21T22:07:06.444 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:06.444 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:06.444 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:06.444 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:06.037332+0000", 2024-09-21T22:07:06.444 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5154999999999999e-05 2024-09-21T22:07:06.444 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:06.444 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:06.444 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:06.444 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:06.037419+0000", 2024-09-21T22:07:06.444 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.6310000000000005e-05 2024-09-21T22:07:06.444 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:06.444 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:06.444 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:06.444 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:06.037430+0000", 2024-09-21T22:07:06.444 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0832000000000001e-05 2024-09-21T22:07:06.444 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:06.445 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:06.445 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:06.445 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:06.039072+0000", 2024-09-21T22:07:06.445 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001642571 2024-09-21T22:07:06.445 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:06.445 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:06.445 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:06.445 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:06.445 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:06.445 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(1.0 50/49 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 15'117, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@15'117, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T22:07:06.445 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:06.041121+0000", 2024-09-21T22:07:06.445 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.39485056099999999, 2024-09-21T22:07:06.446 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0022693829999999998, 2024-09-21T22:07:06.446 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:06.446 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:06.446 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:06.446 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:06.446 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:06.446 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:06.446 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:06.041121+0000", 2024-09-21T22:07:06.446 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:06.446 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:06.446 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:06.446 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:06.446 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:06.041121+0000", 2024-09-21T22:07:06.446 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:06.447 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:06.447 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:06.447 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:06.447 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:06.041122+0000", 2024-09-21T22:07:06.447 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.3500000000000005e-07 2024-09-21T22:07:06.447 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:06.447 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:06.447 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:06.447 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:06.041449+0000", 2024-09-21T22:07:06.447 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00032708299999999999 2024-09-21T22:07:06.447 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:06.447 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:06.447 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:06.447 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:06.041475+0000", 2024-09-21T22:07:06.448 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.6245e-05 2024-09-21T22:07:06.448 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:06.448 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:06.448 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:06.448 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:06.041498+0000", 2024-09-21T22:07:06.448 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.2802e-05 2024-09-21T22:07:06.448 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:06.448 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:06.448 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:06.448 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:06.041579+0000", 2024-09-21T22:07:06.448 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.1203000000000002e-05 2024-09-21T22:07:06.448 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:06.448 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:06.448 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:06.449 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:06.041587+0000", 2024-09-21T22:07:06.449 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.9230000000000002e-06 2024-09-21T22:07:06.449 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:06.449 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:06.449 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:06.449 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:06.043390+0000", 2024-09-21T22:07:06.449 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0018031919999999999 2024-09-21T22:07:06.449 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:06.449 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:06.449 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:06.449 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:06.449 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:06.449 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:07:06.451 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T22:07:06.451 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:06.443+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:06.552 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:06.547+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:06.587 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:07:06.588 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:07:06.588 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-09-21T22:07:06.588 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:07:06.602 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T22:07:06.652 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:06.648+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:06.721 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:07:06.721 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:07:06.721 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-21T22:07:06.721 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-21T22:07:06.721 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:07:06.735 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T22:07:06.753 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:06.747+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:06.854 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:06.848+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:06.911 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:07:06.911 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-21T22:07:06.911 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-21T22:07:06.911 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-21T22:07:06.911 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.911 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:07:06.911 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.036355+0000", 2024-09-21T22:07:06.911 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 64.870810524999996, 2024-09-21T22:07:06.911 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.062441934999999997, 2024-09-21T22:07:06.911 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:06.912 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:06.912 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:06.912 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:06.912 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:07:06.912 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:07:06.912 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1 2024-09-21T22:07:06.912 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.912 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:06.912 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.912 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:06.912 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036355+0000", 2024-09-21T22:07:06.912 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:06.912 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.912 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.913 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:06.913 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036355+0000", 2024-09-21T22:07:06.913 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:06.913 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.913 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.913 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:06.913 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036357+0000", 2024-09-21T22:07:06.913 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.446e-06 2024-09-21T22:07:06.913 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.913 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.913 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:06.913 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036373+0000", 2024-09-21T22:07:06.913 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5447e-05 2024-09-21T22:07:06.913 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.914 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.914 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:06.914 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036399+0000", 2024-09-21T22:07:06.914 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6423000000000001e-05 2024-09-21T22:07:06.914 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.914 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.914 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:06.914 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036420+0000", 2024-09-21T22:07:06.914 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1129999999999999e-05 2024-09-21T22:07:06.914 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.914 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.914 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:06.914 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036557+0000", 2024-09-21T22:07:06.914 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013686899999999999 2024-09-21T22:07:06.915 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.915 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.915 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-21T22:07:06.915 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036585+0000", 2024-09-21T22:07:06.915 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8362999999999999e-05 2024-09-21T22:07:06.915 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.915 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.915 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:06.915 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.095068+0000", 2024-09-21T22:07:06.915 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058482972000000001 2024-09-21T22:07:06.915 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.915 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.915 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:06.915 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.095516+0000", 2024-09-21T22:07:06.915 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00044802999999999998 2024-09-21T22:07:06.916 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.916 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.916 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:06.916 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.095869+0000", 2024-09-21T22:07:06.916 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000352104 2024-09-21T22:07:06.916 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.916 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.916 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:06.916 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.097055+0000", 2024-09-21T22:07:06.916 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011865599999999999 2024-09-21T22:07:06.916 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.916 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.916 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:06.916 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.098637+0000", 2024-09-21T22:07:06.917 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001581458 2024-09-21T22:07:06.917 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.917 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.917 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:06.917 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.098728+0000", 2024-09-21T22:07:06.917 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1043000000000002e-05 2024-09-21T22:07:06.917 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.917 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.917 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:06.917 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.098797+0000", 2024-09-21T22:07:06.917 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9090000000000004e-05 2024-09-21T22:07:06.917 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:06.917 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:06.917 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:06.918 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.918 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.918 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:07:06.918 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.161242+0000", 2024-09-21T22:07:06.918 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 64.745923613000002, 2024-09-21T22:07:06.918 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014477078000000001, 2024-09-21T22:07:06.918 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:06.918 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:06.918 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:06.918 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:06.918 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:07:06.918 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:07:06.918 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 35 2024-09-21T22:07:06.919 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.919 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:06.919 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.919 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:06.919 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161242+0000", 2024-09-21T22:07:06.919 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:06.919 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.919 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.919 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:06.919 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161242+0000", 2024-09-21T22:07:06.919 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:06.919 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.919 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.919 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:06.919 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161243+0000", 2024-09-21T22:07:06.920 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4729999999999999e-06 2024-09-21T22:07:06.920 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.920 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.920 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:06.920 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161385+0000", 2024-09-21T22:07:06.920 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014226899999999999 2024-09-21T22:07:06.920 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.920 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.920 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:06.920 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161417+0000", 2024-09-21T22:07:06.920 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1406999999999999e-05 2024-09-21T22:07:06.920 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.920 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.920 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:06.920 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161434+0000", 2024-09-21T22:07:06.921 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7623000000000001e-05 2024-09-21T22:07:06.921 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.921 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.921 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:06.921 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.169728+0000", 2024-09-21T22:07:06.921 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0082933090000000004 2024-09-21T22:07:06.921 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.921 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.921 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:06.921 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.169861+0000", 2024-09-21T22:07:06.921 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013367300000000001 2024-09-21T22:07:06.921 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.921 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.921 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:06.921 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.170186+0000", 2024-09-21T22:07:06.921 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00032466400000000001 2024-09-21T22:07:06.922 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.922 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.922 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:06.922 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.174759+0000", 2024-09-21T22:07:06.922 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0045727880000000004 2024-09-21T22:07:06.922 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.922 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.922 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:06.922 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.175497+0000", 2024-09-21T22:07:06.922 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00073803400000000004 2024-09-21T22:07:06.922 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.922 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.922 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:06.923 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.175607+0000", 2024-09-21T22:07:06.923 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010987199999999999 2024-09-21T22:07:06.923 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.923 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.923 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:06.923 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.175719+0000", 2024-09-21T22:07:06.923 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011196600000000001 2024-09-21T22:07:06.923 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:06.923 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:06.923 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:06.923 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.923 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.923 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:07:06.923 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.161483+0000", 2024-09-21T22:07:06.923 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 64.745682102000004, 2024-09-21T22:07:06.924 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016155732999999999, 2024-09-21T22:07:06.924 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:06.924 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:06.924 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:06.924 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:06.924 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:07:06.924 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:07:06.924 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 36 2024-09-21T22:07:06.924 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.924 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:06.924 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.924 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:06.924 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161483+0000", 2024-09-21T22:07:06.924 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:06.924 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.925 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.925 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:06.925 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161483+0000", 2024-09-21T22:07:06.925 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:06.925 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.925 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.925 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:06.925 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161485+0000", 2024-09-21T22:07:06.925 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5999999999999999e-06 2024-09-21T22:07:06.925 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.925 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.925 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:06.925 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161498+0000", 2024-09-21T22:07:06.925 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3166999999999999e-05 2024-09-21T22:07:06.925 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.926 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.926 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:06.926 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161526+0000", 2024-09-21T22:07:06.926 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7786000000000001e-05 2024-09-21T22:07:06.926 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.926 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.926 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:06.926 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161545+0000", 2024-09-21T22:07:06.926 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8899999999999999e-05 2024-09-21T22:07:06.926 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.926 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.926 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:06.926 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.172615+0000", 2024-09-21T22:07:06.927 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011070465 2024-09-21T22:07:06.927 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.927 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.927 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:06.927 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.172759+0000", 2024-09-21T22:07:06.927 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014416900000000001 2024-09-21T22:07:06.927 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.927 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.927 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:06.927 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.172967+0000", 2024-09-21T22:07:06.927 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020758500000000001 2024-09-21T22:07:06.927 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.927 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.927 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:06.928 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.174775+0000", 2024-09-21T22:07:06.928 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0018080990000000001 2024-09-21T22:07:06.928 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.928 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.928 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:06.928 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.177447+0000", 2024-09-21T22:07:06.928 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026717429999999999 2024-09-21T22:07:06.928 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.928 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.928 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:06.928 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.177525+0000", 2024-09-21T22:07:06.928 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7846e-05 2024-09-21T22:07:06.928 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.928 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.929 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:06.929 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.177639+0000", 2024-09-21T22:07:06.929 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000114373 2024-09-21T22:07:06.929 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:06.929 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:06.929 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:06.929 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.929 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.929 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:07:06.929 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.161577+0000", 2024-09-21T22:07:06.929 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 64.745588569000006, 2024-09-21T22:07:06.929 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016652991999999998, 2024-09-21T22:07:06.929 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:06.929 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:06.930 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:06.930 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:06.930 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:07:06.930 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:07:06.930 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 37 2024-09-21T22:07:06.930 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.930 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:06.930 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.930 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:06.930 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161577+0000", 2024-09-21T22:07:06.930 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:06.930 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.930 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.930 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:06.931 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161577+0000", 2024-09-21T22:07:06.931 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:06.931 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.931 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.931 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:06.931 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161578+0000", 2024-09-21T22:07:06.931 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5269999999999999e-06 2024-09-21T22:07:06.931 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.931 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.931 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:06.931 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161591+0000", 2024-09-21T22:07:06.931 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.296e-05 2024-09-21T22:07:06.931 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.931 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.932 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:06.932 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161629+0000", 2024-09-21T22:07:06.932 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7799000000000002e-05 2024-09-21T22:07:06.932 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.932 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.932 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:06.932 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161644+0000", 2024-09-21T22:07:06.932 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5333e-05 2024-09-21T22:07:06.932 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.932 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.932 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:06.932 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.173636+0000", 2024-09-21T22:07:06.932 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011992078 2024-09-21T22:07:06.932 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.933 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.933 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:06.933 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.173862+0000", 2024-09-21T22:07:06.933 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022556799999999999 2024-09-21T22:07:06.933 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.933 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.933 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:06.933 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.174193+0000", 2024-09-21T22:07:06.933 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000331088 2024-09-21T22:07:06.933 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.933 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.933 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:06.933 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.175740+0000", 2024-09-21T22:07:06.933 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015467910000000001 2024-09-21T22:07:06.933 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.934 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.934 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:06.934 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.178073+0000", 2024-09-21T22:07:06.934 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002333356 2024-09-21T22:07:06.934 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.934 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.934 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:06.934 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.178162+0000", 2024-09-21T22:07:06.934 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8306000000000005e-05 2024-09-21T22:07:06.934 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.934 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.934 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:06.934 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.178230+0000", 2024-09-21T22:07:06.935 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8186000000000001e-05 2024-09-21T22:07:06.935 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:06.935 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:06.935 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:06.935 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.935 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.935 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:07:06.935 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.280540+0000", 2024-09-21T22:07:06.935 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 64.626624996999993, 2024-09-21T22:07:06.935 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013246212, 2024-09-21T22:07:06.935 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:06.935 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:06.935 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:06.935 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:06.935 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:07:06.936 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:07:06.936 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 88 2024-09-21T22:07:06.936 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.936 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:06.936 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.936 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:06.936 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.280540+0000", 2024-09-21T22:07:06.936 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:06.936 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.936 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.936 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:06.936 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.280540+0000", 2024-09-21T22:07:06.936 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:06.936 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.937 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.937 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:06.937 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.280541+0000", 2024-09-21T22:07:06.937 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.54e-07 2024-09-21T22:07:06.937 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.937 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.937 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:06.937 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.280549+0000", 2024-09-21T22:07:06.937 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3329999999999997e-06 2024-09-21T22:07:06.937 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.937 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.937 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:06.937 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.280560+0000", 2024-09-21T22:07:06.937 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1003e-05 2024-09-21T22:07:06.938 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.938 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.938 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:06.938 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.280569+0000", 2024-09-21T22:07:06.938 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2430000000000005e-06 2024-09-21T22:07:06.938 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.938 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.938 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:06.938 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.291095+0000", 2024-09-21T22:07:06.938 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010526259 2024-09-21T22:07:06.938 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.938 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.938 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:06.938 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.291231+0000", 2024-09-21T22:07:06.939 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013561899999999999 2024-09-21T22:07:06.939 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.939 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.939 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:06.939 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.291457+0000", 2024-09-21T22:07:06.939 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022686200000000001 2024-09-21T22:07:06.939 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.939 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.939 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:06.939 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.293117+0000", 2024-09-21T22:07:06.939 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001659474 2024-09-21T22:07:06.939 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.939 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.939 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:06.940 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.293524+0000", 2024-09-21T22:07:06.940 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00040701300000000001 2024-09-21T22:07:06.940 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.940 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.940 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:06.940 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.293673+0000", 2024-09-21T22:07:06.940 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000148782 2024-09-21T22:07:06.940 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.940 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.940 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:06.940 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.293786+0000", 2024-09-21T22:07:06.940 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011377 2024-09-21T22:07:06.940 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:06.940 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:06.940 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:06.941 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.941 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.941 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:07:06.941 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.466485+0000", 2024-09-21T22:07:06.941 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 64.440680266000001, 2024-09-21T22:07:06.941 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014443981, 2024-09-21T22:07:06.941 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:06.941 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:06.941 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:06.941 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:06.941 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:07:06.941 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:07:06.941 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 107 2024-09-21T22:07:06.941 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.942 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:06.942 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.942 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:06.942 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.466485+0000", 2024-09-21T22:07:06.942 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:06.942 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.942 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.942 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:06.942 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.466485+0000", 2024-09-21T22:07:06.942 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:06.942 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.942 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.942 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:06.942 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.466486+0000", 2024-09-21T22:07:06.943 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6299999999999993e-07 2024-09-21T22:07:06.943 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.943 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.943 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:06.943 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.466493+0000", 2024-09-21T22:07:06.943 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9639999999999997e-06 2024-09-21T22:07:06.943 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.943 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.943 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:06.943 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.466507+0000", 2024-09-21T22:07:06.943 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4216000000000001e-05 2024-09-21T22:07:06.943 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.943 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.943 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:06.943 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.466515+0000", 2024-09-21T22:07:06.944 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.157e-06 2024-09-21T22:07:06.944 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.944 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.944 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:06.944 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.472033+0000", 2024-09-21T22:07:06.944 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0055175839999999999 2024-09-21T22:07:06.944 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.944 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.944 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:06.944 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.472174+0000", 2024-09-21T22:07:06.944 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000141612 2024-09-21T22:07:06.944 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.944 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.944 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:06.944 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.472381+0000", 2024-09-21T22:07:06.945 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000206961 2024-09-21T22:07:06.945 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.945 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.945 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:06.945 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.473924+0000", 2024-09-21T22:07:06.945 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015427119999999999 2024-09-21T22:07:06.945 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.945 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.945 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:06.945 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.480666+0000", 2024-09-21T22:07:06.945 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0067415410000000002 2024-09-21T22:07:06.945 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.945 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.945 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:06.946 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.480811+0000", 2024-09-21T22:07:06.946 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000145422 2024-09-21T22:07:06.946 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.946 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.946 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:06.946 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.480929+0000", 2024-09-21T22:07:06.946 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000117849 2024-09-21T22:07:06.946 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:06.946 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:06.946 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:06.946 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.946 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.946 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:07:06.946 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.467057+0000", 2024-09-21T22:07:06.946 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 64.440108007000006, 2024-09-21T22:07:06.947 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018350195, 2024-09-21T22:07:06.947 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:06.947 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:06.947 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:06.947 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:06.947 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:07:06.947 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:07:06.947 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 108 2024-09-21T22:07:06.947 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.947 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:06.947 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.947 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:06.947 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467057+0000", 2024-09-21T22:07:06.947 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:06.948 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.948 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.948 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:06.948 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467057+0000", 2024-09-21T22:07:06.948 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:06.948 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.948 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.948 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:06.948 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467059+0000", 2024-09-21T22:07:06.948 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.826e-06 2024-09-21T22:07:06.948 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.948 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.948 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:06.948 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467071+0000", 2024-09-21T22:07:06.948 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2224e-05 2024-09-21T22:07:06.949 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.949 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.949 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:06.949 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467098+0000", 2024-09-21T22:07:06.949 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6953000000000001e-05 2024-09-21T22:07:06.949 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.949 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.949 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:06.949 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467113+0000", 2024-09-21T22:07:06.949 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5119999999999999e-05 2024-09-21T22:07:06.949 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.949 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.949 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:06.949 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.474060+0000", 2024-09-21T22:07:06.950 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0069462919999999997 2024-09-21T22:07:06.950 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.950 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.950 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:06.950 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.474283+0000", 2024-09-21T22:07:06.950 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000223315 2024-09-21T22:07:06.950 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.950 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.950 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:06.950 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.474660+0000", 2024-09-21T22:07:06.950 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00037729400000000002 2024-09-21T22:07:06.950 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.950 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.950 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:06.950 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.480622+0000", 2024-09-21T22:07:06.951 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0059613369999999997 2024-09-21T22:07:06.951 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.951 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.951 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:06.951 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.485145+0000", 2024-09-21T22:07:06.951 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0045231960000000002 2024-09-21T22:07:06.951 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.951 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.951 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:06.951 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.485288+0000", 2024-09-21T22:07:06.951 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000143232 2024-09-21T22:07:06.951 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.951 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.951 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:06.952 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.485407+0000", 2024-09-21T22:07:06.952 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000119406 2024-09-21T22:07:06.952 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:06.952 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:06.952 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:06.952 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.952 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.952 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:07:06.952 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.467144+0000", 2024-09-21T22:07:06.952 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 64.440020787999998, 2024-09-21T22:07:06.952 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018907146999999999, 2024-09-21T22:07:06.952 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:06.952 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:06.952 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:06.953 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:06.953 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:07:06.953 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:07:06.953 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 109 2024-09-21T22:07:06.953 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.953 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:06.953 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.953 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:06.953 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467144+0000", 2024-09-21T22:07:06.953 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:06.953 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.953 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.953 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:06.953 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467144+0000", 2024-09-21T22:07:06.954 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:06.954 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.954 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.954 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:06.954 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467145+0000", 2024-09-21T22:07:06.954 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9999999999999996e-07 2024-09-21T22:07:06.954 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.954 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.954 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:06.954 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467331+0000", 2024-09-21T22:07:06.954 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018518899999999999 2024-09-21T22:07:06.954 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.954 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.954 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:06.955 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467350+0000", 2024-09-21T22:07:06.955 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8966000000000001e-05 2024-09-21T22:07:06.955 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.955 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.955 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:06.955 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467361+0000", 2024-09-21T22:07:06.955 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1233000000000001e-05 2024-09-21T22:07:06.955 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.955 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.955 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:06.955 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.475613+0000", 2024-09-21T22:07:06.955 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0082518629999999999 2024-09-21T22:07:06.955 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.955 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.955 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:06.956 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.475859+0000", 2024-09-21T22:07:06.956 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024633800000000003 2024-09-21T22:07:06.956 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.956 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.956 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:06.956 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.476223+0000", 2024-09-21T22:07:06.956 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00036408700000000003 2024-09-21T22:07:06.956 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.956 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.956 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:06.956 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.481046+0000", 2024-09-21T22:07:06.956 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.004823356 2024-09-21T22:07:06.956 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.956 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.956 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:06.957 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.485722+0000", 2024-09-21T22:07:06.957 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0046753640000000004 2024-09-21T22:07:06.957 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.957 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.957 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:06.957 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.485908+0000", 2024-09-21T22:07:06.957 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000185859 2024-09-21T22:07:06.957 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.957 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.957 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:06.957 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.486052+0000", 2024-09-21T22:07:06.957 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014399200000000001 2024-09-21T22:07:06.957 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:06.957 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:06.958 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:06.958 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.958 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.958 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:07:06.958 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.467384+0000", 2024-09-21T22:07:06.958 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 64.439780959999993, 2024-09-21T22:07:06.958 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.01894144, 2024-09-21T22:07:06.958 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:06.958 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:06.958 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:06.958 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:06.958 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:07:06.958 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:07:06.958 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 110 2024-09-21T22:07:06.958 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.959 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:06.959 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.959 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:06.959 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467384+0000", 2024-09-21T22:07:06.959 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:06.959 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.959 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.959 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:06.959 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467384+0000", 2024-09-21T22:07:06.959 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:06.959 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.959 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.959 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:06.959 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467385+0000", 2024-09-21T22:07:06.960 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0470000000000001e-06 2024-09-21T22:07:06.960 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.960 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.960 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:06.960 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467393+0000", 2024-09-21T22:07:06.960 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3869999999999997e-06 2024-09-21T22:07:06.960 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.960 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.960 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:06.960 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467413+0000", 2024-09-21T22:07:06.960 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9953e-05 2024-09-21T22:07:06.960 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.960 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.960 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:06.961 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467421+0000", 2024-09-21T22:07:06.961 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2400000000000007e-06 2024-09-21T22:07:06.961 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.961 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.961 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:06.961 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.479542+0000", 2024-09-21T22:07:06.961 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012120832 2024-09-21T22:07:06.961 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.961 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.961 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:06.961 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.479705+0000", 2024-09-21T22:07:06.961 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016362899999999999 2024-09-21T22:07:06.961 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.961 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.961 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:06.962 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.479914+0000", 2024-09-21T22:07:06.962 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020814499999999999 2024-09-21T22:07:06.962 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.962 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.962 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:06.962 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.485536+0000", 2024-09-21T22:07:06.962 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0056222800000000003 2024-09-21T22:07:06.962 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.962 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.962 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:06.962 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.486077+0000", 2024-09-21T22:07:06.962 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00054079900000000001 2024-09-21T22:07:06.962 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.962 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.963 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:06.963 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.486213+0000", 2024-09-21T22:07:06.963 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000136682 2024-09-21T22:07:06.963 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.963 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.963 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:06.963 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.486326+0000", 2024-09-21T22:07:06.963 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000112446 2024-09-21T22:07:06.963 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:06.963 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:06.963 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:06.963 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.963 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.963 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:07:06.963 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.467436+0000", 2024-09-21T22:07:06.964 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 64.439729459999995, 2024-09-21T22:07:06.964 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019152168000000001, 2024-09-21T22:07:06.964 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:06.964 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:06.964 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:06.964 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:06.964 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:07:06.964 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:07:06.964 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 111 2024-09-21T22:07:06.964 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.964 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:06.964 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.964 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:06.964 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467436+0000", 2024-09-21T22:07:06.965 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:06.965 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.965 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.965 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:06.965 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467436+0000", 2024-09-21T22:07:06.965 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:06.965 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.965 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.965 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:06.965 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467437+0000", 2024-09-21T22:07:06.965 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0699999999999996e-07 2024-09-21T22:07:06.965 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.965 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.965 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:06.966 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467445+0000", 2024-09-21T22:07:06.966 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.789e-06 2024-09-21T22:07:06.966 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.966 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.966 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:06.966 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467467+0000", 2024-09-21T22:07:06.966 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1253000000000001e-05 2024-09-21T22:07:06.966 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.966 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.966 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:06.966 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467475+0000", 2024-09-21T22:07:06.966 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0039999999999999e-06 2024-09-21T22:07:06.966 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.966 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.967 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:06.967 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.481164+0000", 2024-09-21T22:07:06.967 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013689114 2024-09-21T22:07:06.967 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.967 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.967 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:06.967 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.481394+0000", 2024-09-21T22:07:06.967 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000229931 2024-09-21T22:07:06.967 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.967 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.967 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:06.967 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.481779+0000", 2024-09-21T22:07:06.967 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00038545400000000003 2024-09-21T22:07:06.967 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.968 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.968 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:06.968 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.485675+0000", 2024-09-21T22:07:06.968 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0038958370000000001 2024-09-21T22:07:06.968 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.968 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.968 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:06.968 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.486346+0000", 2024-09-21T22:07:06.968 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00067119100000000002 2024-09-21T22:07:06.968 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.968 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.968 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:06.968 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.486478+0000", 2024-09-21T22:07:06.968 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013182600000000001 2024-09-21T22:07:06.968 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.969 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.969 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:06.969 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.486588+0000", 2024-09-21T22:07:06.969 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000110062 2024-09-21T22:07:06.969 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:06.969 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:06.969 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:06.969 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.969 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.969 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:07:06.969 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.467488+0000", 2024-09-21T22:07:06.969 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 64.439676833999997, 2024-09-21T22:07:06.969 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021038304000000001, 2024-09-21T22:07:06.969 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:06.969 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:06.970 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:06.970 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:06.970 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:07:06.970 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:07:06.970 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 112 2024-09-21T22:07:06.970 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.970 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:06.970 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.970 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:06.970 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467488+0000", 2024-09-21T22:07:06.970 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:06.970 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.970 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.970 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:06.971 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467488+0000", 2024-09-21T22:07:06.971 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:06.971 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.971 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.971 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:06.971 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467489+0000", 2024-09-21T22:07:06.971 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7999999999999995e-07 2024-09-21T22:07:06.971 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.971 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.971 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:06.971 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467687+0000", 2024-09-21T22:07:06.971 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019776200000000001 2024-09-21T22:07:06.971 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.971 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.972 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:06.972 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467704+0000", 2024-09-21T22:07:06.972 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7193e-05 2024-09-21T22:07:06.972 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.972 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.972 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:06.972 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467719+0000", 2024-09-21T22:07:06.972 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4703e-05 2024-09-21T22:07:06.972 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.972 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.972 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:06.972 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.482745+0000", 2024-09-21T22:07:06.972 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015025801 2024-09-21T22:07:06.972 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.972 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.973 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:06.973 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.483015+0000", 2024-09-21T22:07:06.973 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026999400000000002 2024-09-21T22:07:06.973 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.973 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.973 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:06.973 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.483440+0000", 2024-09-21T22:07:06.973 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000425697 2024-09-21T22:07:06.973 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.973 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.973 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:06.973 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.486701+0000", 2024-09-21T22:07:06.973 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0032611950000000002 2024-09-21T22:07:06.973 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.974 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.974 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:06.974 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.488262+0000", 2024-09-21T22:07:06.974 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015610439999999999 2024-09-21T22:07:06.974 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.974 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.974 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:06.974 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.488406+0000", 2024-09-21T22:07:06.974 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000143232 2024-09-21T22:07:06.974 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.974 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.974 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:06.974 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.488527+0000", 2024-09-21T22:07:06.974 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012100299999999999 2024-09-21T22:07:06.975 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:06.975 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:06.975 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:06.975 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.975 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.975 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:07:06.975 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.467786+0000", 2024-09-21T22:07:06.975 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 64.439379408999997, 2024-09-21T22:07:06.975 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026339741999999999, 2024-09-21T22:07:06.975 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:06.975 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:06.975 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:06.975 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:06.975 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:07:06.976 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:07:06.976 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 113 2024-09-21T22:07:06.976 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.976 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:06.976 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.976 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:06.976 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467786+0000", 2024-09-21T22:07:06.976 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:06.976 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.976 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.976 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:06.976 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467786+0000", 2024-09-21T22:07:06.976 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:06.976 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.976 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.977 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:06.977 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467787+0000", 2024-09-21T22:07:06.977 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.446e-06 2024-09-21T22:07:06.977 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.977 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.977 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:06.977 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467802+0000", 2024-09-21T22:07:06.977 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4769999999999999e-05 2024-09-21T22:07:06.977 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.977 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.977 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:06.977 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467826+0000", 2024-09-21T22:07:06.977 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3430000000000001e-05 2024-09-21T22:07:06.977 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.978 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.978 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:06.978 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467843+0000", 2024-09-21T22:07:06.978 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7065999999999999e-05 2024-09-21T22:07:06.978 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.978 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.978 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:06.978 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.486820+0000", 2024-09-21T22:07:06.978 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018977233999999999 2024-09-21T22:07:06.978 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.978 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.978 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:06.978 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.487065+0000", 2024-09-21T22:07:06.978 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024558799999999998 2024-09-21T22:07:06.979 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.979 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.979 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:06.979 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.487442+0000", 2024-09-21T22:07:06.979 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000376707 2024-09-21T22:07:06.979 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.979 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.979 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:06.979 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.492256+0000", 2024-09-21T22:07:06.979 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0048141620000000003 2024-09-21T22:07:06.979 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.979 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.979 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:06.979 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.493865+0000", 2024-09-21T22:07:06.980 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016090150000000001 2024-09-21T22:07:06.980 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.980 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.980 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:06.980 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.494007+0000", 2024-09-21T22:07:06.980 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000141359 2024-09-21T22:07:06.980 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.980 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.980 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:06.980 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.494126+0000", 2024-09-21T22:07:06.980 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000118965 2024-09-21T22:07:06.980 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:06.980 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:06.980 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:06.981 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.981 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.981 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:07:06.981 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.467878+0000", 2024-09-21T22:07:06.981 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 64.439286920000001, 2024-09-21T22:07:06.981 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.027783958000000001, 2024-09-21T22:07:06.981 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:06.981 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:06.981 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:06.981 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:06.981 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:07:06.981 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:07:06.981 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 114 2024-09-21T22:07:06.981 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.981 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:06.982 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.982 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:06.982 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467878+0000", 2024-09-21T22:07:06.982 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:06.982 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.982 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.982 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:06.982 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467878+0000", 2024-09-21T22:07:06.982 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:06.982 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.982 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.982 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:06.982 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467880+0000", 2024-09-21T22:07:06.982 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.223e-06 2024-09-21T22:07:06.983 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.983 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.983 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:06.983 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467891+0000", 2024-09-21T22:07:06.983 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1527e-05 2024-09-21T22:07:06.983 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.983 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.983 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:06.983 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467913+0000", 2024-09-21T22:07:06.983 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.143e-05 2024-09-21T22:07:06.983 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.983 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.983 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:06.983 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467930+0000", 2024-09-21T22:07:06.984 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7620000000000001e-05 2024-09-21T22:07:06.984 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.984 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.984 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:06.984 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.488690+0000", 2024-09-21T22:07:06.984 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020759588999999998 2024-09-21T22:07:06.984 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.984 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.984 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:06.984 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.488882+0000", 2024-09-21T22:07:06.984 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019236200000000001 2024-09-21T22:07:06.984 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.984 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.984 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:06.985 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.489093+0000", 2024-09-21T22:07:06.985 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000211318 2024-09-21T22:07:06.985 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.985 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.985 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:06.985 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.494152+0000", 2024-09-21T22:07:06.985 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0050582769999999999 2024-09-21T22:07:06.985 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.985 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.985 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:06.985 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.495373+0000", 2024-09-21T22:07:06.985 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001221747 2024-09-21T22:07:06.985 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.985 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.986 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:06.986 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.495543+0000", 2024-09-21T22:07:06.986 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000169259 2024-09-21T22:07:06.986 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.986 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.986 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:06.986 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.495662+0000", 2024-09-21T22:07:06.986 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011960600000000001 2024-09-21T22:07:06.986 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:06.986 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:06.986 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:06.986 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.986 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.986 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:07:06.986 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.468118+0000", 2024-09-21T22:07:06.987 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 64.439047178999999, 2024-09-21T22:07:06.987 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.027938877000000001, 2024-09-21T22:07:06.987 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:06.987 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:06.987 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:06.987 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:06.987 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:07:06.987 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:07:06.987 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 115 2024-09-21T22:07:06.987 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.987 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:06.987 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.987 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:06.987 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468118+0000", 2024-09-21T22:07:06.988 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:06.988 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.988 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.988 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:06.988 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468118+0000", 2024-09-21T22:07:06.988 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:06.988 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.988 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.988 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:06.988 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468120+0000", 2024-09-21T22:07:06.988 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7069999999999999e-06 2024-09-21T22:07:06.988 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.988 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.988 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:06.988 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468381+0000", 2024-09-21T22:07:06.989 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026135399999999999 2024-09-21T22:07:06.989 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.989 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.989 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:06.989 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468409+0000", 2024-09-21T22:07:06.989 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.781e-05 2024-09-21T22:07:06.989 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.989 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.989 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:06.989 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468433+0000", 2024-09-21T22:07:06.989 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4343999999999999e-05 2024-09-21T22:07:06.989 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.989 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.989 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:06.990 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.489785+0000", 2024-09-21T22:07:06.990 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021351221 2024-09-21T22:07:06.990 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.990 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.990 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:06.990 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.490018+0000", 2024-09-21T22:07:06.990 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023317800000000001 2024-09-21T22:07:06.990 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.990 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.990 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:06.990 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.490412+0000", 2024-09-21T22:07:06.990 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00039417400000000001 2024-09-21T22:07:06.990 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.990 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.990 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:06.991 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.495694+0000", 2024-09-21T22:07:06.991 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.005282335 2024-09-21T22:07:06.991 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.991 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.991 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:06.991 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.495852+0000", 2024-09-21T22:07:06.991 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015777200000000001 2024-09-21T22:07:06.991 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.991 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.991 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:06.991 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.495938+0000", 2024-09-21T22:07:06.991 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6493e-05 2024-09-21T22:07:06.991 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.991 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.992 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:06.992 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.496057+0000", 2024-09-21T22:07:06.992 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000118489 2024-09-21T22:07:06.992 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:06.992 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:06.992 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:06.992 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.992 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.992 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:07:06.992 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.468474+0000", 2024-09-21T22:07:06.992 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 64.438691418000005, 2024-09-21T22:07:06.992 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.028015895999999998, 2024-09-21T22:07:06.992 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:06.992 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:06.993 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:06.993 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:06.993 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:07:06.993 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:07:06.993 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 116 2024-09-21T22:07:06.993 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.993 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:06.993 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.993 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:06.993 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468474+0000", 2024-09-21T22:07:06.993 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:06.993 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.993 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.993 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:06.993 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468474+0000", 2024-09-21T22:07:06.994 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:06.994 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.994 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.994 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:06.994 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468476+0000", 2024-09-21T22:07:06.994 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7060000000000001e-06 2024-09-21T22:07:06.994 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.994 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.994 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:06.994 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468488+0000", 2024-09-21T22:07:06.994 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.234e-05 2024-09-21T22:07:06.994 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.994 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.994 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:06.994 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468517+0000", 2024-09-21T22:07:06.995 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.898e-05 2024-09-21T22:07:06.995 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.995 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.995 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:06.995 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468536+0000", 2024-09-21T22:07:06.995 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8899999999999999e-05 2024-09-21T22:07:06.995 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.995 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.995 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:06.995 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.492376+0000", 2024-09-21T22:07:06.995 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023840659 2024-09-21T22:07:06.995 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.995 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.995 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:06.996 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.492621+0000", 2024-09-21T22:07:06.996 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000244794 2024-09-21T22:07:06.996 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.996 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.996 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:06.996 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.493008+0000", 2024-09-21T22:07:06.996 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00038629099999999998 2024-09-21T22:07:06.996 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.996 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.996 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:06.996 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.496192+0000", 2024-09-21T22:07:06.996 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0031847220000000001 2024-09-21T22:07:06.996 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.996 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.996 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:06.997 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.496233+0000", 2024-09-21T22:07:06.997 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0410000000000001e-05 2024-09-21T22:07:06.997 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.997 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.997 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:06.997 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.496374+0000", 2024-09-21T22:07:06.997 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140872 2024-09-21T22:07:06.997 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.997 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.997 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:06.997 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.496490+0000", 2024-09-21T22:07:06.997 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000116222 2024-09-21T22:07:06.997 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:06.997 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:06.998 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:06.998 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.998 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.998 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:07:06.998 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.468568+0000", 2024-09-21T22:07:06.998 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 64.438597298999994, 2024-09-21T22:07:06.998 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.028181578999999998, 2024-09-21T22:07:06.998 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:06.998 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:06.998 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:06.998 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:06.998 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:07:06.998 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:07:06.998 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 117 2024-09-21T22:07:06.998 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.999 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:06.999 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.999 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:06.999 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468568+0000", 2024-09-21T22:07:06.999 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:06.999 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.999 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.999 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:06.999 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468568+0000", 2024-09-21T22:07:06.999 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:06.999 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:06.999 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:06.999 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:06.999 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468569+0000", 2024-09-21T22:07:07.000 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4139999999999999e-06 2024-09-21T22:07:07.000 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:07.000 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:07.000 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:07.000 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468583+0000", 2024-09-21T22:07:07.000 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3409e-05 2024-09-21T22:07:07.000 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:07.000 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:07.000 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:07.000 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468608+0000", 2024-09-21T22:07:07.000 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5559999999999999e-05 2024-09-21T22:07:07.000 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:07.000 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:07.000 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:07.000 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468626+0000", 2024-09-21T22:07:07.001 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7863000000000002e-05 2024-09-21T22:07:07.001 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:07.001 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:07.001 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:07.001 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.494260+0000", 2024-09-21T22:07:07.001 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025633744999999999 2024-09-21T22:07:07.001 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:07.001 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:07.001 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:07.001 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.494519+0000", 2024-09-21T22:07:07.001 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025924800000000001 2024-09-21T22:07:07.001 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:07.001 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:07.001 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:07.001 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.494747+0000", 2024-09-21T22:07:07.002 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022792799999999999 2024-09-21T22:07:07.002 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:07.002 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:07.002 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:07.002 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.496519+0000", 2024-09-21T22:07:07.002 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017720030000000001 2024-09-21T22:07:07.002 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:07.002 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:07.002 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:07.002 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.496607+0000", 2024-09-21T22:07:07.002 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7873000000000004e-05 2024-09-21T22:07:07.002 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:07.002 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:07.002 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:07.003 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.496691+0000", 2024-09-21T22:07:07.003 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4009000000000002e-05 2024-09-21T22:07:07.003 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:07.003 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:07.003 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:07.003 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.496750+0000", 2024-09-21T22:07:07.003 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8526999999999997e-05 2024-09-21T22:07:07.003 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:07.003 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:07.003 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:07.003 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:07.003 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:07.003 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:07:07.003 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.898244+0000", 2024-09-21T22:07:07.003 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 64.008920996000001, 2024-09-21T22:07:07.004 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014749464, 2024-09-21T22:07:07.004 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:07.004 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:07.004 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:07.004 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:07.004 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:07:07.004 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:07:07.004 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 140 2024-09-21T22:07:07.004 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:07.004 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:07.004 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:07.004 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:07.004 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898244+0000", 2024-09-21T22:07:07.004 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:07.005 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:07.005 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:07.005 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:07.005 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898244+0000", 2024-09-21T22:07:07.005 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:07.005 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:07.005 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:07.005 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:07.005 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898245+0000", 2024-09-21T22:07:07.005 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3000000000000002e-07 2024-09-21T22:07:07.005 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:07.005 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:07.005 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:07.005 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898308+0000", 2024-09-21T22:07:07.005 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3468999999999998e-05 2024-09-21T22:07:07.006 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:07.006 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:07.006 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:07.006 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898315+0000", 2024-09-21T22:07:07.006 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.725e-06 2024-09-21T22:07:07.006 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:07.006 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:07.006 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:07.006 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898321+0000", 2024-09-21T22:07:07.006 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2090000000000002e-06 2024-09-21T22:07:07.006 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:07.006 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:07.006 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:07.006 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.905452+0000", 2024-09-21T22:07:07.007 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0071305989999999996 2024-09-21T22:07:07.007 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:07.007 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:07.007 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:07.007 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.905504+0000", 2024-09-21T22:07:07.007 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2638e-05 2024-09-21T22:07:07.007 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:07.007 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:07.007 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:07.007 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.905590+0000", 2024-09-21T22:07:07.007 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5333000000000004e-05 2024-09-21T22:07:07.007 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:07.007 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:07.007 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:07.007 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.911542+0000", 2024-09-21T22:07:07.008 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0059519819999999998 2024-09-21T22:07:07.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:07.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:07.008 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:07.008 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.912908+0000", 2024-09-21T22:07:07.008 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0013666640000000001 2024-09-21T22:07:07.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:07.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:07.008 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:07.008 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.912957+0000", 2024-09-21T22:07:07.008 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9203000000000003e-05 2024-09-21T22:07:07.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:07.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:07.008 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:07.009 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.912994+0000", 2024-09-21T22:07:07.009 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6312000000000001e-05 2024-09-21T22:07:07.009 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:07.009 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:07.009 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:07.009 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:07.009 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:07.009 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:07:07.009 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.898342+0000", 2024-09-21T22:07:07.009 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 64.008822858000002, 2024-09-21T22:07:07.009 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014787918000000001, 2024-09-21T22:07:07.009 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:07.009 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:07.009 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:07.010 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:07.010 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:07:07.010 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:07:07.010 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 141 2024-09-21T22:07:07.010 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:07.010 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:07.010 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:07.010 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:07.010 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898342+0000", 2024-09-21T22:07:07.010 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:07.010 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:07.010 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:07.010 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:07.010 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898342+0000", 2024-09-21T22:07:07.011 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:07.011 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:07.011 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:07.011 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:07.011 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898343+0000", 2024-09-21T22:07:07.011 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3000000000000001e-07 2024-09-21T22:07:07.011 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:07.011 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:07.011 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:07.011 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898346+0000", 2024-09-21T22:07:07.011 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6009999999999999e-06 2024-09-21T22:07:07.011 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:07.011 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:07.011 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:07.012 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898352+0000", 2024-09-21T22:07:07.012 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8089999999999998e-06 2024-09-21T22:07:07.012 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:07.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:07.012 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:07.012 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898358+0000", 2024-09-21T22:07:07.012 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4349999999999996e-06 2024-09-21T22:07:07.012 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:07.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:07.012 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:07.012 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.909387+0000", 2024-09-21T22:07:07.012 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011029105000000001 2024-09-21T22:07:07.012 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:07.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:07.012 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:07.013 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.909462+0000", 2024-09-21T22:07:07.013 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5192999999999996e-05 2024-09-21T22:07:07.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:07.013 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:07.013 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:07.013 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.909551+0000", 2024-09-21T22:07:07.013 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9046000000000001e-05 2024-09-21T22:07:07.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:07.013 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:07.013 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:07.013 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.911852+0000", 2024-09-21T22:07:07.013 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0023005579999999999 2024-09-21T22:07:07.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:07.013 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:07.014 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:07.014 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.913002+0000", 2024-09-21T22:07:07.014 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011502929999999999 2024-09-21T22:07:07.014 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:07.014 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:07.014 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:07.014 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.913060+0000", 2024-09-21T22:07:07.014 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8427000000000002e-05 2024-09-21T22:07:07.014 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:07.014 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:07.014 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:07.014 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.913130+0000", 2024-09-21T22:07:07.014 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0020999999999999e-05 2024-09-21T22:07:07.014 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:07.015 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:07.015 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:07.015 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:07.015 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:07.015 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:158 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T22:07:07.015 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:03.511201+0000", 2024-09-21T22:07:07.015 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 63.395964374999998, 2024-09-21T22:07:07.015 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013353538999999999, 2024-09-21T22:07:07.015 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:07.015 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:07.015 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:07.015 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:07.015 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:07:07.015 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:07:07.015 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 158 2024-09-21T22:07:07.016 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:07.016 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:07.016 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:07.016 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:07.016 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511201+0000", 2024-09-21T22:07:07.016 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:07.016 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:07.016 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:07.016 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:07.016 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511201+0000", 2024-09-21T22:07:07.016 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:07.016 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:07.016 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:07.016 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:07.016 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511202+0000", 2024-09-21T22:07:07.017 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9299999999999997e-07 2024-09-21T22:07:07.017 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:07.017 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:07.017 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:07.017 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511315+0000", 2024-09-21T22:07:07.017 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000113909 2024-09-21T22:07:07.017 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:07.017 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:07.017 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:07.017 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511333+0000", 2024-09-21T22:07:07.017 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7062999999999999e-05 2024-09-21T22:07:07.017 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:07.017 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:07.017 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:07.018 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511344+0000", 2024-09-21T22:07:07.018 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1177e-05 2024-09-21T22:07:07.018 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:07.018 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:07.018 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:07.018 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.517415+0000", 2024-09-21T22:07:07.018 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0060709830000000003 2024-09-21T22:07:07.018 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:07.018 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:07.018 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:07.018 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.517539+0000", 2024-09-21T22:07:07.018 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000124432 2024-09-21T22:07:07.018 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:07.018 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:07.019 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:07.019 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.517732+0000", 2024-09-21T22:07:07.019 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019262200000000001 2024-09-21T22:07:07.019 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:07.019 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:07.019 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:07.019 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.520709+0000", 2024-09-21T22:07:07.019 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0029767470000000001 2024-09-21T22:07:07.019 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:07.019 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:07.019 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:07.019 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.524389+0000", 2024-09-21T22:07:07.019 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003680975 2024-09-21T22:07:07.019 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:07.019 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:07.020 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:07.020 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.524486+0000", 2024-09-21T22:07:07.020 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6575000000000003e-05 2024-09-21T22:07:07.020 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:07.020 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:07.020 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:07.020 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.524554+0000", 2024-09-21T22:07:07.020 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8362999999999996e-05 2024-09-21T22:07:07.020 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:07.020 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:07.020 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:07.020 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:07.020 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:07.020 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T22:07:07.020 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:03.511368+0000", 2024-09-21T22:07:07.021 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 63.395797473000002, 2024-09-21T22:07:07.021 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013312133, 2024-09-21T22:07:07.021 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:07.021 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:07.021 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:07.021 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:07.021 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:07:07.021 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:07:07.021 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 159 2024-09-21T22:07:07.021 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:07.021 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:07.021 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:07.021 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:07.021 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511368+0000", 2024-09-21T22:07:07.022 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:07.022 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:07.022 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:07.022 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:07.022 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511368+0000", 2024-09-21T22:07:07.022 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:07.022 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:07.022 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:07.022 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:07.022 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511369+0000", 2024-09-21T22:07:07.022 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9599999999999998e-07 2024-09-21T22:07:07.022 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:07.022 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:07.022 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:07.022 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511382+0000", 2024-09-21T22:07:07.023 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3487e-05 2024-09-21T22:07:07.023 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:07.023 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:07.023 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:07.023 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511394+0000", 2024-09-21T22:07:07.023 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1337e-05 2024-09-21T22:07:07.023 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:07.023 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:07.023 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:07.023 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511402+0000", 2024-09-21T22:07:07.023 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0059999999999992e-06 2024-09-21T22:07:07.023 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:07.023 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:07.023 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:07.023 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.519376+0000", 2024-09-21T22:07:07.024 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0079746520000000005 2024-09-21T22:07:07.024 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:07.024 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:07.024 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:07.024 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.519498+0000", 2024-09-21T22:07:07.024 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012212900000000001 2024-09-21T22:07:07.024 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:07.024 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:07.024 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:07.024 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.519728+0000", 2024-09-21T22:07:07.024 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022985799999999999 2024-09-21T22:07:07.024 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:07.024 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:07.024 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:07.025 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.521344+0000", 2024-09-21T22:07:07.025 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016163169999999999 2024-09-21T22:07:07.025 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:07.025 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:07.025 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:07.025 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.524568+0000", 2024-09-21T22:07:07.025 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0032234220000000001 2024-09-21T22:07:07.025 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:07.025 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:07.025 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:07.025 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.524633+0000", 2024-09-21T22:07:07.025 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5126e-05 2024-09-21T22:07:07.025 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:07.025 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:07.025 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:07.025 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.524680+0000", 2024-09-21T22:07:07.026 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6903000000000001e-05 2024-09-21T22:07:07.026 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:07.026 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:07.026 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:07.026 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:07.026 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:07.026 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:07:07.027 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T22:07:07.028 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:07.023+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:07.046 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-21T22:07:07.128 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:07.123+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:07.202 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:07:07.202 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:07:07.202 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-09-21T22:07:07.202 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:07:07.216 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T22:07:07.263 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:07.260+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:07.330 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:07.323+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:07.340 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:07:07.340 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:07:07.340 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-21T22:07:07.340 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-21T22:07:07.341 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:07:07.356 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T22:07:07.431 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:07.427+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:07.528 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:07:07.528 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-21T22:07:07.528 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-21T22:07:07.528 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [] 2024-09-21T22:07:07.529 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:07:07.532 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:07.527+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:07.543 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T22:07:07.580 INFO:tasks.workunit.client.0.smithi157.stderr:Note: switching to '4e45e6b490e6cbe082cdf155fea780438ec53fc7'. 2024-09-21T22:07:07.580 INFO:tasks.workunit.client.0.smithi157.stderr: 2024-09-21T22:07:07.580 INFO:tasks.workunit.client.0.smithi157.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-21T22:07:07.580 INFO:tasks.workunit.client.0.smithi157.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-21T22:07:07.580 INFO:tasks.workunit.client.0.smithi157.stderr:state without impacting any branches by switching back to a branch. 2024-09-21T22:07:07.580 INFO:tasks.workunit.client.0.smithi157.stderr: 2024-09-21T22:07:07.580 INFO:tasks.workunit.client.0.smithi157.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-21T22:07:07.581 INFO:tasks.workunit.client.0.smithi157.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-21T22:07:07.581 INFO:tasks.workunit.client.0.smithi157.stderr: 2024-09-21T22:07:07.581 INFO:tasks.workunit.client.0.smithi157.stderr: git switch -c 2024-09-21T22:07:07.581 INFO:tasks.workunit.client.0.smithi157.stderr: 2024-09-21T22:07:07.581 INFO:tasks.workunit.client.0.smithi157.stderr:Or undo this operation with: 2024-09-21T22:07:07.581 INFO:tasks.workunit.client.0.smithi157.stderr: 2024-09-21T22:07:07.581 INFO:tasks.workunit.client.0.smithi157.stderr: git switch - 2024-09-21T22:07:07.581 INFO:tasks.workunit.client.0.smithi157.stderr: 2024-09-21T22:07:07.581 INFO:tasks.workunit.client.0.smithi157.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-21T22:07:07.581 INFO:tasks.workunit.client.0.smithi157.stderr: 2024-09-21T22:07:07.581 INFO:tasks.workunit.client.0.smithi157.stderr:HEAD is now at 4e45e6b490e Merge pull request #59907 from zdover23/wip-doc-2024-09-21-backport-59904-to-squid 2024-09-21T22:07:07.587 DEBUG:teuthology.orchestra.run.smithi157:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-21T22:07:07.633 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:07.632+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:07.634 INFO:tasks.workunit.client.0.smithi157.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-21T22:07:07.636 INFO:tasks.workunit.client.0.smithi157.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-21T22:07:07.636 INFO:tasks.workunit.client.0.smithi157.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-21T22:07:07.675 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:07:07.675 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:07:07.675 INFO:teuthology.orchestra.run.smithi039.stdout: "num_ops": 0 2024-09-21T22:07:07.675 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:07:07.689 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T22:07:07.735 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:07.731+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:07.816 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:07:07.816 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:07:07.816 INFO:teuthology.orchestra.run.smithi039.stdout: "complaint_time": 30, 2024-09-21T22:07:07.816 INFO:teuthology.orchestra.run.smithi039.stdout: "num_blocked_ops": 0 2024-09-21T22:07:07.816 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:07:07.831 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T22:07:07.878 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:07.871+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:07.946 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:07.939+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:08.020 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:07:08.020 INFO:teuthology.orchestra.run.smithi039.stdout: "size": 20, 2024-09-21T22:07:08.020 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 600, 2024-09-21T22:07:08.020 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [ 2024-09-21T22:07:08.020 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.020 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:20 1.0 e14/13 1:30306672:devicehealth::main.db.0000000000000000:head v 14'10, mlcod=14'8)", 2024-09-21T22:07:08.020 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.137639+0000", 2024-09-21T22:07:08.021 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 65.870558353000007, 2024-09-21T22:07:08.021 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0039902100000000001, 2024-09-21T22:07:08.021 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:08.021 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:08.021 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:08.021 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:08.021 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.021 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:08.021 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.137639+0000", 2024-09-21T22:07:08.021 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:08.021 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.021 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.021 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:08.022 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.137639+0000", 2024-09-21T22:07:08.022 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:08.022 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.022 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.022 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:08.022 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.137641+0000", 2024-09-21T22:07:08.022 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.686e-06 2024-09-21T22:07:08.022 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.022 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.022 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:08.022 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.137934+0000", 2024-09-21T22:07:08.022 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00029319099999999999 2024-09-21T22:07:08.022 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.023 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.023 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:08.023 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.137963+0000", 2024-09-21T22:07:08.023 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9502999999999999e-05 2024-09-21T22:07:08.023 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.023 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.023 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:08.023 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.137989+0000", 2024-09-21T22:07:08.023 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5573000000000001e-05 2024-09-21T22:07:08.023 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.023 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.023 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:08.023 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.138086+0000", 2024-09-21T22:07:08.024 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.6968999999999999e-05 2024-09-21T22:07:08.024 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.024 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.024 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:08.024 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.138120+0000", 2024-09-21T22:07:08.024 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.4560000000000001e-05 2024-09-21T22:07:08.024 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.024 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.024 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:08.024 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.141539+0000", 2024-09-21T22:07:08.024 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0034183289999999999 2024-09-21T22:07:08.024 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.024 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.024 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:08.025 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.141629+0000", 2024-09-21T22:07:08.025 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.0399e-05 2024-09-21T22:07:08.025 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:08.025 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:08.025 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:08.025 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.025 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.025 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:23 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'13, mlcod=14'11)", 2024-09-21T22:07:08.025 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.144914+0000", 2024-09-21T22:07:08.025 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 65.863283034999995, 2024-09-21T22:07:08.026 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0040223230000000004, 2024-09-21T22:07:08.026 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:08.026 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:08.026 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:08.026 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:08.026 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.026 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:08.026 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.144914+0000", 2024-09-21T22:07:08.026 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:08.026 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.026 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.026 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:08.026 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.144914+0000", 2024-09-21T22:07:08.027 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:08.027 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.027 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.027 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:08.027 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.144916+0000", 2024-09-21T22:07:08.027 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6470000000000001e-06 2024-09-21T22:07:08.027 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.027 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.027 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:08.027 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.144929+0000", 2024-09-21T22:07:08.027 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2903e-05 2024-09-21T22:07:08.027 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.028 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.028 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:08.028 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.144958+0000", 2024-09-21T22:07:08.028 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9280000000000001e-05 2024-09-21T22:07:08.028 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.028 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.028 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:08.028 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.144971+0000", 2024-09-21T22:07:08.028 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3407000000000001e-05 2024-09-21T22:07:08.028 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.028 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.029 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:08.029 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.145106+0000", 2024-09-21T22:07:08.029 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013440500000000001 2024-09-21T22:07:08.029 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.029 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.029 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:08.029 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.145170+0000", 2024-09-21T22:07:08.029 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.4597000000000005e-05 2024-09-21T22:07:08.029 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.029 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.029 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:08.029 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.148854+0000", 2024-09-21T22:07:08.029 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0036833249999999999 2024-09-21T22:07:08.029 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.030 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.030 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:08.030 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.148937+0000", 2024-09-21T22:07:08.030 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.2758999999999998e-05 2024-09-21T22:07:08.030 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:08.030 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:08.030 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:08.030 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.030 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.030 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:31 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'17, mlcod=14'14)", 2024-09-21T22:07:08.030 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.163958+0000", 2024-09-21T22:07:08.030 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 65.844238767999997, 2024-09-21T22:07:08.030 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0033891149999999998, 2024-09-21T22:07:08.031 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:08.031 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:08.031 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:08.031 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:08.031 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.031 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:08.031 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.163958+0000", 2024-09-21T22:07:08.031 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:08.031 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.031 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.031 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:08.031 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.163958+0000", 2024-09-21T22:07:08.031 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:08.031 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.032 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.032 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:08.032 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.163960+0000", 2024-09-21T22:07:08.032 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.64e-06 2024-09-21T22:07:08.032 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.032 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.032 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:08.032 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.163971+0000", 2024-09-21T22:07:08.032 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0797e-05 2024-09-21T22:07:08.032 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.033 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.033 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:08.033 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.163995+0000", 2024-09-21T22:07:08.033 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3876e-05 2024-09-21T22:07:08.033 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.033 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.033 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:08.033 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.164008+0000", 2024-09-21T22:07:08.033 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3674e-05 2024-09-21T22:07:08.033 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.033 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.034 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:08.034 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.164203+0000", 2024-09-21T22:07:08.034 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00019457799999999999 2024-09-21T22:07:08.034 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.034 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.034 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:08.034 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.164265+0000", 2024-09-21T22:07:08.034 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.2415999999999993e-05 2024-09-21T22:07:08.034 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.034 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.034 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:08.035 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.167209+0000", 2024-09-21T22:07:08.035 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0029436380000000002 2024-09-21T22:07:08.035 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.035 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.035 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:08.035 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.167348+0000", 2024-09-21T22:07:08.035 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000138496 2024-09-21T22:07:08.035 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:08.035 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:08.035 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:08.035 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.035 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.035 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:32 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'18, mlcod=14'14)", 2024-09-21T22:07:08.036 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.165319+0000", 2024-09-21T22:07:08.036 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 65.842877917999999, 2024-09-21T22:07:08.036 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0033722280000000001, 2024-09-21T22:07:08.036 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:08.036 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:08.036 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:08.036 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:08.036 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.036 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:08.036 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.165319+0000", 2024-09-21T22:07:08.036 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:08.036 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.036 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.037 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:08.037 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.165319+0000", 2024-09-21T22:07:08.037 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:08.037 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.037 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.037 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:08.037 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.165321+0000", 2024-09-21T22:07:08.037 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5999999999999999e-06 2024-09-21T22:07:08.037 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.037 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.037 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:08.037 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.165377+0000", 2024-09-21T22:07:08.037 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.5877000000000001e-05 2024-09-21T22:07:08.038 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.038 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.038 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:08.038 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.165407+0000", 2024-09-21T22:07:08.038 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.0263000000000001e-05 2024-09-21T22:07:08.038 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.038 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.038 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:08.038 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.165441+0000", 2024-09-21T22:07:08.038 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.3966e-05 2024-09-21T22:07:08.038 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.038 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.039 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:08.039 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.165574+0000", 2024-09-21T22:07:08.039 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013327599999999999 2024-09-21T22:07:08.039 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.039 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.039 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:08.039 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.165635+0000", 2024-09-21T22:07:08.039 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.0949000000000002e-05 2024-09-21T22:07:08.039 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.039 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.039 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:08.039 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.168619+0000", 2024-09-21T22:07:08.039 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0029836979999999999 2024-09-21T22:07:08.039 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.040 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.040 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:08.040 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.168692+0000", 2024-09-21T22:07:08.040 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.2599000000000001e-05 2024-09-21T22:07:08.040 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:08.040 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:08.040 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:08.040 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.040 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.040 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:38 1.0 e14/13 1:30306672:devicehealth::main.db.0000000000000000:head v 14'24, mlcod=14'22)", 2024-09-21T22:07:08.040 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.180244+0000", 2024-09-21T22:07:08.040 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 65.827953273000006, 2024-09-21T22:07:08.040 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0041712219999999996, 2024-09-21T22:07:08.041 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:08.041 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:08.041 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:08.041 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:08.041 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.041 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:08.041 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.180244+0000", 2024-09-21T22:07:08.041 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:08.041 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.041 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.041 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:08.041 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.180244+0000", 2024-09-21T22:07:08.042 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:08.042 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.042 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.042 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:08.042 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.180246+0000", 2024-09-21T22:07:08.042 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6959999999999999e-06 2024-09-21T22:07:08.042 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.042 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.042 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:08.042 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.180508+0000", 2024-09-21T22:07:08.042 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00026275199999999999 2024-09-21T22:07:08.042 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.042 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.043 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:08.043 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.180538+0000", 2024-09-21T22:07:08.043 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9142999999999999e-05 2024-09-21T22:07:08.043 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.043 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.043 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:08.043 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.180552+0000", 2024-09-21T22:07:08.043 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4466e-05 2024-09-21T22:07:08.043 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.043 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.043 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:08.043 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.180677+0000", 2024-09-21T22:07:08.043 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000125413 2024-09-21T22:07:08.043 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.044 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.044 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:08.044 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.180739+0000", 2024-09-21T22:07:08.044 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.1122000000000002e-05 2024-09-21T22:07:08.044 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.044 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.044 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:08.044 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.184291+0000", 2024-09-21T22:07:08.044 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0035525510000000001 2024-09-21T22:07:08.044 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.044 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.044 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:08.044 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.184415+0000", 2024-09-21T22:07:08.045 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000124079 2024-09-21T22:07:08.045 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:08.045 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:08.045 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:08.045 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.045 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.045 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:54 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'32, mlcod=14'29)", 2024-09-21T22:07:08.045 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.214326+0000", 2024-09-21T22:07:08.045 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 65.793871035999999, 2024-09-21T22:07:08.045 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0035226569999999998, 2024-09-21T22:07:08.045 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:08.045 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:08.045 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:08.045 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:08.046 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.046 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:08.046 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.214326+0000", 2024-09-21T22:07:08.046 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:08.046 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.046 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.046 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:08.046 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.214326+0000", 2024-09-21T22:07:08.046 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:08.046 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.046 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.046 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:08.046 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.214328+0000", 2024-09-21T22:07:08.047 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6270000000000001e-06 2024-09-21T22:07:08.047 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.047 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.047 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:08.047 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.214340+0000", 2024-09-21T22:07:08.047 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2286000000000001e-05 2024-09-21T22:07:08.047 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.047 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.047 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:08.047 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.214369+0000", 2024-09-21T22:07:08.047 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9297000000000001e-05 2024-09-21T22:07:08.047 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.047 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.047 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:08.048 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.214383+0000", 2024-09-21T22:07:08.048 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3773e-05 2024-09-21T22:07:08.048 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.048 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.048 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:08.048 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.214513+0000", 2024-09-21T22:07:08.048 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000130056 2024-09-21T22:07:08.048 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.048 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.048 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:08.048 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.214568+0000", 2024-09-21T22:07:08.048 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.4873000000000001e-05 2024-09-21T22:07:08.048 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.048 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.049 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:08.049 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.217739+0000", 2024-09-21T22:07:08.049 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0031712720000000002 2024-09-21T22:07:08.049 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.049 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.049 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:08.049 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.217849+0000", 2024-09-21T22:07:08.049 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000109473 2024-09-21T22:07:08.049 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:08.049 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:08.049 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:08.049 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.049 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.049 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:57 1.0 e14/13 1:30306672:devicehealth::main.db.0000000000000000:head v 14'35, mlcod=14'33)", 2024-09-21T22:07:08.050 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.221549+0000", 2024-09-21T22:07:08.050 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 65.786648537000005, 2024-09-21T22:07:08.050 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0041720860000000002, 2024-09-21T22:07:08.050 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:08.050 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:08.050 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:08.050 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:08.050 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.050 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:08.050 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.221549+0000", 2024-09-21T22:07:08.050 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:08.050 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.050 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.050 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:08.051 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.221549+0000", 2024-09-21T22:07:08.051 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:08.051 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.051 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.051 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:08.051 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.221550+0000", 2024-09-21T22:07:08.051 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.717e-06 2024-09-21T22:07:08.051 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.051 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.051 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:08.051 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.221673+0000", 2024-09-21T22:07:08.051 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000122612 2024-09-21T22:07:08.051 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.052 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.052 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:08.052 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.221703+0000", 2024-09-21T22:07:08.052 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9539999999999998e-05 2024-09-21T22:07:08.052 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.052 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.052 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:08.052 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.221716+0000", 2024-09-21T22:07:08.052 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3149999999999999e-05 2024-09-21T22:07:08.052 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.052 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.052 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:08.052 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.221848+0000", 2024-09-21T22:07:08.053 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013201900000000001 2024-09-21T22:07:08.053 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.053 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.053 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:08.053 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.221908+0000", 2024-09-21T22:07:08.053 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.0452999999999999e-05 2024-09-21T22:07:08.053 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.053 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.053 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:08.053 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.225573+0000", 2024-09-21T22:07:08.053 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.003664676 2024-09-21T22:07:08.053 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.053 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.054 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:08.054 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.225721+0000", 2024-09-21T22:07:08.054 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00014791899999999999 2024-09-21T22:07:08.054 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:08.054 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:08.054 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:08.054 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.054 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.054 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:68 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'42, mlcod=14'39)", 2024-09-21T22:07:08.054 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.247413+0000", 2024-09-21T22:07:08.054 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 65.760784188000002, 2024-09-21T22:07:08.054 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0039572339999999996, 2024-09-21T22:07:08.054 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:08.055 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:08.055 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:08.055 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:08.055 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.055 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:08.055 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.247413+0000", 2024-09-21T22:07:08.055 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:08.055 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.055 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.055 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:08.055 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.247413+0000", 2024-09-21T22:07:08.055 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:08.055 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.056 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.056 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:08.056 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.247415+0000", 2024-09-21T22:07:08.056 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.883e-06 2024-09-21T22:07:08.056 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.056 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.056 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:08.056 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.247427+0000", 2024-09-21T22:07:08.056 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2557e-05 2024-09-21T22:07:08.056 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.056 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.056 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:08.056 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.247457+0000", 2024-09-21T22:07:08.057 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9966000000000001e-05 2024-09-21T22:07:08.057 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.057 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.057 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:08.057 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.247471+0000", 2024-09-21T22:07:08.057 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3747000000000001e-05 2024-09-21T22:07:08.057 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.057 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.057 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:08.057 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.247603+0000", 2024-09-21T22:07:08.057 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013171600000000001 2024-09-21T22:07:08.058 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.058 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.058 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:08.058 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.247656+0000", 2024-09-21T22:07:08.058 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.3529999999999997e-05 2024-09-21T22:07:08.058 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.058 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.058 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:08.058 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.251225+0000", 2024-09-21T22:07:08.058 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00356865 2024-09-21T22:07:08.058 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.058 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.059 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:08.059 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.251370+0000", 2024-09-21T22:07:08.059 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00014518500000000001 2024-09-21T22:07:08.059 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:08.059 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:08.059 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:08.059 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.059 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.059 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:69 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'43, mlcod=14'39)", 2024-09-21T22:07:08.059 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.248839+0000", 2024-09-21T22:07:08.059 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 65.759358258000006, 2024-09-21T22:07:08.059 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0034954470000000001, 2024-09-21T22:07:08.059 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:08.060 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:08.060 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:08.060 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:08.060 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.060 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:08.060 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.248839+0000", 2024-09-21T22:07:08.060 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:08.060 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.060 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.061 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:08.061 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.248839+0000", 2024-09-21T22:07:08.061 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:08.061 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.061 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.061 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:08.061 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.248841+0000", 2024-09-21T22:07:08.061 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6759999999999999e-06 2024-09-21T22:07:08.061 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.061 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.061 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:08.061 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.249058+0000", 2024-09-21T22:07:08.061 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000217455 2024-09-21T22:07:08.062 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.062 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.062 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:08.062 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.249085+0000", 2024-09-21T22:07:08.062 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.7002999999999999e-05 2024-09-21T22:07:08.062 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.062 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.063 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:08.063 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.249099+0000", 2024-09-21T22:07:08.063 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.402e-05 2024-09-21T22:07:08.063 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.063 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.063 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:08.063 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.249241+0000", 2024-09-21T22:07:08.063 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00014194899999999999 2024-09-21T22:07:08.063 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.063 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.063 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:08.063 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.249298+0000", 2024-09-21T22:07:08.063 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.6687000000000001e-05 2024-09-21T22:07:08.063 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.064 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.064 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:08.064 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.252266+0000", 2024-09-21T22:07:08.064 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.002968348 2024-09-21T22:07:08.064 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.064 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.064 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:08.064 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.252334+0000", 2024-09-21T22:07:08.064 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.8308999999999999e-05 2024-09-21T22:07:08.064 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:08.064 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:08.064 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:08.065 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.065 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.065 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:71 1.0 e14/13 1:30306672:devicehealth::main.db.0000000000000000:head v 14'45, mlcod=14'43)", 2024-09-21T22:07:08.065 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.256068+0000", 2024-09-21T22:07:08.065 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 65.752128786, 2024-09-21T22:07:08.065 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0041712579999999997, 2024-09-21T22:07:08.065 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:08.065 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:08.065 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:08.065 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:08.065 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.065 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:08.065 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.256068+0000", 2024-09-21T22:07:08.065 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:08.066 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.066 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.066 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:08.066 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.256068+0000", 2024-09-21T22:07:08.066 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:08.066 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.066 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.066 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:08.066 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.256070+0000", 2024-09-21T22:07:08.066 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.61e-06 2024-09-21T22:07:08.066 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.066 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.066 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:08.067 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.256129+0000", 2024-09-21T22:07:08.067 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.8702999999999997e-05 2024-09-21T22:07:08.067 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.067 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.067 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:08.067 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.256158+0000", 2024-09-21T22:07:08.067 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9533e-05 2024-09-21T22:07:08.067 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.067 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.067 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:08.067 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.256173+0000", 2024-09-21T22:07:08.067 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4267000000000001e-05 2024-09-21T22:07:08.068 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.068 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.068 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:08.068 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.256304+0000", 2024-09-21T22:07:08.068 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000131352 2024-09-21T22:07:08.068 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.068 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.068 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:08.068 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.256357+0000", 2024-09-21T22:07:08.068 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.3492999999999998e-05 2024-09-21T22:07:08.068 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.068 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.068 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:08.068 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.260147+0000", 2024-09-21T22:07:08.069 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0037900410000000001 2024-09-21T22:07:08.069 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.069 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.069 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:08.069 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.260240+0000", 2024-09-21T22:07:08.069 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.2258999999999999e-05 2024-09-21T22:07:08.069 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:08.069 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:08.069 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:08.069 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.069 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.069 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:72 1.0 e14/13 1:30306672:devicehealth::main.db.0000000000000000:head v 14'46, mlcod=14'43)", 2024-09-21T22:07:08.069 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.258358+0000", 2024-09-21T22:07:08.070 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 65.749838792999995, 2024-09-21T22:07:08.070 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0041492919999999997, 2024-09-21T22:07:08.070 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:08.070 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:08.070 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:08.070 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:08.070 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.070 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:08.070 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.258358+0000", 2024-09-21T22:07:08.070 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:08.070 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.070 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.070 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:08.071 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.258358+0000", 2024-09-21T22:07:08.071 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:08.071 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.071 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.071 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:08.071 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.258360+0000", 2024-09-21T22:07:08.071 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.623e-06 2024-09-21T22:07:08.071 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.071 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.071 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:08.071 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.258415+0000", 2024-09-21T22:07:08.071 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.4756000000000002e-05 2024-09-21T22:07:08.071 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.071 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.072 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:08.072 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.258444+0000", 2024-09-21T22:07:08.072 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9164000000000001e-05 2024-09-21T22:07:08.072 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.072 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.072 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:08.072 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.258459+0000", 2024-09-21T22:07:08.072 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4996e-05 2024-09-21T22:07:08.072 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.072 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.072 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:08.072 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.258637+0000", 2024-09-21T22:07:08.072 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00017796500000000001 2024-09-21T22:07:08.073 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.073 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.073 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:08.073 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.258695+0000", 2024-09-21T22:07:08.073 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.8207e-05 2024-09-21T22:07:08.073 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.073 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.073 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:08.073 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.262421+0000", 2024-09-21T22:07:08.073 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0037261550000000001 2024-09-21T22:07:08.073 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.073 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.073 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:08.074 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.262508+0000", 2024-09-21T22:07:08.074 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.6426000000000002e-05 2024-09-21T22:07:08.074 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:08.074 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:08.074 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:08.074 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.074 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.074 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:87 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'57, mlcod=14'52)", 2024-09-21T22:07:08.074 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.290115+0000", 2024-09-21T22:07:08.074 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 65.718082347999996, 2024-09-21T22:07:08.074 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0030478200000000001, 2024-09-21T22:07:08.075 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:08.075 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:08.075 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:08.075 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:08.075 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.075 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:08.075 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.290115+0000", 2024-09-21T22:07:08.075 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:08.075 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.075 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.075 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:08.075 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.290115+0000", 2024-09-21T22:07:08.075 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:08.076 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.076 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.076 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:08.076 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.290116+0000", 2024-09-21T22:07:08.076 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3829999999999999e-06 2024-09-21T22:07:08.076 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.076 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.076 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:08.076 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.290191+0000", 2024-09-21T22:07:08.076 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.4562999999999997e-05 2024-09-21T22:07:08.076 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.076 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.076 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:08.076 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.290216+0000", 2024-09-21T22:07:08.077 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.565e-05 2024-09-21T22:07:08.077 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.077 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.077 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:08.077 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.290236+0000", 2024-09-21T22:07:08.077 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9326000000000001e-05 2024-09-21T22:07:08.077 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.077 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.077 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:08.077 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.290340+0000", 2024-09-21T22:07:08.077 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0001047 2024-09-21T22:07:08.077 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.077 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.078 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:08.078 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.290377+0000", 2024-09-21T22:07:08.078 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.6356000000000002e-05 2024-09-21T22:07:08.078 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.078 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.078 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:08.078 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.293096+0000", 2024-09-21T22:07:08.078 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.002718996 2024-09-21T22:07:08.078 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.078 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.078 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:08.078 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.293163+0000", 2024-09-21T22:07:08.078 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.6846000000000003e-05 2024-09-21T22:07:08.079 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:08.079 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:08.079 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:08.079 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.079 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.079 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:114 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'76, mlcod=14'73)", 2024-09-21T22:07:08.079 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.490030+0000", 2024-09-21T22:07:08.079 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 65.518166909000001, 2024-09-21T22:07:08.079 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.004227873, 2024-09-21T22:07:08.079 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:08.079 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:08.079 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:08.079 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:08.079 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.080 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:08.080 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.490030+0000", 2024-09-21T22:07:08.080 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:08.080 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.080 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.080 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:08.080 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.490030+0000", 2024-09-21T22:07:08.080 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:08.080 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.080 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.080 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:08.080 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.490031+0000", 2024-09-21T22:07:08.080 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.3300000000000001e-07 2024-09-21T22:07:08.080 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.081 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.081 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:08.081 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.490042+0000", 2024-09-21T22:07:08.081 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.1095e-05 2024-09-21T22:07:08.081 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.081 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.081 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:08.081 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.490053+0000", 2024-09-21T22:07:08.081 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0501e-05 2024-09-21T22:07:08.081 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.081 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.081 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:08.081 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.490060+0000", 2024-09-21T22:07:08.081 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.9829999999999999e-06 2024-09-21T22:07:08.082 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.082 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.082 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:08.082 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.490107+0000", 2024-09-21T22:07:08.082 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.7278999999999999e-05 2024-09-21T22:07:08.082 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.082 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.082 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:08.082 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.490124+0000", 2024-09-21T22:07:08.082 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7190999999999999e-05 2024-09-21T22:07:08.082 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.082 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.082 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:08.083 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.494229+0000", 2024-09-21T22:07:08.083 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0041050440000000004 2024-09-21T22:07:08.083 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.083 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.083 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:08.083 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.494258+0000", 2024-09-21T22:07:08.083 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9047000000000002e-05 2024-09-21T22:07:08.083 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:08.083 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:08.083 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:08.083 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.083 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.083 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:115 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'77, mlcod=14'73)", 2024-09-21T22:07:08.084 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.491380+0000", 2024-09-21T22:07:08.084 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 65.516817286999995, 2024-09-21T22:07:08.084 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0031550889999999998, 2024-09-21T22:07:08.084 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:08.084 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:08.084 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:08.084 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:08.084 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.084 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:08.084 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.491380+0000", 2024-09-21T22:07:08.084 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:08.084 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.084 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.084 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:08.085 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.491380+0000", 2024-09-21T22:07:08.085 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:08.085 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.085 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.085 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:08.085 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.491381+0000", 2024-09-21T22:07:08.085 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.1399999999999996e-07 2024-09-21T22:07:08.085 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.085 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.085 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:08.085 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.491435+0000", 2024-09-21T22:07:08.085 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.3797e-05 2024-09-21T22:07:08.085 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.085 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.086 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:08.086 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.491447+0000", 2024-09-21T22:07:08.086 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2642e-05 2024-09-21T22:07:08.086 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.086 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.086 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:08.086 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.491453+0000", 2024-09-21T22:07:08.086 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.4820000000000002e-06 2024-09-21T22:07:08.086 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.086 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.086 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:08.086 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.491502+0000", 2024-09-21T22:07:08.086 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.9038e-05 2024-09-21T22:07:08.087 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.087 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.087 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:08.087 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.491514+0000", 2024-09-21T22:07:08.087 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2483999999999999e-05 2024-09-21T22:07:08.087 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.087 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.087 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:08.087 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.494486+0000", 2024-09-21T22:07:08.087 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.002971338 2024-09-21T22:07:08.087 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.087 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.087 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:08.088 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.494535+0000", 2024-09-21T22:07:08.088 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.9493999999999997e-05 2024-09-21T22:07:08.088 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:08.088 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:08.088 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:08.088 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.088 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.088 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:118 1.0 e14/13 1:30306672:devicehealth::main.db.0000000000000000:head v 14'80, mlcod=14'78)", 2024-09-21T22:07:08.088 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.499045+0000", 2024-09-21T22:07:08.088 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 65.509152284999999, 2024-09-21T22:07:08.088 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0029889589999999998, 2024-09-21T22:07:08.088 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:08.088 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:08.088 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:08.089 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:08.089 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.089 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:08.089 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.499045+0000", 2024-09-21T22:07:08.089 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:08.089 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.089 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.089 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:08.089 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.499045+0000", 2024-09-21T22:07:08.089 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:08.089 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.089 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.089 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:08.089 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.499046+0000", 2024-09-21T22:07:08.090 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.0299999999999998e-07 2024-09-21T22:07:08.090 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.090 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.090 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:08.090 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.499110+0000", 2024-09-21T22:07:08.090 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.4688000000000001e-05 2024-09-21T22:07:08.090 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.090 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.090 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:08.090 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.499125+0000", 2024-09-21T22:07:08.090 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4387e-05 2024-09-21T22:07:08.090 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.090 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.091 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:08.091 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.499131+0000", 2024-09-21T22:07:08.091 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.2129999999999996e-06 2024-09-21T22:07:08.091 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.091 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.091 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:08.091 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.499255+0000", 2024-09-21T22:07:08.091 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00012424199999999999 2024-09-21T22:07:08.091 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.091 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.091 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:08.091 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.499312+0000", 2024-09-21T22:07:08.091 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.6709999999999997e-05 2024-09-21T22:07:08.091 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.092 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.092 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:08.092 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.502005+0000", 2024-09-21T22:07:08.092 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0026927169999999999 2024-09-21T22:07:08.092 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.092 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.092 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:08.092 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.502034+0000", 2024-09-21T22:07:08.092 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9198999999999999e-05 2024-09-21T22:07:08.092 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:08.092 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:08.092 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:08.092 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.092 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.093 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:136 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'94, mlcod=14'89)", 2024-09-21T22:07:08.093 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.903810+0000", 2024-09-21T22:07:08.093 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 65.104387313999993, 2024-09-21T22:07:08.093 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0062746829999999996, 2024-09-21T22:07:08.093 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:08.093 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:08.093 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:08.093 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:08.093 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.093 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:08.093 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.903810+0000", 2024-09-21T22:07:08.093 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:08.093 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.094 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.094 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:08.094 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.903810+0000", 2024-09-21T22:07:08.094 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:08.094 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.094 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.094 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:08.094 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.903810+0000", 2024-09-21T22:07:08.094 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.03e-07 2024-09-21T22:07:08.094 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.094 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.094 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:08.095 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.903814+0000", 2024-09-21T22:07:08.095 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.676e-06 2024-09-21T22:07:08.095 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.095 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.095 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:08.095 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.903822+0000", 2024-09-21T22:07:08.095 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.2879999999999993e-06 2024-09-21T22:07:08.095 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.095 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.095 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:08.095 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.903829+0000", 2024-09-21T22:07:08.095 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.2920000000000001e-06 2024-09-21T22:07:08.095 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.095 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.096 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:08.096 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.903879+0000", 2024-09-21T22:07:08.096 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.0253999999999997e-05 2024-09-21T22:07:08.096 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.096 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.096 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:08.096 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.903897+0000", 2024-09-21T22:07:08.096 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8468999999999999e-05 2024-09-21T22:07:08.096 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.096 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.096 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:08.096 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.910044+0000", 2024-09-21T22:07:08.096 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0061468950000000003 2024-09-21T22:07:08.096 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.097 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.097 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:08.097 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.910085+0000", 2024-09-21T22:07:08.097 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.0405999999999999e-05 2024-09-21T22:07:08.097 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:08.097 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:08.097 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:08.097 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.097 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.097 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:137 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'95, mlcod=14'90)", 2024-09-21T22:07:08.097 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.904439+0000", 2024-09-21T22:07:08.097 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 65.103757917999999, 2024-09-21T22:07:08.097 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0072625019999999997, 2024-09-21T22:07:08.097 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:08.098 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:08.098 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:08.098 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:08.098 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.098 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:08.098 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.904439+0000", 2024-09-21T22:07:08.098 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:08.098 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.098 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.098 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:08.098 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.904439+0000", 2024-09-21T22:07:08.098 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:08.098 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.099 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.099 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:08.099 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.904440+0000", 2024-09-21T22:07:08.099 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8400000000000005e-07 2024-09-21T22:07:08.099 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.099 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.099 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:08.099 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.904498+0000", 2024-09-21T22:07:08.099 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.8320999999999999e-05 2024-09-21T22:07:08.099 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.099 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.099 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:08.099 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.904507+0000", 2024-09-21T22:07:08.100 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.7790000000000001e-06 2024-09-21T22:07:08.100 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.100 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.100 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:08.100 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.904511+0000", 2024-09-21T22:07:08.100 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.9700000000000001e-06 2024-09-21T22:07:08.100 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.100 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.100 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:08.100 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.904551+0000", 2024-09-21T22:07:08.100 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.9910000000000002e-05 2024-09-21T22:07:08.100 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.100 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.100 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:08.100 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.904567+0000", 2024-09-21T22:07:08.101 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5903999999999999e-05 2024-09-21T22:07:08.101 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.101 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.101 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:08.101 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.911666+0000", 2024-09-21T22:07:08.101 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0070996929999999998 2024-09-21T22:07:08.101 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.101 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.101 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:08.101 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.911702+0000", 2024-09-21T22:07:08.101 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.5441000000000002e-05 2024-09-21T22:07:08.101 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:08.101 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:08.101 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:08.102 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.102 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.102 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:138 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'96, mlcod=14'90)", 2024-09-21T22:07:08.102 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.905402+0000", 2024-09-21T22:07:08.102 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 65.102794940999999, 2024-09-21T22:07:08.102 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0063366890000000004, 2024-09-21T22:07:08.102 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:08.102 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:08.102 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:08.102 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:08.102 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.102 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:08.102 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905402+0000", 2024-09-21T22:07:08.103 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:08.103 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.103 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.103 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:08.103 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905402+0000", 2024-09-21T22:07:08.103 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:08.103 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.103 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.103 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:08.103 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905403+0000", 2024-09-21T22:07:08.103 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.9200000000000002e-07 2024-09-21T22:07:08.103 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.103 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.103 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:08.103 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905406+0000", 2024-09-21T22:07:08.104 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.326e-06 2024-09-21T22:07:08.104 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.104 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.104 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:08.104 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905413+0000", 2024-09-21T22:07:08.104 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.3320000000000003e-06 2024-09-21T22:07:08.104 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.104 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.104 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:08.104 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905417+0000", 2024-09-21T22:07:08.104 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.7139999999999999e-06 2024-09-21T22:07:08.104 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.105 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.106 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:08.106 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.910115+0000", 2024-09-21T22:07:08.106 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0046983939999999998 2024-09-21T22:07:08.106 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.106 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.106 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:08.106 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.910140+0000", 2024-09-21T22:07:08.106 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4689999999999999e-05 2024-09-21T22:07:08.106 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.106 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.106 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:08.106 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.911706+0000", 2024-09-21T22:07:08.106 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0015658479999999999 2024-09-21T22:07:08.106 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.107 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.107 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:08.107 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.911739+0000", 2024-09-21T22:07:08.107 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.2993000000000001e-05 2024-09-21T22:07:08.107 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:08.107 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:08.107 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:08.107 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.107 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.107 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:139 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'97, mlcod=14'90)", 2024-09-21T22:07:08.107 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.905983+0000", 2024-09-21T22:07:08.107 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 65.102214023000002, 2024-09-21T22:07:08.107 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0057906030000000001, 2024-09-21T22:07:08.107 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:08.108 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:08.108 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:08.108 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:08.108 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.108 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:08.108 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905983+0000", 2024-09-21T22:07:08.108 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:08.108 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.108 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.108 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:08.108 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905983+0000", 2024-09-21T22:07:08.109 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:08.109 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.109 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.109 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:08.109 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905984+0000", 2024-09-21T22:07:08.109 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.15e-07 2024-09-21T22:07:08.109 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.109 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.109 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:08.109 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905986+0000", 2024-09-21T22:07:08.109 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.852e-06 2024-09-21T22:07:08.110 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.110 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.110 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:08.110 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905993+0000", 2024-09-21T22:07:08.110 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.652e-06 2024-09-21T22:07:08.110 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.110 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.110 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:08.110 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905996+0000", 2024-09-21T22:07:08.110 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.3519999999999998e-06 2024-09-21T22:07:08.110 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.110 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.110 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:08.111 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.910519+0000", 2024-09-21T22:07:08.111 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0045230820000000003 2024-09-21T22:07:08.111 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.111 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.111 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:08.111 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.910548+0000", 2024-09-21T22:07:08.111 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8399999999999999e-05 2024-09-21T22:07:08.111 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.111 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.111 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:08.111 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.911742+0000", 2024-09-21T22:07:08.111 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011939909999999999 2024-09-21T22:07:08.111 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.112 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.112 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:08.112 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.911774+0000", 2024-09-21T22:07:08.112 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.1958999999999997e-05 2024-09-21T22:07:08.112 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:08.112 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:08.112 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:08.112 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.112 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.112 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:140 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'98, mlcod=14'90)", 2024-09-21T22:07:08.112 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.906441+0000", 2024-09-21T22:07:08.112 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 65.101755893999993, 2024-09-21T22:07:08.112 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0057052989999999996, 2024-09-21T22:07:08.113 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:08.113 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:08.113 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:08.113 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:08.113 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.113 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:08.113 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.906441+0000", 2024-09-21T22:07:08.113 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:08.113 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.113 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.113 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:08.113 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.906441+0000", 2024-09-21T22:07:08.114 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:08.114 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.114 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.114 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:08.114 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.906442+0000", 2024-09-21T22:07:08.114 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.7e-07 2024-09-21T22:07:08.114 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.114 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.114 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:08.114 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.906511+0000", 2024-09-21T22:07:08.114 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.8857999999999997e-05 2024-09-21T22:07:08.114 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.115 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.115 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:08.115 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.906518+0000", 2024-09-21T22:07:08.115 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.1690000000000003e-06 2024-09-21T22:07:08.115 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.115 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.115 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:08.115 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.906521+0000", 2024-09-21T22:07:08.115 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.4829999999999999e-06 2024-09-21T22:07:08.115 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.115 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.115 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:08.115 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.910939+0000", 2024-09-21T22:07:08.116 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0044176809999999997 2024-09-21T22:07:08.116 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.116 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.116 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:08.116 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.910959+0000", 2024-09-21T22:07:08.116 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9898000000000002e-05 2024-09-21T22:07:08.116 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.116 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.116 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:08.116 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.912113+0000", 2024-09-21T22:07:08.116 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001153811 2024-09-21T22:07:08.116 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:08.116 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:08.116 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:08.117 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.912147+0000", 2024-09-21T22:07:08.117 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.4029000000000002e-05 2024-09-21T22:07:08.117 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:08.117 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:08.117 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:08.117 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:08.117 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:08.117 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:07:08.117 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T22:07:08.119 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T22:07:08.120 INFO:teuthology.orchestra.run.smithi039.stderr:noscrub is unset 2024-09-21T22:07:08.125 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:08.116+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:08.220 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:08.215+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:08.267 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:07:08.268 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:07:08.268 INFO:teuthology.orchestra.run.smithi039.stdout: "num_ops": 0 2024-09-21T22:07:08.268 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:07:08.277 INFO:teuthology.orchestra.run.smithi039.stdout:osd.0: {} 2024-09-21T22:07:08.277 INFO:teuthology.orchestra.run.smithi039.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T22:07:08.284 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T22:07:08.295 INFO:teuthology.orchestra.run.smithi039.stdout:osd.1: {} 2024-09-21T22:07:08.295 INFO:teuthology.orchestra.run.smithi039.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T22:07:08.313 INFO:teuthology.orchestra.run.smithi039.stdout:osd.2: {} 2024-09-21T22:07:08.313 INFO:teuthology.orchestra.run.smithi039.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T22:07:08.322 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:08.316+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:08.333 INFO:teuthology.orchestra.run.smithi039.stdout:osd.3: {} 2024-09-21T22:07:08.333 INFO:teuthology.orchestra.run.smithi039.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T22:07:08.354 INFO:teuthology.orchestra.run.smithi039.stdout:osd.4: {} 2024-09-21T22:07:08.355 INFO:teuthology.orchestra.run.smithi039.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T22:07:08.355 INFO:teuthology.orchestra.run.smithi039.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-21T22:07:08.358 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T22:07:08.359 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T22:07:08.416 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:08.422 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:08.423 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:08.415+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:08.431 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T22:07:08.450 INFO:tasks.workunit.client.0.smithi157.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-21T22:07:08.526 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:08.519+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:08.534 INFO:tasks.workunit.client.0.smithi157.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-21T22:07:08.592 INFO:tasks.workunit.client.0.smithi157.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-21T22:07:08.594 INFO:tasks.workunit.client.0.smithi157.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-21T22:07:08.594 INFO:tasks.workunit.client.0.smithi157.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-21T22:07:08.604 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:08.605 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:08.619 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T22:07:08.626 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:08.620+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:08.639 INFO:tasks.workunit.client.0.smithi157.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-21T22:07:08.643 DEBUG:teuthology.orchestra.run.smithi157:> set -ex 2024-09-21T22:07:08.643 DEBUG:teuthology.orchestra.run.smithi157:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-21T22:07:08.690 INFO:tasks.workunit:Running workunits matching rbd/rbd-nbd.sh on client.0... 2024-09-21T22:07:08.692 INFO:tasks.workunit:Running workunit rbd/rbd-nbd.sh... 2024-09-21T22:07:08.692 DEBUG:teuthology.orchestra.run.smithi157:workunit test rbd/rbd-nbd.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4e45e6b490e6cbe082cdf155fea780438ec53fc7 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/rbd-nbd.sh 2024-09-21T22:07:08.726 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:08.719+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:08.743 INFO:tasks.workunit.client.0.smithi157.stderr:++ dirname /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/rbd-nbd.sh 2024-09-21T22:07:08.744 INFO:tasks.workunit.client.0.smithi157.stderr:+ . /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/../../standalone/ceph-helpers.sh 2024-09-21T22:07:08.744 INFO:tasks.workunit.client.0.smithi157.stderr:++ TIMEOUT=300 2024-09-21T22:07:08.744 INFO:tasks.workunit.client.0.smithi157.stderr:++ WAIT_FOR_CLEAN_TIMEOUT=90 2024-09-21T22:07:08.744 INFO:tasks.workunit.client.0.smithi157.stderr:++ MAX_TIMEOUT=15 2024-09-21T22:07:08.744 INFO:tasks.workunit.client.0.smithi157.stderr:++ PG_NUM=4 2024-09-21T22:07:08.744 INFO:tasks.workunit.client.0.smithi157.stderr:++ TMPDIR=/tmp 2024-09-21T22:07:08.744 INFO:tasks.workunit.client.0.smithi157.stderr:++ CEPH_BUILD_VIRTUALENV=/tmp 2024-09-21T22:07:08.745 INFO:tasks.workunit.client.0.smithi157.stderr:++ TESTDIR=/home/ubuntu/cephtest 2024-09-21T22:07:08.745 INFO:tasks.workunit.client.0.smithi157.stderr:++ type xmlstarlet 2024-09-21T22:07:08.745 INFO:tasks.workunit.client.0.smithi157.stderr:++ XMLSTARLET=xmlstarlet 2024-09-21T22:07:08.745 INFO:tasks.workunit.client.0.smithi157.stderr:+++ uname 2024-09-21T22:07:08.745 INFO:tasks.workunit.client.0.smithi157.stderr:++ '[' Linux = FreeBSD ']' 2024-09-21T22:07:08.745 INFO:tasks.workunit.client.0.smithi157.stderr:++ SED=sed 2024-09-21T22:07:08.746 INFO:tasks.workunit.client.0.smithi157.stderr:++ AWK=awk 2024-09-21T22:07:08.746 INFO:tasks.workunit.client.0.smithi157.stderr:+++ stty -a 2024-09-21T22:07:08.746 INFO:tasks.workunit.client.0.smithi157.stderr:+++ head -1 2024-09-21T22:07:08.746 INFO:tasks.workunit.client.0.smithi157.stderr:+++ sed -e 's/.*columns \([0-9]*\).*/\1/' 2024-09-21T22:07:08.761 INFO:tasks.workunit.client.0.smithi157.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-21T22:07:08.761 INFO:tasks.workunit.client.0.smithi157.stderr:++ termwidth= 2024-09-21T22:07:08.761 INFO:tasks.workunit.client.0.smithi157.stderr:++ '[' -n '' -a '' '!=' 0 ']' 2024-09-21T22:07:08.761 INFO:tasks.workunit.client.0.smithi157.stderr:++ DIFFCOLOPTS='-y ' 2024-09-21T22:07:08.761 INFO:tasks.workunit.client.0.smithi157.stderr:++ KERNCORE=kernel.core_pattern 2024-09-21T22:07:08.762 INFO:tasks.workunit.client.0.smithi157.stderr:++ EXTRA_OPTS= 2024-09-21T22:07:08.765 INFO:tasks.workunit.client.0.smithi157.stderr:++ test '' = TESTS 2024-09-21T22:07:08.765 INFO:tasks.workunit.client.0.smithi157.stderr:+ POOL=rbd 2024-09-21T22:07:08.765 INFO:tasks.workunit.client.0.smithi157.stderr:+ ANOTHER_POOL=new_default_pool21553 2024-09-21T22:07:08.765 INFO:tasks.workunit.client.0.smithi157.stderr:+ NS=ns 2024-09-21T22:07:08.765 INFO:tasks.workunit.client.0.smithi157.stderr:+ IMAGE=testrbdnbd21553 2024-09-21T22:07:08.765 INFO:tasks.workunit.client.0.smithi157.stderr:+ SIZE=64 2024-09-21T22:07:08.766 INFO:tasks.workunit.client.0.smithi157.stderr:+ DATA= 2024-09-21T22:07:08.766 INFO:tasks.workunit.client.0.smithi157.stderr:+ DEV= 2024-09-21T22:07:08.766 INFO:tasks.workunit.client.0.smithi157.stderr:+ setup 2024-09-21T22:07:08.766 INFO:tasks.workunit.client.0.smithi157.stderr:+ local ns x 2024-09-21T22:07:08.766 INFO:tasks.workunit.client.0.smithi157.stderr:+ '[' -e CMakeCache.txt ']' 2024-09-21T22:07:08.766 INFO:tasks.workunit.client.0.smithi157.stderr:+ _sudo echo test sudo 2024-09-21T22:07:08.766 INFO:tasks.workunit.client.0.smithi157.stderr:+ local cmd 2024-09-21T22:07:08.766 INFO:tasks.workunit.client.0.smithi157.stderr:++ id -u 2024-09-21T22:07:08.767 INFO:tasks.workunit.client.0.smithi157.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T22:07:08.767 INFO:tasks.workunit.client.0.smithi157.stderr:++ which echo 2024-09-21T22:07:08.768 INFO:tasks.workunit.client.0.smithi157.stderr:+ cmd=/usr/bin/echo 2024-09-21T22:07:08.768 INFO:tasks.workunit.client.0.smithi157.stderr:+ shift 2024-09-21T22:07:08.768 INFO:tasks.workunit.client.0.smithi157.stderr:+ sudo -nE /usr/bin/echo test sudo 2024-09-21T22:07:08.774 INFO:tasks.workunit.client.0.smithi157.stdout:test sudo 2024-09-21T22:07:08.775 INFO:tasks.workunit.client.0.smithi157.stderr:+ trap cleanup INT TERM EXIT 2024-09-21T22:07:08.775 INFO:tasks.workunit.client.0.smithi157.stderr:++ mktemp -d 2024-09-21T22:07:08.776 INFO:tasks.workunit.client.0.smithi157.stderr:+ TEMPDIR=/tmp/tmp.GqKuv7rsZi 2024-09-21T22:07:08.776 INFO:tasks.workunit.client.0.smithi157.stderr:+ DATA=/tmp/tmp.GqKuv7rsZi/data 2024-09-21T22:07:08.776 INFO:tasks.workunit.client.0.smithi157.stderr:+ dd if=/dev/urandom of=/tmp/tmp.GqKuv7rsZi/data bs=1M count=64 2024-09-21T22:07:08.793 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:08.793 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:08.808 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T22:07:08.856 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:08.851+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:08.928 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:08.924+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:08.975 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:08.975 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:08.989 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T22:07:08.991 INFO:tasks.workunit.client.0.smithi157.stderr:64+0 records in 2024-09-21T22:07:08.991 INFO:tasks.workunit.client.0.smithi157.stderr:64+0 records out 2024-09-21T22:07:08.991 INFO:tasks.workunit.client.0.smithi157.stderr:67108864 bytes (67 MB, 64 MiB) copied, 0.213171 s, 315 MB/s 2024-09-21T22:07:08.991 INFO:tasks.workunit.client.0.smithi157.stderr:+ rbd namespace create rbd/ns 2024-09-21T22:07:09.036 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:09.031+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:09.071 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 0] out_osds: [] dead_osds: [5] live_osds: [0, 1, 2, 3, 4] 2024-09-21T22:07:09.071 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T22:07:09.071 INFO:tasks.thrashosds.thrasher:Removing osd 2, in_osds are: [1, 2, 3, 4, 5, 0] 2024-09-21T22:07:09.071 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 2 2024-09-21T22:07:09.121 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:09.121 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:09.130 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:09.124+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:09.136 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T22:07:09.232 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:09.227+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:09.272 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:09.272 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:09.275 INFO:tasks.workunit.client.0.smithi157.stderr:+ for ns in '' ${NS} 2024-09-21T22:07:09.275 INFO:tasks.workunit.client.0.smithi157.stderr:+ rbd --dest-pool rbd --dest-namespace '' --no-progress import /tmp/tmp.GqKuv7rsZi/data testrbdnbd21553 2024-09-21T22:07:09.287 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T22:07:09.333 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:09.327+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:09.416 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:09.416 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:09.416 INFO:tasks.ceph.mon.a.smithi039.stderr:2024-09-21T22:07:09.411+0000 7f9967196640 -1 mon.a@0(leader).osd e52 definitely_dead 0 2024-09-21T22:07:09.431 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T22:07:09.434 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:09.428+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:09.534 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:09.528+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:09.552 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:09.553 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:09.567 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T22:07:09.635 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:09.631+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:09.736 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:09.732+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:09.767 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:09.767 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:09.783 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T22:07:09.837 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:09.831+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:09.938 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:09.931+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:09.969 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:09.970 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:09.987 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T22:07:10.039 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:10.036+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:10.054 INFO:teuthology.orchestra.run.smithi039.stderr:marked out osd.2. 2024-09-21T22:07:10.105 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:10.105 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:10.119 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T22:07:10.120 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-21T22:07:10.167 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:10.163+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:10.236 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:10.236 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:10.240 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:10.235+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:10.251 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T22:07:10.342 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:10.335+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:10.360 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T22:07:10.402 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:10.402 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:10.416 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T22:07:10.459 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:10.455+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:10.507 INFO:teuthology.orchestra.run.smithi039.stdout:osd.0: {} 2024-09-21T22:07:10.507 INFO:teuthology.orchestra.run.smithi039.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T22:07:10.523 INFO:teuthology.orchestra.run.smithi039.stdout:osd.1: {} 2024-09-21T22:07:10.524 INFO:teuthology.orchestra.run.smithi039.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T22:07:10.542 INFO:teuthology.orchestra.run.smithi039.stdout:osd.2: {} 2024-09-21T22:07:10.542 INFO:teuthology.orchestra.run.smithi039.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T22:07:10.543 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:10.540+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:10.562 INFO:teuthology.orchestra.run.smithi039.stdout:osd.3: {} 2024-09-21T22:07:10.562 INFO:teuthology.orchestra.run.smithi039.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T22:07:10.582 INFO:teuthology.orchestra.run.smithi039.stdout:osd.4: {} 2024-09-21T22:07:10.583 INFO:teuthology.orchestra.run.smithi039.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T22:07:10.583 INFO:teuthology.orchestra.run.smithi039.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-21T22:07:10.586 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T22:07:10.586 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T22:07:10.589 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:07:10.589 INFO:teuthology.orchestra.run.smithi039.stdout: "size": 20, 2024-09-21T22:07:10.589 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 600, 2024-09-21T22:07:10.589 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [ 2024-09-21T22:07:10.589 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.589 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:20 1.0 e14/13 1:30306672:devicehealth::main.db.0000000000000000:head v 14'10, mlcod=14'8)", 2024-09-21T22:07:10.589 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.137639+0000", 2024-09-21T22:07:10.589 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 68.449261036999999, 2024-09-21T22:07:10.589 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0039902100000000001, 2024-09-21T22:07:10.589 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:10.589 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:10.589 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:10.590 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:10.590 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.590 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:10.590 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.137639+0000", 2024-09-21T22:07:10.590 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:10.590 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.590 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.590 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:10.590 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.137639+0000", 2024-09-21T22:07:10.590 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:10.590 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.590 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.590 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:10.591 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.137641+0000", 2024-09-21T22:07:10.591 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.686e-06 2024-09-21T22:07:10.591 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.591 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.591 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:10.591 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.137934+0000", 2024-09-21T22:07:10.591 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00029319099999999999 2024-09-21T22:07:10.591 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.591 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.591 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:10.591 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.137963+0000", 2024-09-21T22:07:10.591 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9502999999999999e-05 2024-09-21T22:07:10.591 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.592 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.592 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:10.592 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.137989+0000", 2024-09-21T22:07:10.592 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5573000000000001e-05 2024-09-21T22:07:10.592 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.592 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.592 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:10.592 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.138086+0000", 2024-09-21T22:07:10.592 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.6968999999999999e-05 2024-09-21T22:07:10.592 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.592 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.592 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:10.592 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.138120+0000", 2024-09-21T22:07:10.592 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.4560000000000001e-05 2024-09-21T22:07:10.593 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.593 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.593 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:10.593 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.141539+0000", 2024-09-21T22:07:10.593 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0034183289999999999 2024-09-21T22:07:10.593 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.593 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.593 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:10.593 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.141629+0000", 2024-09-21T22:07:10.593 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.0399e-05 2024-09-21T22:07:10.593 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:10.593 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:10.593 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:10.593 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.594 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.594 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:23 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'13, mlcod=14'11)", 2024-09-21T22:07:10.594 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.144914+0000", 2024-09-21T22:07:10.594 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 68.441985719000002, 2024-09-21T22:07:10.594 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0040223230000000004, 2024-09-21T22:07:10.594 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:10.594 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:10.594 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:10.594 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:10.594 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.594 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:10.594 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.144914+0000", 2024-09-21T22:07:10.594 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:10.594 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.595 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.595 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:10.595 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.144914+0000", 2024-09-21T22:07:10.595 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:10.595 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.595 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.595 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:10.595 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.144916+0000", 2024-09-21T22:07:10.595 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6470000000000001e-06 2024-09-21T22:07:10.595 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.595 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.595 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:10.595 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.144929+0000", 2024-09-21T22:07:10.595 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2903e-05 2024-09-21T22:07:10.596 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.596 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.596 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:10.596 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.144958+0000", 2024-09-21T22:07:10.596 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9280000000000001e-05 2024-09-21T22:07:10.596 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.596 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.596 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:10.596 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.144971+0000", 2024-09-21T22:07:10.596 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3407000000000001e-05 2024-09-21T22:07:10.596 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.596 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.596 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:10.597 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.145106+0000", 2024-09-21T22:07:10.597 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013440500000000001 2024-09-21T22:07:10.597 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.597 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.597 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:10.597 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.145170+0000", 2024-09-21T22:07:10.597 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.4597000000000005e-05 2024-09-21T22:07:10.597 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.597 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.597 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:10.597 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.148854+0000", 2024-09-21T22:07:10.597 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0036833249999999999 2024-09-21T22:07:10.597 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.597 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.598 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:10.598 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.148937+0000", 2024-09-21T22:07:10.598 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.2758999999999998e-05 2024-09-21T22:07:10.598 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:10.598 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:10.598 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:10.598 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.598 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.598 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:31 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'17, mlcod=14'14)", 2024-09-21T22:07:10.598 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.163958+0000", 2024-09-21T22:07:10.598 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 68.422941452000003, 2024-09-21T22:07:10.598 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0033891149999999998, 2024-09-21T22:07:10.598 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:10.598 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:10.599 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:10.599 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:10.599 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.599 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:10.599 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.163958+0000", 2024-09-21T22:07:10.599 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:10.599 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.599 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.599 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:10.599 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.163958+0000", 2024-09-21T22:07:10.599 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:10.599 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.599 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.599 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:10.600 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.163960+0000", 2024-09-21T22:07:10.600 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.64e-06 2024-09-21T22:07:10.600 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.600 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.600 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:10.600 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.163971+0000", 2024-09-21T22:07:10.600 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0797e-05 2024-09-21T22:07:10.600 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.600 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.600 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:10.600 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.163995+0000", 2024-09-21T22:07:10.600 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3876e-05 2024-09-21T22:07:10.600 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.600 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.601 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:10.601 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.164008+0000", 2024-09-21T22:07:10.601 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3674e-05 2024-09-21T22:07:10.601 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.601 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.601 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:10.601 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.164203+0000", 2024-09-21T22:07:10.601 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00019457799999999999 2024-09-21T22:07:10.601 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.601 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.601 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:10.601 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.164265+0000", 2024-09-21T22:07:10.601 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.2415999999999993e-05 2024-09-21T22:07:10.601 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.602 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.602 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:10.602 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.167209+0000", 2024-09-21T22:07:10.602 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0029436380000000002 2024-09-21T22:07:10.602 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.602 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.602 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:10.602 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.167348+0000", 2024-09-21T22:07:10.602 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000138496 2024-09-21T22:07:10.602 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:10.602 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:10.602 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:10.602 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.602 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.602 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:32 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'18, mlcod=14'14)", 2024-09-21T22:07:10.603 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.165319+0000", 2024-09-21T22:07:10.603 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 68.421580602000006, 2024-09-21T22:07:10.603 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0033722280000000001, 2024-09-21T22:07:10.603 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:10.603 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:10.603 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:10.603 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:10.603 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.603 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:10.603 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.165319+0000", 2024-09-21T22:07:10.603 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:10.603 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.603 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.603 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:10.604 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.165319+0000", 2024-09-21T22:07:10.604 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:10.604 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.604 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.604 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:10.604 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.165321+0000", 2024-09-21T22:07:10.604 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5999999999999999e-06 2024-09-21T22:07:10.604 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.604 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.604 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:10.604 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.165377+0000", 2024-09-21T22:07:10.604 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.5877000000000001e-05 2024-09-21T22:07:10.604 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.604 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.605 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:10.605 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.165407+0000", 2024-09-21T22:07:10.605 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.0263000000000001e-05 2024-09-21T22:07:10.605 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.605 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.605 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:10.605 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.165441+0000", 2024-09-21T22:07:10.605 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.3966e-05 2024-09-21T22:07:10.605 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.605 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.605 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:10.605 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.165574+0000", 2024-09-21T22:07:10.605 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013327599999999999 2024-09-21T22:07:10.605 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.605 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.606 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:10.606 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.165635+0000", 2024-09-21T22:07:10.606 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.0949000000000002e-05 2024-09-21T22:07:10.606 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.606 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.606 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:10.606 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.168619+0000", 2024-09-21T22:07:10.606 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0029836979999999999 2024-09-21T22:07:10.606 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.606 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.606 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:10.606 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.168692+0000", 2024-09-21T22:07:10.606 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.2599000000000001e-05 2024-09-21T22:07:10.607 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:10.607 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:10.607 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:10.607 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.607 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.607 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:38 1.0 e14/13 1:30306672:devicehealth::main.db.0000000000000000:head v 14'24, mlcod=14'22)", 2024-09-21T22:07:10.607 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.180244+0000", 2024-09-21T22:07:10.607 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 68.406655956999998, 2024-09-21T22:07:10.607 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0041712219999999996, 2024-09-21T22:07:10.607 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:10.607 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:10.607 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:10.607 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:10.607 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.607 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:10.608 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.180244+0000", 2024-09-21T22:07:10.608 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:10.608 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.608 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.608 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:10.608 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.180244+0000", 2024-09-21T22:07:10.608 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:10.608 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.608 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.608 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:10.608 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.180246+0000", 2024-09-21T22:07:10.608 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6959999999999999e-06 2024-09-21T22:07:10.608 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.608 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.609 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:10.609 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.180508+0000", 2024-09-21T22:07:10.609 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00026275199999999999 2024-09-21T22:07:10.609 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.609 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.609 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:10.609 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.180538+0000", 2024-09-21T22:07:10.609 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9142999999999999e-05 2024-09-21T22:07:10.609 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.609 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.609 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:10.609 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.180552+0000", 2024-09-21T22:07:10.609 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4466e-05 2024-09-21T22:07:10.609 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.610 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.610 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:10.610 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.180677+0000", 2024-09-21T22:07:10.610 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000125413 2024-09-21T22:07:10.610 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.610 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.610 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:10.610 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.180739+0000", 2024-09-21T22:07:10.610 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.1122000000000002e-05 2024-09-21T22:07:10.610 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.610 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.610 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:10.610 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.184291+0000", 2024-09-21T22:07:10.610 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0035525510000000001 2024-09-21T22:07:10.611 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.611 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.611 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:10.611 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.184415+0000", 2024-09-21T22:07:10.611 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000124079 2024-09-21T22:07:10.611 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:10.611 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:10.611 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:10.611 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.611 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.611 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:54 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'32, mlcod=14'29)", 2024-09-21T22:07:10.611 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.214326+0000", 2024-09-21T22:07:10.611 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 68.372573720000005, 2024-09-21T22:07:10.611 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0035226569999999998, 2024-09-21T22:07:10.612 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:10.612 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:10.612 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:10.612 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:10.612 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.612 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:10.612 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.214326+0000", 2024-09-21T22:07:10.612 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:10.612 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.612 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.612 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:10.612 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.214326+0000", 2024-09-21T22:07:10.612 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:10.613 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.613 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.613 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:10.613 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.214328+0000", 2024-09-21T22:07:10.613 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6270000000000001e-06 2024-09-21T22:07:10.613 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.613 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.613 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:10.613 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.214340+0000", 2024-09-21T22:07:10.613 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2286000000000001e-05 2024-09-21T22:07:10.613 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.613 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.613 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:10.613 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.214369+0000", 2024-09-21T22:07:10.614 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9297000000000001e-05 2024-09-21T22:07:10.614 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.614 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.614 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:10.614 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.214383+0000", 2024-09-21T22:07:10.614 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3773e-05 2024-09-21T22:07:10.614 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.614 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.614 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:10.614 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.214513+0000", 2024-09-21T22:07:10.614 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000130056 2024-09-21T22:07:10.614 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.614 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.614 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:10.615 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.214568+0000", 2024-09-21T22:07:10.615 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.4873000000000001e-05 2024-09-21T22:07:10.615 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.615 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.615 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:10.615 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.217739+0000", 2024-09-21T22:07:10.615 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0031712720000000002 2024-09-21T22:07:10.615 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.615 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.615 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:10.615 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.217849+0000", 2024-09-21T22:07:10.615 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000109473 2024-09-21T22:07:10.615 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:10.615 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:10.616 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:10.616 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.616 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.616 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:57 1.0 e14/13 1:30306672:devicehealth::main.db.0000000000000000:head v 14'35, mlcod=14'33)", 2024-09-21T22:07:10.616 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.221549+0000", 2024-09-21T22:07:10.616 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 68.365351220999997, 2024-09-21T22:07:10.616 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0041720860000000002, 2024-09-21T22:07:10.616 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:10.616 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:10.616 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:10.616 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:10.616 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.616 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:10.616 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.221549+0000", 2024-09-21T22:07:10.617 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:10.617 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.617 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.617 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:10.617 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.221549+0000", 2024-09-21T22:07:10.617 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:10.617 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.617 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.617 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:10.617 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.221550+0000", 2024-09-21T22:07:10.617 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.717e-06 2024-09-21T22:07:10.617 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.617 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.618 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:10.618 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.221673+0000", 2024-09-21T22:07:10.618 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000122612 2024-09-21T22:07:10.618 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.618 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.618 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:10.618 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.221703+0000", 2024-09-21T22:07:10.618 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9539999999999998e-05 2024-09-21T22:07:10.618 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.618 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.618 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:10.618 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.221716+0000", 2024-09-21T22:07:10.618 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3149999999999999e-05 2024-09-21T22:07:10.618 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.619 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.619 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:10.619 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.221848+0000", 2024-09-21T22:07:10.619 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013201900000000001 2024-09-21T22:07:10.619 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.619 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.619 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:10.619 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.221908+0000", 2024-09-21T22:07:10.619 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.0452999999999999e-05 2024-09-21T22:07:10.619 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.619 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.619 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:10.619 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.225573+0000", 2024-09-21T22:07:10.619 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.003664676 2024-09-21T22:07:10.620 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.620 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.620 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:10.620 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.225721+0000", 2024-09-21T22:07:10.620 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00014791899999999999 2024-09-21T22:07:10.620 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:10.620 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:10.620 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:10.620 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.620 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.620 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:68 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'42, mlcod=14'39)", 2024-09-21T22:07:10.620 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.247413+0000", 2024-09-21T22:07:10.620 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 68.339486871999995, 2024-09-21T22:07:10.620 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0039572339999999996, 2024-09-21T22:07:10.621 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:10.621 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:10.621 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:10.621 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:10.621 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.621 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:10.621 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.247413+0000", 2024-09-21T22:07:10.621 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:10.621 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.621 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.621 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:10.621 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.247413+0000", 2024-09-21T22:07:10.621 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:10.621 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.621 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.622 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:10.622 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.247415+0000", 2024-09-21T22:07:10.622 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.883e-06 2024-09-21T22:07:10.622 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.622 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.622 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:10.622 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.247427+0000", 2024-09-21T22:07:10.622 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2557e-05 2024-09-21T22:07:10.622 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.622 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.622 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:10.622 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.247457+0000", 2024-09-21T22:07:10.622 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9966000000000001e-05 2024-09-21T22:07:10.622 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.623 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.623 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:10.623 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.247471+0000", 2024-09-21T22:07:10.623 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3747000000000001e-05 2024-09-21T22:07:10.623 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.623 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.623 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:10.623 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.247603+0000", 2024-09-21T22:07:10.623 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013171600000000001 2024-09-21T22:07:10.623 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.623 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.623 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:10.623 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.247656+0000", 2024-09-21T22:07:10.623 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.3529999999999997e-05 2024-09-21T22:07:10.624 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.624 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.624 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:10.624 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.251225+0000", 2024-09-21T22:07:10.624 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00356865 2024-09-21T22:07:10.624 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.624 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.624 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:10.624 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.251370+0000", 2024-09-21T22:07:10.624 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00014518500000000001 2024-09-21T22:07:10.624 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:10.624 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:10.624 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:10.624 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.625 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.625 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:69 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'43, mlcod=14'39)", 2024-09-21T22:07:10.625 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.248839+0000", 2024-09-21T22:07:10.625 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 68.338060941999998, 2024-09-21T22:07:10.625 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0034954470000000001, 2024-09-21T22:07:10.625 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:10.625 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:10.625 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:10.625 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:10.625 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.625 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:10.625 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.248839+0000", 2024-09-21T22:07:10.625 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:10.625 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.626 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.626 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:10.626 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.248839+0000", 2024-09-21T22:07:10.626 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:10.626 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.626 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.626 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:10.626 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.248841+0000", 2024-09-21T22:07:10.626 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6759999999999999e-06 2024-09-21T22:07:10.626 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.626 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.626 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:10.626 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.249058+0000", 2024-09-21T22:07:10.626 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000217455 2024-09-21T22:07:10.627 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.627 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.627 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:10.627 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.249085+0000", 2024-09-21T22:07:10.627 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.7002999999999999e-05 2024-09-21T22:07:10.627 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.627 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.627 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:10.627 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.249099+0000", 2024-09-21T22:07:10.627 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.402e-05 2024-09-21T22:07:10.627 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.627 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.627 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:10.628 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.249241+0000", 2024-09-21T22:07:10.628 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00014194899999999999 2024-09-21T22:07:10.628 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.628 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.628 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:10.628 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.249298+0000", 2024-09-21T22:07:10.628 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.6687000000000001e-05 2024-09-21T22:07:10.628 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.628 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.628 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:10.628 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.252266+0000", 2024-09-21T22:07:10.628 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.002968348 2024-09-21T22:07:10.628 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.628 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.629 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:10.629 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.252334+0000", 2024-09-21T22:07:10.629 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.8308999999999999e-05 2024-09-21T22:07:10.629 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:10.629 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:10.629 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:10.629 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.629 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.629 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:71 1.0 e14/13 1:30306672:devicehealth::main.db.0000000000000000:head v 14'45, mlcod=14'43)", 2024-09-21T22:07:10.629 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.256068+0000", 2024-09-21T22:07:10.629 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 68.330831470000007, 2024-09-21T22:07:10.629 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0041712579999999997, 2024-09-21T22:07:10.629 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:10.629 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:10.630 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:10.630 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:10.630 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.630 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:10.630 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.256068+0000", 2024-09-21T22:07:10.630 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:10.630 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.630 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.630 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:10.630 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.256068+0000", 2024-09-21T22:07:10.630 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:10.630 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.630 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.630 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:10.630 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.256070+0000", 2024-09-21T22:07:10.631 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.61e-06 2024-09-21T22:07:10.631 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.631 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.631 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:10.631 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.256129+0000", 2024-09-21T22:07:10.631 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.8702999999999997e-05 2024-09-21T22:07:10.631 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.631 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.631 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:10.631 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.256158+0000", 2024-09-21T22:07:10.631 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9533e-05 2024-09-21T22:07:10.631 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.631 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.631 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:10.632 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.256173+0000", 2024-09-21T22:07:10.632 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4267000000000001e-05 2024-09-21T22:07:10.632 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.632 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.632 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:10.632 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.256304+0000", 2024-09-21T22:07:10.632 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000131352 2024-09-21T22:07:10.632 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.632 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.632 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:10.632 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.256357+0000", 2024-09-21T22:07:10.632 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.3492999999999998e-05 2024-09-21T22:07:10.632 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.632 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.633 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:10.633 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.260147+0000", 2024-09-21T22:07:10.633 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0037900410000000001 2024-09-21T22:07:10.633 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.633 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.633 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:10.633 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.260240+0000", 2024-09-21T22:07:10.633 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.2258999999999999e-05 2024-09-21T22:07:10.633 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:10.633 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:10.633 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:10.633 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.633 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.633 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:72 1.0 e14/13 1:30306672:devicehealth::main.db.0000000000000000:head v 14'46, mlcod=14'43)", 2024-09-21T22:07:10.634 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.258358+0000", 2024-09-21T22:07:10.634 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 68.328541477000002, 2024-09-21T22:07:10.634 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0041492919999999997, 2024-09-21T22:07:10.634 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:10.634 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:10.634 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:10.634 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:10.634 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.634 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:10.634 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.258358+0000", 2024-09-21T22:07:10.634 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:10.634 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.634 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.634 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:10.635 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.258358+0000", 2024-09-21T22:07:10.635 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:10.635 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.635 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.635 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:10.635 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.258360+0000", 2024-09-21T22:07:10.635 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.623e-06 2024-09-21T22:07:10.635 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.635 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.635 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:10.635 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.258415+0000", 2024-09-21T22:07:10.635 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.4756000000000002e-05 2024-09-21T22:07:10.635 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.635 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.636 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:10.636 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.258444+0000", 2024-09-21T22:07:10.636 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9164000000000001e-05 2024-09-21T22:07:10.636 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.636 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.636 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:10.636 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.258459+0000", 2024-09-21T22:07:10.636 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4996e-05 2024-09-21T22:07:10.636 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.636 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.636 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:10.636 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.258637+0000", 2024-09-21T22:07:10.636 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00017796500000000001 2024-09-21T22:07:10.636 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.636 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.637 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:10.637 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.258695+0000", 2024-09-21T22:07:10.637 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.8207e-05 2024-09-21T22:07:10.637 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.637 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.637 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:10.637 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.262421+0000", 2024-09-21T22:07:10.637 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0037261550000000001 2024-09-21T22:07:10.637 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.637 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.637 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:10.637 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.262508+0000", 2024-09-21T22:07:10.637 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.6426000000000002e-05 2024-09-21T22:07:10.637 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:10.638 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:10.638 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:10.638 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.638 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.638 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:87 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'57, mlcod=14'52)", 2024-09-21T22:07:10.638 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.290115+0000", 2024-09-21T22:07:10.638 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 68.296785032000003, 2024-09-21T22:07:10.638 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0030478200000000001, 2024-09-21T22:07:10.638 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:10.638 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:10.638 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:10.638 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:10.638 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.638 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:10.639 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.290115+0000", 2024-09-21T22:07:10.639 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:10.639 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.639 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.639 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:10.639 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.290115+0000", 2024-09-21T22:07:10.639 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:10.639 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.639 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.639 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:10.639 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.290116+0000", 2024-09-21T22:07:10.639 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3829999999999999e-06 2024-09-21T22:07:10.639 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.639 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.640 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:10.640 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.290191+0000", 2024-09-21T22:07:10.640 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.4562999999999997e-05 2024-09-21T22:07:10.640 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.640 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.640 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:10.640 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.290216+0000", 2024-09-21T22:07:10.640 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.565e-05 2024-09-21T22:07:10.640 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.640 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.640 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:10.640 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.290236+0000", 2024-09-21T22:07:10.640 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9326000000000001e-05 2024-09-21T22:07:10.640 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.641 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.641 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:10.641 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.290340+0000", 2024-09-21T22:07:10.641 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0001047 2024-09-21T22:07:10.641 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.641 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.641 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:10.641 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.290377+0000", 2024-09-21T22:07:10.641 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.6356000000000002e-05 2024-09-21T22:07:10.641 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.641 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.641 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:10.641 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.293096+0000", 2024-09-21T22:07:10.641 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.002718996 2024-09-21T22:07:10.642 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.642 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.642 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:10.642 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.293163+0000", 2024-09-21T22:07:10.642 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.6846000000000003e-05 2024-09-21T22:07:10.642 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:10.642 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:10.642 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:10.642 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.642 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.642 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:114 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'76, mlcod=14'73)", 2024-09-21T22:07:10.642 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.490030+0000", 2024-09-21T22:07:10.642 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 68.096869592999994, 2024-09-21T22:07:10.642 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.004227873, 2024-09-21T22:07:10.642 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:10.643 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:10.643 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:10.643 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:10.643 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.643 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:10.643 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.490030+0000", 2024-09-21T22:07:10.643 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:10.643 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.643 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.643 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:10.643 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.490030+0000", 2024-09-21T22:07:10.643 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:10.643 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.643 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.644 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:10.644 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.490031+0000", 2024-09-21T22:07:10.644 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.3300000000000001e-07 2024-09-21T22:07:10.644 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.644 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.644 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:10.644 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.490042+0000", 2024-09-21T22:07:10.644 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.1095e-05 2024-09-21T22:07:10.644 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.644 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.644 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:10.644 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.490053+0000", 2024-09-21T22:07:10.644 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0501e-05 2024-09-21T22:07:10.644 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.644 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.645 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:10.645 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.490060+0000", 2024-09-21T22:07:10.645 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.9829999999999999e-06 2024-09-21T22:07:10.645 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.645 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.645 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:10.645 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.490107+0000", 2024-09-21T22:07:10.645 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.7278999999999999e-05 2024-09-21T22:07:10.645 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.645 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.645 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:10.645 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.490124+0000", 2024-09-21T22:07:10.645 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7190999999999999e-05 2024-09-21T22:07:10.645 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.646 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.646 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:10.646 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.494229+0000", 2024-09-21T22:07:10.646 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0041050440000000004 2024-09-21T22:07:10.646 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.646 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.646 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:10.646 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.494258+0000", 2024-09-21T22:07:10.646 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9047000000000002e-05 2024-09-21T22:07:10.646 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:10.646 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:10.646 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:10.646 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.647 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.647 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:115 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'77, mlcod=14'73)", 2024-09-21T22:07:10.647 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.491380+0000", 2024-09-21T22:07:10.647 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 68.095519971000002, 2024-09-21T22:07:10.647 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0031550889999999998, 2024-09-21T22:07:10.647 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:10.647 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:10.647 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:10.647 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:10.647 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.647 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:10.647 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.491380+0000", 2024-09-21T22:07:10.647 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:10.647 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.647 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.648 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:10.648 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.491380+0000", 2024-09-21T22:07:10.648 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:10.648 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.648 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.648 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:10.648 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.491381+0000", 2024-09-21T22:07:10.648 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.1399999999999996e-07 2024-09-21T22:07:10.648 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.648 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.648 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:10.648 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.491435+0000", 2024-09-21T22:07:10.648 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.3797e-05 2024-09-21T22:07:10.648 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.649 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.649 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:10.649 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.491447+0000", 2024-09-21T22:07:10.649 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2642e-05 2024-09-21T22:07:10.649 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.649 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.649 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:10.649 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.491453+0000", 2024-09-21T22:07:10.649 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.4820000000000002e-06 2024-09-21T22:07:10.649 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.649 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.649 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:10.649 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.491502+0000", 2024-09-21T22:07:10.649 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.9038e-05 2024-09-21T22:07:10.649 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.650 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.650 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:10.650 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.491514+0000", 2024-09-21T22:07:10.650 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2483999999999999e-05 2024-09-21T22:07:10.650 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.650 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.650 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:10.650 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.494486+0000", 2024-09-21T22:07:10.650 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.002971338 2024-09-21T22:07:10.650 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.650 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.650 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:10.650 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.494535+0000", 2024-09-21T22:07:10.650 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.9493999999999997e-05 2024-09-21T22:07:10.651 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:10.651 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:10.651 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:10.651 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.651 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.651 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:118 1.0 e14/13 1:30306672:devicehealth::main.db.0000000000000000:head v 14'80, mlcod=14'78)", 2024-09-21T22:07:10.651 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.499045+0000", 2024-09-21T22:07:10.651 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 68.087854969000006, 2024-09-21T22:07:10.651 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0029889589999999998, 2024-09-21T22:07:10.651 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:10.651 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:10.651 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:10.651 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:10.651 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.651 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:10.652 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.499045+0000", 2024-09-21T22:07:10.652 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:10.652 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.652 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.652 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:10.652 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.499045+0000", 2024-09-21T22:07:10.652 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:10.652 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.652 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.652 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:10.652 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.499046+0000", 2024-09-21T22:07:10.652 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.0299999999999998e-07 2024-09-21T22:07:10.652 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.652 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.653 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:10.653 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.499110+0000", 2024-09-21T22:07:10.653 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.4688000000000001e-05 2024-09-21T22:07:10.653 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.653 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.653 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:10.653 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.499125+0000", 2024-09-21T22:07:10.653 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4387e-05 2024-09-21T22:07:10.653 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.653 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.653 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:10.653 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.499131+0000", 2024-09-21T22:07:10.653 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.2129999999999996e-06 2024-09-21T22:07:10.653 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.653 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.654 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:10.654 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.499255+0000", 2024-09-21T22:07:10.654 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00012424199999999999 2024-09-21T22:07:10.654 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.654 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.654 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:10.654 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.499312+0000", 2024-09-21T22:07:10.654 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.6709999999999997e-05 2024-09-21T22:07:10.654 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.654 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.654 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:10.654 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.502005+0000", 2024-09-21T22:07:10.654 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0026927169999999999 2024-09-21T22:07:10.654 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.655 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.655 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:10.655 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.502034+0000", 2024-09-21T22:07:10.655 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9198999999999999e-05 2024-09-21T22:07:10.655 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:10.655 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:10.655 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:10.655 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.655 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.655 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:136 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'94, mlcod=14'89)", 2024-09-21T22:07:10.655 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.903810+0000", 2024-09-21T22:07:10.655 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 67.683089998, 2024-09-21T22:07:10.655 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0062746829999999996, 2024-09-21T22:07:10.655 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:10.656 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:10.656 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:10.656 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:10.656 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.656 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:10.656 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.903810+0000", 2024-09-21T22:07:10.656 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:10.656 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.656 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.656 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:10.656 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.903810+0000", 2024-09-21T22:07:10.656 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:10.656 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.656 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.656 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:10.657 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.903810+0000", 2024-09-21T22:07:10.657 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.03e-07 2024-09-21T22:07:10.657 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.657 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.657 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:10.657 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.903814+0000", 2024-09-21T22:07:10.657 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.676e-06 2024-09-21T22:07:10.657 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.657 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.657 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:10.657 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.903822+0000", 2024-09-21T22:07:10.657 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.2879999999999993e-06 2024-09-21T22:07:10.657 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.657 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.657 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:10.658 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.903829+0000", 2024-09-21T22:07:10.658 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.2920000000000001e-06 2024-09-21T22:07:10.658 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.658 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.658 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:10.658 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.903879+0000", 2024-09-21T22:07:10.658 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.0253999999999997e-05 2024-09-21T22:07:10.658 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.658 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.658 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:10.658 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.903897+0000", 2024-09-21T22:07:10.658 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8468999999999999e-05 2024-09-21T22:07:10.658 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.659 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.659 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:10.659 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.910044+0000", 2024-09-21T22:07:10.659 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0061468950000000003 2024-09-21T22:07:10.659 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.659 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.659 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:10.659 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.910085+0000", 2024-09-21T22:07:10.659 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.0405999999999999e-05 2024-09-21T22:07:10.659 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:10.659 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:10.659 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:10.659 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.659 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.659 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:137 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'95, mlcod=14'90)", 2024-09-21T22:07:10.660 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.904439+0000", 2024-09-21T22:07:10.660 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 67.682460602000006, 2024-09-21T22:07:10.660 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0072625019999999997, 2024-09-21T22:07:10.660 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:10.660 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:10.660 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:10.660 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:10.660 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.660 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:10.660 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.904439+0000", 2024-09-21T22:07:10.660 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:10.660 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.660 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.660 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:10.661 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.904439+0000", 2024-09-21T22:07:10.661 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:10.661 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.661 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.661 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:10.661 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.904440+0000", 2024-09-21T22:07:10.661 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8400000000000005e-07 2024-09-21T22:07:10.661 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.661 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.661 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:10.661 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.904498+0000", 2024-09-21T22:07:10.661 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.8320999999999999e-05 2024-09-21T22:07:10.661 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.661 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.661 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:10.662 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.904507+0000", 2024-09-21T22:07:10.662 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.7790000000000001e-06 2024-09-21T22:07:10.662 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.662 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.662 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:10.662 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.904511+0000", 2024-09-21T22:07:10.662 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.9700000000000001e-06 2024-09-21T22:07:10.662 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.662 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.662 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:10.662 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.904551+0000", 2024-09-21T22:07:10.662 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.9910000000000002e-05 2024-09-21T22:07:10.662 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.662 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.663 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:10.663 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.904567+0000", 2024-09-21T22:07:10.663 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5903999999999999e-05 2024-09-21T22:07:10.663 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.663 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.663 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:10.663 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.911666+0000", 2024-09-21T22:07:10.663 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0070996929999999998 2024-09-21T22:07:10.663 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.663 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.663 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:10.663 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.911702+0000", 2024-09-21T22:07:10.663 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.5441000000000002e-05 2024-09-21T22:07:10.663 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:10.663 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:10.664 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:10.664 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.664 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.664 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:138 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'96, mlcod=14'90)", 2024-09-21T22:07:10.664 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.905402+0000", 2024-09-21T22:07:10.664 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 67.681497625000006, 2024-09-21T22:07:10.664 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0063366890000000004, 2024-09-21T22:07:10.664 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:10.664 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:10.664 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:10.664 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:10.664 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.664 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:10.664 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905402+0000", 2024-09-21T22:07:10.665 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:10.665 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.665 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.665 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:10.665 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905402+0000", 2024-09-21T22:07:10.665 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:10.665 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.665 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.665 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:10.665 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905403+0000", 2024-09-21T22:07:10.665 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.9200000000000002e-07 2024-09-21T22:07:10.665 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.665 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.665 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:10.665 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905406+0000", 2024-09-21T22:07:10.666 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.326e-06 2024-09-21T22:07:10.666 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.666 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.666 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:10.666 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905413+0000", 2024-09-21T22:07:10.666 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.3320000000000003e-06 2024-09-21T22:07:10.666 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.666 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.666 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:10.666 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905417+0000", 2024-09-21T22:07:10.666 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.7139999999999999e-06 2024-09-21T22:07:10.666 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.666 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.666 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:10.667 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.910115+0000", 2024-09-21T22:07:10.667 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0046983939999999998 2024-09-21T22:07:10.667 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.667 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.667 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:10.667 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.910140+0000", 2024-09-21T22:07:10.667 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4689999999999999e-05 2024-09-21T22:07:10.667 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.667 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.667 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:10.667 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.911706+0000", 2024-09-21T22:07:10.667 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0015658479999999999 2024-09-21T22:07:10.667 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.667 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.668 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:10.668 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.911739+0000", 2024-09-21T22:07:10.668 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.2993000000000001e-05 2024-09-21T22:07:10.668 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:10.668 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:10.668 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:10.668 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.668 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.668 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:139 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'97, mlcod=14'90)", 2024-09-21T22:07:10.668 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.905983+0000", 2024-09-21T22:07:10.668 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 67.680916706999994, 2024-09-21T22:07:10.668 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0057906030000000001, 2024-09-21T22:07:10.668 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:10.668 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:10.668 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:10.669 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:10.669 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.669 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:10.669 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905983+0000", 2024-09-21T22:07:10.669 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:10.669 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.669 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.669 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:10.669 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905983+0000", 2024-09-21T22:07:10.669 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:10.669 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.669 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.669 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:10.669 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905984+0000", 2024-09-21T22:07:10.670 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.15e-07 2024-09-21T22:07:10.670 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.670 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.670 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:10.670 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905986+0000", 2024-09-21T22:07:10.670 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.852e-06 2024-09-21T22:07:10.670 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.670 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.670 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:10.670 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905993+0000", 2024-09-21T22:07:10.670 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.652e-06 2024-09-21T22:07:10.670 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.670 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.670 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:10.671 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905996+0000", 2024-09-21T22:07:10.671 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.3519999999999998e-06 2024-09-21T22:07:10.671 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.671 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.671 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:10.671 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.910519+0000", 2024-09-21T22:07:10.671 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0045230820000000003 2024-09-21T22:07:10.671 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.671 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.671 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:10.671 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.910548+0000", 2024-09-21T22:07:10.671 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8399999999999999e-05 2024-09-21T22:07:10.671 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.671 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.671 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:10.672 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.911742+0000", 2024-09-21T22:07:10.672 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011939909999999999 2024-09-21T22:07:10.672 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.672 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.672 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:10.672 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.911774+0000", 2024-09-21T22:07:10.672 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.1958999999999997e-05 2024-09-21T22:07:10.672 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:10.672 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:10.672 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:10.672 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.672 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.672 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:140 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'98, mlcod=14'90)", 2024-09-21T22:07:10.672 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.906441+0000", 2024-09-21T22:07:10.673 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 67.680458578, 2024-09-21T22:07:10.673 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0057052989999999996, 2024-09-21T22:07:10.673 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:10.673 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:10.673 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:10.673 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:10.673 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.673 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:10.673 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.906441+0000", 2024-09-21T22:07:10.673 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:10.673 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.673 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.673 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:10.673 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.906441+0000", 2024-09-21T22:07:10.673 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:10.674 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.674 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.674 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:10.674 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.906442+0000", 2024-09-21T22:07:10.674 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.7e-07 2024-09-21T22:07:10.674 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.674 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.674 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:10.674 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.906511+0000", 2024-09-21T22:07:10.674 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.8857999999999997e-05 2024-09-21T22:07:10.674 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.674 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.674 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:10.674 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.906518+0000", 2024-09-21T22:07:10.675 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.1690000000000003e-06 2024-09-21T22:07:10.675 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.675 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.675 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:10.675 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.906521+0000", 2024-09-21T22:07:10.675 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.4829999999999999e-06 2024-09-21T22:07:10.675 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.675 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.675 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:10.675 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.910939+0000", 2024-09-21T22:07:10.675 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0044176809999999997 2024-09-21T22:07:10.675 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.675 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.675 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:10.675 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.910959+0000", 2024-09-21T22:07:10.676 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9898000000000002e-05 2024-09-21T22:07:10.676 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.676 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.676 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:10.676 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.912113+0000", 2024-09-21T22:07:10.676 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001153811 2024-09-21T22:07:10.676 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:10.676 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:10.676 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:10.676 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.912147+0000", 2024-09-21T22:07:10.676 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.4029000000000002e-05 2024-09-21T22:07:10.676 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:10.676 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:10.676 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:10.676 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:10.677 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:10.677 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:07:10.678 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T22:07:10.678 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:10.676+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:10.779 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:10.776+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:10.849 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:07:10.849 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:07:10.849 INFO:teuthology.orchestra.run.smithi039.stdout: "num_ops": 0 2024-09-21T22:07:10.849 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:07:10.863 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T22:07:10.908 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:10.903+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:10.980 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:10.975+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:10.981 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:07:10.981 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:07:10.982 INFO:teuthology.orchestra.run.smithi039.stdout: "complaint_time": 30, 2024-09-21T22:07:10.982 INFO:teuthology.orchestra.run.smithi039.stdout: "num_blocked_ops": 0 2024-09-21T22:07:10.982 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:07:10.996 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T22:07:11.056 INFO:teuthology.orchestra.run.smithi039.stderr:nodeep-scrub is unset 2024-09-21T22:07:11.081 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:11.075+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:11.164 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:07:11.164 INFO:teuthology.orchestra.run.smithi039.stdout: "size": 20, 2024-09-21T22:07:11.164 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 600, 2024-09-21T22:07:11.164 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [ 2024-09-21T22:07:11.164 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:11.164 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4224.0:5 2.2 e18/16 2:5cea7035:::rbd_info:head v 18'1, mlcod=0'0)", 2024-09-21T22:07:11.164 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:06.114955+0000", 2024-09-21T22:07:11.164 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 65.048527389, 2024-09-21T22:07:11.164 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0017716170000000001, 2024-09-21T22:07:11.164 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:11.164 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:11.164 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:11.165 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:11.165 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:11.165 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:11.165 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:06.114955+0000", 2024-09-21T22:07:11.165 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:11.165 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:11.165 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:11.165 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:11.165 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:06.114955+0000", 2024-09-21T22:07:11.165 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:11.165 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:11.165 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:11.165 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:11.165 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:06.114956+0000", 2024-09-21T22:07:11.166 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.9299999999999997e-07 2024-09-21T22:07:11.166 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:11.166 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:11.166 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:11.166 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:06.114964+0000", 2024-09-21T22:07:11.166 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.9840000000000001e-06 2024-09-21T22:07:11.166 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:11.166 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:11.166 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:11.166 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:06.115001+0000", 2024-09-21T22:07:11.166 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.7719999999999998e-05 2024-09-21T22:07:11.166 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:11.166 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:11.166 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:11.167 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:06.115019+0000", 2024-09-21T22:07:11.167 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7943000000000001e-05 2024-09-21T22:07:11.167 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:11.167 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:11.167 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:11.167 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:06.115168+0000", 2024-09-21T22:07:11.167 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000148589 2024-09-21T22:07:11.167 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:11.167 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:11.167 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:11.167 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:06.115225+0000", 2024-09-21T22:07:11.167 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.6975999999999998e-05 2024-09-21T22:07:11.167 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:11.167 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:11.167 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:11.168 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:06.116585+0000", 2024-09-21T22:07:11.168 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00136038 2024-09-21T22:07:11.168 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:11.168 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:11.168 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:11.168 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:06.116726+0000", 2024-09-21T22:07:11.168 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00014133200000000001 2024-09-21T22:07:11.168 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:11.168 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:11.168 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:11.168 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:11.168 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:11.168 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4224.0:7 2.2 e19/16 2:5cea7035:::rbd_info:head v 19'2, mlcod=0'0)", 2024-09-21T22:07:11.168 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:07.129567+0000", 2024-09-21T22:07:11.168 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 64.033915379999996, 2024-09-21T22:07:11.169 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0018460239999999999, 2024-09-21T22:07:11.169 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:11.169 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:11.169 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:11.169 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:11.169 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:11.169 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:11.169 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:07.129567+0000", 2024-09-21T22:07:11.169 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:11.169 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:11.169 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:11.169 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:11.169 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:07.129567+0000", 2024-09-21T22:07:11.169 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:11.169 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:11.170 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:11.170 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:11.170 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:07.129567+0000", 2024-09-21T22:07:11.170 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.9499999999999998e-07 2024-09-21T22:07:11.170 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:11.170 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:11.170 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:11.170 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:07.129580+0000", 2024-09-21T22:07:11.170 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2335e-05 2024-09-21T22:07:11.170 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:11.170 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:11.170 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:11.170 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:07.129606+0000", 2024-09-21T22:07:11.170 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.6528000000000001e-05 2024-09-21T22:07:11.171 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:11.171 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:11.171 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:11.171 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:07.129628+0000", 2024-09-21T22:07:11.171 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1549999999999999e-05 2024-09-21T22:07:11.171 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:11.171 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:11.171 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:11.171 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:07.129744+0000", 2024-09-21T22:07:11.171 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000116551 2024-09-21T22:07:11.171 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:11.171 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:11.171 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:11.171 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:07.129780+0000", 2024-09-21T22:07:11.171 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.5447000000000001e-05 2024-09-21T22:07:11.172 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:11.172 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:11.172 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:11.172 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:07.131348+0000", 2024-09-21T22:07:11.172 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001568631 2024-09-21T22:07:11.172 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:11.172 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:11.172 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:11.172 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:07.131413+0000", 2024-09-21T22:07:11.172 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.4586999999999997e-05 2024-09-21T22:07:11.173 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:11.173 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:11.173 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:11.173 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:11.173 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:11.173 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:07:11.180 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T22:07:11.182 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:11.175+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:11.283 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:11.275+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:11.294 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:07:11.294 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:07:11.294 INFO:teuthology.orchestra.run.smithi039.stdout: "num_ops": 0 2024-09-21T22:07:11.295 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:07:11.311 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T22:07:11.399 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:11.395+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:11.483 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:07:11.483 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:07:11.483 INFO:teuthology.orchestra.run.smithi039.stdout: "complaint_time": 30, 2024-09-21T22:07:11.483 INFO:teuthology.orchestra.run.smithi039.stdout: "num_blocked_ops": 0 2024-09-21T22:07:11.483 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:07:11.485 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:11.480+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:11.498 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T22:07:11.585 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:11.579+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:11.664 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:07:11.664 INFO:teuthology.orchestra.run.smithi039.stdout: "size": 20, 2024-09-21T22:07:11.664 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 600, 2024-09-21T22:07:11.665 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [ 2024-09-21T22:07:11.665 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:11.665 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4224.0:1 2.3 e17/16 2:c4c92e5a:::rbd_trash:head v 17'1, mlcod=0'0)", 2024-09-21T22:07:11.665 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:05.107810+0000", 2024-09-21T22:07:11.665 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 66.555683755999993, 2024-09-21T22:07:11.665 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0017679200000000001, 2024-09-21T22:07:11.665 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:11.665 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:11.665 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:11.665 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:11.665 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:11.665 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:11.665 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.107810+0000", 2024-09-21T22:07:11.666 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:11.666 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:11.666 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:11.666 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:11.666 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.107810+0000", 2024-09-21T22:07:11.666 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:11.666 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:11.666 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:11.666 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:11.666 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.107811+0000", 2024-09-21T22:07:11.666 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.6700000000000003e-07 2024-09-21T22:07:11.666 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:11.666 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:11.666 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:11.667 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.107822+0000", 2024-09-21T22:07:11.667 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0443e-05 2024-09-21T22:07:11.667 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:11.667 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:11.667 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:11.667 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.107853+0000", 2024-09-21T22:07:11.667 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.1046000000000001e-05 2024-09-21T22:07:11.667 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:11.667 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:11.667 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:11.667 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.107876+0000", 2024-09-21T22:07:11.667 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3547e-05 2024-09-21T22:07:11.667 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:11.667 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:11.667 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:11.668 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.108002+0000", 2024-09-21T22:07:11.668 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000125599 2024-09-21T22:07:11.668 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:11.668 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:11.668 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:11.668 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.108069+0000", 2024-09-21T22:07:11.668 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.7162999999999994e-05 2024-09-21T22:07:11.668 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:11.668 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:11.668 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:11.668 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.109385+0000", 2024-09-21T22:07:11.668 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0013162899999999999 2024-09-21T22:07:11.668 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:11.668 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:11.669 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:11.669 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.109578+0000", 2024-09-21T22:07:11.669 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000193065 2024-09-21T22:07:11.669 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:11.669 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:11.669 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:11.669 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:11.669 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:11.669 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(1.0 50/49 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 15'116, data_included: [0~262688], data_size: 262688, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@15'116, size: 262688, copy_subset: [(0, 262688)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 262688, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T22:07:11.669 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:06.036813+0000", 2024-09-21T22:07:11.669 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 5.626681177, 2024-09-21T22:07:11.669 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0022591270000000001, 2024-09-21T22:07:11.669 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:11.669 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:11.670 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:11.670 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:11.670 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:11.670 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:11.670 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:06.036813+0000", 2024-09-21T22:07:11.670 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:11.670 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:11.670 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:11.670 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:11.670 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:06.036813+0000", 2024-09-21T22:07:11.670 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:11.670 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:11.670 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:11.670 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:11.671 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:06.036814+0000", 2024-09-21T22:07:11.671 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.0400000000000005e-07 2024-09-21T22:07:11.671 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:11.671 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:11.671 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:11.671 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:06.037279+0000", 2024-09-21T22:07:11.671 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00046499600000000002 2024-09-21T22:07:11.671 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:11.671 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:11.671 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:11.671 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:06.037307+0000", 2024-09-21T22:07:11.671 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8359000000000001e-05 2024-09-21T22:07:11.671 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:11.671 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:11.672 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:11.672 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:06.037332+0000", 2024-09-21T22:07:11.672 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5154999999999999e-05 2024-09-21T22:07:11.672 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:11.672 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:11.672 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:11.672 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:06.037419+0000", 2024-09-21T22:07:11.672 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.6310000000000005e-05 2024-09-21T22:07:11.672 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:11.672 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:11.672 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:11.672 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:06.037430+0000", 2024-09-21T22:07:11.672 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0832000000000001e-05 2024-09-21T22:07:11.672 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:11.673 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:11.673 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:11.673 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:06.039072+0000", 2024-09-21T22:07:11.673 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001642571 2024-09-21T22:07:11.673 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:11.673 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:11.673 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:11.673 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:11.673 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:11.673 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(1.0 50/49 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 15'117, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@15'117, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T22:07:11.673 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:06.041121+0000", 2024-09-21T22:07:11.673 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 5.6223735960000001, 2024-09-21T22:07:11.673 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0022693829999999998, 2024-09-21T22:07:11.673 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:11.674 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:11.674 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:11.674 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:11.674 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:11.674 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:11.674 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:06.041121+0000", 2024-09-21T22:07:11.674 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:11.674 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:11.674 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:11.674 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:11.674 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:06.041121+0000", 2024-09-21T22:07:11.674 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:11.675 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:11.675 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:11.675 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:11.675 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:06.041122+0000", 2024-09-21T22:07:11.675 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.3500000000000005e-07 2024-09-21T22:07:11.675 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:11.675 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:11.675 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:11.675 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:06.041449+0000", 2024-09-21T22:07:11.675 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00032708299999999999 2024-09-21T22:07:11.675 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:11.675 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:11.675 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:11.676 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:06.041475+0000", 2024-09-21T22:07:11.676 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.6245e-05 2024-09-21T22:07:11.676 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:11.676 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:11.676 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:11.676 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:06.041498+0000", 2024-09-21T22:07:11.676 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.2802e-05 2024-09-21T22:07:11.676 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:11.676 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:11.676 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:11.676 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:06.041579+0000", 2024-09-21T22:07:11.676 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.1203000000000002e-05 2024-09-21T22:07:11.676 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:11.676 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:11.677 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:11.677 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:06.041587+0000", 2024-09-21T22:07:11.677 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.9230000000000002e-06 2024-09-21T22:07:11.677 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:11.677 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:11.677 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:11.677 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:06.043390+0000", 2024-09-21T22:07:11.677 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0018031919999999999 2024-09-21T22:07:11.677 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:11.677 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:11.677 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:11.677 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:11.678 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:11.678 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:07:11.679 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T22:07:11.720 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:11.719+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:11.786 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:11.784+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:11.816 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:07:11.816 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:07:11.816 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-09-21T22:07:11.816 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:07:11.831 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T22:07:11.887 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:11.883+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:11.988 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:11.987+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:11.999 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:07:12.000 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:07:12.000 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-21T22:07:12.000 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-21T22:07:12.000 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:07:12.013 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T22:07:12.089 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:12.083+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:12.139 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:07:12.140 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-21T22:07:12.140 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-21T22:07:12.140 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-21T22:07:12.140 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.140 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:07:12.140 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.036355+0000", 2024-09-21T22:07:12.140 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 70.099934125999994, 2024-09-21T22:07:12.140 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.062441934999999997, 2024-09-21T22:07:12.140 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:12.140 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:12.140 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:12.140 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:12.141 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:07:12.141 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:07:12.141 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1 2024-09-21T22:07:12.141 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.141 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:12.141 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.141 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:12.141 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036355+0000", 2024-09-21T22:07:12.141 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:12.141 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.141 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.141 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:12.141 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036355+0000", 2024-09-21T22:07:12.142 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:12.142 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.142 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.142 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:12.142 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036357+0000", 2024-09-21T22:07:12.142 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.446e-06 2024-09-21T22:07:12.142 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.142 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.142 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:12.142 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036373+0000", 2024-09-21T22:07:12.142 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5447e-05 2024-09-21T22:07:12.142 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.142 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.143 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:12.143 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036399+0000", 2024-09-21T22:07:12.143 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6423000000000001e-05 2024-09-21T22:07:12.143 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.143 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.143 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:12.143 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036420+0000", 2024-09-21T22:07:12.143 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1129999999999999e-05 2024-09-21T22:07:12.143 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.143 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.143 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:12.143 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036557+0000", 2024-09-21T22:07:12.143 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013686899999999999 2024-09-21T22:07:12.143 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.144 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.144 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-21T22:07:12.144 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036585+0000", 2024-09-21T22:07:12.144 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8362999999999999e-05 2024-09-21T22:07:12.144 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.144 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.144 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:12.144 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.095068+0000", 2024-09-21T22:07:12.144 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058482972000000001 2024-09-21T22:07:12.144 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.144 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.144 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:12.144 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.095516+0000", 2024-09-21T22:07:12.145 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00044802999999999998 2024-09-21T22:07:12.145 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.145 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.145 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:12.145 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.095869+0000", 2024-09-21T22:07:12.145 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000352104 2024-09-21T22:07:12.145 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.145 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.145 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:12.145 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.097055+0000", 2024-09-21T22:07:12.145 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011865599999999999 2024-09-21T22:07:12.145 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.145 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.145 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:12.145 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.098637+0000", 2024-09-21T22:07:12.146 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001581458 2024-09-21T22:07:12.146 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.146 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.146 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:12.146 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.098728+0000", 2024-09-21T22:07:12.146 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1043000000000002e-05 2024-09-21T22:07:12.146 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.146 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.146 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:12.146 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.098797+0000", 2024-09-21T22:07:12.146 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9090000000000004e-05 2024-09-21T22:07:12.146 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:12.146 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:12.146 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:12.147 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.147 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.147 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:07:12.147 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.161242+0000", 2024-09-21T22:07:12.147 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 69.975047214, 2024-09-21T22:07:12.147 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014477078000000001, 2024-09-21T22:07:12.147 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:12.147 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:12.147 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:12.147 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:12.147 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:07:12.147 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:07:12.147 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 35 2024-09-21T22:07:12.148 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.148 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:12.148 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.148 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:12.148 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161242+0000", 2024-09-21T22:07:12.148 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:12.148 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.148 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.148 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:12.148 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161242+0000", 2024-09-21T22:07:12.148 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:12.148 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.148 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.148 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:12.148 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161243+0000", 2024-09-21T22:07:12.149 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4729999999999999e-06 2024-09-21T22:07:12.149 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.149 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.149 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:12.149 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161385+0000", 2024-09-21T22:07:12.149 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014226899999999999 2024-09-21T22:07:12.149 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.149 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.149 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:12.149 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161417+0000", 2024-09-21T22:07:12.149 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1406999999999999e-05 2024-09-21T22:07:12.149 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.149 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.149 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:12.150 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161434+0000", 2024-09-21T22:07:12.150 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7623000000000001e-05 2024-09-21T22:07:12.150 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.150 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.150 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:12.150 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.169728+0000", 2024-09-21T22:07:12.150 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0082933090000000004 2024-09-21T22:07:12.150 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.150 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.150 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:12.150 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.169861+0000", 2024-09-21T22:07:12.150 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013367300000000001 2024-09-21T22:07:12.150 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.151 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.151 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:12.151 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.170186+0000", 2024-09-21T22:07:12.151 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00032466400000000001 2024-09-21T22:07:12.151 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.151 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.151 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:12.151 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.174759+0000", 2024-09-21T22:07:12.151 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0045727880000000004 2024-09-21T22:07:12.151 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.151 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.151 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:12.151 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.175497+0000", 2024-09-21T22:07:12.151 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00073803400000000004 2024-09-21T22:07:12.151 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.152 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.152 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:12.152 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.175607+0000", 2024-09-21T22:07:12.152 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010987199999999999 2024-09-21T22:07:12.152 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.152 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.152 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:12.152 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.175719+0000", 2024-09-21T22:07:12.152 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011196600000000001 2024-09-21T22:07:12.152 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:12.152 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:12.152 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:12.152 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.152 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.152 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:07:12.153 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.161483+0000", 2024-09-21T22:07:12.153 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 69.974805703000001, 2024-09-21T22:07:12.153 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016155732999999999, 2024-09-21T22:07:12.153 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:12.153 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:12.153 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:12.153 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:12.153 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:07:12.153 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:07:12.153 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 36 2024-09-21T22:07:12.153 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.153 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:12.153 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.153 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:12.154 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161483+0000", 2024-09-21T22:07:12.154 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:12.154 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.154 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.154 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:12.154 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161483+0000", 2024-09-21T22:07:12.154 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:12.154 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.154 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.154 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:12.154 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161485+0000", 2024-09-21T22:07:12.154 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5999999999999999e-06 2024-09-21T22:07:12.154 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.155 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.155 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:12.155 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161498+0000", 2024-09-21T22:07:12.155 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3166999999999999e-05 2024-09-21T22:07:12.155 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.155 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.155 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:12.155 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161526+0000", 2024-09-21T22:07:12.155 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7786000000000001e-05 2024-09-21T22:07:12.155 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.155 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.155 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:12.155 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161545+0000", 2024-09-21T22:07:12.155 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8899999999999999e-05 2024-09-21T22:07:12.155 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.156 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.156 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:12.156 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.172615+0000", 2024-09-21T22:07:12.156 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011070465 2024-09-21T22:07:12.156 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.156 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.156 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:12.156 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.172759+0000", 2024-09-21T22:07:12.156 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014416900000000001 2024-09-21T22:07:12.156 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.156 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.156 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:12.156 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.172967+0000", 2024-09-21T22:07:12.156 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020758500000000001 2024-09-21T22:07:12.157 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.157 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.157 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:12.157 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.174775+0000", 2024-09-21T22:07:12.157 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0018080990000000001 2024-09-21T22:07:12.157 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.157 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.157 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:12.157 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.177447+0000", 2024-09-21T22:07:12.157 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026717429999999999 2024-09-21T22:07:12.157 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.157 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.157 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:12.158 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.177525+0000", 2024-09-21T22:07:12.158 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7846e-05 2024-09-21T22:07:12.158 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.158 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.158 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:12.158 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.177639+0000", 2024-09-21T22:07:12.158 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000114373 2024-09-21T22:07:12.158 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:12.158 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:12.158 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:12.158 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.158 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.158 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:07:12.158 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.161577+0000", 2024-09-21T22:07:12.159 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 69.974712170000004, 2024-09-21T22:07:12.159 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016652991999999998, 2024-09-21T22:07:12.159 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:12.159 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:12.159 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:12.159 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:12.159 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:07:12.159 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:07:12.159 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 37 2024-09-21T22:07:12.159 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.159 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:12.159 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.159 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:12.159 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161577+0000", 2024-09-21T22:07:12.160 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:12.160 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.160 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.160 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:12.160 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161577+0000", 2024-09-21T22:07:12.160 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:12.160 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.160 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.160 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:12.160 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161578+0000", 2024-09-21T22:07:12.160 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5269999999999999e-06 2024-09-21T22:07:12.160 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.160 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.160 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:12.160 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161591+0000", 2024-09-21T22:07:12.161 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.296e-05 2024-09-21T22:07:12.161 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.161 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.161 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:12.161 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161629+0000", 2024-09-21T22:07:12.161 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7799000000000002e-05 2024-09-21T22:07:12.161 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.161 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.161 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:12.161 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161644+0000", 2024-09-21T22:07:12.161 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5333e-05 2024-09-21T22:07:12.161 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.161 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.161 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:12.161 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.173636+0000", 2024-09-21T22:07:12.162 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011992078 2024-09-21T22:07:12.162 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.162 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.162 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:12.162 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.173862+0000", 2024-09-21T22:07:12.162 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022556799999999999 2024-09-21T22:07:12.162 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.162 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.162 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:12.162 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.174193+0000", 2024-09-21T22:07:12.162 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000331088 2024-09-21T22:07:12.162 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.162 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.162 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:12.163 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.175740+0000", 2024-09-21T22:07:12.163 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015467910000000001 2024-09-21T22:07:12.163 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.163 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.163 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:12.163 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.178073+0000", 2024-09-21T22:07:12.163 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002333356 2024-09-21T22:07:12.163 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.163 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.163 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:12.163 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.178162+0000", 2024-09-21T22:07:12.163 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8306000000000005e-05 2024-09-21T22:07:12.163 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.163 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.163 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:12.164 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.178230+0000", 2024-09-21T22:07:12.164 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8186000000000001e-05 2024-09-21T22:07:12.164 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:12.164 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:12.164 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:12.164 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.164 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.164 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:07:12.164 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.280540+0000", 2024-09-21T22:07:12.164 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 69.855748598000005, 2024-09-21T22:07:12.164 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013246212, 2024-09-21T22:07:12.164 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:12.164 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:12.164 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:12.164 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:12.165 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:07:12.165 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:07:12.165 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 88 2024-09-21T22:07:12.165 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.165 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:12.165 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.165 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:12.165 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.280540+0000", 2024-09-21T22:07:12.165 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:12.165 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.165 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.165 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:12.165 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.280540+0000", 2024-09-21T22:07:12.165 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:12.165 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.166 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.166 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:12.166 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.280541+0000", 2024-09-21T22:07:12.166 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.54e-07 2024-09-21T22:07:12.166 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.166 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.166 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:12.166 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.280549+0000", 2024-09-21T22:07:12.166 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3329999999999997e-06 2024-09-21T22:07:12.166 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.166 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.166 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:12.166 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.280560+0000", 2024-09-21T22:07:12.166 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1003e-05 2024-09-21T22:07:12.167 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.167 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.167 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:12.167 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.280569+0000", 2024-09-21T22:07:12.167 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2430000000000005e-06 2024-09-21T22:07:12.167 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.167 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.167 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:12.167 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.291095+0000", 2024-09-21T22:07:12.167 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010526259 2024-09-21T22:07:12.167 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.167 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.167 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:12.167 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.291231+0000", 2024-09-21T22:07:12.168 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013561899999999999 2024-09-21T22:07:12.168 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.168 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.168 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:12.168 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.291457+0000", 2024-09-21T22:07:12.168 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022686200000000001 2024-09-21T22:07:12.168 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.168 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.168 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:12.168 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.293117+0000", 2024-09-21T22:07:12.168 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001659474 2024-09-21T22:07:12.168 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.168 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.168 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:12.168 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.293524+0000", 2024-09-21T22:07:12.169 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00040701300000000001 2024-09-21T22:07:12.169 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.169 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.169 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:12.169 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.293673+0000", 2024-09-21T22:07:12.169 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000148782 2024-09-21T22:07:12.169 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.169 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.169 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:12.169 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.293786+0000", 2024-09-21T22:07:12.169 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011377 2024-09-21T22:07:12.169 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:12.169 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:12.169 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:12.170 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.170 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.170 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:07:12.170 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.466485+0000", 2024-09-21T22:07:12.170 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 69.669803866999999, 2024-09-21T22:07:12.170 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014443981, 2024-09-21T22:07:12.170 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:12.170 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:12.170 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:12.170 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:12.170 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:07:12.170 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:07:12.170 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 107 2024-09-21T22:07:12.170 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.171 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:12.171 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.171 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:12.171 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.466485+0000", 2024-09-21T22:07:12.171 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:12.171 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.171 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.171 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:12.171 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.466485+0000", 2024-09-21T22:07:12.171 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:12.171 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.171 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.171 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:12.171 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.466486+0000", 2024-09-21T22:07:12.171 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6299999999999993e-07 2024-09-21T22:07:12.172 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.172 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.172 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:12.172 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.466493+0000", 2024-09-21T22:07:12.172 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9639999999999997e-06 2024-09-21T22:07:12.172 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.172 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.172 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:12.172 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.466507+0000", 2024-09-21T22:07:12.172 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4216000000000001e-05 2024-09-21T22:07:12.172 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.172 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.172 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:12.172 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.466515+0000", 2024-09-21T22:07:12.173 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.157e-06 2024-09-21T22:07:12.173 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.173 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.173 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:12.173 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.472033+0000", 2024-09-21T22:07:12.173 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0055175839999999999 2024-09-21T22:07:12.173 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.173 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.173 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:12.173 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.472174+0000", 2024-09-21T22:07:12.173 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000141612 2024-09-21T22:07:12.173 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.173 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.173 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:12.174 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.472381+0000", 2024-09-21T22:07:12.174 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000206961 2024-09-21T22:07:12.174 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.174 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.174 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:12.174 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.473924+0000", 2024-09-21T22:07:12.174 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015427119999999999 2024-09-21T22:07:12.174 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.174 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.174 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:12.174 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.480666+0000", 2024-09-21T22:07:12.174 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0067415410000000002 2024-09-21T22:07:12.174 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.174 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.175 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:12.175 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.480811+0000", 2024-09-21T22:07:12.175 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000145422 2024-09-21T22:07:12.175 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.175 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.175 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:12.175 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.480929+0000", 2024-09-21T22:07:12.175 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000117849 2024-09-21T22:07:12.175 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:12.175 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:12.175 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:12.176 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.176 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.176 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:07:12.176 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.467057+0000", 2024-09-21T22:07:12.176 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 69.669231608000004, 2024-09-21T22:07:12.176 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018350195, 2024-09-21T22:07:12.176 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:12.176 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:12.176 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:12.176 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:12.176 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:07:12.176 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:07:12.176 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 108 2024-09-21T22:07:12.176 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.176 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:12.177 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.177 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:12.177 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467057+0000", 2024-09-21T22:07:12.177 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:12.177 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.177 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.177 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:12.177 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467057+0000", 2024-09-21T22:07:12.177 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:12.177 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.177 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.177 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:12.177 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467059+0000", 2024-09-21T22:07:12.178 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.826e-06 2024-09-21T22:07:12.178 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.178 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.178 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:12.178 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467071+0000", 2024-09-21T22:07:12.178 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2224e-05 2024-09-21T22:07:12.178 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.178 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.178 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:12.178 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467098+0000", 2024-09-21T22:07:12.178 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6953000000000001e-05 2024-09-21T22:07:12.178 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.178 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.178 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:12.178 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467113+0000", 2024-09-21T22:07:12.179 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5119999999999999e-05 2024-09-21T22:07:12.179 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.179 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.179 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:12.179 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.474060+0000", 2024-09-21T22:07:12.179 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0069462919999999997 2024-09-21T22:07:12.179 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.179 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.179 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:12.179 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.474283+0000", 2024-09-21T22:07:12.179 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000223315 2024-09-21T22:07:12.179 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.179 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.180 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:12.180 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.474660+0000", 2024-09-21T22:07:12.180 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00037729400000000002 2024-09-21T22:07:12.180 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.180 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.180 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:12.180 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.480622+0000", 2024-09-21T22:07:12.180 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0059613369999999997 2024-09-21T22:07:12.180 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.180 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.180 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:12.180 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.485145+0000", 2024-09-21T22:07:12.180 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0045231960000000002 2024-09-21T22:07:12.180 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.181 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.181 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:12.181 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.485288+0000", 2024-09-21T22:07:12.181 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000143232 2024-09-21T22:07:12.181 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.181 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.181 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:12.181 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.485407+0000", 2024-09-21T22:07:12.181 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000119406 2024-09-21T22:07:12.181 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:12.181 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:12.181 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:12.181 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.181 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.181 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:07:12.182 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.467144+0000", 2024-09-21T22:07:12.182 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 69.669144388999996, 2024-09-21T22:07:12.182 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018907146999999999, 2024-09-21T22:07:12.182 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:12.182 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:12.182 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:12.182 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:12.182 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:07:12.182 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:07:12.182 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 109 2024-09-21T22:07:12.182 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.182 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:12.183 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.183 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:12.183 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467144+0000", 2024-09-21T22:07:12.183 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:12.183 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.183 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.183 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:12.183 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467144+0000", 2024-09-21T22:07:12.183 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:12.183 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.183 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.183 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:12.183 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467145+0000", 2024-09-21T22:07:12.183 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9999999999999996e-07 2024-09-21T22:07:12.184 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.184 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.184 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:12.184 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467331+0000", 2024-09-21T22:07:12.184 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018518899999999999 2024-09-21T22:07:12.184 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.184 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.184 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:12.184 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467350+0000", 2024-09-21T22:07:12.184 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8966000000000001e-05 2024-09-21T22:07:12.184 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.184 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.184 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:12.184 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467361+0000", 2024-09-21T22:07:12.185 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1233000000000001e-05 2024-09-21T22:07:12.185 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.185 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.185 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:12.185 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.475613+0000", 2024-09-21T22:07:12.185 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0082518629999999999 2024-09-21T22:07:12.185 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.185 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.185 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:12.185 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.475859+0000", 2024-09-21T22:07:12.185 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024633800000000003 2024-09-21T22:07:12.185 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.185 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.185 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:12.186 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.476223+0000", 2024-09-21T22:07:12.186 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00036408700000000003 2024-09-21T22:07:12.186 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.186 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.186 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:12.186 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.481046+0000", 2024-09-21T22:07:12.186 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.004823356 2024-09-21T22:07:12.186 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.186 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.186 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:12.186 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.485722+0000", 2024-09-21T22:07:12.186 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0046753640000000004 2024-09-21T22:07:12.186 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.186 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.187 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:12.187 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.485908+0000", 2024-09-21T22:07:12.187 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000185859 2024-09-21T22:07:12.187 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.187 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.187 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:12.187 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.486052+0000", 2024-09-21T22:07:12.187 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014399200000000001 2024-09-21T22:07:12.187 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:12.187 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:12.187 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:12.187 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.187 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.187 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:07:12.188 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.467384+0000", 2024-09-21T22:07:12.188 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 69.668904561000005, 2024-09-21T22:07:12.188 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.01894144, 2024-09-21T22:07:12.188 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:12.188 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:12.188 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:12.188 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:12.188 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:07:12.188 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:07:12.188 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 110 2024-09-21T22:07:12.188 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.188 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:12.188 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.189 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:12.189 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467384+0000", 2024-09-21T22:07:12.189 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:12.189 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.189 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.189 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:12.189 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467384+0000", 2024-09-21T22:07:12.189 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:12.189 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.189 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.189 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:12.189 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467385+0000", 2024-09-21T22:07:12.189 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0470000000000001e-06 2024-09-21T22:07:12.189 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.189 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.190 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:12.190 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467393+0000", 2024-09-21T22:07:12.190 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3869999999999997e-06 2024-09-21T22:07:12.190 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.190 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.190 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:12.190 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467413+0000", 2024-09-21T22:07:12.190 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9953e-05 2024-09-21T22:07:12.190 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.190 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.190 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:12.190 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467421+0000", 2024-09-21T22:07:12.190 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2400000000000007e-06 2024-09-21T22:07:12.191 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.191 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.191 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:12.191 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.479542+0000", 2024-09-21T22:07:12.191 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012120832 2024-09-21T22:07:12.191 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.191 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.191 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:12.191 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.479705+0000", 2024-09-21T22:07:12.191 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016362899999999999 2024-09-21T22:07:12.191 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.191 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.191 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:12.191 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.479914+0000", 2024-09-21T22:07:12.192 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020814499999999999 2024-09-21T22:07:12.192 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.192 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.192 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:12.192 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.485536+0000", 2024-09-21T22:07:12.192 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0056222800000000003 2024-09-21T22:07:12.192 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.192 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.192 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:12.192 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.486077+0000", 2024-09-21T22:07:12.192 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00054079900000000001 2024-09-21T22:07:12.192 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.192 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.192 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:12.192 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.486213+0000", 2024-09-21T22:07:12.193 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000136682 2024-09-21T22:07:12.193 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.193 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.193 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:12.193 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.486326+0000", 2024-09-21T22:07:12.193 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000112446 2024-09-21T22:07:12.193 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:12.193 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:12.193 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:12.193 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.193 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.193 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:07:12.193 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.467436+0000", 2024-09-21T22:07:12.193 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 69.668853060999993, 2024-09-21T22:07:12.194 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019152168000000001, 2024-09-21T22:07:12.194 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:12.194 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:12.194 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:12.194 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:12.194 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:07:12.194 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:07:12.194 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 111 2024-09-21T22:07:12.194 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.194 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:12.194 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.194 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:12.194 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467436+0000", 2024-09-21T22:07:12.195 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:12.195 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.195 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.195 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:12.195 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467436+0000", 2024-09-21T22:07:12.195 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:12.195 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.195 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.195 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:12.195 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467437+0000", 2024-09-21T22:07:12.195 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0699999999999996e-07 2024-09-21T22:07:12.195 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.195 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.195 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:12.195 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467445+0000", 2024-09-21T22:07:12.196 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.789e-06 2024-09-21T22:07:12.196 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.196 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.196 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:12.196 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467467+0000", 2024-09-21T22:07:12.196 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1253000000000001e-05 2024-09-21T22:07:12.196 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.196 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.196 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:12.196 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467475+0000", 2024-09-21T22:07:12.196 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0039999999999999e-06 2024-09-21T22:07:12.196 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.196 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.196 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:12.196 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.481164+0000", 2024-09-21T22:07:12.197 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013689114 2024-09-21T22:07:12.197 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.197 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.197 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:12.197 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.481394+0000", 2024-09-21T22:07:12.197 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000229931 2024-09-21T22:07:12.197 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.197 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.197 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:12.197 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.481779+0000", 2024-09-21T22:07:12.197 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00038545400000000003 2024-09-21T22:07:12.197 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.197 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.197 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:12.197 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.485675+0000", 2024-09-21T22:07:12.198 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0038958370000000001 2024-09-21T22:07:12.198 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.198 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.198 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:12.198 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.486346+0000", 2024-09-21T22:07:12.198 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00067119100000000002 2024-09-21T22:07:12.198 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.198 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.198 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:12.198 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.486478+0000", 2024-09-21T22:07:12.198 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013182600000000001 2024-09-21T22:07:12.198 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.198 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.198 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:12.199 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.486588+0000", 2024-09-21T22:07:12.199 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000110062 2024-09-21T22:07:12.199 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:12.199 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:12.199 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:12.199 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.199 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.199 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:07:12.199 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.467488+0000", 2024-09-21T22:07:12.199 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 69.668800434999994, 2024-09-21T22:07:12.199 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021038304000000001, 2024-09-21T22:07:12.199 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:12.199 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:12.199 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:12.200 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:12.200 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:07:12.200 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:07:12.200 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 112 2024-09-21T22:07:12.200 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.200 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:12.200 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.200 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:12.200 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467488+0000", 2024-09-21T22:07:12.200 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:12.200 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.200 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.200 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:12.200 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467488+0000", 2024-09-21T22:07:12.201 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:12.201 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.201 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.201 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:12.201 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467489+0000", 2024-09-21T22:07:12.201 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7999999999999995e-07 2024-09-21T22:07:12.201 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.201 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.201 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:12.201 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467687+0000", 2024-09-21T22:07:12.201 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019776200000000001 2024-09-21T22:07:12.201 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.201 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.201 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:12.201 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467704+0000", 2024-09-21T22:07:12.202 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7193e-05 2024-09-21T22:07:12.202 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.202 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.202 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:12.202 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467719+0000", 2024-09-21T22:07:12.202 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4703e-05 2024-09-21T22:07:12.202 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.202 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.202 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:12.202 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.482745+0000", 2024-09-21T22:07:12.202 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015025801 2024-09-21T22:07:12.202 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.202 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.202 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:12.202 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.483015+0000", 2024-09-21T22:07:12.203 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026999400000000002 2024-09-21T22:07:12.203 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.203 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.203 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:12.203 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.483440+0000", 2024-09-21T22:07:12.203 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000425697 2024-09-21T22:07:12.203 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.203 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.203 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:12.203 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.486701+0000", 2024-09-21T22:07:12.203 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0032611950000000002 2024-09-21T22:07:12.203 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.203 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.203 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:12.204 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.488262+0000", 2024-09-21T22:07:12.204 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015610439999999999 2024-09-21T22:07:12.204 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.204 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.204 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:12.204 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.488406+0000", 2024-09-21T22:07:12.204 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000143232 2024-09-21T22:07:12.204 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.204 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.204 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:12.204 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.488527+0000", 2024-09-21T22:07:12.204 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012100299999999999 2024-09-21T22:07:12.204 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:12.204 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:12.205 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:12.205 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.205 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.205 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:07:12.205 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.467786+0000", 2024-09-21T22:07:12.205 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 69.668503009999995, 2024-09-21T22:07:12.205 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026339741999999999, 2024-09-21T22:07:12.205 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:12.205 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:12.205 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:12.205 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:12.205 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:07:12.205 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:07:12.205 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 113 2024-09-21T22:07:12.206 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.206 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:12.206 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.206 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:12.206 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467786+0000", 2024-09-21T22:07:12.206 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:12.206 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.206 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.206 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:12.206 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467786+0000", 2024-09-21T22:07:12.206 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:12.206 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.206 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.206 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:12.207 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467787+0000", 2024-09-21T22:07:12.207 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.446e-06 2024-09-21T22:07:12.207 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.207 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.207 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:12.207 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467802+0000", 2024-09-21T22:07:12.207 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4769999999999999e-05 2024-09-21T22:07:12.207 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.207 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.207 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:12.207 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467826+0000", 2024-09-21T22:07:12.207 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3430000000000001e-05 2024-09-21T22:07:12.207 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.207 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.207 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:12.208 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467843+0000", 2024-09-21T22:07:12.208 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7065999999999999e-05 2024-09-21T22:07:12.208 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.208 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.208 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:12.208 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.486820+0000", 2024-09-21T22:07:12.208 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018977233999999999 2024-09-21T22:07:12.208 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.208 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.208 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:12.208 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.487065+0000", 2024-09-21T22:07:12.208 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024558799999999998 2024-09-21T22:07:12.208 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.208 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.209 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:12.209 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.487442+0000", 2024-09-21T22:07:12.209 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000376707 2024-09-21T22:07:12.209 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.209 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.209 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:12.209 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.492256+0000", 2024-09-21T22:07:12.209 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0048141620000000003 2024-09-21T22:07:12.209 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.209 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.209 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:12.209 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.493865+0000", 2024-09-21T22:07:12.209 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016090150000000001 2024-09-21T22:07:12.210 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.210 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.210 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:12.210 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.494007+0000", 2024-09-21T22:07:12.210 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000141359 2024-09-21T22:07:12.210 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.210 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.210 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:12.210 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.494126+0000", 2024-09-21T22:07:12.210 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000118965 2024-09-21T22:07:12.210 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:12.210 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:12.210 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:12.210 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.211 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.211 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:07:12.211 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.467878+0000", 2024-09-21T22:07:12.211 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 69.668410520999998, 2024-09-21T22:07:12.211 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.027783958000000001, 2024-09-21T22:07:12.211 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:12.211 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:12.211 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:12.211 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:12.211 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:07:12.211 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:07:12.211 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 114 2024-09-21T22:07:12.211 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.211 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:12.212 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.212 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:12.212 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467878+0000", 2024-09-21T22:07:12.212 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:12.212 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.212 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.212 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:12.212 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467878+0000", 2024-09-21T22:07:12.212 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:12.212 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.212 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.212 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:12.212 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467880+0000", 2024-09-21T22:07:12.212 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.223e-06 2024-09-21T22:07:12.213 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.213 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.213 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:12.213 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467891+0000", 2024-09-21T22:07:12.213 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1527e-05 2024-09-21T22:07:12.213 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.213 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.213 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:12.213 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467913+0000", 2024-09-21T22:07:12.213 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.143e-05 2024-09-21T22:07:12.213 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.213 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.213 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:12.213 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467930+0000", 2024-09-21T22:07:12.214 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7620000000000001e-05 2024-09-21T22:07:12.214 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.214 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.214 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:12.214 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.488690+0000", 2024-09-21T22:07:12.214 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020759588999999998 2024-09-21T22:07:12.214 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.214 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.214 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:12.214 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.488882+0000", 2024-09-21T22:07:12.214 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019236200000000001 2024-09-21T22:07:12.214 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.214 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.214 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:12.215 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.489093+0000", 2024-09-21T22:07:12.215 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000211318 2024-09-21T22:07:12.215 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.215 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.215 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:12.215 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.494152+0000", 2024-09-21T22:07:12.215 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0050582769999999999 2024-09-21T22:07:12.215 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.215 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.215 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:12.215 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.495373+0000", 2024-09-21T22:07:12.215 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001221747 2024-09-21T22:07:12.215 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.215 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.216 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:12.216 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.495543+0000", 2024-09-21T22:07:12.216 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000169259 2024-09-21T22:07:12.216 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.216 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.216 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:12.216 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.495662+0000", 2024-09-21T22:07:12.216 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011960600000000001 2024-09-21T22:07:12.216 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:12.216 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:12.216 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:12.216 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.216 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.216 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:07:12.217 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.468118+0000", 2024-09-21T22:07:12.217 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 69.668170779999997, 2024-09-21T22:07:12.217 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.027938877000000001, 2024-09-21T22:07:12.217 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:12.217 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:12.217 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:12.217 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:12.217 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:07:12.217 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:07:12.217 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 115 2024-09-21T22:07:12.217 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.217 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:12.217 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.217 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:12.217 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468118+0000", 2024-09-21T22:07:12.218 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:12.218 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.218 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.218 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:12.218 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468118+0000", 2024-09-21T22:07:12.218 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:12.218 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.218 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.218 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:12.218 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468120+0000", 2024-09-21T22:07:12.218 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7069999999999999e-06 2024-09-21T22:07:12.218 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.218 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.218 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:12.219 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468381+0000", 2024-09-21T22:07:12.219 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026135399999999999 2024-09-21T22:07:12.219 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.219 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.219 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:12.219 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468409+0000", 2024-09-21T22:07:12.219 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.781e-05 2024-09-21T22:07:12.219 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.219 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.219 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:12.219 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468433+0000", 2024-09-21T22:07:12.219 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4343999999999999e-05 2024-09-21T22:07:12.219 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.219 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.219 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:12.220 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.489785+0000", 2024-09-21T22:07:12.220 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021351221 2024-09-21T22:07:12.220 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.220 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.220 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:12.220 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.490018+0000", 2024-09-21T22:07:12.220 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023317800000000001 2024-09-21T22:07:12.220 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.220 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.220 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:12.220 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.490412+0000", 2024-09-21T22:07:12.220 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00039417400000000001 2024-09-21T22:07:12.220 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.220 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.221 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:12.221 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.495694+0000", 2024-09-21T22:07:12.221 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.005282335 2024-09-21T22:07:12.221 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.221 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.221 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:12.221 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.495852+0000", 2024-09-21T22:07:12.221 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015777200000000001 2024-09-21T22:07:12.221 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.221 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.221 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:12.221 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.495938+0000", 2024-09-21T22:07:12.221 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6493e-05 2024-09-21T22:07:12.221 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.221 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.222 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:12.222 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.496057+0000", 2024-09-21T22:07:12.222 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000118489 2024-09-21T22:07:12.222 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:12.222 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:12.222 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:12.222 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.222 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.222 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:07:12.222 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.468474+0000", 2024-09-21T22:07:12.222 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 69.667815019000003, 2024-09-21T22:07:12.222 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.028015895999999998, 2024-09-21T22:07:12.222 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:12.222 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:12.223 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:12.223 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:12.223 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:07:12.223 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:07:12.223 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 116 2024-09-21T22:07:12.223 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.223 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:12.223 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.223 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:12.223 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468474+0000", 2024-09-21T22:07:12.223 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:12.223 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.223 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.224 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:12.224 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468474+0000", 2024-09-21T22:07:12.224 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:12.224 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.224 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.224 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:12.224 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468476+0000", 2024-09-21T22:07:12.224 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7060000000000001e-06 2024-09-21T22:07:12.224 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.224 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.224 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:12.224 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468488+0000", 2024-09-21T22:07:12.224 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.234e-05 2024-09-21T22:07:12.224 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.224 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.225 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:12.225 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468517+0000", 2024-09-21T22:07:12.225 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.898e-05 2024-09-21T22:07:12.225 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.225 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.225 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:12.225 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468536+0000", 2024-09-21T22:07:12.225 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8899999999999999e-05 2024-09-21T22:07:12.225 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.225 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.225 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:12.225 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.492376+0000", 2024-09-21T22:07:12.225 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023840659 2024-09-21T22:07:12.225 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.226 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.226 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:12.226 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.492621+0000", 2024-09-21T22:07:12.226 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000244794 2024-09-21T22:07:12.226 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.226 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.226 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:12.226 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.493008+0000", 2024-09-21T22:07:12.226 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00038629099999999998 2024-09-21T22:07:12.226 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.226 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.226 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:12.226 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.496192+0000", 2024-09-21T22:07:12.226 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0031847220000000001 2024-09-21T22:07:12.227 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.227 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.227 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:12.227 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.496233+0000", 2024-09-21T22:07:12.227 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0410000000000001e-05 2024-09-21T22:07:12.227 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.227 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.227 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:12.227 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.496374+0000", 2024-09-21T22:07:12.227 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140872 2024-09-21T22:07:12.227 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.227 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.227 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:12.227 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.496490+0000", 2024-09-21T22:07:12.227 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000116222 2024-09-21T22:07:12.228 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:12.228 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:12.228 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:12.228 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.228 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.228 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:07:12.228 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.468568+0000", 2024-09-21T22:07:12.228 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 69.667720900000006, 2024-09-21T22:07:12.228 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.028181578999999998, 2024-09-21T22:07:12.228 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:12.228 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:12.228 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:12.228 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:12.228 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:07:12.229 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:07:12.229 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 117 2024-09-21T22:07:12.229 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.229 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:12.229 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.229 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:12.229 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468568+0000", 2024-09-21T22:07:12.229 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:12.229 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.229 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.229 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:12.229 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468568+0000", 2024-09-21T22:07:12.229 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:12.229 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.230 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.230 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:12.230 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468569+0000", 2024-09-21T22:07:12.230 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4139999999999999e-06 2024-09-21T22:07:12.230 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.230 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.230 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:12.230 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468583+0000", 2024-09-21T22:07:12.230 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3409e-05 2024-09-21T22:07:12.230 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.230 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.230 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:12.230 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468608+0000", 2024-09-21T22:07:12.230 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5559999999999999e-05 2024-09-21T22:07:12.231 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.231 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.231 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:12.231 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468626+0000", 2024-09-21T22:07:12.231 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7863000000000002e-05 2024-09-21T22:07:12.231 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.231 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.231 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:12.231 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.494260+0000", 2024-09-21T22:07:12.231 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025633744999999999 2024-09-21T22:07:12.231 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.231 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.231 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:12.231 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.494519+0000", 2024-09-21T22:07:12.232 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025924800000000001 2024-09-21T22:07:12.232 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.232 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.232 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:12.232 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.494747+0000", 2024-09-21T22:07:12.232 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022792799999999999 2024-09-21T22:07:12.232 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.232 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.232 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:12.232 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.496519+0000", 2024-09-21T22:07:12.232 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017720030000000001 2024-09-21T22:07:12.232 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.232 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.232 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:12.232 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.496607+0000", 2024-09-21T22:07:12.232 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7873000000000004e-05 2024-09-21T22:07:12.233 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.233 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.233 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:12.233 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.496691+0000", 2024-09-21T22:07:12.233 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4009000000000002e-05 2024-09-21T22:07:12.233 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.233 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.233 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:12.233 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.496750+0000", 2024-09-21T22:07:12.233 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8526999999999997e-05 2024-09-21T22:07:12.233 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:12.233 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:12.233 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:12.233 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.233 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.233 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:07:12.234 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.898244+0000", 2024-09-21T22:07:12.234 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 69.238044596999998, 2024-09-21T22:07:12.234 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014749464, 2024-09-21T22:07:12.234 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:12.234 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:12.234 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:12.234 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:12.234 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:07:12.234 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:07:12.234 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 140 2024-09-21T22:07:12.234 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.234 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:12.234 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.235 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:12.235 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898244+0000", 2024-09-21T22:07:12.235 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:12.235 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.235 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.235 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:12.235 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898244+0000", 2024-09-21T22:07:12.235 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:12.235 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.235 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.235 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:12.235 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898245+0000", 2024-09-21T22:07:12.235 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3000000000000002e-07 2024-09-21T22:07:12.235 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.236 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.236 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:12.236 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898308+0000", 2024-09-21T22:07:12.236 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3468999999999998e-05 2024-09-21T22:07:12.236 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.236 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.236 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:12.236 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898315+0000", 2024-09-21T22:07:12.236 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.725e-06 2024-09-21T22:07:12.236 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.236 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.236 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:12.236 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898321+0000", 2024-09-21T22:07:12.236 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2090000000000002e-06 2024-09-21T22:07:12.237 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.237 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.237 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:12.237 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.905452+0000", 2024-09-21T22:07:12.237 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0071305989999999996 2024-09-21T22:07:12.237 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.237 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.237 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:12.237 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.905504+0000", 2024-09-21T22:07:12.237 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2638e-05 2024-09-21T22:07:12.237 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.237 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.237 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:12.238 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.905590+0000", 2024-09-21T22:07:12.238 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5333000000000004e-05 2024-09-21T22:07:12.238 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.238 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.238 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:12.238 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.911542+0000", 2024-09-21T22:07:12.238 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0059519819999999998 2024-09-21T22:07:12.238 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.238 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.238 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:12.238 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.912908+0000", 2024-09-21T22:07:12.238 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0013666640000000001 2024-09-21T22:07:12.238 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.238 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.239 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:12.239 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.912957+0000", 2024-09-21T22:07:12.239 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9203000000000003e-05 2024-09-21T22:07:12.239 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.239 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.239 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:12.239 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.912994+0000", 2024-09-21T22:07:12.239 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6312000000000001e-05 2024-09-21T22:07:12.239 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:12.239 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:12.239 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:12.239 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.239 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.239 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:07:12.240 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.898342+0000", 2024-09-21T22:07:12.240 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 69.237946459, 2024-09-21T22:07:12.240 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014787918000000001, 2024-09-21T22:07:12.240 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:12.240 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:12.240 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:12.240 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:12.240 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:07:12.240 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:07:12.240 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 141 2024-09-21T22:07:12.240 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.240 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:12.240 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.240 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:12.241 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898342+0000", 2024-09-21T22:07:12.241 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:12.241 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.241 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.241 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:12.241 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898342+0000", 2024-09-21T22:07:12.241 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:12.241 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.241 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.241 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:12.241 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898343+0000", 2024-09-21T22:07:12.241 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3000000000000001e-07 2024-09-21T22:07:12.241 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.241 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.242 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:12.242 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898346+0000", 2024-09-21T22:07:12.242 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6009999999999999e-06 2024-09-21T22:07:12.242 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.242 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.242 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:12.242 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898352+0000", 2024-09-21T22:07:12.242 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8089999999999998e-06 2024-09-21T22:07:12.242 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.242 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.242 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:12.242 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898358+0000", 2024-09-21T22:07:12.242 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4349999999999996e-06 2024-09-21T22:07:12.242 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.243 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.243 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:12.243 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.909387+0000", 2024-09-21T22:07:12.243 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011029105000000001 2024-09-21T22:07:12.243 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.243 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.243 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:12.243 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.909462+0000", 2024-09-21T22:07:12.243 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5192999999999996e-05 2024-09-21T22:07:12.243 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.243 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.243 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:12.243 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.909551+0000", 2024-09-21T22:07:12.243 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9046000000000001e-05 2024-09-21T22:07:12.243 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.244 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.244 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:12.244 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.911852+0000", 2024-09-21T22:07:12.244 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0023005579999999999 2024-09-21T22:07:12.244 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.244 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.244 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:12.244 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.913002+0000", 2024-09-21T22:07:12.244 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011502929999999999 2024-09-21T22:07:12.244 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.244 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.244 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:12.244 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.913060+0000", 2024-09-21T22:07:12.244 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8427000000000002e-05 2024-09-21T22:07:12.245 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.245 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.245 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:12.245 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.913130+0000", 2024-09-21T22:07:12.245 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0020999999999999e-05 2024-09-21T22:07:12.245 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:12.245 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:12.245 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:12.245 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.245 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.245 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:158 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T22:07:12.245 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:03.511201+0000", 2024-09-21T22:07:12.245 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 68.625087976000003, 2024-09-21T22:07:12.245 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013353538999999999, 2024-09-21T22:07:12.245 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:12.246 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:12.246 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:12.246 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:12.246 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:07:12.246 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:07:12.246 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 158 2024-09-21T22:07:12.246 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.246 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:12.246 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.246 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:12.246 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511201+0000", 2024-09-21T22:07:12.246 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:12.246 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.246 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.247 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:12.247 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511201+0000", 2024-09-21T22:07:12.247 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:12.247 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.247 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.247 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:12.247 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511202+0000", 2024-09-21T22:07:12.247 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9299999999999997e-07 2024-09-21T22:07:12.247 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.247 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.247 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:12.247 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511315+0000", 2024-09-21T22:07:12.247 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000113909 2024-09-21T22:07:12.247 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.247 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.248 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:12.248 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511333+0000", 2024-09-21T22:07:12.248 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7062999999999999e-05 2024-09-21T22:07:12.248 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.248 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.248 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:12.248 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511344+0000", 2024-09-21T22:07:12.248 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1177e-05 2024-09-21T22:07:12.248 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.248 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.248 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:12.248 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.517415+0000", 2024-09-21T22:07:12.248 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0060709830000000003 2024-09-21T22:07:12.248 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.248 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.249 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:12.249 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.517539+0000", 2024-09-21T22:07:12.249 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000124432 2024-09-21T22:07:12.249 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.249 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.249 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:12.249 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.517732+0000", 2024-09-21T22:07:12.249 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019262200000000001 2024-09-21T22:07:12.249 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.249 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.249 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:12.249 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.520709+0000", 2024-09-21T22:07:12.249 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0029767470000000001 2024-09-21T22:07:12.249 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.250 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.250 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:12.250 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.524389+0000", 2024-09-21T22:07:12.250 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003680975 2024-09-21T22:07:12.250 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.250 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.250 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:12.250 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.524486+0000", 2024-09-21T22:07:12.250 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6575000000000003e-05 2024-09-21T22:07:12.250 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.250 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.250 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:12.250 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.524554+0000", 2024-09-21T22:07:12.250 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8362999999999996e-05 2024-09-21T22:07:12.251 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:12.251 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:12.251 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:12.251 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.251 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.251 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T22:07:12.251 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:03.511368+0000", 2024-09-21T22:07:12.251 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 68.624921074, 2024-09-21T22:07:12.251 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013312133, 2024-09-21T22:07:12.251 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:12.251 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:12.251 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:12.251 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:12.251 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:07:12.251 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:07:12.252 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 159 2024-09-21T22:07:12.252 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.252 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:12.252 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.252 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:12.252 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511368+0000", 2024-09-21T22:07:12.252 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:12.252 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.252 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.252 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:12.252 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511368+0000", 2024-09-21T22:07:12.252 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:12.252 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.252 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.252 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:12.253 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511369+0000", 2024-09-21T22:07:12.253 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9599999999999998e-07 2024-09-21T22:07:12.253 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.253 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.253 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:12.253 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511382+0000", 2024-09-21T22:07:12.253 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3487e-05 2024-09-21T22:07:12.253 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.253 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.253 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:12.253 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511394+0000", 2024-09-21T22:07:12.253 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1337e-05 2024-09-21T22:07:12.253 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.253 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.254 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:12.254 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511402+0000", 2024-09-21T22:07:12.254 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0059999999999992e-06 2024-09-21T22:07:12.254 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.254 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.254 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:12.254 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.519376+0000", 2024-09-21T22:07:12.254 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0079746520000000005 2024-09-21T22:07:12.254 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.254 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.254 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:12.254 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.519498+0000", 2024-09-21T22:07:12.254 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012212900000000001 2024-09-21T22:07:12.254 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.254 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.255 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:12.255 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.519728+0000", 2024-09-21T22:07:12.255 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022985799999999999 2024-09-21T22:07:12.255 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.255 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.255 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:12.255 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.521344+0000", 2024-09-21T22:07:12.255 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016163169999999999 2024-09-21T22:07:12.255 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.255 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.255 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:12.255 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.524568+0000", 2024-09-21T22:07:12.255 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0032234220000000001 2024-09-21T22:07:12.255 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.256 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.256 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:12.256 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.524633+0000", 2024-09-21T22:07:12.256 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5126e-05 2024-09-21T22:07:12.256 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:12.256 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:12.256 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:12.256 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.524680+0000", 2024-09-21T22:07:12.256 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6903000000000001e-05 2024-09-21T22:07:12.256 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:12.256 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:12.256 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:12.256 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:12.256 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:12.257 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:07:12.258 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T22:07:12.258 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:12.251+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:12.359 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:12.356+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:12.390 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:07:12.390 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:07:12.390 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-09-21T22:07:12.390 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:07:12.404 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T22:07:12.459 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:12.455+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:12.560 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:12.556+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:12.573 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:07:12.573 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:07:12.573 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-21T22:07:12.573 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-21T22:07:12.573 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:07:12.588 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T22:07:12.589 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T22:07:12.675 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:12.672+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:12.746 INFO:teuthology.orchestra.run.smithi039.stdout:osd.0: {} 2024-09-21T22:07:12.746 INFO:teuthology.orchestra.run.smithi039.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T22:07:12.758 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:07:12.759 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-21T22:07:12.759 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-21T22:07:12.759 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [] 2024-09-21T22:07:12.759 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:07:12.762 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:12.759+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:12.762 INFO:teuthology.orchestra.run.smithi039.stdout:osd.1: {} 2024-09-21T22:07:12.763 INFO:teuthology.orchestra.run.smithi039.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T22:07:12.773 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T22:07:12.779 INFO:teuthology.orchestra.run.smithi039.stdout:osd.2: {} 2024-09-21T22:07:12.780 INFO:teuthology.orchestra.run.smithi039.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T22:07:12.800 INFO:teuthology.orchestra.run.smithi039.stdout:osd.3: {} 2024-09-21T22:07:12.800 INFO:teuthology.orchestra.run.smithi039.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T22:07:12.819 INFO:teuthology.orchestra.run.smithi039.stdout:osd.4: {} 2024-09-21T22:07:12.819 INFO:teuthology.orchestra.run.smithi039.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T22:07:12.820 INFO:teuthology.orchestra.run.smithi039.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-21T22:07:12.823 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T22:07:12.824 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T22:07:12.863 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:12.860+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:12.894 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:12.894 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:12.908 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T22:07:12.963 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:12.959+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:13.034 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:13.034 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:13.048 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T22:07:13.085 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-21T22:07:13.092 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:13.091+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:13.165 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:13.163+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:13.215 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:13.216 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:13.232 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T22:07:13.266 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:13.264+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:13.367 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:13.363+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:13.397 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:13.398 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:13.412 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T22:07:13.469 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:13.464+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:13.569 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:13.567+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:13.582 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:13.582 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:13.597 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T22:07:13.669 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:13.664+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:13.768 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:13.768 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:13.770 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:13.768+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:13.782 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T22:07:13.871 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:13.871+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:13.912 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:13.913 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:13.927 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T22:07:13.972 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:13.971+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:14.073 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:14.071+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:14.090 INFO:teuthology.orchestra.run.smithi039.stderr:noscrub is set 2024-09-21T22:07:14.105 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:14.105 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:14.121 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T22:07:14.208 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:14.207+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:14.274 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:14.271+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:14.282 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:14.282 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:14.296 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T22:07:14.354 INFO:tasks.workunit.client.0.smithi157.stderr:+ for ns in '' ${NS} 2024-09-21T22:07:14.354 INFO:tasks.workunit.client.0.smithi157.stderr:+ rbd --dest-pool rbd --dest-namespace ns --no-progress import /tmp/tmp.GqKuv7rsZi/data testrbdnbd21553 2024-09-21T22:07:14.375 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:14.372+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:14.433 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:14.433 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:14.447 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T22:07:14.477 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:14.475+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:14.577 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:14.575+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:14.612 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:14.612 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:14.627 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T22:07:14.678 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:14.676+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:14.779 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:14.775+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:14.793 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:14.793 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:14.808 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T22:07:14.825 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T22:07:14.899 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:14.896+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:14.979 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:14.979 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:14.980 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:14.979+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:14.996 INFO:teuthology.orchestra.run.smithi039.stdout:osd.0: {} 2024-09-21T22:07:14.996 INFO:teuthology.orchestra.run.smithi039.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T22:07:14.999 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T22:07:15.079 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 5, 0] out_osds: [2] dead_osds: [5] live_osds: [0, 1, 2, 3, 4] 2024-09-21T22:07:15.080 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T22:07:15.080 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2024-09-21T22:07:15.080 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-21T22:07:15.080 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-21T22:07:15.080 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok config set filestore_inject_stall 3 2024-09-21T22:07:15.082 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:15.079+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:15.093 INFO:teuthology.orchestra.run.smithi039.stdout:osd.1: {} 2024-09-21T22:07:15.093 INFO:teuthology.orchestra.run.smithi039.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T22:07:15.110 INFO:teuthology.orchestra.run.smithi039.stdout:osd.2: {} 2024-09-21T22:07:15.110 INFO:teuthology.orchestra.run.smithi039.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T22:07:15.131 INFO:teuthology.orchestra.run.smithi039.stdout:osd.3: {} 2024-09-21T22:07:15.131 INFO:teuthology.orchestra.run.smithi039.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T22:07:15.145 INFO:tasks.workunit.client.0.smithi157.stderr:+ ceph osd pool create new_default_pool21553 8 2024-09-21T22:07:15.148 INFO:teuthology.orchestra.run.smithi039.stdout:osd.4: {} 2024-09-21T22:07:15.149 INFO:teuthology.orchestra.run.smithi039.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T22:07:15.149 INFO:teuthology.orchestra.run.smithi039.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-21T22:07:15.152 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T22:07:15.152 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T22:07:15.165 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:07:15.165 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:07:15.165 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-21T22:07:15.165 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-21T22:07:15.165 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:07:15.179 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T22:07:15.199 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:15.199+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:15.202 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:07:15.202 INFO:teuthology.orchestra.run.smithi039.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-21T22:07:15.202 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:07:15.283 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:15.280+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:15.300 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:07:15.300 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-21T22:07:15.301 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-21T22:07:15.301 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [] 2024-09-21T22:07:15.301 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:07:15.315 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T22:07:15.384 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:15.383+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:15.447 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:07:15.447 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:07:15.447 INFO:teuthology.orchestra.run.smithi039.stdout: "num_ops": 0 2024-09-21T22:07:15.447 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:07:15.463 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T22:07:15.504 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:15.503+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:15.586 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:15.583+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:15.634 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:07:15.634 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:07:15.634 INFO:teuthology.orchestra.run.smithi039.stdout: "complaint_time": 30, 2024-09-21T22:07:15.634 INFO:teuthology.orchestra.run.smithi039.stdout: "num_blocked_ops": 0 2024-09-21T22:07:15.634 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:07:15.649 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T22:07:15.692 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:15.691+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:15.788 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:15.787+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:15.821 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:07:15.821 INFO:teuthology.orchestra.run.smithi039.stdout: "size": 20, 2024-09-21T22:07:15.821 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 600, 2024-09-21T22:07:15.821 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [ 2024-09-21T22:07:15.821 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.821 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:38 1.0 e14/13 1:30306672:devicehealth::main.db.0000000000000000:head v 14'24, mlcod=14'22)", 2024-09-21T22:07:15.821 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.180244+0000", 2024-09-21T22:07:15.821 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 73.638326297000006, 2024-09-21T22:07:15.821 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0041712219999999996, 2024-09-21T22:07:15.821 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:15.821 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:15.821 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:15.822 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:15.822 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.822 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:15.822 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.180244+0000", 2024-09-21T22:07:15.822 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:15.822 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.822 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.822 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:15.822 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.180244+0000", 2024-09-21T22:07:15.822 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:15.823 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.823 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.823 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:15.823 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.180246+0000", 2024-09-21T22:07:15.823 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6959999999999999e-06 2024-09-21T22:07:15.823 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.823 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.823 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:15.823 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.180508+0000", 2024-09-21T22:07:15.823 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00026275199999999999 2024-09-21T22:07:15.823 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.823 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.823 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:15.823 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.180538+0000", 2024-09-21T22:07:15.824 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9142999999999999e-05 2024-09-21T22:07:15.824 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.824 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.824 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:15.824 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.180552+0000", 2024-09-21T22:07:15.824 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4466e-05 2024-09-21T22:07:15.824 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.824 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.824 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:15.824 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.180677+0000", 2024-09-21T22:07:15.824 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000125413 2024-09-21T22:07:15.824 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.824 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.824 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:15.824 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.180739+0000", 2024-09-21T22:07:15.825 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.1122000000000002e-05 2024-09-21T22:07:15.825 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.825 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.825 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:15.825 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.184291+0000", 2024-09-21T22:07:15.825 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0035525510000000001 2024-09-21T22:07:15.825 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.825 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.825 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:15.825 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.184415+0000", 2024-09-21T22:07:15.825 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000124079 2024-09-21T22:07:15.825 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:15.825 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:15.825 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:15.826 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.826 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.826 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:57 1.0 e14/13 1:30306672:devicehealth::main.db.0000000000000000:head v 14'35, mlcod=14'33)", 2024-09-21T22:07:15.826 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.221549+0000", 2024-09-21T22:07:15.826 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 73.597021561000005, 2024-09-21T22:07:15.826 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0041720860000000002, 2024-09-21T22:07:15.826 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:15.826 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:15.826 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:15.826 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:15.826 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.826 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:15.826 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.221549+0000", 2024-09-21T22:07:15.826 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:15.827 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.827 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.827 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:15.827 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.221549+0000", 2024-09-21T22:07:15.827 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:15.827 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.827 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.827 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:15.827 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.221550+0000", 2024-09-21T22:07:15.827 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.717e-06 2024-09-21T22:07:15.827 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.827 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.827 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:15.827 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.221673+0000", 2024-09-21T22:07:15.827 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000122612 2024-09-21T22:07:15.828 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.828 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.828 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:15.828 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.221703+0000", 2024-09-21T22:07:15.828 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9539999999999998e-05 2024-09-21T22:07:15.828 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.828 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.828 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:15.828 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.221716+0000", 2024-09-21T22:07:15.828 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3149999999999999e-05 2024-09-21T22:07:15.828 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.828 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.828 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:15.828 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.221848+0000", 2024-09-21T22:07:15.828 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013201900000000001 2024-09-21T22:07:15.829 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.829 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.829 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:15.829 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.221908+0000", 2024-09-21T22:07:15.829 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.0452999999999999e-05 2024-09-21T22:07:15.829 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.829 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.829 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:15.829 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.225573+0000", 2024-09-21T22:07:15.829 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.003664676 2024-09-21T22:07:15.829 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.829 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.829 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:15.829 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.225721+0000", 2024-09-21T22:07:15.830 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00014791899999999999 2024-09-21T22:07:15.830 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:15.830 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:15.830 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:15.830 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.830 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.830 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:71 1.0 e14/13 1:30306672:devicehealth::main.db.0000000000000000:head v 14'45, mlcod=14'43)", 2024-09-21T22:07:15.830 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.256068+0000", 2024-09-21T22:07:15.830 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 73.562501810000001, 2024-09-21T22:07:15.830 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0041712579999999997, 2024-09-21T22:07:15.830 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:15.830 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:15.830 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:15.830 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:15.831 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.831 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:15.831 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.256068+0000", 2024-09-21T22:07:15.831 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:15.831 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.831 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.831 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:15.831 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.256068+0000", 2024-09-21T22:07:15.831 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:15.831 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.831 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.831 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:15.831 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.256070+0000", 2024-09-21T22:07:15.831 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.61e-06 2024-09-21T22:07:15.831 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.832 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.832 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:15.832 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.256129+0000", 2024-09-21T22:07:15.832 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.8702999999999997e-05 2024-09-21T22:07:15.832 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.832 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.832 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:15.832 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.256158+0000", 2024-09-21T22:07:15.832 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9533e-05 2024-09-21T22:07:15.832 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.832 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.832 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:15.832 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.256173+0000", 2024-09-21T22:07:15.832 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4267000000000001e-05 2024-09-21T22:07:15.833 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.833 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.833 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:15.833 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.256304+0000", 2024-09-21T22:07:15.833 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000131352 2024-09-21T22:07:15.833 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.833 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.833 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:15.833 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.256357+0000", 2024-09-21T22:07:15.833 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.3492999999999998e-05 2024-09-21T22:07:15.833 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.833 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.833 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:15.833 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.260147+0000", 2024-09-21T22:07:15.833 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0037900410000000001 2024-09-21T22:07:15.834 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.834 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.834 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:15.834 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.260240+0000", 2024-09-21T22:07:15.834 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.2258999999999999e-05 2024-09-21T22:07:15.834 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:15.834 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:15.834 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:15.834 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.834 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.834 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:72 1.0 e14/13 1:30306672:devicehealth::main.db.0000000000000000:head v 14'46, mlcod=14'43)", 2024-09-21T22:07:15.834 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.258358+0000", 2024-09-21T22:07:15.834 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 73.560211816999995, 2024-09-21T22:07:15.834 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0041492919999999997, 2024-09-21T22:07:15.835 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:15.835 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:15.835 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:15.835 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:15.835 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.835 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:15.835 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.258358+0000", 2024-09-21T22:07:15.835 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:15.835 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.835 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.835 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:15.835 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.258358+0000", 2024-09-21T22:07:15.835 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:15.835 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.836 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.836 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:15.836 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.258360+0000", 2024-09-21T22:07:15.836 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.623e-06 2024-09-21T22:07:15.836 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.836 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.836 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:15.836 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.258415+0000", 2024-09-21T22:07:15.836 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.4756000000000002e-05 2024-09-21T22:07:15.836 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.836 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.836 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:15.836 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.258444+0000", 2024-09-21T22:07:15.836 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9164000000000001e-05 2024-09-21T22:07:15.837 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.837 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.837 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:15.837 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.258459+0000", 2024-09-21T22:07:15.837 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4996e-05 2024-09-21T22:07:15.837 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.837 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.837 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:15.837 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.258637+0000", 2024-09-21T22:07:15.837 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00017796500000000001 2024-09-21T22:07:15.837 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.837 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.837 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:15.837 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.258695+0000", 2024-09-21T22:07:15.837 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.8207e-05 2024-09-21T22:07:15.838 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.838 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.838 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:15.838 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.262421+0000", 2024-09-21T22:07:15.838 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0037261550000000001 2024-09-21T22:07:15.838 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.838 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.838 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:15.838 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.262508+0000", 2024-09-21T22:07:15.838 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.6426000000000002e-05 2024-09-21T22:07:15.838 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:15.838 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:15.838 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:15.838 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.839 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.839 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:114 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'76, mlcod=14'73)", 2024-09-21T22:07:15.839 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.490030+0000", 2024-09-21T22:07:15.839 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 73.328539933000002, 2024-09-21T22:07:15.839 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.004227873, 2024-09-21T22:07:15.839 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:15.839 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:15.839 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:15.839 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:15.839 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.839 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:15.839 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.490030+0000", 2024-09-21T22:07:15.839 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:15.839 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.840 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.840 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:15.840 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.490030+0000", 2024-09-21T22:07:15.840 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:15.840 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.840 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.840 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:15.840 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.490031+0000", 2024-09-21T22:07:15.840 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.3300000000000001e-07 2024-09-21T22:07:15.840 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.840 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.840 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:15.840 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.490042+0000", 2024-09-21T22:07:15.840 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.1095e-05 2024-09-21T22:07:15.841 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.841 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.841 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:15.841 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.490053+0000", 2024-09-21T22:07:15.841 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0501e-05 2024-09-21T22:07:15.841 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.841 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.841 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:15.841 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.490060+0000", 2024-09-21T22:07:15.841 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.9829999999999999e-06 2024-09-21T22:07:15.841 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.841 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.841 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:15.841 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.490107+0000", 2024-09-21T22:07:15.841 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.7278999999999999e-05 2024-09-21T22:07:15.842 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.842 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.842 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:15.842 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.490124+0000", 2024-09-21T22:07:15.842 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7190999999999999e-05 2024-09-21T22:07:15.842 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.842 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.842 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:15.842 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.494229+0000", 2024-09-21T22:07:15.842 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0041050440000000004 2024-09-21T22:07:15.842 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.842 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.842 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:15.843 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.494258+0000", 2024-09-21T22:07:15.843 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9047000000000002e-05 2024-09-21T22:07:15.843 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:15.843 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:15.843 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:15.843 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.843 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.843 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:136 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'94, mlcod=14'89)", 2024-09-21T22:07:15.843 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.903810+0000", 2024-09-21T22:07:15.843 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 72.914760337999994, 2024-09-21T22:07:15.843 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0062746829999999996, 2024-09-21T22:07:15.843 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:15.843 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:15.843 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:15.843 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:15.844 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.844 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:15.844 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.903810+0000", 2024-09-21T22:07:15.844 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:15.844 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.844 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.844 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:15.844 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.903810+0000", 2024-09-21T22:07:15.844 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:15.844 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.844 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.844 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:15.844 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.903810+0000", 2024-09-21T22:07:15.844 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.03e-07 2024-09-21T22:07:15.845 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.845 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.845 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:15.845 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.903814+0000", 2024-09-21T22:07:15.845 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.676e-06 2024-09-21T22:07:15.845 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.845 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.845 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:15.845 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.903822+0000", 2024-09-21T22:07:15.845 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.2879999999999993e-06 2024-09-21T22:07:15.845 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.845 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.845 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:15.845 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.903829+0000", 2024-09-21T22:07:15.846 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.2920000000000001e-06 2024-09-21T22:07:15.846 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.846 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.846 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:15.846 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.903879+0000", 2024-09-21T22:07:15.846 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.0253999999999997e-05 2024-09-21T22:07:15.846 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.846 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.846 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:15.846 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.903897+0000", 2024-09-21T22:07:15.846 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8468999999999999e-05 2024-09-21T22:07:15.846 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.846 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.846 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:15.846 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.910044+0000", 2024-09-21T22:07:15.847 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0061468950000000003 2024-09-21T22:07:15.847 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.847 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.847 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:15.847 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.910085+0000", 2024-09-21T22:07:15.847 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.0405999999999999e-05 2024-09-21T22:07:15.847 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:15.847 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:15.847 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:15.847 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.847 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.847 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:137 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'95, mlcod=14'90)", 2024-09-21T22:07:15.847 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.904439+0000", 2024-09-21T22:07:15.847 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 72.914130942, 2024-09-21T22:07:15.848 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0072625019999999997, 2024-09-21T22:07:15.848 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:15.848 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:15.848 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:15.848 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:15.848 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.848 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:15.848 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.904439+0000", 2024-09-21T22:07:15.848 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:15.848 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.848 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.848 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:15.848 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.904439+0000", 2024-09-21T22:07:15.848 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:15.848 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.849 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.849 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:15.849 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.904440+0000", 2024-09-21T22:07:15.849 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8400000000000005e-07 2024-09-21T22:07:15.849 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.849 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.849 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:15.849 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.904498+0000", 2024-09-21T22:07:15.849 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.8320999999999999e-05 2024-09-21T22:07:15.849 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.849 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.849 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:15.849 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.904507+0000", 2024-09-21T22:07:15.849 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.7790000000000001e-06 2024-09-21T22:07:15.850 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.850 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.850 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:15.850 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.904511+0000", 2024-09-21T22:07:15.850 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.9700000000000001e-06 2024-09-21T22:07:15.850 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.850 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.850 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:15.850 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.904551+0000", 2024-09-21T22:07:15.850 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.9910000000000002e-05 2024-09-21T22:07:15.850 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.850 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.850 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:15.850 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.904567+0000", 2024-09-21T22:07:15.851 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5903999999999999e-05 2024-09-21T22:07:15.851 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.851 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.851 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:15.851 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.911666+0000", 2024-09-21T22:07:15.851 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0070996929999999998 2024-09-21T22:07:15.851 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.851 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.851 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:15.851 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.911702+0000", 2024-09-21T22:07:15.851 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.5441000000000002e-05 2024-09-21T22:07:15.851 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:15.851 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:15.851 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:15.852 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.852 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.852 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:138 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'96, mlcod=14'90)", 2024-09-21T22:07:15.852 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.905402+0000", 2024-09-21T22:07:15.852 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 72.913167965, 2024-09-21T22:07:15.852 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0063366890000000004, 2024-09-21T22:07:15.852 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:15.852 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:15.852 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:15.852 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:15.852 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.852 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:15.852 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905402+0000", 2024-09-21T22:07:15.852 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:15.853 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.853 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.853 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:15.853 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905402+0000", 2024-09-21T22:07:15.853 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:15.853 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.853 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.853 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:15.853 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905403+0000", 2024-09-21T22:07:15.853 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.9200000000000002e-07 2024-09-21T22:07:15.853 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.853 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.853 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:15.853 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905406+0000", 2024-09-21T22:07:15.854 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.326e-06 2024-09-21T22:07:15.854 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.854 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.854 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:15.854 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905413+0000", 2024-09-21T22:07:15.854 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.3320000000000003e-06 2024-09-21T22:07:15.854 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.854 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.854 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:15.854 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905417+0000", 2024-09-21T22:07:15.854 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.7139999999999999e-06 2024-09-21T22:07:15.854 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.854 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.854 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:15.855 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.910115+0000", 2024-09-21T22:07:15.855 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0046983939999999998 2024-09-21T22:07:15.855 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.855 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.855 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:15.855 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.910140+0000", 2024-09-21T22:07:15.855 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4689999999999999e-05 2024-09-21T22:07:15.855 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.855 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.855 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:15.855 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.911706+0000", 2024-09-21T22:07:15.855 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0015658479999999999 2024-09-21T22:07:15.855 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.856 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.856 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:15.856 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.911739+0000", 2024-09-21T22:07:15.856 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.2993000000000001e-05 2024-09-21T22:07:15.856 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:15.856 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:15.856 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:15.856 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.856 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.856 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:139 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'97, mlcod=14'90)", 2024-09-21T22:07:15.856 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.905983+0000", 2024-09-21T22:07:15.856 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 72.912587047000002, 2024-09-21T22:07:15.856 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0057906030000000001, 2024-09-21T22:07:15.856 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:15.857 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:15.857 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:15.857 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:15.857 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.857 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:15.857 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905983+0000", 2024-09-21T22:07:15.857 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:15.857 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.857 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.857 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:15.857 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905983+0000", 2024-09-21T22:07:15.857 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:15.857 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.857 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.857 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:15.858 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905984+0000", 2024-09-21T22:07:15.858 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.15e-07 2024-09-21T22:07:15.858 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.858 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.858 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:15.858 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905986+0000", 2024-09-21T22:07:15.858 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.852e-06 2024-09-21T22:07:15.858 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.858 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.858 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:15.858 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905993+0000", 2024-09-21T22:07:15.859 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.652e-06 2024-09-21T22:07:15.859 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.859 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.860 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:15.860 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905996+0000", 2024-09-21T22:07:15.860 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.3519999999999998e-06 2024-09-21T22:07:15.860 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.860 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.860 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:15.860 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.910519+0000", 2024-09-21T22:07:15.860 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0045230820000000003 2024-09-21T22:07:15.860 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.860 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.860 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:15.860 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.910548+0000", 2024-09-21T22:07:15.860 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8399999999999999e-05 2024-09-21T22:07:15.860 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.861 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.861 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:15.861 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.911742+0000", 2024-09-21T22:07:15.861 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011939909999999999 2024-09-21T22:07:15.861 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.861 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.861 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:15.861 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.911774+0000", 2024-09-21T22:07:15.861 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.1958999999999997e-05 2024-09-21T22:07:15.861 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:15.861 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:15.861 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:15.861 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.861 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.861 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:140 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'98, mlcod=14'90)", 2024-09-21T22:07:15.862 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.906441+0000", 2024-09-21T22:07:15.862 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 72.912128917999993, 2024-09-21T22:07:15.862 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0057052989999999996, 2024-09-21T22:07:15.862 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:15.862 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:15.862 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:15.862 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:15.862 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.862 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:15.862 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.906441+0000", 2024-09-21T22:07:15.862 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:15.862 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.862 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.862 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:15.862 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.906441+0000", 2024-09-21T22:07:15.863 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:15.863 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.863 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.863 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:15.863 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.906442+0000", 2024-09-21T22:07:15.863 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.7e-07 2024-09-21T22:07:15.863 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.863 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.863 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:15.863 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.906511+0000", 2024-09-21T22:07:15.863 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.8857999999999997e-05 2024-09-21T22:07:15.863 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.863 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.863 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:15.864 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.906518+0000", 2024-09-21T22:07:15.864 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.1690000000000003e-06 2024-09-21T22:07:15.864 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.864 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.864 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:15.864 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.906521+0000", 2024-09-21T22:07:15.864 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.4829999999999999e-06 2024-09-21T22:07:15.864 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.864 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.864 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:15.864 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.910939+0000", 2024-09-21T22:07:15.864 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0044176809999999997 2024-09-21T22:07:15.864 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.864 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.864 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:15.865 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.910959+0000", 2024-09-21T22:07:15.865 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9898000000000002e-05 2024-09-21T22:07:15.865 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.865 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.865 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:15.865 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.912113+0000", 2024-09-21T22:07:15.865 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001153811 2024-09-21T22:07:15.865 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.865 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.865 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:15.865 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.912147+0000", 2024-09-21T22:07:15.865 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.4029000000000002e-05 2024-09-21T22:07:15.865 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:15.865 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:15.865 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:15.866 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.866 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.866 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.1 54/53 [PushOp(2:837ddd83:::rbd_data.126483d75f2d.000000000000000d:head, version: 52'1, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:837ddd83:::rbd_data.126483d75f2d.000000000000000d:head@52'1, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T22:07:15.866 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:11.071135+0000", 2024-09-21T22:07:15.866 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 4.7474348070000003, 2024-09-21T22:07:15.866 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.025278432, 2024-09-21T22:07:15.866 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:15.866 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:15.866 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:15.866 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:15.866 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.866 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:15.866 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.071135+0000", 2024-09-21T22:07:15.866 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:15.867 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.867 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.867 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:15.867 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.071135+0000", 2024-09-21T22:07:15.867 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:15.867 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.867 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.867 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:15.867 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.071137+0000", 2024-09-21T22:07:15.867 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.0430000000000002e-06 2024-09-21T22:07:15.867 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.867 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.867 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:15.867 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.071783+0000", 2024-09-21T22:07:15.867 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00064543399999999996 2024-09-21T22:07:15.868 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.868 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.868 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:15.868 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.071820+0000", 2024-09-21T22:07:15.868 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.7588e-05 2024-09-21T22:07:15.868 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.868 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.868 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:15.868 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.071846+0000", 2024-09-21T22:07:15.868 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5673999999999998e-05 2024-09-21T22:07:15.868 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.868 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.868 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:15.868 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.071967+0000", 2024-09-21T22:07:15.869 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00012071899999999999 2024-09-21T22:07:15.869 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.869 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.869 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:15.869 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.071983+0000", 2024-09-21T22:07:15.869 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6013999999999999e-05 2024-09-21T22:07:15.869 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.869 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.869 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:15.869 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.096414+0000", 2024-09-21T22:07:15.869 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.024430960000000002 2024-09-21T22:07:15.869 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:15.869 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:15.869 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:15.869 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.870 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.870 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.1 54/53 [PushOp(2:837ddd83:::rbd_data.126483d75f2d.000000000000000d:head, version: 52'1, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:837ddd83:::rbd_data.126483d75f2d.000000000000000d:head@52'1, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:07:15.870 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:11.099645+0000", 2024-09-21T22:07:15.870 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 4.7189252459999995, 2024-09-21T22:07:15.870 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.026929337000000001, 2024-09-21T22:07:15.870 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:15.870 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:15.870 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:15.870 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:15.870 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.870 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:15.870 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.099645+0000", 2024-09-21T22:07:15.870 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:15.870 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.871 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.871 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:15.871 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.099645+0000", 2024-09-21T22:07:15.871 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:15.871 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.871 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.871 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:15.871 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.099646+0000", 2024-09-21T22:07:15.871 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.221e-06 2024-09-21T22:07:15.871 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.871 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.871 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:15.871 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.100121+0000", 2024-09-21T22:07:15.871 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00047480500000000002 2024-09-21T22:07:15.871 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.872 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.872 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:15.872 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.100144+0000", 2024-09-21T22:07:15.872 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.2812000000000001e-05 2024-09-21T22:07:15.872 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.872 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.872 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:15.872 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.100178+0000", 2024-09-21T22:07:15.872 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.4379999999999999e-05 2024-09-21T22:07:15.872 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.872 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.872 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:15.872 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.100313+0000", 2024-09-21T22:07:15.872 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013434499999999999 2024-09-21T22:07:15.872 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.873 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.873 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:15.873 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.100323+0000", 2024-09-21T22:07:15.873 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0930999999999999e-05 2024-09-21T22:07:15.873 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.873 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.873 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:15.873 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.126574+0000", 2024-09-21T22:07:15.873 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.026250842999999999 2024-09-21T22:07:15.873 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:15.873 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:15.873 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:15.873 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.873 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.873 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.1 54/53 [PushOp(2:837ddd83:::rbd_data.126483d75f2d.000000000000000d:head, version: 52'1, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:837ddd83:::rbd_data.126483d75f2d.000000000000000d:head@52'1, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:07:15.874 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:11.129410+0000", 2024-09-21T22:07:15.874 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 4.6891600630000001, 2024-09-21T22:07:15.874 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.027274217, 2024-09-21T22:07:15.874 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:15.874 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:15.874 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:15.874 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:15.874 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.874 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:15.874 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.129410+0000", 2024-09-21T22:07:15.874 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:15.874 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.874 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.874 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:15.875 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.129410+0000", 2024-09-21T22:07:15.875 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:15.875 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.875 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.875 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:15.875 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.129412+0000", 2024-09-21T22:07:15.875 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6160000000000001e-06 2024-09-21T22:07:15.875 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.875 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.875 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:15.875 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.130407+0000", 2024-09-21T22:07:15.875 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00099554299999999999 2024-09-21T22:07:15.875 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.875 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.876 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:15.876 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.130454+0000", 2024-09-21T22:07:15.876 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.6545000000000002e-05 2024-09-21T22:07:15.876 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.876 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.876 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:15.876 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.130508+0000", 2024-09-21T22:07:15.876 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.3940999999999997e-05 2024-09-21T22:07:15.876 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.876 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.876 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:15.876 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.130658+0000", 2024-09-21T22:07:15.876 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000149732 2024-09-21T22:07:15.876 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.876 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.877 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:15.877 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.130673+0000", 2024-09-21T22:07:15.877 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5608e-05 2024-09-21T22:07:15.877 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.877 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.877 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:15.877 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.156684+0000", 2024-09-21T22:07:15.877 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.026011231999999999 2024-09-21T22:07:15.877 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:15.877 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:15.877 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:15.877 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.877 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.877 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.1 54/53 [PushOp(2:837ddd83:::rbd_data.126483d75f2d.000000000000000d:head, version: 52'1, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:837ddd83:::rbd_data.126483d75f2d.000000000000000d:head@52'1, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 4194304, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:07:15.877 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:11.159382+0000", 2024-09-21T22:07:15.878 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 4.6591886750000002, 2024-09-21T22:07:15.878 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.026585683999999998, 2024-09-21T22:07:15.878 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:15.878 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:15.878 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:15.878 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:15.878 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.878 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:15.878 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.159382+0000", 2024-09-21T22:07:15.878 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:15.878 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.878 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.878 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:15.879 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.159382+0000", 2024-09-21T22:07:15.879 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:15.879 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.879 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.879 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:15.879 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.159383+0000", 2024-09-21T22:07:15.879 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.544e-06 2024-09-21T22:07:15.879 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.879 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.879 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:15.879 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.160369+0000", 2024-09-21T22:07:15.879 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00098620700000000001 2024-09-21T22:07:15.879 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.879 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.879 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:15.880 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.160415+0000", 2024-09-21T22:07:15.880 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.6074e-05 2024-09-21T22:07:15.880 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.880 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.880 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:15.880 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.160463+0000", 2024-09-21T22:07:15.880 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8087999999999998e-05 2024-09-21T22:07:15.880 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.880 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.880 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:15.880 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.160577+0000", 2024-09-21T22:07:15.880 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000113893 2024-09-21T22:07:15.880 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.880 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.880 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:15.881 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.160592+0000", 2024-09-21T22:07:15.881 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.465e-05 2024-09-21T22:07:15.881 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.881 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.881 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:15.881 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.185967+0000", 2024-09-21T22:07:15.881 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.025375228 2024-09-21T22:07:15.881 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:15.881 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:15.881 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:15.881 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.881 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.881 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.1 54/53 [PushOp(2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head, version: 52'2, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head@52'2, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T22:07:15.881 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:11.189894+0000", 2024-09-21T22:07:15.881 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 4.6286767099999997, 2024-09-21T22:07:15.882 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.025252653, 2024-09-21T22:07:15.882 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:15.882 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:15.882 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:15.882 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:15.882 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.882 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:15.882 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.189894+0000", 2024-09-21T22:07:15.882 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:15.882 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.882 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.882 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:15.882 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.189894+0000", 2024-09-21T22:07:15.883 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:15.883 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.883 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.883 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:15.883 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.189895+0000", 2024-09-21T22:07:15.883 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.916e-06 2024-09-21T22:07:15.883 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.883 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.883 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:15.883 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.190890+0000", 2024-09-21T22:07:15.883 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00099456900000000005 2024-09-21T22:07:15.883 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.883 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.883 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:15.883 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.190939+0000", 2024-09-21T22:07:15.884 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.9132000000000003e-05 2024-09-21T22:07:15.884 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.884 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.884 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:15.884 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.190989+0000", 2024-09-21T22:07:15.884 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.9913999999999997e-05 2024-09-21T22:07:15.884 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.884 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.884 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:15.884 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.191122+0000", 2024-09-21T22:07:15.884 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013264 2024-09-21T22:07:15.884 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.884 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.884 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:15.884 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.191137+0000", 2024-09-21T22:07:15.885 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4871999999999999e-05 2024-09-21T22:07:15.885 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.885 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.885 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:15.885 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.215146+0000", 2024-09-21T22:07:15.885 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.024009610000000001 2024-09-21T22:07:15.885 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:15.885 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:15.885 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:15.885 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.885 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.885 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.1 54/53 [PushOp(2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head, version: 52'2, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head@52'2, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:07:15.885 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:11.217669+0000", 2024-09-21T22:07:15.885 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 4.6009012689999995, 2024-09-21T22:07:15.886 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.024843860999999998, 2024-09-21T22:07:15.886 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:15.886 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:15.886 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:15.886 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:15.886 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.886 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:15.886 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.217669+0000", 2024-09-21T22:07:15.886 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:15.886 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.886 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.886 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:15.886 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.217669+0000", 2024-09-21T22:07:15.886 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:15.887 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.887 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.887 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:15.887 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.217671+0000", 2024-09-21T22:07:15.887 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7129999999999999e-06 2024-09-21T22:07:15.887 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.887 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.887 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:15.887 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.218680+0000", 2024-09-21T22:07:15.887 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0010089630000000001 2024-09-21T22:07:15.887 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.887 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.887 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:15.887 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.218726+0000", 2024-09-21T22:07:15.887 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.6307999999999998e-05 2024-09-21T22:07:15.888 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.888 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.888 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:15.888 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.218779+0000", 2024-09-21T22:07:15.888 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.3344999999999998e-05 2024-09-21T22:07:15.888 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.888 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.888 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:15.888 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.218907+0000", 2024-09-21T22:07:15.888 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00012746799999999999 2024-09-21T22:07:15.888 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.888 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.888 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:15.888 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.218921+0000", 2024-09-21T22:07:15.888 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4581e-05 2024-09-21T22:07:15.889 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.889 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.889 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:15.889 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.242513+0000", 2024-09-21T22:07:15.889 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.023591483 2024-09-21T22:07:15.889 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:15.889 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:15.889 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:15.889 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.889 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.889 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.1 54/53 [PushOp(2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head, version: 52'2, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head@52'2, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:07:15.889 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:11.245267+0000", 2024-09-21T22:07:15.889 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 4.5733034870000004, 2024-09-21T22:07:15.889 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.025143901, 2024-09-21T22:07:15.890 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:15.890 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:15.890 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:15.890 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:15.890 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.890 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:15.890 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.245267+0000", 2024-09-21T22:07:15.890 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:15.890 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.890 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.890 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:15.890 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.245267+0000", 2024-09-21T22:07:15.890 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:15.890 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.891 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.891 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:15.891 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.245269+0000", 2024-09-21T22:07:15.891 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5299999999999999e-06 2024-09-21T22:07:15.891 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.891 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.891 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:15.891 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.246497+0000", 2024-09-21T22:07:15.891 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00122766 2024-09-21T22:07:15.891 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.891 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.891 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:15.891 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.246545+0000", 2024-09-21T22:07:15.891 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.7757999999999999e-05 2024-09-21T22:07:15.892 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.892 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.892 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:15.892 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.246579+0000", 2024-09-21T22:07:15.892 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.4186000000000001e-05 2024-09-21T22:07:15.892 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.892 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.892 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:15.892 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.246712+0000", 2024-09-21T22:07:15.892 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013312099999999999 2024-09-21T22:07:15.892 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.892 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.892 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:15.892 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.246722+0000", 2024-09-21T22:07:15.892 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.7869999999999996e-06 2024-09-21T22:07:15.893 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.893 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.893 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:15.893 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.270411+0000", 2024-09-21T22:07:15.893 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.023688859 2024-09-21T22:07:15.893 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:15.893 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:15.893 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:15.893 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.893 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.893 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.1 54/53 [PushOp(2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head, version: 52'2, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head@52'2, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 4194304, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:07:15.893 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:11.273163+0000", 2024-09-21T22:07:15.893 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 4.5454067980000001, 2024-09-21T22:07:15.893 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.025286874000000001, 2024-09-21T22:07:15.894 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:15.894 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:15.894 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:15.894 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:15.894 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.894 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:15.894 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.273163+0000", 2024-09-21T22:07:15.894 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:15.894 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.894 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.894 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:15.894 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.273163+0000", 2024-09-21T22:07:15.894 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:15.894 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.894 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.895 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:15.895 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.273166+0000", 2024-09-21T22:07:15.895 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5969999999999999e-06 2024-09-21T22:07:15.895 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.895 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.895 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:15.895 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.274383+0000", 2024-09-21T22:07:15.895 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001217274 2024-09-21T22:07:15.895 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.895 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.895 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:15.895 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.274430+0000", 2024-09-21T22:07:15.895 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.6848000000000003e-05 2024-09-21T22:07:15.895 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.896 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.896 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:15.896 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.274465+0000", 2024-09-21T22:07:15.896 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.5231999999999997e-05 2024-09-21T22:07:15.896 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.896 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.896 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:15.896 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.274561+0000", 2024-09-21T22:07:15.896 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.5962e-05 2024-09-21T22:07:15.896 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.896 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.896 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:15.896 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.274571+0000", 2024-09-21T22:07:15.896 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.5449999999999992e-06 2024-09-21T22:07:15.896 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.897 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.897 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:15.897 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.298450+0000", 2024-09-21T22:07:15.897 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.023879416000000001 2024-09-21T22:07:15.897 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:15.897 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:15.897 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:15.897 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.897 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.897 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(1.0 55/54 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 15'116, data_included: [0~262688], data_size: 262688, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@15'116, size: 262688, copy_subset: [(0, 262688)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 262688, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T22:07:15.897 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.155955+0000", 2024-09-21T22:07:15.897 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 3.6626148970000001, 2024-09-21T22:07:15.897 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.004903368, 2024-09-21T22:07:15.897 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:15.897 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:15.898 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:15.898 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:15.898 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.898 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:15.898 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.155955+0000", 2024-09-21T22:07:15.898 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:15.898 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.898 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.898 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:15.898 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.155955+0000", 2024-09-21T22:07:15.898 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:15.898 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.898 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.898 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:15.899 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.155957+0000", 2024-09-21T22:07:15.899 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5829999999999999e-06 2024-09-21T22:07:15.899 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.899 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.899 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:15.899 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.156375+0000", 2024-09-21T22:07:15.899 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00041813700000000001 2024-09-21T22:07:15.899 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.899 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.899 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:15.899 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.156420+0000", 2024-09-21T22:07:15.899 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.4734999999999998e-05 2024-09-21T22:07:15.899 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.899 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.900 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:15.900 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.156452+0000", 2024-09-21T22:07:15.900 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.1985e-05 2024-09-21T22:07:15.900 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.900 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.900 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:15.900 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.156629+0000", 2024-09-21T22:07:15.900 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000177107 2024-09-21T22:07:15.900 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.900 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.900 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:15.900 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.156657+0000", 2024-09-21T22:07:15.900 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.7871000000000001e-05 2024-09-21T22:07:15.900 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.900 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.901 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:15.901 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.160859+0000", 2024-09-21T22:07:15.901 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0042019500000000003 2024-09-21T22:07:15.901 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:15.901 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:15.901 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:15.901 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.901 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.901 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(1.0 55/54 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 15'117, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@15'117, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T22:07:15.901 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.162929+0000", 2024-09-21T22:07:15.901 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 3.6556416729999999, 2024-09-21T22:07:15.901 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0056463110000000002, 2024-09-21T22:07:15.901 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:15.901 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:15.902 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:15.902 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:15.902 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.902 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:15.902 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.162929+0000", 2024-09-21T22:07:15.902 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:15.902 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.902 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.902 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:15.902 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.162929+0000", 2024-09-21T22:07:15.902 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:15.902 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.902 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.902 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:15.903 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.162930+0000", 2024-09-21T22:07:15.903 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.77e-06 2024-09-21T22:07:15.903 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.903 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.903 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:15.903 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.163405+0000", 2024-09-21T22:07:15.903 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00047464499999999999 2024-09-21T22:07:15.903 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.903 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.903 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:15.903 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.163453+0000", 2024-09-21T22:07:15.903 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8211999999999999e-05 2024-09-21T22:07:15.903 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.903 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.904 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:15.904 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.163485+0000", 2024-09-21T22:07:15.904 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.2011000000000002e-05 2024-09-21T22:07:15.904 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.904 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.904 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:15.904 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.163662+0000", 2024-09-21T22:07:15.904 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000177059 2024-09-21T22:07:15.904 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.904 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.904 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:15.904 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.163688+0000", 2024-09-21T22:07:15.904 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5831000000000001e-05 2024-09-21T22:07:15.904 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:15.904 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:15.905 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:15.905 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.168575+0000", 2024-09-21T22:07:15.905 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0048867829999999996 2024-09-21T22:07:15.905 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:15.905 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:15.905 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:15.905 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:15.905 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:15.905 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:07:15.906 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T22:07:15.907 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:15.904+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:16.007 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:16.007+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:16.079 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:07:16.079 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:07:16.079 INFO:teuthology.orchestra.run.smithi039.stdout: "num_ops": 0 2024-09-21T22:07:16.079 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:07:16.094 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T22:07:16.114 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-21T22:07:16.140 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:16.139+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:16.209 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:16.207+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:16.214 INFO:tasks.workunit.client.0.smithi157.stderr:pool 'new_default_pool21553' already exists 2024-09-21T22:07:16.230 INFO:tasks.workunit.client.0.smithi157.stderr:+ rbd pool init new_default_pool21553 2024-09-21T22:07:16.265 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:07:16.265 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:07:16.265 INFO:teuthology.orchestra.run.smithi039.stdout: "complaint_time": 30, 2024-09-21T22:07:16.265 INFO:teuthology.orchestra.run.smithi039.stdout: "num_blocked_ops": 0 2024-09-21T22:07:16.265 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:07:16.281 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T22:07:16.328 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:16.327+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:16.410 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:16.407+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:16.448 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:07:16.448 INFO:teuthology.orchestra.run.smithi039.stdout: "size": 20, 2024-09-21T22:07:16.448 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 600, 2024-09-21T22:07:16.448 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [ 2024-09-21T22:07:16.448 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.449 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4224.0:5 2.2 e18/16 2:5cea7035:::rbd_info:head v 18'1, mlcod=0'0)", 2024-09-21T22:07:16.449 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:06.114955+0000", 2024-09-21T22:07:16.449 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 70.329162041000004, 2024-09-21T22:07:16.449 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0017716170000000001, 2024-09-21T22:07:16.449 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:16.449 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:16.449 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:16.450 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:16.451 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.451 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:16.451 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:06.114955+0000", 2024-09-21T22:07:16.451 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:16.451 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.451 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.452 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:16.452 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:06.114955+0000", 2024-09-21T22:07:16.452 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:16.452 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.452 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.452 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:16.452 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:06.114956+0000", 2024-09-21T22:07:16.453 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.9299999999999997e-07 2024-09-21T22:07:16.453 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.453 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.453 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:16.453 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:06.114964+0000", 2024-09-21T22:07:16.453 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.9840000000000001e-06 2024-09-21T22:07:16.454 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.454 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.454 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:16.454 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:06.115001+0000", 2024-09-21T22:07:16.454 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.7719999999999998e-05 2024-09-21T22:07:16.454 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.454 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.454 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:16.455 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:06.115019+0000", 2024-09-21T22:07:16.455 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7943000000000001e-05 2024-09-21T22:07:16.455 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.455 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.455 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:16.455 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:06.115168+0000", 2024-09-21T22:07:16.455 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000148589 2024-09-21T22:07:16.455 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.455 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.455 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:16.456 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:06.115225+0000", 2024-09-21T22:07:16.456 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.6975999999999998e-05 2024-09-21T22:07:16.456 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.456 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.456 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:16.456 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:06.116585+0000", 2024-09-21T22:07:16.456 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00136038 2024-09-21T22:07:16.457 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.457 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.457 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:16.457 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:06.116726+0000", 2024-09-21T22:07:16.457 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00014133200000000001 2024-09-21T22:07:16.457 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:16.457 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:16.457 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:16.457 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.457 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.457 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4224.0:7 2.2 e19/16 2:5cea7035:::rbd_info:head v 19'2, mlcod=0'0)", 2024-09-21T22:07:16.457 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:07.129567+0000", 2024-09-21T22:07:16.457 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 69.314550032, 2024-09-21T22:07:16.458 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0018460239999999999, 2024-09-21T22:07:16.458 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:16.458 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:16.458 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:16.458 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:16.458 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.458 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:16.458 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:07.129567+0000", 2024-09-21T22:07:16.458 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:16.458 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.458 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.459 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:16.459 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:07.129567+0000", 2024-09-21T22:07:16.459 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:16.459 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.459 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.459 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:16.459 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:07.129567+0000", 2024-09-21T22:07:16.459 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.9499999999999998e-07 2024-09-21T22:07:16.459 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.459 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.459 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:16.459 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:07.129580+0000", 2024-09-21T22:07:16.459 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2335e-05 2024-09-21T22:07:16.459 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.460 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.460 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:16.460 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:07.129606+0000", 2024-09-21T22:07:16.460 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.6528000000000001e-05 2024-09-21T22:07:16.460 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.460 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.460 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:16.460 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:07.129628+0000", 2024-09-21T22:07:16.460 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1549999999999999e-05 2024-09-21T22:07:16.460 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.460 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.460 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:16.460 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:07.129744+0000", 2024-09-21T22:07:16.460 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000116551 2024-09-21T22:07:16.461 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.461 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.461 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:16.461 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:07.129780+0000", 2024-09-21T22:07:16.462 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.5447000000000001e-05 2024-09-21T22:07:16.462 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.462 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.462 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:16.462 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:07.131348+0000", 2024-09-21T22:07:16.462 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001568631 2024-09-21T22:07:16.462 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.462 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.462 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:16.462 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:07.131413+0000", 2024-09-21T22:07:16.462 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.4586999999999997e-05 2024-09-21T22:07:16.462 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:16.462 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:16.463 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:16.463 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.463 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.463 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPushReply(2.1 54/53 [PushReplyOp(2:837ddd83:::rbd_data.126483d75f2d.000000000000000d:head)])", 2024-09-21T22:07:16.463 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:11.097439+0000", 2024-09-21T22:07:16.463 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 5.3466779029999998, 2024-09-21T22:07:16.463 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0020810300000000002, 2024-09-21T22:07:16.463 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:16.463 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:16.463 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-21T22:07:16.463 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:16.463 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.463 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:16.463 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.097439+0000", 2024-09-21T22:07:16.464 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:16.464 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.464 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.464 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:16.464 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.097439+0000", 2024-09-21T22:07:16.464 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:16.464 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.464 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.464 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:16.464 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.097441+0000", 2024-09-21T22:07:16.464 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.804e-06 2024-09-21T22:07:16.464 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.464 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.464 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:16.464 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.097449+0000", 2024-09-21T22:07:16.465 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.439e-06 2024-09-21T22:07:16.465 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.465 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.465 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:16.465 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.097469+0000", 2024-09-21T22:07:16.465 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9351000000000002e-05 2024-09-21T22:07:16.465 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.465 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.465 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:16.465 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.097480+0000", 2024-09-21T22:07:16.465 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.1215e-05 2024-09-21T22:07:16.465 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.465 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.465 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:16.465 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.097558+0000", 2024-09-21T22:07:16.466 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.8675000000000001e-05 2024-09-21T22:07:16.466 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.466 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.466 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:16.466 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.099520+0000", 2024-09-21T22:07:16.466 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0019615460000000002 2024-09-21T22:07:16.466 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:16.466 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:16.466 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:16.466 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.466 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.466 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPushReply(2.1 54/53 [PushReplyOp(2:837ddd83:::rbd_data.126483d75f2d.000000000000000d:head)])", 2024-09-21T22:07:16.466 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:11.127573+0000", 2024-09-21T22:07:16.466 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 5.3165434600000001, 2024-09-21T22:07:16.467 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001713153, 2024-09-21T22:07:16.467 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:16.467 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:16.467 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-21T22:07:16.467 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:16.467 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.467 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:16.467 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.127573+0000", 2024-09-21T22:07:16.467 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:16.467 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.467 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.467 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:16.467 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.127573+0000", 2024-09-21T22:07:16.467 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:16.468 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.468 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.468 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:16.468 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.127575+0000", 2024-09-21T22:07:16.468 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5319999999999999e-06 2024-09-21T22:07:16.468 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.468 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.468 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:16.468 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.127583+0000", 2024-09-21T22:07:16.468 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.7579999999999992e-06 2024-09-21T22:07:16.468 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.468 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.468 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:16.468 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.127601+0000", 2024-09-21T22:07:16.469 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.836e-05 2024-09-21T22:07:16.469 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.469 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.469 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:16.469 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.127611+0000", 2024-09-21T22:07:16.469 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.9550000000000004e-06 2024-09-21T22:07:16.469 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.469 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.469 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:16.469 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.127697+0000", 2024-09-21T22:07:16.469 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.6397000000000006e-05 2024-09-21T22:07:16.470 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.470 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.470 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:16.470 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.129287+0000", 2024-09-21T22:07:16.470 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001589151 2024-09-21T22:07:16.470 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:16.470 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:16.470 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:16.470 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.470 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.470 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPushReply(2.1 54/53 [PushReplyOp(2:837ddd83:::rbd_data.126483d75f2d.000000000000000d:head)])", 2024-09-21T22:07:16.470 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:11.157614+0000", 2024-09-21T22:07:16.470 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 5.2865027800000002, 2024-09-21T22:07:16.471 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001640238, 2024-09-21T22:07:16.471 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:16.471 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:16.471 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-21T22:07:16.471 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:16.471 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.471 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:16.471 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.157614+0000", 2024-09-21T22:07:16.471 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:16.471 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.471 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.471 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:16.471 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.157614+0000", 2024-09-21T22:07:16.472 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:16.472 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.472 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.472 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:16.472 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.157616+0000", 2024-09-21T22:07:16.472 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.815e-06 2024-09-21T22:07:16.472 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.472 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.472 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:16.472 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.157624+0000", 2024-09-21T22:07:16.472 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.8739999999999995e-06 2024-09-21T22:07:16.472 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.472 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.472 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:16.472 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.157640+0000", 2024-09-21T22:07:16.473 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6461e-05 2024-09-21T22:07:16.473 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.473 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.473 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:16.473 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.157649+0000", 2024-09-21T22:07:16.473 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.9809999999999992e-06 2024-09-21T22:07:16.473 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.473 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.473 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:16.473 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.157714+0000", 2024-09-21T22:07:16.473 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.4802999999999996e-05 2024-09-21T22:07:16.473 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.473 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.473 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:16.474 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.159254+0000", 2024-09-21T22:07:16.474 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0015403039999999999 2024-09-21T22:07:16.474 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:16.474 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:16.474 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:16.474 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.474 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.474 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPushReply(2.1 54/53 [PushReplyOp(2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head)])", 2024-09-21T22:07:16.474 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:11.215949+0000", 2024-09-21T22:07:16.474 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 5.2281676619999997, 2024-09-21T22:07:16.474 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0016095669999999999, 2024-09-21T22:07:16.474 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:16.474 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:16.475 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-21T22:07:16.475 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:16.475 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.475 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:16.475 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.215949+0000", 2024-09-21T22:07:16.475 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:16.475 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.475 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.475 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:16.475 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.215949+0000", 2024-09-21T22:07:16.475 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:16.475 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.475 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.475 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:16.476 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.215951+0000", 2024-09-21T22:07:16.476 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3680000000000001e-06 2024-09-21T22:07:16.476 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.476 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.476 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:16.476 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.215958+0000", 2024-09-21T22:07:16.476 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.8350000000000004e-06 2024-09-21T22:07:16.476 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.476 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.476 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:16.476 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.215977+0000", 2024-09-21T22:07:16.476 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8420999999999999e-05 2024-09-21T22:07:16.476 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.476 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.476 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:16.477 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.215986+0000", 2024-09-21T22:07:16.477 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.6150000000000003e-06 2024-09-21T22:07:16.477 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.477 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.477 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:16.477 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.216058+0000", 2024-09-21T22:07:16.477 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.1205999999999996e-05 2024-09-21T22:07:16.477 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.477 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.477 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:16.477 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.217559+0000", 2024-09-21T22:07:16.477 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0015011219999999999 2024-09-21T22:07:16.477 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:16.477 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:16.477 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:16.478 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.478 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.478 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPushReply(2.1 54/53 [PushReplyOp(2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head)])", 2024-09-21T22:07:16.478 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:11.243439+0000", 2024-09-21T22:07:16.478 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 5.2006780739999998, 2024-09-21T22:07:16.478 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001592937, 2024-09-21T22:07:16.478 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:16.478 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:16.478 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-21T22:07:16.478 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:16.478 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.478 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:16.478 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.243439+0000", 2024-09-21T22:07:16.478 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:16.479 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.479 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.479 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:16.479 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.243439+0000", 2024-09-21T22:07:16.479 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:16.479 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.479 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.479 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:16.479 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.243441+0000", 2024-09-21T22:07:16.479 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.0370000000000001e-06 2024-09-21T22:07:16.479 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.479 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.479 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:16.479 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.243454+0000", 2024-09-21T22:07:16.479 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2789000000000001e-05 2024-09-21T22:07:16.480 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.480 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.480 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:16.480 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.243487+0000", 2024-09-21T22:07:16.480 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.2966000000000003e-05 2024-09-21T22:07:16.480 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.480 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.480 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:16.480 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.243505+0000", 2024-09-21T22:07:16.480 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8204999999999999e-05 2024-09-21T22:07:16.480 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.480 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.480 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:16.480 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.243590+0000", 2024-09-21T22:07:16.480 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.5649999999999995e-05 2024-09-21T22:07:16.481 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.481 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.481 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:16.481 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.245032+0000", 2024-09-21T22:07:16.481 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00144129 2024-09-21T22:07:16.481 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:16.481 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:16.481 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:16.481 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.481 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.481 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPushReply(2.1 54/53 [PushReplyOp(2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head)])", 2024-09-21T22:07:16.481 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:11.271310+0000", 2024-09-21T22:07:16.481 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 5.1728066369999999, 2024-09-21T22:07:16.481 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001608749, 2024-09-21T22:07:16.482 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:16.482 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:16.482 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-21T22:07:16.482 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:16.482 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.482 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:16.482 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.271310+0000", 2024-09-21T22:07:16.482 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:16.482 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.482 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.482 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:16.482 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.271310+0000", 2024-09-21T22:07:16.482 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:16.483 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.483 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.483 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:16.483 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.271313+0000", 2024-09-21T22:07:16.483 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3499999999999999e-06 2024-09-21T22:07:16.483 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.483 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.483 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:16.483 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.271342+0000", 2024-09-21T22:07:16.483 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9856e-05 2024-09-21T22:07:16.483 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.483 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.483 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:16.483 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.271380+0000", 2024-09-21T22:07:16.484 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.7431000000000001e-05 2024-09-21T22:07:16.484 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.484 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.484 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:16.484 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.271401+0000", 2024-09-21T22:07:16.484 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1376e-05 2024-09-21T22:07:16.484 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.484 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.484 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:16.484 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.271489+0000", 2024-09-21T22:07:16.484 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.8183000000000006e-05 2024-09-21T22:07:16.484 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.484 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.484 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:16.485 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.272919+0000", 2024-09-21T22:07:16.485 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0014295530000000001 2024-09-21T22:07:16.485 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:16.485 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:16.485 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:16.485 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.485 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.485 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.3 55/54 [PushOp(2:c4c92e5a:::rbd_trash:head, version: 17'1, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:c4c92e5a:::rbd_trash:head@17'1, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 0, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T22:07:16.485 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.072860+0000", 2024-09-21T22:07:16.485 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 4.3712565410000002, 2024-09-21T22:07:16.485 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001018885, 2024-09-21T22:07:16.485 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:16.485 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:16.485 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:16.486 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:16.486 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.486 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:16.486 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.072860+0000", 2024-09-21T22:07:16.486 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:16.486 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.486 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.486 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:16.486 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.072860+0000", 2024-09-21T22:07:16.486 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:16.486 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.486 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.486 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:16.486 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.072861+0000", 2024-09-21T22:07:16.487 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.8599999999999997e-07 2024-09-21T22:07:16.487 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.487 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.487 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:16.487 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.072882+0000", 2024-09-21T22:07:16.487 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.0329000000000001e-05 2024-09-21T22:07:16.487 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.487 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.487 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:16.487 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.072917+0000", 2024-09-21T22:07:16.487 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.5432000000000002e-05 2024-09-21T22:07:16.487 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.487 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.487 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:16.488 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.072963+0000", 2024-09-21T22:07:16.488 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.6421999999999997e-05 2024-09-21T22:07:16.488 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.488 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.488 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:16.488 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.073103+0000", 2024-09-21T22:07:16.488 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013912199999999999 2024-09-21T22:07:16.488 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.488 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.488 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:16.488 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.073123+0000", 2024-09-21T22:07:16.488 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.0636999999999999e-05 2024-09-21T22:07:16.488 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.488 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.489 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:16.489 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.073879+0000", 2024-09-21T22:07:16.489 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00075605699999999997 2024-09-21T22:07:16.489 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:16.489 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:16.489 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:16.489 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.489 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.489 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.3 55/54 [PushOp(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head, version: 52'2, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head@52'2, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T22:07:16.489 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.084379+0000", 2024-09-21T22:07:16.489 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 4.3597380699999997, 2024-09-21T22:07:16.489 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.051020674000000002, 2024-09-21T22:07:16.489 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:16.490 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:16.490 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:16.490 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:16.490 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.490 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:16.490 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.084379+0000", 2024-09-21T22:07:16.490 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:16.490 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.490 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.490 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:16.490 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.084379+0000", 2024-09-21T22:07:16.490 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:16.490 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.490 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.491 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:16.491 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.084381+0000", 2024-09-21T22:07:16.491 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.73e-06 2024-09-21T22:07:16.491 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.491 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.491 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:16.491 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.085494+0000", 2024-09-21T22:07:16.491 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011131559999999999 2024-09-21T22:07:16.491 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.491 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.491 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:16.491 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.085542+0000", 2024-09-21T22:07:16.491 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8334999999999997e-05 2024-09-21T22:07:16.492 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.492 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.492 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:16.492 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.085591+0000", 2024-09-21T22:07:16.492 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.9165000000000001e-05 2024-09-21T22:07:16.492 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.493 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.493 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:16.493 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.085777+0000", 2024-09-21T22:07:16.494 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000186152 2024-09-21T22:07:16.494 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.494 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.494 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:16.494 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.085796+0000", 2024-09-21T22:07:16.494 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8612999999999999e-05 2024-09-21T22:07:16.494 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.494 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.494 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:16.494 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.135399+0000", 2024-09-21T22:07:16.494 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.049603522999999997 2024-09-21T22:07:16.494 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:16.494 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:16.494 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:16.495 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.495 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.495 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.3 55/54 [PushOp(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head, version: 52'2, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head@52'2, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:07:16.495 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.139554+0000", 2024-09-21T22:07:16.495 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 4.3045629669999999, 2024-09-21T22:07:16.495 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.056215389999999997, 2024-09-21T22:07:16.495 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:16.495 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:16.495 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:16.495 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:16.495 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.495 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:16.495 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.139554+0000", 2024-09-21T22:07:16.495 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:16.496 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.496 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.496 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:16.496 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.139554+0000", 2024-09-21T22:07:16.496 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:16.496 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.496 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.496 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:16.496 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.139556+0000", 2024-09-21T22:07:16.496 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7999999999999999e-06 2024-09-21T22:07:16.496 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.496 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.496 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:16.496 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.140671+0000", 2024-09-21T22:07:16.497 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001115612 2024-09-21T22:07:16.497 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.497 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.497 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:16.497 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.140720+0000", 2024-09-21T22:07:16.497 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8631999999999998e-05 2024-09-21T22:07:16.497 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.497 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.497 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:16.497 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.140752+0000", 2024-09-21T22:07:16.497 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.2199000000000001e-05 2024-09-21T22:07:16.497 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.497 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.497 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:16.497 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.140900+0000", 2024-09-21T22:07:16.498 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00014786000000000001 2024-09-21T22:07:16.498 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.498 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.498 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:16.498 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.140917+0000", 2024-09-21T22:07:16.498 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6753999999999999e-05 2024-09-21T22:07:16.498 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.498 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.498 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:16.498 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.195769+0000", 2024-09-21T22:07:16.498 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.054852533000000002 2024-09-21T22:07:16.498 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:16.498 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:16.498 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:16.499 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.499 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.499 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.3 55/54 [PushOp(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head, version: 52'2, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head@52'2, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:07:16.499 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.199187+0000", 2024-09-21T22:07:16.499 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 4.2449294230000003, 2024-09-21T22:07:16.499 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.051543975999999998, 2024-09-21T22:07:16.499 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:16.499 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:16.499 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:16.499 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:16.499 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.499 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:16.499 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.199187+0000", 2024-09-21T22:07:16.499 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:16.500 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.500 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.500 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:16.500 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.199187+0000", 2024-09-21T22:07:16.500 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:16.500 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.500 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.500 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:16.500 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.199189+0000", 2024-09-21T22:07:16.500 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7969999999999999e-06 2024-09-21T22:07:16.500 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.500 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.500 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:16.500 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.200355+0000", 2024-09-21T22:07:16.501 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011660139999999999 2024-09-21T22:07:16.501 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.501 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.501 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:16.501 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.200405+0000", 2024-09-21T22:07:16.501 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.0124000000000003e-05 2024-09-21T22:07:16.501 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.501 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.501 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:16.501 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.200457+0000", 2024-09-21T22:07:16.501 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.1177999999999997e-05 2024-09-21T22:07:16.501 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.501 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.501 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:16.502 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.200635+0000", 2024-09-21T22:07:16.502 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00017841299999999999 2024-09-21T22:07:16.502 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.502 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.502 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:16.502 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.200653+0000", 2024-09-21T22:07:16.502 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7944999999999998e-05 2024-09-21T22:07:16.502 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.502 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.502 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:16.502 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.250731+0000", 2024-09-21T22:07:16.502 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.050078505000000002 2024-09-21T22:07:16.502 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:16.502 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:16.503 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:16.503 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.503 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.503 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.3 55/54 [PushOp(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head, version: 52'2, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head@52'2, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 4194304, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:07:16.503 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.253966+0000", 2024-09-21T22:07:16.503 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 4.1901510929999999, 2024-09-21T22:07:16.503 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.041100606999999997, 2024-09-21T22:07:16.503 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:16.503 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:16.503 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:16.503 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:16.503 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.503 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:16.503 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.253966+0000", 2024-09-21T22:07:16.504 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:16.504 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.504 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.504 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:16.504 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.253966+0000", 2024-09-21T22:07:16.504 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:16.504 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.504 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.504 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:16.504 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.253967+0000", 2024-09-21T22:07:16.504 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.423e-06 2024-09-21T22:07:16.504 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.504 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.504 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:16.505 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.255117+0000", 2024-09-21T22:07:16.505 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011496709999999999 2024-09-21T22:07:16.505 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.505 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.505 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:16.505 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.255167+0000", 2024-09-21T22:07:16.505 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.0315000000000002e-05 2024-09-21T22:07:16.505 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.505 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.505 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:16.505 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.255218+0000", 2024-09-21T22:07:16.505 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.0390999999999999e-05 2024-09-21T22:07:16.505 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.505 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.506 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:16.506 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.255391+0000", 2024-09-21T22:07:16.506 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00017332799999999999 2024-09-21T22:07:16.506 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.506 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.506 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:16.506 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.255405+0000", 2024-09-21T22:07:16.506 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3723999999999999e-05 2024-09-21T22:07:16.506 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.506 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.506 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:16.506 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.295066+0000", 2024-09-21T22:07:16.506 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.039661755 2024-09-21T22:07:16.506 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:16.507 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:16.507 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:16.507 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.507 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.507 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.3 55/54 [PushOp(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head, version: 52'3, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head@52'3, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T22:07:16.507 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.299195+0000", 2024-09-21T22:07:16.507 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 4.1449223279999998, 2024-09-21T22:07:16.507 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.040955077999999999, 2024-09-21T22:07:16.507 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:16.507 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:16.507 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:16.507 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:16.507 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.507 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:16.508 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.299195+0000", 2024-09-21T22:07:16.508 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:16.508 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.508 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.508 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:16.508 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.299195+0000", 2024-09-21T22:07:16.508 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:16.508 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.508 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.508 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:16.508 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.299196+0000", 2024-09-21T22:07:16.508 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.604e-06 2024-09-21T22:07:16.508 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.508 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.509 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:16.509 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.300356+0000", 2024-09-21T22:07:16.509 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011602699999999999 2024-09-21T22:07:16.509 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.509 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.509 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:16.509 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.300406+0000", 2024-09-21T22:07:16.509 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.9144999999999998e-05 2024-09-21T22:07:16.509 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.509 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.509 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:16.509 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.300435+0000", 2024-09-21T22:07:16.509 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9886000000000002e-05 2024-09-21T22:07:16.509 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.510 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.510 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:16.510 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.300590+0000", 2024-09-21T22:07:16.510 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00015431200000000001 2024-09-21T22:07:16.510 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.510 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.510 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:16.510 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.300604+0000", 2024-09-21T22:07:16.510 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4559000000000001e-05 2024-09-21T22:07:16.510 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.510 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.510 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:16.510 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.340150+0000", 2024-09-21T22:07:16.510 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.039545301999999997 2024-09-21T22:07:16.510 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:16.511 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:16.511 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:16.511 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.511 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.511 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.3 55/54 [PushOp(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head, version: 52'3, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head@52'3, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:07:16.511 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.343754+0000", 2024-09-21T22:07:16.511 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 4.1003627119999999, 2024-09-21T22:07:16.511 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.063560607000000005, 2024-09-21T22:07:16.511 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:16.511 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:16.511 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:16.511 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:16.511 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.511 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:16.511 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.343754+0000", 2024-09-21T22:07:16.512 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:16.512 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.512 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.512 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:16.512 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.343754+0000", 2024-09-21T22:07:16.512 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:16.512 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.512 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.512 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:16.512 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.343756+0000", 2024-09-21T22:07:16.512 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7069999999999999e-06 2024-09-21T22:07:16.512 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.512 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.512 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:16.513 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.344864+0000", 2024-09-21T22:07:16.513 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011085159999999999 2024-09-21T22:07:16.513 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.513 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.513 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:16.513 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.344914+0000", 2024-09-21T22:07:16.513 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.9462000000000002e-05 2024-09-21T22:07:16.513 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.513 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.513 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:16.513 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.344961+0000", 2024-09-21T22:07:16.513 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.7633999999999998e-05 2024-09-21T22:07:16.513 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.513 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.514 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:16.514 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.345171+0000", 2024-09-21T22:07:16.514 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000209652 2024-09-21T22:07:16.514 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.514 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.514 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:16.514 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.345193+0000", 2024-09-21T22:07:16.514 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1985999999999999e-05 2024-09-21T22:07:16.514 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.514 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.514 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:16.514 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.407315+0000", 2024-09-21T22:07:16.514 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.06212165 2024-09-21T22:07:16.514 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:16.515 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:16.515 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:16.515 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.515 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.515 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.3 55/54 [PushOp(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head, version: 52'3, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head@52'3, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:07:16.515 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.410551+0000", 2024-09-21T22:07:16.515 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 4.0335655810000004, 2024-09-21T22:07:16.515 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.054517522999999998, 2024-09-21T22:07:16.515 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:16.515 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:16.515 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:16.515 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:16.515 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.515 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:16.516 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.410551+0000", 2024-09-21T22:07:16.516 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:16.516 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.516 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.516 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:16.516 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.410551+0000", 2024-09-21T22:07:16.516 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:16.516 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.516 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.516 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:16.516 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.410553+0000", 2024-09-21T22:07:16.516 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5200000000000001e-06 2024-09-21T22:07:16.516 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.516 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.516 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:16.517 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.411747+0000", 2024-09-21T22:07:16.517 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011946089999999999 2024-09-21T22:07:16.517 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.517 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.517 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:16.517 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.411796+0000", 2024-09-21T22:07:16.517 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8717999999999997e-05 2024-09-21T22:07:16.517 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.517 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.517 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:16.517 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.411844+0000", 2024-09-21T22:07:16.517 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8291999999999998e-05 2024-09-21T22:07:16.517 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.517 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.518 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:16.518 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.412066+0000", 2024-09-21T22:07:16.518 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00022115899999999999 2024-09-21T22:07:16.518 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.518 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.518 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:16.518 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.412081+0000", 2024-09-21T22:07:16.518 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5654e-05 2024-09-21T22:07:16.518 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.518 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.518 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:16.518 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.465069+0000", 2024-09-21T22:07:16.518 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.052987570999999997 2024-09-21T22:07:16.518 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:16.519 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:16.519 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:16.519 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.519 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.519 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.3 55/54 [PushOp(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head, version: 52'3, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head@52'3, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 4194304, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:07:16.519 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.468339+0000", 2024-09-21T22:07:16.519 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 3.9757779170000003, 2024-09-21T22:07:16.519 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.064010024999999998, 2024-09-21T22:07:16.519 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:16.519 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:16.519 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:16.519 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:16.519 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.519 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:16.520 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.468339+0000", 2024-09-21T22:07:16.520 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:16.520 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.520 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.520 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:16.520 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.468339+0000", 2024-09-21T22:07:16.520 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:16.520 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.520 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.520 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:16.520 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.468340+0000", 2024-09-21T22:07:16.520 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4330000000000001e-06 2024-09-21T22:07:16.520 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.520 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.521 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:16.521 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.469484+0000", 2024-09-21T22:07:16.521 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001143535 2024-09-21T22:07:16.521 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.521 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.521 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:16.521 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.469533+0000", 2024-09-21T22:07:16.521 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8980999999999997e-05 2024-09-21T22:07:16.521 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.521 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.521 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:16.521 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.469581+0000", 2024-09-21T22:07:16.521 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8551999999999999e-05 2024-09-21T22:07:16.521 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.522 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.522 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:16.522 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.469804+0000", 2024-09-21T22:07:16.522 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00022287699999999999 2024-09-21T22:07:16.522 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.522 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.522 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:16.522 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.469829+0000", 2024-09-21T22:07:16.522 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4766e-05 2024-09-21T22:07:16.522 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.522 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.522 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:16.522 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.532349+0000", 2024-09-21T22:07:16.523 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.062519880999999999 2024-09-21T22:07:16.523 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:16.523 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:16.523 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:16.523 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.523 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.523 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4735.0:62 2.4 2:2d455e74:ns::rbd_header.127f79bc213d:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T22:07:16.523 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:15.126097+0000", 2024-09-21T22:07:16.523 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 1.318019668, 2024-09-21T22:07:16.523 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001198934, 2024-09-21T22:07:16.523 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:16.523 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:16.523 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:16.523 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-21T22:07:16.524 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4735", 2024-09-21T22:07:16.524 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.157:0/4210461984", 2024-09-21T22:07:16.524 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 62 2024-09-21T22:07:16.524 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.524 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:16.524 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.524 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:16.524 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:15.126097+0000", 2024-09-21T22:07:16.524 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:16.524 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.524 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.524 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:16.524 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:15.126097+0000", 2024-09-21T22:07:16.524 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:16.524 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.525 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.525 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:16.525 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:15.126099+0000", 2024-09-21T22:07:16.525 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4169999999999999e-06 2024-09-21T22:07:16.525 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.525 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.525 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:16.525 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:15.126104+0000", 2024-09-21T22:07:16.525 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.4770000000000002e-06 2024-09-21T22:07:16.525 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.525 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.525 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:16.525 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:15.126117+0000", 2024-09-21T22:07:16.525 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2585000000000001e-05 2024-09-21T22:07:16.526 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.526 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.526 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:16.526 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:15.126123+0000", 2024-09-21T22:07:16.526 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.8599999999999998e-06 2024-09-21T22:07:16.526 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.526 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.526 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:16.526 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:15.126173+0000", 2024-09-21T22:07:16.526 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.0005e-05 2024-09-21T22:07:16.526 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.526 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.526 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:16.526 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:15.126238+0000", 2024-09-21T22:07:16.527 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.5388000000000005e-05 2024-09-21T22:07:16.527 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.527 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.527 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:16.527 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:15.126354+0000", 2024-09-21T22:07:16.527 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000115633 2024-09-21T22:07:16.527 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.527 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.527 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-21T22:07:16.527 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:15.126864+0000", 2024-09-21T22:07:16.527 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00051046200000000005 2024-09-21T22:07:16.527 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.527 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.527 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:16.528 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:15.127242+0000", 2024-09-21T22:07:16.528 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00037800799999999999 2024-09-21T22:07:16.528 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.528 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.528 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:16.528 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:15.127271+0000", 2024-09-21T22:07:16.528 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9382000000000001e-05 2024-09-21T22:07:16.528 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.528 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.528 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:16.528 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:15.127296+0000", 2024-09-21T22:07:16.528 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4717000000000001e-05 2024-09-21T22:07:16.528 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:16.528 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:16.528 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:16.529 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.529 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.529 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4735.0:64 2.4 2:2d455e74:ns::rbd_header.127f79bc213d:head [watch unwatch cookie 140037073179952] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-09-21T22:07:16.529 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:15.127776+0000", 2024-09-21T22:07:16.529 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 1.316340412, 2024-09-21T22:07:16.529 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001131738, 2024-09-21T22:07:16.529 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:16.529 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:16.529 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:16.529 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-21T22:07:16.529 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4735", 2024-09-21T22:07:16.529 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.157:0/4210461984", 2024-09-21T22:07:16.529 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 64 2024-09-21T22:07:16.530 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.530 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:16.530 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.530 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:16.530 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:15.127776+0000", 2024-09-21T22:07:16.530 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:16.530 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.530 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.530 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:16.530 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:15.127776+0000", 2024-09-21T22:07:16.530 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:16.530 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.530 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.530 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:16.531 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:15.127777+0000", 2024-09-21T22:07:16.531 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.8100000000000003e-07 2024-09-21T22:07:16.531 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.531 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.531 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:16.531 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:15.127780+0000", 2024-09-21T22:07:16.531 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.1240000000000001e-06 2024-09-21T22:07:16.531 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.531 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.531 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:16.531 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:15.127793+0000", 2024-09-21T22:07:16.531 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2407e-05 2024-09-21T22:07:16.531 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.531 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.532 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:16.532 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:15.127799+0000", 2024-09-21T22:07:16.532 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.9939999999999997e-06 2024-09-21T22:07:16.532 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.532 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.532 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:16.532 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:15.127930+0000", 2024-09-21T22:07:16.532 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013156299999999999 2024-09-21T22:07:16.532 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.532 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.532 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:16.532 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:15.127971+0000", 2024-09-21T22:07:16.532 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.0485999999999998e-05 2024-09-21T22:07:16.532 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.533 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.533 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:16.533 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:15.128060+0000", 2024-09-21T22:07:16.533 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.8982999999999998e-05 2024-09-21T22:07:16.533 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.533 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.533 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-21T22:07:16.533 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:15.128494+0000", 2024-09-21T22:07:16.533 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00043474099999999999 2024-09-21T22:07:16.533 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.533 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.533 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:16.533 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:15.128827+0000", 2024-09-21T22:07:16.533 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00033299600000000001 2024-09-21T22:07:16.534 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.534 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.534 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:16.534 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:15.128857+0000", 2024-09-21T22:07:16.534 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9890999999999999e-05 2024-09-21T22:07:16.534 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.534 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.534 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:16.534 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:15.128908+0000", 2024-09-21T22:07:16.534 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.0971999999999999e-05 2024-09-21T22:07:16.534 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:16.534 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:16.534 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:16.534 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.535 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.535 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4735.0:65 2.4 2:2d455e74:ns::rbd_header.127f79bc213d:head [notify-ack] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-09-21T22:07:16.535 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:15.128110+0000", 2024-09-21T22:07:16.535 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 1.3160070150000001, 2024-09-21T22:07:16.535 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00090383200000000003, 2024-09-21T22:07:16.535 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:16.535 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:16.535 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:16.535 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-21T22:07:16.535 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4735", 2024-09-21T22:07:16.535 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.157:0/4210461984", 2024-09-21T22:07:16.535 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 65 2024-09-21T22:07:16.535 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.536 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:16.536 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.536 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:16.536 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:15.128110+0000", 2024-09-21T22:07:16.536 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:16.536 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.536 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.536 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:16.536 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:15.128110+0000", 2024-09-21T22:07:16.536 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:16.536 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.536 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.536 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:16.536 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:15.128110+0000", 2024-09-21T22:07:16.537 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.9500000000000002e-07 2024-09-21T22:07:16.537 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.537 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.537 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:16.537 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:15.128114+0000", 2024-09-21T22:07:16.537 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.45e-06 2024-09-21T22:07:16.537 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.537 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.537 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:16.537 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:15.128123+0000", 2024-09-21T22:07:16.537 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.6410000000000008e-06 2024-09-21T22:07:16.537 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.537 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.537 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:16.537 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:15.128128+0000", 2024-09-21T22:07:16.538 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.9960000000000001e-06 2024-09-21T22:07:16.538 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.538 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.538 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:16.538 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:15.128235+0000", 2024-09-21T22:07:16.538 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000107938 2024-09-21T22:07:16.538 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.538 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.538 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for rw locks", 2024-09-21T22:07:16.538 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:15.128325+0000", 2024-09-21T22:07:16.538 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.9702000000000003e-05 2024-09-21T22:07:16.538 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.538 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.538 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:16.539 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:15.128934+0000", 2024-09-21T22:07:16.539 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00060853399999999998 2024-09-21T22:07:16.539 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.539 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.539 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:16.539 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:15.128970+0000", 2024-09-21T22:07:16.539 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.6449999999999998e-05 2024-09-21T22:07:16.539 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:16.539 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:16.539 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:16.539 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:15.129014+0000", 2024-09-21T22:07:16.539 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.3526000000000003e-05 2024-09-21T22:07:16.539 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:16.539 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:16.539 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:16.540 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:16.540 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:16.540 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:07:16.541 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T22:07:16.541 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:16.539+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:16.642 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:16.639+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:16.715 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:07:16.715 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:07:16.715 INFO:teuthology.orchestra.run.smithi039.stdout: "num_ops": 0 2024-09-21T22:07:16.715 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:07:16.730 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T22:07:16.772 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:16.771+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:16.844 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:16.843+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:16.895 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:07:16.896 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:07:16.896 INFO:teuthology.orchestra.run.smithi039.stdout: "complaint_time": 30, 2024-09-21T22:07:16.896 INFO:teuthology.orchestra.run.smithi039.stdout: "num_blocked_ops": 0 2024-09-21T22:07:16.896 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:07:16.910 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T22:07:16.952 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:16.951+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:17.045 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:17.043+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:17.080 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:07:17.080 INFO:teuthology.orchestra.run.smithi039.stdout: "size": 20, 2024-09-21T22:07:17.080 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 600, 2024-09-21T22:07:17.080 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [ 2024-09-21T22:07:17.081 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.081 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4224.0:1 2.3 e17/16 2:c4c92e5a:::rbd_trash:head v 17'1, mlcod=0'0)", 2024-09-21T22:07:17.081 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:05.107810+0000", 2024-09-21T22:07:17.081 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 71.970539771999995, 2024-09-21T22:07:17.081 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0017679200000000001, 2024-09-21T22:07:17.081 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:17.081 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:17.081 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:17.081 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:17.081 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.081 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:17.081 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.107810+0000", 2024-09-21T22:07:17.081 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:17.081 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:17.082 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.082 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:17.082 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.107810+0000", 2024-09-21T22:07:17.082 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:17.082 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:17.082 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.082 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:17.082 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.107811+0000", 2024-09-21T22:07:17.082 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.6700000000000003e-07 2024-09-21T22:07:17.082 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:17.082 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.082 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:17.082 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.107822+0000", 2024-09-21T22:07:17.083 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0443e-05 2024-09-21T22:07:17.083 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:17.083 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.083 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:17.083 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.107853+0000", 2024-09-21T22:07:17.083 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.1046000000000001e-05 2024-09-21T22:07:17.083 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:17.083 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.083 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:17.083 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.107876+0000", 2024-09-21T22:07:17.083 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3547e-05 2024-09-21T22:07:17.083 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:17.083 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.084 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:17.084 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.108002+0000", 2024-09-21T22:07:17.084 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000125599 2024-09-21T22:07:17.084 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:17.084 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.084 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:17.084 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.108069+0000", 2024-09-21T22:07:17.084 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.7162999999999994e-05 2024-09-21T22:07:17.084 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:17.085 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.085 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:17.085 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.109385+0000", 2024-09-21T22:07:17.085 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0013162899999999999 2024-09-21T22:07:17.085 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:17.085 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.085 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:17.085 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:05.109578+0000", 2024-09-21T22:07:17.085 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000193065 2024-09-21T22:07:17.085 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:17.085 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:17.085 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:17.085 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:17.086 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.086 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(1.0 50/49 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 15'116, data_included: [0~262688], data_size: 262688, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@15'116, size: 262688, copy_subset: [(0, 262688)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 262688, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T22:07:17.086 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:06.036813+0000", 2024-09-21T22:07:17.086 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 11.041537193, 2024-09-21T22:07:17.086 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0022591270000000001, 2024-09-21T22:07:17.086 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:17.086 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:17.086 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:17.086 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:17.087 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.087 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:17.087 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:06.036813+0000", 2024-09-21T22:07:17.087 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:17.087 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:17.087 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.087 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:17.087 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:06.036813+0000", 2024-09-21T22:07:17.087 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:17.087 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:17.087 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.087 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:17.087 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:06.036814+0000", 2024-09-21T22:07:17.087 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.0400000000000005e-07 2024-09-21T22:07:17.088 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:17.088 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.088 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:17.088 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:06.037279+0000", 2024-09-21T22:07:17.088 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00046499600000000002 2024-09-21T22:07:17.088 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:17.088 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.088 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:17.088 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:06.037307+0000", 2024-09-21T22:07:17.089 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8359000000000001e-05 2024-09-21T22:07:17.089 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:17.089 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.089 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:17.089 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:06.037332+0000", 2024-09-21T22:07:17.089 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5154999999999999e-05 2024-09-21T22:07:17.089 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:17.089 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.089 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:17.089 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:06.037419+0000", 2024-09-21T22:07:17.089 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.6310000000000005e-05 2024-09-21T22:07:17.089 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:17.089 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.089 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:17.090 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:06.037430+0000", 2024-09-21T22:07:17.090 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0832000000000001e-05 2024-09-21T22:07:17.090 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:17.090 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.090 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:17.090 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:06.039072+0000", 2024-09-21T22:07:17.090 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001642571 2024-09-21T22:07:17.090 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:17.090 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:17.090 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:17.090 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:17.091 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.091 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(1.0 50/49 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 15'117, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@15'117, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T22:07:17.091 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:06.041121+0000", 2024-09-21T22:07:17.091 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 11.037229612000001, 2024-09-21T22:07:17.091 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0022693829999999998, 2024-09-21T22:07:17.091 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:17.091 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:17.091 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:17.091 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:17.091 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.091 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:17.091 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:06.041121+0000", 2024-09-21T22:07:17.092 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:17.092 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:17.092 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.092 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:17.092 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:06.041121+0000", 2024-09-21T22:07:17.092 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:17.092 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:17.092 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.092 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:17.092 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:06.041122+0000", 2024-09-21T22:07:17.092 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.3500000000000005e-07 2024-09-21T22:07:17.093 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:17.093 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.093 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:17.093 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:06.041449+0000", 2024-09-21T22:07:17.093 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00032708299999999999 2024-09-21T22:07:17.093 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:17.093 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.093 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:17.093 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:06.041475+0000", 2024-09-21T22:07:17.093 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.6245e-05 2024-09-21T22:07:17.093 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:17.093 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.094 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:17.094 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:06.041498+0000", 2024-09-21T22:07:17.094 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.2802e-05 2024-09-21T22:07:17.094 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:17.094 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.094 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:17.094 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:06.041579+0000", 2024-09-21T22:07:17.094 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.1203000000000002e-05 2024-09-21T22:07:17.094 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:17.094 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.095 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:17.095 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:06.041587+0000", 2024-09-21T22:07:17.095 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.9230000000000002e-06 2024-09-21T22:07:17.095 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:17.095 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.095 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:17.095 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:06.043390+0000", 2024-09-21T22:07:17.095 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0018031919999999999 2024-09-21T22:07:17.095 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:17.095 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:17.096 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:17.096 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:17.096 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.096 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPull(2.3 e55/54 cost 1048576)", 2024-09-21T22:07:17.096 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.072555+0000", 2024-09-21T22:07:17.096 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 5.0057956380000004, 2024-09-21T22:07:17.096 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000253047, 2024-09-21T22:07:17.096 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:17.096 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:17.096 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-21T22:07:17.096 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:17.096 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.096 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:17.096 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.072555+0000", 2024-09-21T22:07:17.097 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:17.097 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:17.097 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.097 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:17.097 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.072555+0000", 2024-09-21T22:07:17.097 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:17.097 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:17.097 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.097 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:17.097 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.072556+0000", 2024-09-21T22:07:17.097 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.358e-06 2024-09-21T22:07:17.097 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:17.097 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.097 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:17.098 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.072573+0000", 2024-09-21T22:07:17.098 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7000999999999999e-05 2024-09-21T22:07:17.098 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:17.098 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.098 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:17.098 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.072595+0000", 2024-09-21T22:07:17.098 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1704000000000001e-05 2024-09-21T22:07:17.098 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:17.098 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.098 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:17.098 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.072607+0000", 2024-09-21T22:07:17.098 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2238e-05 2024-09-21T22:07:17.098 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:17.099 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.099 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:17.099 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.072659+0000", 2024-09-21T22:07:17.099 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.2206000000000001e-05 2024-09-21T22:07:17.099 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:17.099 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.099 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:17.099 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.072808+0000", 2024-09-21T22:07:17.099 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00014854000000000001 2024-09-21T22:07:17.099 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:17.099 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:17.100 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:17.100 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:17.100 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.100 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPull(2.3 e55/54 cost 1048576)", 2024-09-21T22:07:17.100 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.081116+0000", 2024-09-21T22:07:17.100 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 4.997234186, 2024-09-21T22:07:17.100 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0030214159999999999, 2024-09-21T22:07:17.100 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:17.100 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:17.100 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-21T22:07:17.100 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:17.100 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.100 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:17.101 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.081116+0000", 2024-09-21T22:07:17.101 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:17.101 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:17.101 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.101 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:17.101 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.081116+0000", 2024-09-21T22:07:17.101 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:17.101 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:17.101 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.101 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:17.101 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.081118+0000", 2024-09-21T22:07:17.101 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.545e-06 2024-09-21T22:07:17.101 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:17.101 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.102 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:17.102 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.081133+0000", 2024-09-21T22:07:17.102 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5642e-05 2024-09-21T22:07:17.102 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:17.102 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.102 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:17.102 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.081155+0000", 2024-09-21T22:07:17.102 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1858e-05 2024-09-21T22:07:17.102 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:17.102 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.102 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:17.102 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.081171+0000", 2024-09-21T22:07:17.102 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5804e-05 2024-09-21T22:07:17.102 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:17.103 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.103 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:17.103 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.081291+0000", 2024-09-21T22:07:17.103 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000120553 2024-09-21T22:07:17.103 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:17.103 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.103 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:17.103 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.084137+0000", 2024-09-21T22:07:17.103 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.002846014 2024-09-21T22:07:17.103 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:17.103 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:17.103 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:17.103 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:17.103 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.104 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPull(2.3 e55/54 cost 1048576)", 2024-09-21T22:07:17.104 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.136357+0000", 2024-09-21T22:07:17.104 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 4.9419931200000002, 2024-09-21T22:07:17.104 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.002843904, 2024-09-21T22:07:17.104 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:17.104 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:17.104 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-21T22:07:17.104 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:17.104 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.104 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:17.104 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.136357+0000", 2024-09-21T22:07:17.104 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:17.104 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:17.104 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.105 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:17.105 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.136357+0000", 2024-09-21T22:07:17.105 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:17.105 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:17.105 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.105 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:17.105 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.136359+0000", 2024-09-21T22:07:17.105 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.0669999999999999e-06 2024-09-21T22:07:17.105 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:17.105 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.105 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:17.105 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.136381+0000", 2024-09-21T22:07:17.105 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.2104999999999999e-05 2024-09-21T22:07:17.105 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:17.105 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.106 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:17.106 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.136412+0000", 2024-09-21T22:07:17.106 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.0602999999999998e-05 2024-09-21T22:07:17.106 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:17.106 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.106 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:17.106 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.136427+0000", 2024-09-21T22:07:17.106 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4832e-05 2024-09-21T22:07:17.106 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:17.106 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.106 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:17.106 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.136569+0000", 2024-09-21T22:07:17.106 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000142702 2024-09-21T22:07:17.106 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:17.107 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.107 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:17.107 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.139201+0000", 2024-09-21T22:07:17.107 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.002631595 2024-09-21T22:07:17.107 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:17.107 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:17.107 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:17.107 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:17.107 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.107 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPull(2.3 e55/54 cost 1048576)", 2024-09-21T22:07:17.107 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.196896+0000", 2024-09-21T22:07:17.107 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 4.8814539650000004, 2024-09-21T22:07:17.107 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.002057279, 2024-09-21T22:07:17.107 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:17.108 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:17.108 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-21T22:07:17.109 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:17.109 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.109 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:17.109 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.196896+0000", 2024-09-21T22:07:17.109 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:17.110 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:17.110 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.110 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:17.110 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.196896+0000", 2024-09-21T22:07:17.110 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:17.110 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:17.110 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.110 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:17.110 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.196898+0000", 2024-09-21T22:07:17.110 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.0729999999999999e-06 2024-09-21T22:07:17.110 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:17.110 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.110 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:17.110 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.196923+0000", 2024-09-21T22:07:17.110 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4922e-05 2024-09-21T22:07:17.111 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:17.111 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.111 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:17.111 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.196956+0000", 2024-09-21T22:07:17.111 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.3212e-05 2024-09-21T22:07:17.111 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:17.111 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.111 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:17.111 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.196978+0000", 2024-09-21T22:07:17.111 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1746000000000002e-05 2024-09-21T22:07:17.111 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:17.111 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.111 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:17.111 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.197088+0000", 2024-09-21T22:07:17.112 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000110222 2024-09-21T22:07:17.112 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:17.112 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.112 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:17.112 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.198954+0000", 2024-09-21T22:07:17.112 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001865104 2024-09-21T22:07:17.112 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:17.112 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:17.112 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:17.112 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:17.112 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.112 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPull(2.3 e55/54 cost 1048576)", 2024-09-21T22:07:17.112 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.251826+0000", 2024-09-21T22:07:17.112 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 4.8265244689999998, 2024-09-21T22:07:17.113 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001903825, 2024-09-21T22:07:17.113 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:17.113 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:17.113 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-21T22:07:17.113 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:17.113 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.113 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:17.113 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.251826+0000", 2024-09-21T22:07:17.113 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:17.113 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:17.113 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.113 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:17.113 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.251826+0000", 2024-09-21T22:07:17.113 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:17.114 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:17.114 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.114 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:17.114 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.251828+0000", 2024-09-21T22:07:17.114 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9300000000000002e-06 2024-09-21T22:07:17.114 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:17.114 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.114 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:17.114 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.251847+0000", 2024-09-21T22:07:17.114 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9602999999999998e-05 2024-09-21T22:07:17.114 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:17.114 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.114 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:17.114 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.251877+0000", 2024-09-21T22:07:17.115 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.0179e-05 2024-09-21T22:07:17.115 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:17.115 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.115 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:17.115 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.251897+0000", 2024-09-21T22:07:17.115 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9465000000000001e-05 2024-09-21T22:07:17.115 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:17.115 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.115 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:17.115 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.251985+0000", 2024-09-21T22:07:17.115 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.8126000000000003e-05 2024-09-21T22:07:17.115 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:17.115 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.115 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:17.116 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.253730+0000", 2024-09-21T22:07:17.116 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0017445220000000001 2024-09-21T22:07:17.116 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:17.116 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:17.116 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:17.116 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:17.116 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.116 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPull(2.3 e55/54 cost 1048576)", 2024-09-21T22:07:17.116 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.296534+0000", 2024-09-21T22:07:17.116 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 4.7818157890000004, 2024-09-21T22:07:17.116 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0023727290000000001, 2024-09-21T22:07:17.116 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:17.116 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:17.116 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-21T22:07:17.116 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:17.117 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.117 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:17.117 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.296534+0000", 2024-09-21T22:07:17.117 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:17.117 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:17.117 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.117 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:17.117 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.296534+0000", 2024-09-21T22:07:17.117 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:17.117 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:17.117 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.117 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:17.117 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.296537+0000", 2024-09-21T22:07:17.118 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1299999999999999e-06 2024-09-21T22:07:17.118 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:17.118 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.118 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:17.118 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.296567+0000", 2024-09-21T22:07:17.118 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.0812000000000002e-05 2024-09-21T22:07:17.118 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:17.118 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.118 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:17.118 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.296598+0000", 2024-09-21T22:07:17.118 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.0803000000000003e-05 2024-09-21T22:07:17.118 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:17.118 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.118 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:17.118 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.296625+0000", 2024-09-21T22:07:17.119 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.6452e-05 2024-09-21T22:07:17.119 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:17.119 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.119 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:17.119 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.296717+0000", 2024-09-21T22:07:17.119 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.1953000000000005e-05 2024-09-21T22:07:17.119 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:17.119 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.119 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:17.119 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.298907+0000", 2024-09-21T22:07:17.119 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0021905789999999998 2024-09-21T22:07:17.119 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:17.119 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:17.120 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:17.120 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:17.120 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.120 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPull(2.3 e55/54 cost 1048576)", 2024-09-21T22:07:17.120 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.341173+0000", 2024-09-21T22:07:17.120 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 4.7371767919999996, 2024-09-21T22:07:17.120 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.002347933, 2024-09-21T22:07:17.120 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:17.120 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:17.120 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-21T22:07:17.121 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:17.121 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.121 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:17.121 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.341173+0000", 2024-09-21T22:07:17.121 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:17.121 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:17.121 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.121 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:17.121 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.341173+0000", 2024-09-21T22:07:17.121 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:17.121 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:17.121 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.121 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:17.121 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.341175+0000", 2024-09-21T22:07:17.122 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4479999999999999e-06 2024-09-21T22:07:17.122 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:17.122 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.122 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:17.122 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.341187+0000", 2024-09-21T22:07:17.122 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.221e-05 2024-09-21T22:07:17.122 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:17.122 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.122 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:17.122 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.341203+0000", 2024-09-21T22:07:17.123 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6099e-05 2024-09-21T22:07:17.123 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:17.123 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.123 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:17.123 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.341214+0000", 2024-09-21T22:07:17.123 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0461e-05 2024-09-21T22:07:17.123 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:17.123 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.123 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:17.123 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.341307+0000", 2024-09-21T22:07:17.123 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.3772999999999998e-05 2024-09-21T22:07:17.123 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:17.123 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.123 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:17.124 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.343521+0000", 2024-09-21T22:07:17.124 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.002213942 2024-09-21T22:07:17.124 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:17.124 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:17.124 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:17.124 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:17.124 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.124 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPull(2.3 e55/54 cost 1048576)", 2024-09-21T22:07:17.124 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.408344+0000", 2024-09-21T22:07:17.125 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 4.6700060060000004, 2024-09-21T22:07:17.125 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0019795059999999998, 2024-09-21T22:07:17.125 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:17.125 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:17.125 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-21T22:07:17.125 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:17.125 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.125 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:17.125 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.408344+0000", 2024-09-21T22:07:17.125 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:17.125 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:17.125 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.125 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:17.125 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.408344+0000", 2024-09-21T22:07:17.126 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:17.126 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:17.126 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.126 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:17.126 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.408347+0000", 2024-09-21T22:07:17.126 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.306e-06 2024-09-21T22:07:17.126 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:17.126 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.126 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:17.126 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.408367+0000", 2024-09-21T22:07:17.127 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.0545999999999999e-05 2024-09-21T22:07:17.127 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:17.127 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.127 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:17.127 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.408396+0000", 2024-09-21T22:07:17.127 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8642000000000001e-05 2024-09-21T22:07:17.127 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:17.127 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.127 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:17.127 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.408410+0000", 2024-09-21T22:07:17.127 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4185999999999999e-05 2024-09-21T22:07:17.127 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:17.127 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.128 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:17.128 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.408500+0000", 2024-09-21T22:07:17.128 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.0135000000000004e-05 2024-09-21T22:07:17.128 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:17.128 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.128 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:17.128 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.410324+0000", 2024-09-21T22:07:17.128 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0018236909999999999 2024-09-21T22:07:17.128 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:17.129 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:17.129 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:17.129 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:17.129 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.129 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPull(2.3 e55/54 cost 1048576)", 2024-09-21T22:07:17.129 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.466128+0000", 2024-09-21T22:07:17.129 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 4.6122225989999999, 2024-09-21T22:07:17.129 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0019720779999999999, 2024-09-21T22:07:17.129 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:17.130 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:17.130 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-21T22:07:17.130 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:17.130 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.130 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:17.130 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.466128+0000", 2024-09-21T22:07:17.130 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:17.130 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:17.130 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.130 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:17.131 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.466128+0000", 2024-09-21T22:07:17.131 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:17.131 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:17.131 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.131 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:17.131 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.466129+0000", 2024-09-21T22:07:17.131 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.792e-06 2024-09-21T22:07:17.131 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:17.131 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.132 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:17.132 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.466144+0000", 2024-09-21T22:07:17.132 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4558e-05 2024-09-21T22:07:17.132 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:17.132 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.132 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:17.132 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.466165+0000", 2024-09-21T22:07:17.132 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1211000000000001e-05 2024-09-21T22:07:17.132 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:17.132 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.133 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:17.133 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.466176+0000", 2024-09-21T22:07:17.133 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0844e-05 2024-09-21T22:07:17.133 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:17.133 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.133 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:17.133 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.466266+0000", 2024-09-21T22:07:17.133 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.9865999999999996e-05 2024-09-21T22:07:17.133 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:17.133 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:17.133 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:17.133 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.468100+0000", 2024-09-21T22:07:17.133 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001833807 2024-09-21T22:07:17.133 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:17.133 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:17.134 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:17.134 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:17.134 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:17.134 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:07:17.135 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T22:07:17.147 INFO:teuthology.orchestra.run.smithi039.stderr:nodeep-scrub is set 2024-09-21T22:07:17.154 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T22:07:17.182 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:17.180+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:17.246 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:17.243+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:17.270 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:07:17.270 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:07:17.270 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-09-21T22:07:17.270 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:07:17.285 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T22:07:17.291 INFO:teuthology.orchestra.run.smithi039.stdout:osd.0: {} 2024-09-21T22:07:17.291 INFO:teuthology.orchestra.run.smithi039.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T22:07:17.309 INFO:teuthology.orchestra.run.smithi039.stdout:osd.1: {} 2024-09-21T22:07:17.309 INFO:teuthology.orchestra.run.smithi039.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T22:07:17.326 INFO:teuthology.orchestra.run.smithi039.stdout:osd.2: {} 2024-09-21T22:07:17.326 INFO:teuthology.orchestra.run.smithi039.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T22:07:17.345 INFO:teuthology.orchestra.run.smithi039.stdout:osd.3: {} 2024-09-21T22:07:17.345 INFO:teuthology.orchestra.run.smithi039.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T22:07:17.346 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:17.343+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:17.366 INFO:teuthology.orchestra.run.smithi039.stdout:osd.4: {} 2024-09-21T22:07:17.366 INFO:teuthology.orchestra.run.smithi039.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T22:07:17.367 INFO:teuthology.orchestra.run.smithi039.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-21T22:07:17.370 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T22:07:17.370 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T22:07:17.448 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:17.444+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:17.469 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:17.469 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:17.484 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T22:07:17.549 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:17.547+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:17.649 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:17.644+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:17.655 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:17.655 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:17.670 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T22:07:17.755 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:17.752+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:17.829 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:17.829 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:17.844 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T22:07:17.891 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:17.888+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:17.953 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:17.951+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:18.020 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:18.021 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:18.036 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T22:07:18.054 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:18.051+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:18.155 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:18.152+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:18.156 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:18.157 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:18.171 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T22:07:18.258 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:18.255+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:18.308 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:18.308 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:18.323 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T22:07:18.364 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:18.363+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:18.458 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:18.455+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:18.490 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:18.491 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:18.506 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T22:07:18.596 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:18.595+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:18.659 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:18.656+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:18.677 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:18.677 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:18.692 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T22:07:18.784 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:18.783+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:18.861 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:18.856+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:18.866 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:18.867 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:18.886 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T22:07:18.972 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:18.971+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:19.056 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:19.056 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:19.062 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:19.056+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:19.072 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T22:07:19.162 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:19.159+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:19.162 INFO:tasks.workunit.client.0.smithi157.stderr:+ expect_false rbd-nbd 2024-09-21T22:07:19.163 INFO:tasks.workunit.client.0.smithi157.stderr:+ rbd-nbd 2024-09-21T22:07:19.175 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-21T22:07:19.247 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:19.248 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:19.263 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T22:07:19.264 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:19.260+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:19.318 INFO:tasks.workunit.client.0.smithi157.stderr:rbd-nbd: must specify command 2024-09-21T22:07:19.320 INFO:tasks.workunit.client.0.smithi157.stderr:+ return 0 2024-09-21T22:07:19.320 INFO:tasks.workunit.client.0.smithi157.stderr:+ expect_false rbd-nbd INVALIDCMD 2024-09-21T22:07:19.320 INFO:tasks.workunit.client.0.smithi157.stderr:+ rbd-nbd INVALIDCMD 2024-09-21T22:07:19.346 INFO:tasks.workunit.client.0.smithi157.stderr:rbd-nbd: unknown command: INVALIDCMD 2024-09-21T22:07:19.347 INFO:tasks.workunit.client.0.smithi157.stderr:+ return 0 2024-09-21T22:07:19.347 INFO:tasks.workunit.client.0.smithi157.stderr:++ id -u 2024-09-21T22:07:19.349 INFO:tasks.workunit.client.0.smithi157.stderr:+ '[' 1000 -ne 0 ']' 2024-09-21T22:07:19.349 INFO:tasks.workunit.client.0.smithi157.stderr:+ expect_false rbd device --device-type nbd map testrbdnbd21553 2024-09-21T22:07:19.349 INFO:tasks.workunit.client.0.smithi157.stderr:+ rbd device --device-type nbd map testrbdnbd21553 2024-09-21T22:07:19.363 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:19.363+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:19.371 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T22:07:19.382 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:19.382 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:19.399 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T22:07:19.460 INFO:tasks.workunit.client.0.smithi157.stderr:modprobe: ERROR: could not insert 'nbd': Operation not permitted 2024-09-21T22:07:19.465 INFO:tasks.workunit.client.0.smithi157.stderr:rbd-nbd: Netlink interface not supported. Using ioctl interface. 2024-09-21T22:07:19.465 INFO:tasks.workunit.client.0.smithi157.stderr:rbd-nbd: failed to find unused device 2024-09-21T22:07:19.472 INFO:tasks.workunit.client.0.smithi157.stderr:rbd: rbd-nbd failed with error: /usr/bin/rbd-nbd: exit status: 1 2024-09-21T22:07:19.476 INFO:tasks.workunit.client.0.smithi157.stderr:+ return 0 2024-09-21T22:07:19.476 INFO:tasks.workunit.client.0.smithi157.stderr:+ expect_false _sudo rbd device --device-type nbd map INVALIDIMAGE 2024-09-21T22:07:19.476 INFO:tasks.workunit.client.0.smithi157.stderr:+ _sudo rbd device --device-type nbd map INVALIDIMAGE 2024-09-21T22:07:19.476 INFO:tasks.workunit.client.0.smithi157.stderr:+ local cmd 2024-09-21T22:07:19.477 INFO:tasks.workunit.client.0.smithi157.stderr:++ id -u 2024-09-21T22:07:19.477 INFO:tasks.workunit.client.0.smithi157.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T22:07:19.477 INFO:tasks.workunit.client.0.smithi157.stderr:++ which rbd 2024-09-21T22:07:19.479 INFO:tasks.workunit.client.0.smithi157.stderr:+ cmd=/usr/bin/rbd 2024-09-21T22:07:19.479 INFO:tasks.workunit.client.0.smithi157.stderr:+ shift 2024-09-21T22:07:19.479 INFO:tasks.workunit.client.0.smithi157.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd map INVALIDIMAGE 2024-09-21T22:07:19.491 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:19.491+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:19.566 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:19.560+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:19.567 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:19.567 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:19.569 INFO:tasks.workunit.client.0.smithi157.stderr:rbd: rbd-nbd failed with error: /usr/bin/rbd-nbd: exit status: 1 2024-09-21T22:07:19.574 INFO:tasks.workunit.client.0.smithi157.stderr:+ return 0 2024-09-21T22:07:19.574 INFO:tasks.workunit.client.0.smithi157.stderr:+ expect_false _sudo rbd-nbd --device INVALIDDEV map testrbdnbd21553 2024-09-21T22:07:19.574 INFO:tasks.workunit.client.0.smithi157.stderr:+ _sudo rbd-nbd --device INVALIDDEV map testrbdnbd21553 2024-09-21T22:07:19.574 INFO:tasks.workunit.client.0.smithi157.stderr:+ local cmd 2024-09-21T22:07:19.574 INFO:tasks.workunit.client.0.smithi157.stderr:++ id -u 2024-09-21T22:07:19.575 INFO:tasks.workunit.client.0.smithi157.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T22:07:19.575 INFO:tasks.workunit.client.0.smithi157.stderr:++ which rbd-nbd 2024-09-21T22:07:19.576 INFO:tasks.workunit.client.0.smithi157.stderr:+ cmd=/usr/bin/rbd-nbd 2024-09-21T22:07:19.576 INFO:tasks.workunit.client.0.smithi157.stderr:+ shift 2024-09-21T22:07:19.576 INFO:tasks.workunit.client.0.smithi157.stderr:+ sudo -nE /usr/bin/rbd-nbd --device INVALIDDEV map testrbdnbd21553 2024-09-21T22:07:19.582 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T22:07:19.655 INFO:tasks.workunit.client.0.smithi157.stderr:rbd-nbd: invalid device path: INVALIDDEV (expected /dev/nbd{num}) 2024-09-21T22:07:19.666 INFO:tasks.workunit.client.0.smithi157.stderr:+ return 0 2024-09-21T22:07:19.666 INFO:tasks.workunit.client.0.smithi157.stderr:+ expect_false rbd device --device-type nbd --format INVALID list 2024-09-21T22:07:19.666 INFO:tasks.workunit.client.0.smithi157.stderr:+ rbd device --device-type nbd --format INVALID list 2024-09-21T22:07:19.666 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:19.664+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:19.691 INFO:tasks.workunit.client.0.smithi157.stderr:rbd: the argument for option '--format' is invalid 2024-09-21T22:07:19.692 INFO:tasks.workunit.client.0.smithi157.stderr:+ return 0 2024-09-21T22:07:19.693 INFO:tasks.workunit.client.0.smithi157.stderr:+ rbd device --device-type nbd --format json --pretty-format list 2024-09-21T22:07:19.720 INFO:teuthology.orchestra.run.smithi039.stdout:osd.0: {} 2024-09-21T22:07:19.720 INFO:teuthology.orchestra.run.smithi039.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T22:07:19.738 INFO:teuthology.orchestra.run.smithi039.stdout:osd.1: {} 2024-09-21T22:07:19.739 INFO:teuthology.orchestra.run.smithi039.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T22:07:19.742 INFO:tasks.workunit.client.0.smithi157.stdout:[] 2024-09-21T22:07:19.743 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:19.744 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:19.746 INFO:tasks.workunit.client.0.smithi157.stderr:+ rbd device --device-type nbd --format xml list 2024-09-21T22:07:19.757 INFO:teuthology.orchestra.run.smithi039.stdout:osd.2: {} 2024-09-21T22:07:19.757 INFO:teuthology.orchestra.run.smithi039.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T22:07:19.758 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T22:07:19.767 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:19.763+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:19.779 INFO:teuthology.orchestra.run.smithi039.stdout:osd.3: {} 2024-09-21T22:07:19.779 INFO:teuthology.orchestra.run.smithi039.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T22:07:19.796 INFO:tasks.workunit.client.0.smithi157.stderr:++ _sudo rbd device --device-type nbd map rbd/testrbdnbd21553 2024-09-21T22:07:19.797 INFO:tasks.workunit.client.0.smithi157.stderr:++ local cmd 2024-09-21T22:07:19.797 INFO:tasks.workunit.client.0.smithi157.stderr:+++ id -u 2024-09-21T22:07:19.797 INFO:tasks.workunit.client.0.smithi157.stderr:++ '[' 1000 -eq 0 ']' 2024-09-21T22:07:19.798 INFO:tasks.workunit.client.0.smithi157.stderr:+++ which rbd 2024-09-21T22:07:19.799 INFO:tasks.workunit.client.0.smithi157.stderr:++ cmd=/usr/bin/rbd 2024-09-21T22:07:19.799 INFO:tasks.workunit.client.0.smithi157.stderr:++ shift 2024-09-21T22:07:19.799 INFO:tasks.workunit.client.0.smithi157.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map rbd/testrbdnbd21553 2024-09-21T22:07:19.799 INFO:teuthology.orchestra.run.smithi039.stdout:osd.4: {} 2024-09-21T22:07:19.800 INFO:teuthology.orchestra.run.smithi039.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T22:07:19.800 INFO:teuthology.orchestra.run.smithi039.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-21T22:07:19.805 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T22:07:19.806 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T22:07:19.868 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:19.867+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:19.894 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:07:19.894 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:07:19.894 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-09-21T22:07:19.894 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:07:19.909 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T22:07:19.939 INFO:tasks.workunit.client.0.smithi157.stderr:+ DEV=/dev/nbd0 2024-09-21T22:07:19.939 INFO:tasks.workunit.client.0.smithi157.stderr:+ get_pid rbd 2024-09-21T22:07:19.939 INFO:tasks.workunit.client.0.smithi157.stderr:+ local pool=rbd 2024-09-21T22:07:19.939 INFO:tasks.workunit.client.0.smithi157.stderr:+ local ns= 2024-09-21T22:07:19.940 INFO:tasks.workunit.client.0.smithi157.stderr:++ rbd device --device-type nbd --format xml list 2024-09-21T22:07:19.940 INFO:tasks.workunit.client.0.smithi157.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd21553'\''][device='\''/dev/nbd0'\'']/id' 2024-09-21T22:07:19.968 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:19.967+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:20.027 INFO:tasks.workunit.client.0.smithi157.stderr:+ PID=21857 2024-09-21T22:07:20.027 INFO:tasks.workunit.client.0.smithi157.stderr:+ test -n 21857 2024-09-21T22:07:20.027 INFO:tasks.workunit.client.0.smithi157.stderr:+ ps -p 21857 -C rbd-nbd 2024-09-21T22:07:20.035 INFO:tasks.workunit.client.0.smithi157.stdout: PID TTY TIME CMD 2024-09-21T22:07:20.035 INFO:tasks.workunit.client.0.smithi157.stdout: 21857 ? 00:00:00 rbd-nbd 2024-09-21T22:07:20.036 INFO:tasks.workunit.client.0.smithi157.stderr:+ expect_false _sudo rbd-nbd --device /dev/nbd0 map rbd/testrbdnbd21553 2024-09-21T22:07:20.036 INFO:tasks.workunit.client.0.smithi157.stderr:+ _sudo rbd-nbd --device /dev/nbd0 map rbd/testrbdnbd21553 2024-09-21T22:07:20.036 INFO:tasks.workunit.client.0.smithi157.stderr:+ local cmd 2024-09-21T22:07:20.036 INFO:tasks.workunit.client.0.smithi157.stderr:++ id -u 2024-09-21T22:07:20.036 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:07:20.036 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:07:20.036 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-21T22:07:20.036 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-21T22:07:20.036 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:07:20.037 INFO:tasks.workunit.client.0.smithi157.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T22:07:20.037 INFO:tasks.workunit.client.0.smithi157.stderr:++ which rbd-nbd 2024-09-21T22:07:20.038 INFO:tasks.workunit.client.0.smithi157.stderr:+ cmd=/usr/bin/rbd-nbd 2024-09-21T22:07:20.039 INFO:tasks.workunit.client.0.smithi157.stderr:+ shift 2024-09-21T22:07:20.039 INFO:tasks.workunit.client.0.smithi157.stderr:+ sudo -nE /usr/bin/rbd-nbd --device /dev/nbd0 map rbd/testrbdnbd21553 2024-09-21T22:07:20.051 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T22:07:20.098 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:20.096+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:20.119 INFO:tasks.workunit.client.0.smithi157.stderr:rbd-nbd: netlink connect failed: Object busy 2024-09-21T22:07:20.128 INFO:tasks.workunit.client.0.smithi157.stderr:+ return 0 2024-09-21T22:07:20.128 INFO:tasks.workunit.client.0.smithi157.stderr:+ dev1=/dev/nbd0 2024-09-21T22:07:20.129 INFO:tasks.workunit.client.0.smithi157.stderr:+ unmap_device /dev/nbd0 21857 2024-09-21T22:07:20.129 INFO:tasks.workunit.client.0.smithi157.stderr:+ local args=/dev/nbd0 2024-09-21T22:07:20.129 INFO:tasks.workunit.client.0.smithi157.stderr:+ local pid=21857 2024-09-21T22:07:20.129 INFO:tasks.workunit.client.0.smithi157.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd0 2024-09-21T22:07:20.129 INFO:tasks.workunit.client.0.smithi157.stderr:+ local cmd 2024-09-21T22:07:20.129 INFO:tasks.workunit.client.0.smithi157.stderr:++ id -u 2024-09-21T22:07:20.129 INFO:tasks.workunit.client.0.smithi157.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T22:07:20.129 INFO:tasks.workunit.client.0.smithi157.stderr:++ which rbd 2024-09-21T22:07:20.130 INFO:tasks.workunit.client.0.smithi157.stderr:+ cmd=/usr/bin/rbd 2024-09-21T22:07:20.130 INFO:tasks.workunit.client.0.smithi157.stderr:+ shift 2024-09-21T22:07:20.130 INFO:tasks.workunit.client.0.smithi157.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd0 2024-09-21T22:07:20.152 INFO:teuthology.orchestra.run.smithi039.stderr:noscrub is unset 2024-09-21T22:07:20.170 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:20.167+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:20.183 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:07:20.183 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-21T22:07:20.183 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-21T22:07:20.183 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-21T22:07:20.183 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.183 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:07:20.183 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.036355+0000", 2024-09-21T22:07:20.183 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 78.143118275000006, 2024-09-21T22:07:20.183 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.062441934999999997, 2024-09-21T22:07:20.183 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:20.184 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:20.184 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:20.184 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:20.184 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:07:20.184 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:07:20.184 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1 2024-09-21T22:07:20.184 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.184 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:20.184 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.184 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:20.184 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036355+0000", 2024-09-21T22:07:20.184 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:20.184 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.185 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.185 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:20.185 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036355+0000", 2024-09-21T22:07:20.185 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:20.185 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.185 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.185 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:20.185 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036357+0000", 2024-09-21T22:07:20.185 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.446e-06 2024-09-21T22:07:20.185 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.185 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.185 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:20.185 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036373+0000", 2024-09-21T22:07:20.185 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5447e-05 2024-09-21T22:07:20.185 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.186 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.186 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:20.186 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036399+0000", 2024-09-21T22:07:20.186 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6423000000000001e-05 2024-09-21T22:07:20.186 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.186 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.186 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:20.186 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036420+0000", 2024-09-21T22:07:20.186 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1129999999999999e-05 2024-09-21T22:07:20.186 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.186 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.186 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:20.186 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036557+0000", 2024-09-21T22:07:20.187 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013686899999999999 2024-09-21T22:07:20.187 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.187 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.187 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-21T22:07:20.187 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036585+0000", 2024-09-21T22:07:20.187 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8362999999999999e-05 2024-09-21T22:07:20.187 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.187 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.187 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:20.187 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.095068+0000", 2024-09-21T22:07:20.187 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058482972000000001 2024-09-21T22:07:20.187 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.187 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.187 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:20.188 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.095516+0000", 2024-09-21T22:07:20.188 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00044802999999999998 2024-09-21T22:07:20.188 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.188 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.188 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:20.188 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.095869+0000", 2024-09-21T22:07:20.188 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000352104 2024-09-21T22:07:20.188 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.188 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.188 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:20.188 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.097055+0000", 2024-09-21T22:07:20.188 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011865599999999999 2024-09-21T22:07:20.188 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.188 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.189 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:20.189 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.098637+0000", 2024-09-21T22:07:20.189 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001581458 2024-09-21T22:07:20.189 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.189 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.189 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:20.189 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.098728+0000", 2024-09-21T22:07:20.189 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1043000000000002e-05 2024-09-21T22:07:20.189 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.189 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.189 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:20.189 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.098797+0000", 2024-09-21T22:07:20.189 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9090000000000004e-05 2024-09-21T22:07:20.189 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:20.190 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:20.190 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:20.190 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.190 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.190 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:35 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:07:20.190 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.161242+0000", 2024-09-21T22:07:20.190 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 78.018231362999998, 2024-09-21T22:07:20.190 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014477078000000001, 2024-09-21T22:07:20.190 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:20.190 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:20.190 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:20.190 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:20.190 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:07:20.190 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:07:20.191 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 35 2024-09-21T22:07:20.191 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.191 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:20.191 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.191 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:20.191 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161242+0000", 2024-09-21T22:07:20.192 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:20.192 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.192 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.192 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:20.192 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161242+0000", 2024-09-21T22:07:20.192 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:20.192 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.192 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.193 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:20.193 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161243+0000", 2024-09-21T22:07:20.193 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4729999999999999e-06 2024-09-21T22:07:20.193 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.193 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.193 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:20.193 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161385+0000", 2024-09-21T22:07:20.193 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014226899999999999 2024-09-21T22:07:20.193 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.193 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.193 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:20.193 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161417+0000", 2024-09-21T22:07:20.193 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1406999999999999e-05 2024-09-21T22:07:20.194 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.194 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.194 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:20.194 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161434+0000", 2024-09-21T22:07:20.194 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7623000000000001e-05 2024-09-21T22:07:20.194 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.194 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.194 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:20.194 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.169728+0000", 2024-09-21T22:07:20.194 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0082933090000000004 2024-09-21T22:07:20.194 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.194 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.194 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:20.195 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.169861+0000", 2024-09-21T22:07:20.195 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013367300000000001 2024-09-21T22:07:20.195 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.195 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.195 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:20.195 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.170186+0000", 2024-09-21T22:07:20.195 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00032466400000000001 2024-09-21T22:07:20.195 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.195 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.195 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:20.195 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.174759+0000", 2024-09-21T22:07:20.195 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0045727880000000004 2024-09-21T22:07:20.195 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.195 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.196 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:20.196 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.175497+0000", 2024-09-21T22:07:20.196 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00073803400000000004 2024-09-21T22:07:20.196 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.196 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.196 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:20.196 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.175607+0000", 2024-09-21T22:07:20.196 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010987199999999999 2024-09-21T22:07:20.196 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.196 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.196 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:20.196 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.175719+0000", 2024-09-21T22:07:20.196 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011196600000000001 2024-09-21T22:07:20.196 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:20.197 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:20.197 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:20.197 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.197 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.197 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:36 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:07:20.197 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.161483+0000", 2024-09-21T22:07:20.197 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 78.017989851999999, 2024-09-21T22:07:20.197 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016155732999999999, 2024-09-21T22:07:20.197 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:20.197 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:20.197 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:20.197 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:20.197 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:07:20.197 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:07:20.198 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 36 2024-09-21T22:07:20.198 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.198 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:20.198 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.198 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:20.198 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161483+0000", 2024-09-21T22:07:20.198 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:20.198 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.198 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.198 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:20.198 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161483+0000", 2024-09-21T22:07:20.198 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:20.198 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.199 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.199 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:20.199 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161485+0000", 2024-09-21T22:07:20.199 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5999999999999999e-06 2024-09-21T22:07:20.199 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.199 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.199 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:20.199 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161498+0000", 2024-09-21T22:07:20.199 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3166999999999999e-05 2024-09-21T22:07:20.199 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.199 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.199 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:20.199 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161526+0000", 2024-09-21T22:07:20.199 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7786000000000001e-05 2024-09-21T22:07:20.199 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.200 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.200 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:20.200 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161545+0000", 2024-09-21T22:07:20.200 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8899999999999999e-05 2024-09-21T22:07:20.200 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.200 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.200 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:20.200 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.172615+0000", 2024-09-21T22:07:20.200 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011070465 2024-09-21T22:07:20.200 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.200 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.200 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:20.200 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.172759+0000", 2024-09-21T22:07:20.200 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014416900000000001 2024-09-21T22:07:20.201 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.201 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.201 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:20.201 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.172967+0000", 2024-09-21T22:07:20.201 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020758500000000001 2024-09-21T22:07:20.201 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.201 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.201 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:20.201 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.174775+0000", 2024-09-21T22:07:20.201 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0018080990000000001 2024-09-21T22:07:20.201 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.201 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.201 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:20.201 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.177447+0000", 2024-09-21T22:07:20.201 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026717429999999999 2024-09-21T22:07:20.202 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.202 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.202 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:20.202 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.177525+0000", 2024-09-21T22:07:20.202 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7846e-05 2024-09-21T22:07:20.202 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.202 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.202 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:20.202 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.177639+0000", 2024-09-21T22:07:20.202 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000114373 2024-09-21T22:07:20.202 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:20.202 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:20.202 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:20.203 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.203 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.203 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:37 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:07:20.203 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.161577+0000", 2024-09-21T22:07:20.203 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 78.017896319000002, 2024-09-21T22:07:20.203 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016652991999999998, 2024-09-21T22:07:20.203 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:20.203 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:20.203 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:20.203 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:20.203 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:07:20.203 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:07:20.203 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 37 2024-09-21T22:07:20.203 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.204 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:20.204 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.204 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:20.204 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161577+0000", 2024-09-21T22:07:20.204 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:20.204 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.204 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.204 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:20.204 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161577+0000", 2024-09-21T22:07:20.204 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:20.204 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.204 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.204 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:20.204 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161578+0000", 2024-09-21T22:07:20.204 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5269999999999999e-06 2024-09-21T22:07:20.205 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.205 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.205 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:20.205 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161591+0000", 2024-09-21T22:07:20.205 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.296e-05 2024-09-21T22:07:20.205 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.205 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.205 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:20.205 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161629+0000", 2024-09-21T22:07:20.205 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7799000000000002e-05 2024-09-21T22:07:20.205 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.205 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.205 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:20.205 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.161644+0000", 2024-09-21T22:07:20.206 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5333e-05 2024-09-21T22:07:20.206 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.206 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.206 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:20.206 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.173636+0000", 2024-09-21T22:07:20.206 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011992078 2024-09-21T22:07:20.206 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.206 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.206 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:20.206 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.173862+0000", 2024-09-21T22:07:20.206 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022556799999999999 2024-09-21T22:07:20.206 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.206 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.206 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:20.207 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.174193+0000", 2024-09-21T22:07:20.207 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000331088 2024-09-21T22:07:20.207 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.207 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.207 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:20.207 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.175740+0000", 2024-09-21T22:07:20.207 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015467910000000001 2024-09-21T22:07:20.207 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.207 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.207 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:20.207 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.178073+0000", 2024-09-21T22:07:20.207 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002333356 2024-09-21T22:07:20.207 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.207 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.208 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:20.208 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.178162+0000", 2024-09-21T22:07:20.208 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8306000000000005e-05 2024-09-21T22:07:20.208 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.208 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.208 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:20.208 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.178230+0000", 2024-09-21T22:07:20.208 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8186000000000001e-05 2024-09-21T22:07:20.208 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:20.208 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:20.208 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:20.208 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.208 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.208 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:88 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:07:20.209 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.280540+0000", 2024-09-21T22:07:20.209 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 77.898932747000003, 2024-09-21T22:07:20.209 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013246212, 2024-09-21T22:07:20.209 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:20.209 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:20.209 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:20.209 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:20.209 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:07:20.209 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:07:20.209 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 88 2024-09-21T22:07:20.209 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.209 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:20.209 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.209 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:20.210 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.280540+0000", 2024-09-21T22:07:20.210 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:20.210 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.210 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.210 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:20.210 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.280540+0000", 2024-09-21T22:07:20.210 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:20.210 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.210 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.210 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:20.210 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.280541+0000", 2024-09-21T22:07:20.210 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.54e-07 2024-09-21T22:07:20.210 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.210 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.211 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:20.211 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.280549+0000", 2024-09-21T22:07:20.211 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3329999999999997e-06 2024-09-21T22:07:20.211 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.211 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.211 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:20.211 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.280560+0000", 2024-09-21T22:07:20.211 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1003e-05 2024-09-21T22:07:20.211 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.211 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.211 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:20.211 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.280569+0000", 2024-09-21T22:07:20.212 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2430000000000005e-06 2024-09-21T22:07:20.212 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.212 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.212 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:20.212 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.291095+0000", 2024-09-21T22:07:20.212 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010526259 2024-09-21T22:07:20.212 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.212 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.212 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:20.212 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.291231+0000", 2024-09-21T22:07:20.212 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013561899999999999 2024-09-21T22:07:20.212 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.212 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.212 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:20.213 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.291457+0000", 2024-09-21T22:07:20.213 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022686200000000001 2024-09-21T22:07:20.213 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.213 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.213 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:20.213 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.293117+0000", 2024-09-21T22:07:20.213 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001659474 2024-09-21T22:07:20.213 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.213 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.213 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:20.213 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.293524+0000", 2024-09-21T22:07:20.213 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00040701300000000001 2024-09-21T22:07:20.213 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.213 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.214 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:20.214 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.293673+0000", 2024-09-21T22:07:20.214 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000148782 2024-09-21T22:07:20.214 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.214 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.214 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:20.214 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.293786+0000", 2024-09-21T22:07:20.214 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011377 2024-09-21T22:07:20.214 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:20.214 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:20.214 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:20.214 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.214 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.215 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:107 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:07:20.215 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.466485+0000", 2024-09-21T22:07:20.215 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 77.712988015999997, 2024-09-21T22:07:20.215 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014443981, 2024-09-21T22:07:20.215 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:20.215 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:20.215 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:20.215 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:20.215 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:07:20.215 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:07:20.215 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 107 2024-09-21T22:07:20.215 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.215 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:20.215 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.216 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:20.216 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.466485+0000", 2024-09-21T22:07:20.216 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:20.216 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.216 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.216 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:20.216 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.466485+0000", 2024-09-21T22:07:20.216 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:20.216 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.216 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.216 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:20.216 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.466486+0000", 2024-09-21T22:07:20.216 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6299999999999993e-07 2024-09-21T22:07:20.216 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.217 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.217 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:20.217 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.466493+0000", 2024-09-21T22:07:20.217 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9639999999999997e-06 2024-09-21T22:07:20.217 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.217 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.217 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:20.217 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.466507+0000", 2024-09-21T22:07:20.217 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4216000000000001e-05 2024-09-21T22:07:20.217 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.217 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.217 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:20.217 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.466515+0000", 2024-09-21T22:07:20.217 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.157e-06 2024-09-21T22:07:20.218 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.218 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.218 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:20.218 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.472033+0000", 2024-09-21T22:07:20.218 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0055175839999999999 2024-09-21T22:07:20.218 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.218 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.218 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:20.218 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.472174+0000", 2024-09-21T22:07:20.218 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000141612 2024-09-21T22:07:20.218 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.218 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.218 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:20.219 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.472381+0000", 2024-09-21T22:07:20.219 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000206961 2024-09-21T22:07:20.219 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.219 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.219 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:20.219 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.473924+0000", 2024-09-21T22:07:20.219 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015427119999999999 2024-09-21T22:07:20.219 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.219 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.219 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:20.219 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.480666+0000", 2024-09-21T22:07:20.219 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0067415410000000002 2024-09-21T22:07:20.219 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.220 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.220 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:20.220 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.480811+0000", 2024-09-21T22:07:20.220 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000145422 2024-09-21T22:07:20.220 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.220 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.220 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:20.220 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.480929+0000", 2024-09-21T22:07:20.220 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000117849 2024-09-21T22:07:20.220 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:20.220 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:20.220 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:20.220 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.220 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.220 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:07:20.221 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.467057+0000", 2024-09-21T22:07:20.221 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 77.712415757000002, 2024-09-21T22:07:20.221 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018350195, 2024-09-21T22:07:20.221 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:20.221 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:20.221 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:20.221 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:20.221 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:07:20.221 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:07:20.221 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 108 2024-09-21T22:07:20.221 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.221 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:20.221 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.221 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:20.222 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467057+0000", 2024-09-21T22:07:20.222 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:20.222 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.222 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.222 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:20.222 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467057+0000", 2024-09-21T22:07:20.222 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:20.222 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.222 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.222 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:20.222 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467059+0000", 2024-09-21T22:07:20.222 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.826e-06 2024-09-21T22:07:20.222 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.223 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.223 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:20.223 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467071+0000", 2024-09-21T22:07:20.223 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2224e-05 2024-09-21T22:07:20.223 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.223 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.223 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:20.223 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467098+0000", 2024-09-21T22:07:20.223 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6953000000000001e-05 2024-09-21T22:07:20.223 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.223 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.223 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:20.223 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467113+0000", 2024-09-21T22:07:20.223 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5119999999999999e-05 2024-09-21T22:07:20.224 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.224 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.224 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:20.224 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.474060+0000", 2024-09-21T22:07:20.224 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0069462919999999997 2024-09-21T22:07:20.224 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.224 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.224 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:20.224 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.474283+0000", 2024-09-21T22:07:20.224 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000223315 2024-09-21T22:07:20.224 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.224 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.224 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:20.224 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.474660+0000", 2024-09-21T22:07:20.224 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00037729400000000002 2024-09-21T22:07:20.225 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.225 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.225 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:20.225 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.480622+0000", 2024-09-21T22:07:20.225 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0059613369999999997 2024-09-21T22:07:20.225 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.225 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.225 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:20.225 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.485145+0000", 2024-09-21T22:07:20.225 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0045231960000000002 2024-09-21T22:07:20.225 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.225 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.225 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:20.225 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.485288+0000", 2024-09-21T22:07:20.226 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000143232 2024-09-21T22:07:20.226 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.226 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.226 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:20.226 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.485407+0000", 2024-09-21T22:07:20.226 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000119406 2024-09-21T22:07:20.226 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:20.226 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:20.226 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:20.226 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.226 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.226 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:07:20.226 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.467144+0000", 2024-09-21T22:07:20.227 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 77.712328537999994, 2024-09-21T22:07:20.227 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018907146999999999, 2024-09-21T22:07:20.227 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:20.227 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:20.227 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:20.227 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:20.227 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:07:20.227 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:07:20.227 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 109 2024-09-21T22:07:20.227 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.227 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:20.227 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.227 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:20.228 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467144+0000", 2024-09-21T22:07:20.228 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:20.228 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.228 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.228 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:20.228 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467144+0000", 2024-09-21T22:07:20.228 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:20.228 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.228 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.228 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:20.228 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467145+0000", 2024-09-21T22:07:20.228 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9999999999999996e-07 2024-09-21T22:07:20.228 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.229 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.229 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:20.229 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467331+0000", 2024-09-21T22:07:20.229 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018518899999999999 2024-09-21T22:07:20.229 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.229 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.229 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:20.229 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467350+0000", 2024-09-21T22:07:20.229 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8966000000000001e-05 2024-09-21T22:07:20.229 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.229 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.229 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:20.229 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467361+0000", 2024-09-21T22:07:20.229 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1233000000000001e-05 2024-09-21T22:07:20.230 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.230 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.230 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:20.230 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.475613+0000", 2024-09-21T22:07:20.230 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0082518629999999999 2024-09-21T22:07:20.230 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.230 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.230 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:20.230 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.475859+0000", 2024-09-21T22:07:20.230 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024633800000000003 2024-09-21T22:07:20.230 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.230 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.230 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:20.230 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.476223+0000", 2024-09-21T22:07:20.231 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00036408700000000003 2024-09-21T22:07:20.231 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.231 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.231 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:20.231 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.481046+0000", 2024-09-21T22:07:20.231 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.004823356 2024-09-21T22:07:20.231 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.231 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.231 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:20.231 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.485722+0000", 2024-09-21T22:07:20.231 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0046753640000000004 2024-09-21T22:07:20.231 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.231 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.231 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:20.232 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.485908+0000", 2024-09-21T22:07:20.232 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000185859 2024-09-21T22:07:20.232 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.232 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.232 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:20.232 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.486052+0000", 2024-09-21T22:07:20.232 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014399200000000001 2024-09-21T22:07:20.232 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:20.232 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:20.232 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:20.232 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.232 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.232 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:07:20.232 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.467384+0000", 2024-09-21T22:07:20.232 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 77.712088710000003, 2024-09-21T22:07:20.233 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.01894144, 2024-09-21T22:07:20.233 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:20.233 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:20.233 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:20.233 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:20.233 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:07:20.233 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:07:20.233 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 110 2024-09-21T22:07:20.233 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.233 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:20.233 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.233 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:20.233 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467384+0000", 2024-09-21T22:07:20.233 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:20.233 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.234 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.234 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:20.234 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467384+0000", 2024-09-21T22:07:20.234 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:20.234 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.234 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.234 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:20.234 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467385+0000", 2024-09-21T22:07:20.234 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0470000000000001e-06 2024-09-21T22:07:20.234 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.234 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.234 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:20.234 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467393+0000", 2024-09-21T22:07:20.234 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3869999999999997e-06 2024-09-21T22:07:20.235 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.235 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.235 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:20.235 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467413+0000", 2024-09-21T22:07:20.235 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9953e-05 2024-09-21T22:07:20.235 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.235 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.235 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:20.235 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467421+0000", 2024-09-21T22:07:20.235 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2400000000000007e-06 2024-09-21T22:07:20.235 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.235 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.235 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:20.235 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.479542+0000", 2024-09-21T22:07:20.236 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.012120832 2024-09-21T22:07:20.236 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.236 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.236 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:20.236 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.479705+0000", 2024-09-21T22:07:20.236 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016362899999999999 2024-09-21T22:07:20.236 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.236 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.236 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:20.236 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.479914+0000", 2024-09-21T22:07:20.236 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020814499999999999 2024-09-21T22:07:20.236 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.236 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.236 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:20.236 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.485536+0000", 2024-09-21T22:07:20.237 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0056222800000000003 2024-09-21T22:07:20.237 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.237 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.237 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:20.237 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.486077+0000", 2024-09-21T22:07:20.237 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00054079900000000001 2024-09-21T22:07:20.237 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.237 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.237 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:20.237 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.486213+0000", 2024-09-21T22:07:20.237 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000136682 2024-09-21T22:07:20.237 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.237 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.237 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:20.238 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.486326+0000", 2024-09-21T22:07:20.238 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000112446 2024-09-21T22:07:20.238 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:20.238 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:20.238 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:20.238 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.238 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.238 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:07:20.238 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.467436+0000", 2024-09-21T22:07:20.238 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 77.712037210000005, 2024-09-21T22:07:20.238 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019152168000000001, 2024-09-21T22:07:20.238 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:20.238 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:20.238 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:20.239 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:20.239 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:07:20.239 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:07:20.239 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 111 2024-09-21T22:07:20.239 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.239 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:20.239 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.239 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:20.239 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467436+0000", 2024-09-21T22:07:20.239 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:20.239 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.239 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.239 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:20.239 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467436+0000", 2024-09-21T22:07:20.240 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:20.240 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.240 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.240 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:20.240 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467437+0000", 2024-09-21T22:07:20.240 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0699999999999996e-07 2024-09-21T22:07:20.240 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.240 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.240 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:20.240 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467445+0000", 2024-09-21T22:07:20.240 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.789e-06 2024-09-21T22:07:20.240 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.240 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.240 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:20.241 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467467+0000", 2024-09-21T22:07:20.241 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1253000000000001e-05 2024-09-21T22:07:20.241 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.241 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.241 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:20.241 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467475+0000", 2024-09-21T22:07:20.241 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0039999999999999e-06 2024-09-21T22:07:20.241 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.241 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.241 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:20.241 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.481164+0000", 2024-09-21T22:07:20.241 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013689114 2024-09-21T22:07:20.241 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.241 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.242 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:20.242 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.481394+0000", 2024-09-21T22:07:20.242 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000229931 2024-09-21T22:07:20.242 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.242 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.242 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:20.242 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.481779+0000", 2024-09-21T22:07:20.242 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00038545400000000003 2024-09-21T22:07:20.242 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.242 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.242 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:20.242 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.485675+0000", 2024-09-21T22:07:20.242 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0038958370000000001 2024-09-21T22:07:20.242 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.243 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.243 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:20.243 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.486346+0000", 2024-09-21T22:07:20.243 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00067119100000000002 2024-09-21T22:07:20.243 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.243 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.243 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:20.243 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.486478+0000", 2024-09-21T22:07:20.243 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013182600000000001 2024-09-21T22:07:20.243 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.243 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.243 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:20.243 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.486588+0000", 2024-09-21T22:07:20.243 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000110062 2024-09-21T22:07:20.243 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:20.244 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:20.244 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:20.244 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.244 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.244 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:07:20.244 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.467488+0000", 2024-09-21T22:07:20.244 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 77.711984584000007, 2024-09-21T22:07:20.244 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021038304000000001, 2024-09-21T22:07:20.244 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:20.244 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:20.244 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:20.244 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:20.244 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:07:20.244 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:07:20.245 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 112 2024-09-21T22:07:20.245 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.245 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:20.245 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.245 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:20.245 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467488+0000", 2024-09-21T22:07:20.245 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:20.245 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.245 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.245 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:20.245 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467488+0000", 2024-09-21T22:07:20.245 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:20.245 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.245 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.245 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:20.246 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467489+0000", 2024-09-21T22:07:20.246 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7999999999999995e-07 2024-09-21T22:07:20.246 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.246 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.246 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:20.246 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467687+0000", 2024-09-21T22:07:20.246 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019776200000000001 2024-09-21T22:07:20.246 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.246 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.246 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:20.246 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467704+0000", 2024-09-21T22:07:20.246 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7193e-05 2024-09-21T22:07:20.246 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.247 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.247 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:20.247 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467719+0000", 2024-09-21T22:07:20.247 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4703e-05 2024-09-21T22:07:20.247 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.247 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.247 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:20.247 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.482745+0000", 2024-09-21T22:07:20.247 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015025801 2024-09-21T22:07:20.247 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.247 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.247 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:20.247 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.483015+0000", 2024-09-21T22:07:20.247 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026999400000000002 2024-09-21T22:07:20.247 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.248 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.248 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:20.248 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.483440+0000", 2024-09-21T22:07:20.248 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000425697 2024-09-21T22:07:20.248 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.248 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.248 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:20.248 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.486701+0000", 2024-09-21T22:07:20.248 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0032611950000000002 2024-09-21T22:07:20.248 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.248 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.248 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:20.248 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.488262+0000", 2024-09-21T22:07:20.248 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015610439999999999 2024-09-21T22:07:20.249 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.249 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.249 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:20.249 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.488406+0000", 2024-09-21T22:07:20.249 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000143232 2024-09-21T22:07:20.249 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.249 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.249 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:20.249 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.488527+0000", 2024-09-21T22:07:20.249 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012100299999999999 2024-09-21T22:07:20.249 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:20.249 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:20.249 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:20.250 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.250 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.250 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:07:20.250 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.467786+0000", 2024-09-21T22:07:20.250 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 77.711687158999993, 2024-09-21T22:07:20.250 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026339741999999999, 2024-09-21T22:07:20.250 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:20.250 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:20.250 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:20.250 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:20.250 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:07:20.250 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:07:20.250 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 113 2024-09-21T22:07:20.250 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.251 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:20.251 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.251 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:20.251 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467786+0000", 2024-09-21T22:07:20.251 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:20.251 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.251 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.251 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:20.251 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467786+0000", 2024-09-21T22:07:20.251 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:20.251 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.251 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.251 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:20.251 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467787+0000", 2024-09-21T22:07:20.251 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.446e-06 2024-09-21T22:07:20.252 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.252 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.252 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:20.252 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467802+0000", 2024-09-21T22:07:20.252 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4769999999999999e-05 2024-09-21T22:07:20.252 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.252 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.252 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:20.252 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467826+0000", 2024-09-21T22:07:20.252 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3430000000000001e-05 2024-09-21T22:07:20.252 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.252 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.252 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:20.252 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467843+0000", 2024-09-21T22:07:20.253 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7065999999999999e-05 2024-09-21T22:07:20.253 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.253 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.253 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:20.253 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.486820+0000", 2024-09-21T22:07:20.253 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018977233999999999 2024-09-21T22:07:20.253 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.253 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.253 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:20.253 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.487065+0000", 2024-09-21T22:07:20.253 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024558799999999998 2024-09-21T22:07:20.253 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.253 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.253 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:20.253 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.487442+0000", 2024-09-21T22:07:20.254 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000376707 2024-09-21T22:07:20.254 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.254 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.254 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:20.254 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.492256+0000", 2024-09-21T22:07:20.254 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0048141620000000003 2024-09-21T22:07:20.254 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.254 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.254 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:20.254 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.493865+0000", 2024-09-21T22:07:20.254 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016090150000000001 2024-09-21T22:07:20.254 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.254 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.255 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:20.255 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.494007+0000", 2024-09-21T22:07:20.255 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000141359 2024-09-21T22:07:20.255 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.255 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.255 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:20.255 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.494126+0000", 2024-09-21T22:07:20.255 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000118965 2024-09-21T22:07:20.255 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:20.255 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:20.255 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:20.255 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.255 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.255 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:07:20.256 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.467878+0000", 2024-09-21T22:07:20.256 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 77.711594669999997, 2024-09-21T22:07:20.256 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.027783958000000001, 2024-09-21T22:07:20.256 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:20.256 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:20.256 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:20.256 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:20.256 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:07:20.256 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:07:20.256 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 114 2024-09-21T22:07:20.256 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.256 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:20.256 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.256 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:20.257 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467878+0000", 2024-09-21T22:07:20.257 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:20.257 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.257 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.257 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:20.257 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467878+0000", 2024-09-21T22:07:20.257 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:20.257 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.257 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.257 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:20.257 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467880+0000", 2024-09-21T22:07:20.257 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.223e-06 2024-09-21T22:07:20.257 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.257 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.258 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:20.258 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467891+0000", 2024-09-21T22:07:20.258 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1527e-05 2024-09-21T22:07:20.258 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.258 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.258 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:20.258 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467913+0000", 2024-09-21T22:07:20.258 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.143e-05 2024-09-21T22:07:20.258 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.258 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.258 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:20.258 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.467930+0000", 2024-09-21T22:07:20.258 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7620000000000001e-05 2024-09-21T22:07:20.258 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.259 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.259 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:20.259 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.488690+0000", 2024-09-21T22:07:20.259 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020759588999999998 2024-09-21T22:07:20.259 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.259 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.259 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:20.259 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.488882+0000", 2024-09-21T22:07:20.259 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019236200000000001 2024-09-21T22:07:20.259 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.259 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.259 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:20.259 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.489093+0000", 2024-09-21T22:07:20.259 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000211318 2024-09-21T22:07:20.260 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.260 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.260 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:20.260 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.494152+0000", 2024-09-21T22:07:20.260 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0050582769999999999 2024-09-21T22:07:20.260 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.260 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.260 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:20.260 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.495373+0000", 2024-09-21T22:07:20.260 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001221747 2024-09-21T22:07:20.260 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.260 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.260 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:20.260 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.495543+0000", 2024-09-21T22:07:20.261 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000169259 2024-09-21T22:07:20.261 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.261 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.261 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:20.261 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.495662+0000", 2024-09-21T22:07:20.261 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011960600000000001 2024-09-21T22:07:20.261 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:20.261 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:20.261 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:20.261 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.261 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.261 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:07:20.261 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.468118+0000", 2024-09-21T22:07:20.261 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 77.711354928999995, 2024-09-21T22:07:20.262 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.027938877000000001, 2024-09-21T22:07:20.262 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:20.262 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:20.262 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:20.262 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:20.262 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:07:20.262 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:07:20.262 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 115 2024-09-21T22:07:20.262 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.262 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:20.262 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.262 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:20.262 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468118+0000", 2024-09-21T22:07:20.262 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:20.263 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.263 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.263 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:20.263 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468118+0000", 2024-09-21T22:07:20.263 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:20.263 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.263 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.263 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:20.263 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468120+0000", 2024-09-21T22:07:20.263 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7069999999999999e-06 2024-09-21T22:07:20.263 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.263 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.263 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:20.263 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468381+0000", 2024-09-21T22:07:20.264 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026135399999999999 2024-09-21T22:07:20.264 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.264 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.264 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:20.264 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468409+0000", 2024-09-21T22:07:20.264 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.781e-05 2024-09-21T22:07:20.264 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.264 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.264 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:20.264 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468433+0000", 2024-09-21T22:07:20.264 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4343999999999999e-05 2024-09-21T22:07:20.264 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.264 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.264 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:20.265 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.489785+0000", 2024-09-21T22:07:20.265 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021351221 2024-09-21T22:07:20.265 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.265 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.265 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:20.265 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.490018+0000", 2024-09-21T22:07:20.265 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023317800000000001 2024-09-21T22:07:20.265 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.265 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.265 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:20.265 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.490412+0000", 2024-09-21T22:07:20.265 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00039417400000000001 2024-09-21T22:07:20.265 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.265 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.266 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:20.266 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.495694+0000", 2024-09-21T22:07:20.266 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.005282335 2024-09-21T22:07:20.266 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.266 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.266 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:20.266 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.495852+0000", 2024-09-21T22:07:20.266 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015777200000000001 2024-09-21T22:07:20.266 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.266 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.266 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:20.266 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.495938+0000", 2024-09-21T22:07:20.266 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6493e-05 2024-09-21T22:07:20.267 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.267 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.267 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:20.267 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.496057+0000", 2024-09-21T22:07:20.267 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000118489 2024-09-21T22:07:20.267 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:20.267 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:20.267 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:20.267 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.267 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.267 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:07:20.267 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.468474+0000", 2024-09-21T22:07:20.267 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 77.710999168000001, 2024-09-21T22:07:20.268 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.028015895999999998, 2024-09-21T22:07:20.268 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:20.268 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:20.268 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:20.268 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:20.268 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:07:20.268 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:07:20.268 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 116 2024-09-21T22:07:20.268 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.268 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:20.268 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.268 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:20.268 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468474+0000", 2024-09-21T22:07:20.268 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:20.268 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.269 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.269 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:20.269 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468474+0000", 2024-09-21T22:07:20.269 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:20.269 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.269 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.269 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:20.269 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468476+0000", 2024-09-21T22:07:20.269 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7060000000000001e-06 2024-09-21T22:07:20.269 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.269 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.269 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:20.269 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468488+0000", 2024-09-21T22:07:20.269 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.234e-05 2024-09-21T22:07:20.270 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.270 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.270 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:20.270 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468517+0000", 2024-09-21T22:07:20.270 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.898e-05 2024-09-21T22:07:20.270 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.270 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.270 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:20.270 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468536+0000", 2024-09-21T22:07:20.270 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8899999999999999e-05 2024-09-21T22:07:20.270 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.270 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.270 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:20.270 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.492376+0000", 2024-09-21T22:07:20.271 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023840659 2024-09-21T22:07:20.271 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.271 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.271 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:20.271 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.492621+0000", 2024-09-21T22:07:20.271 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000244794 2024-09-21T22:07:20.271 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.271 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.271 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:20.271 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.493008+0000", 2024-09-21T22:07:20.271 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00038629099999999998 2024-09-21T22:07:20.271 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.271 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.271 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:20.271 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.496192+0000", 2024-09-21T22:07:20.272 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0031847220000000001 2024-09-21T22:07:20.272 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.272 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.272 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:20.272 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.496233+0000", 2024-09-21T22:07:20.272 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0410000000000001e-05 2024-09-21T22:07:20.272 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.272 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.272 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:20.272 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.496374+0000", 2024-09-21T22:07:20.272 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140872 2024-09-21T22:07:20.272 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.272 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.272 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:20.273 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.496490+0000", 2024-09-21T22:07:20.273 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000116222 2024-09-21T22:07:20.273 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:20.273 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:20.273 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:20.273 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.273 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.273 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:07:20.273 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.468568+0000", 2024-09-21T22:07:20.273 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 77.710905049000004, 2024-09-21T22:07:20.273 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.028181578999999998, 2024-09-21T22:07:20.273 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:20.273 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:20.273 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:20.274 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:20.274 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:07:20.274 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:07:20.274 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 117 2024-09-21T22:07:20.274 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.274 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:20.274 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.274 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:20.274 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468568+0000", 2024-09-21T22:07:20.274 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:20.274 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.274 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.274 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:20.274 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468568+0000", 2024-09-21T22:07:20.275 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:20.275 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.275 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.275 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:20.275 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468569+0000", 2024-09-21T22:07:20.275 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4139999999999999e-06 2024-09-21T22:07:20.275 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.275 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.275 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:20.275 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468583+0000", 2024-09-21T22:07:20.275 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3409e-05 2024-09-21T22:07:20.275 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.275 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.275 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:20.275 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468608+0000", 2024-09-21T22:07:20.276 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5559999999999999e-05 2024-09-21T22:07:20.276 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.276 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.276 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:20.276 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468626+0000", 2024-09-21T22:07:20.276 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7863000000000002e-05 2024-09-21T22:07:20.276 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.276 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.276 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:20.276 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.494260+0000", 2024-09-21T22:07:20.276 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025633744999999999 2024-09-21T22:07:20.276 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.277 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.277 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:20.277 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.494519+0000", 2024-09-21T22:07:20.277 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025924800000000001 2024-09-21T22:07:20.277 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.277 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.277 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:20.277 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.494747+0000", 2024-09-21T22:07:20.277 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022792799999999999 2024-09-21T22:07:20.277 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.277 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.277 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:20.277 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.496519+0000", 2024-09-21T22:07:20.277 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017720030000000001 2024-09-21T22:07:20.277 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.278 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.278 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:20.278 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.496607+0000", 2024-09-21T22:07:20.278 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7873000000000004e-05 2024-09-21T22:07:20.278 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.278 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.278 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:20.278 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.496691+0000", 2024-09-21T22:07:20.278 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4009000000000002e-05 2024-09-21T22:07:20.278 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.278 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.279 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:20.279 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.496750+0000", 2024-09-21T22:07:20.279 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8526999999999997e-05 2024-09-21T22:07:20.279 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:20.279 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:20.279 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:20.279 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.279 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.279 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:07:20.279 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.898244+0000", 2024-09-21T22:07:20.279 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 77.281228745999996, 2024-09-21T22:07:20.279 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014749464, 2024-09-21T22:07:20.279 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:20.279 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:20.280 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:20.280 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:20.280 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:07:20.280 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:07:20.280 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 140 2024-09-21T22:07:20.280 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.280 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:20.280 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.280 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:20.280 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898244+0000", 2024-09-21T22:07:20.280 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:20.280 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.280 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.280 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:20.281 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898244+0000", 2024-09-21T22:07:20.281 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:20.281 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.281 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.281 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:20.281 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898245+0000", 2024-09-21T22:07:20.281 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3000000000000002e-07 2024-09-21T22:07:20.281 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.281 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.281 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:20.281 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898308+0000", 2024-09-21T22:07:20.281 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3468999999999998e-05 2024-09-21T22:07:20.281 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.281 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.282 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:20.282 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898315+0000", 2024-09-21T22:07:20.282 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.725e-06 2024-09-21T22:07:20.282 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.282 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.282 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:20.282 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898321+0000", 2024-09-21T22:07:20.282 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2090000000000002e-06 2024-09-21T22:07:20.282 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.282 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.282 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:20.282 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.905452+0000", 2024-09-21T22:07:20.282 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0071305989999999996 2024-09-21T22:07:20.282 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.283 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.283 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:20.283 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.905504+0000", 2024-09-21T22:07:20.283 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2638e-05 2024-09-21T22:07:20.283 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.283 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.283 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:20.283 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.905590+0000", 2024-09-21T22:07:20.283 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5333000000000004e-05 2024-09-21T22:07:20.283 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.283 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.283 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:20.283 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.911542+0000", 2024-09-21T22:07:20.283 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0059519819999999998 2024-09-21T22:07:20.283 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.284 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.284 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:20.284 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.912908+0000", 2024-09-21T22:07:20.284 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0013666640000000001 2024-09-21T22:07:20.284 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.284 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.284 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:20.284 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.912957+0000", 2024-09-21T22:07:20.284 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9203000000000003e-05 2024-09-21T22:07:20.284 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.284 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.284 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:20.284 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.912994+0000", 2024-09-21T22:07:20.284 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6312000000000001e-05 2024-09-21T22:07:20.285 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:20.285 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:20.285 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:20.285 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.285 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.285 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:07:20.285 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.898342+0000", 2024-09-21T22:07:20.285 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 77.281130607999998, 2024-09-21T22:07:20.285 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014787918000000001, 2024-09-21T22:07:20.285 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:20.285 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:20.285 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:20.285 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:20.285 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:07:20.286 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:07:20.286 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 141 2024-09-21T22:07:20.286 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.286 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:20.286 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.286 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:20.286 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898342+0000", 2024-09-21T22:07:20.286 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:20.286 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.286 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.286 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:20.286 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898342+0000", 2024-09-21T22:07:20.286 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:20.286 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.286 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.287 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:20.287 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898343+0000", 2024-09-21T22:07:20.287 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3000000000000001e-07 2024-09-21T22:07:20.287 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.287 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.287 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:20.287 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898346+0000", 2024-09-21T22:07:20.287 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6009999999999999e-06 2024-09-21T22:07:20.287 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.287 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.287 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:20.287 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898352+0000", 2024-09-21T22:07:20.287 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8089999999999998e-06 2024-09-21T22:07:20.287 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.288 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.288 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:20.288 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.898358+0000", 2024-09-21T22:07:20.288 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4349999999999996e-06 2024-09-21T22:07:20.288 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.288 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.288 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:20.288 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.909387+0000", 2024-09-21T22:07:20.288 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011029105000000001 2024-09-21T22:07:20.288 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.288 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.288 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:20.288 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.909462+0000", 2024-09-21T22:07:20.288 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5192999999999996e-05 2024-09-21T22:07:20.289 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.289 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.289 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:20.289 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.909551+0000", 2024-09-21T22:07:20.289 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9046000000000001e-05 2024-09-21T22:07:20.289 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.289 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.289 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:20.289 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.911852+0000", 2024-09-21T22:07:20.289 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0023005579999999999 2024-09-21T22:07:20.289 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.289 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.289 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:20.289 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.913002+0000", 2024-09-21T22:07:20.289 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011502929999999999 2024-09-21T22:07:20.290 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.290 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.290 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:20.290 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.913060+0000", 2024-09-21T22:07:20.290 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8427000000000002e-05 2024-09-21T22:07:20.290 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.290 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.290 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:20.290 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.913130+0000", 2024-09-21T22:07:20.290 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0020999999999999e-05 2024-09-21T22:07:20.290 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:20.290 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:20.290 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:20.290 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.291 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.291 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:158 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T22:07:20.291 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:03.511201+0000", 2024-09-21T22:07:20.291 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 76.668272125000001, 2024-09-21T22:07:20.291 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013353538999999999, 2024-09-21T22:07:20.291 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:20.291 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:20.291 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:20.291 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:20.291 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:07:20.291 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:07:20.291 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 158 2024-09-21T22:07:20.291 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.291 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:20.291 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.292 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:20.292 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511201+0000", 2024-09-21T22:07:20.292 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:20.292 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.292 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.292 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:20.292 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511201+0000", 2024-09-21T22:07:20.292 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:20.292 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.292 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.292 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:20.292 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511202+0000", 2024-09-21T22:07:20.292 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9299999999999997e-07 2024-09-21T22:07:20.292 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.293 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.293 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:20.293 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511315+0000", 2024-09-21T22:07:20.293 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000113909 2024-09-21T22:07:20.293 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.293 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.293 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:20.293 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511333+0000", 2024-09-21T22:07:20.293 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7062999999999999e-05 2024-09-21T22:07:20.293 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.293 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.293 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:20.293 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511344+0000", 2024-09-21T22:07:20.293 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1177e-05 2024-09-21T22:07:20.294 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.294 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.294 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:20.294 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.517415+0000", 2024-09-21T22:07:20.294 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0060709830000000003 2024-09-21T22:07:20.294 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.294 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.294 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:20.294 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.517539+0000", 2024-09-21T22:07:20.294 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000124432 2024-09-21T22:07:20.294 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.294 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.294 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:20.294 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.517732+0000", 2024-09-21T22:07:20.294 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019262200000000001 2024-09-21T22:07:20.295 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.295 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.295 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:20.295 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.520709+0000", 2024-09-21T22:07:20.295 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0029767470000000001 2024-09-21T22:07:20.295 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.295 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.295 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:20.295 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.524389+0000", 2024-09-21T22:07:20.295 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003680975 2024-09-21T22:07:20.295 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.295 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.295 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:20.295 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.524486+0000", 2024-09-21T22:07:20.296 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6575000000000003e-05 2024-09-21T22:07:20.296 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.296 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.296 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:20.296 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.524554+0000", 2024-09-21T22:07:20.296 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8362999999999996e-05 2024-09-21T22:07:20.296 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:20.296 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:20.296 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:20.296 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.296 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.296 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T22:07:20.296 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:03.511368+0000", 2024-09-21T22:07:20.296 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 76.668105222999998, 2024-09-21T22:07:20.297 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013312133, 2024-09-21T22:07:20.297 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:20.297 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:20.297 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:20.297 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:20.297 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:07:20.297 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:07:20.297 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 159 2024-09-21T22:07:20.297 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.297 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:20.297 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.297 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:20.297 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511368+0000", 2024-09-21T22:07:20.297 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:20.297 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.298 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.298 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:20.298 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511368+0000", 2024-09-21T22:07:20.298 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:20.298 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.298 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.298 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:20.298 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511369+0000", 2024-09-21T22:07:20.298 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9599999999999998e-07 2024-09-21T22:07:20.298 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.298 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.298 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:20.298 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511382+0000", 2024-09-21T22:07:20.298 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3487e-05 2024-09-21T22:07:20.299 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.299 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.299 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:20.299 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511394+0000", 2024-09-21T22:07:20.299 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1337e-05 2024-09-21T22:07:20.299 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.299 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.299 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:20.299 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.511402+0000", 2024-09-21T22:07:20.299 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0059999999999992e-06 2024-09-21T22:07:20.299 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.299 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.299 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:20.299 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.519376+0000", 2024-09-21T22:07:20.300 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0079746520000000005 2024-09-21T22:07:20.300 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.300 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.300 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:20.300 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.519498+0000", 2024-09-21T22:07:20.300 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012212900000000001 2024-09-21T22:07:20.300 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.300 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.300 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:20.300 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.519728+0000", 2024-09-21T22:07:20.300 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022985799999999999 2024-09-21T22:07:20.300 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.300 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.300 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:20.301 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.521344+0000", 2024-09-21T22:07:20.301 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0016163169999999999 2024-09-21T22:07:20.301 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.301 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.301 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:20.301 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.524568+0000", 2024-09-21T22:07:20.301 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0032234220000000001 2024-09-21T22:07:20.301 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.301 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.301 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:20.301 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.524633+0000", 2024-09-21T22:07:20.301 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5126e-05 2024-09-21T22:07:20.301 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.301 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.301 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:20.302 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:03.524680+0000", 2024-09-21T22:07:20.302 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6903000000000001e-05 2024-09-21T22:07:20.302 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:20.302 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:20.302 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:20.302 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:20.302 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:20.302 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:07:20.302 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 5, 0] out_osds: [2] dead_osds: [5] live_osds: [0, 1, 2, 3, 4] 2024-09-21T22:07:20.302 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T22:07:20.302 INFO:tasks.thrashosds.thrasher:Reviving osd 5 2024-09-21T22:07:20.303 INFO:tasks.ceph.osd.5:Restarting daemon 2024-09-21T22:07:20.303 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-09-21T22:07:20.305 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T22:07:20.305 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:20.300+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:20.306 INFO:tasks.workunit.client.0.smithi157.stderr:+ rbd device --device-type nbd list 2024-09-21T22:07:20.306 INFO:tasks.workunit.client.0.smithi157.stderr:+ expect_false grep '^21857\b' 2024-09-21T22:07:20.306 INFO:tasks.workunit.client.0.smithi157.stderr:+ grep '^21857\b' 2024-09-21T22:07:20.306 INFO:tasks.workunit.client.0.smithi157.stderr:+ return 0 2024-09-21T22:07:20.306 INFO:tasks.workunit.client.0.smithi157.stderr:+ ps -C rbd-nbd 2024-09-21T22:07:20.306 INFO:tasks.workunit.client.0.smithi157.stderr:+ expect_false grep '^ *21857\b' 2024-09-21T22:07:20.306 INFO:tasks.workunit.client.0.smithi157.stderr:+ grep '^ *21857\b' 2024-09-21T22:07:20.306 INFO:tasks.workunit.client.0.smithi157.stderr:+ return 0 2024-09-21T22:07:20.306 INFO:tasks.workunit.client.0.smithi157.stderr:+ sleep 0.5 2024-09-21T22:07:20.348 INFO:tasks.ceph.osd.5:Started 2024-09-21T22:07:20.348 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T22:07:20.405 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:20.403+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:20.468 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-09-21T22:07:20.468 INFO:teuthology.orchestra.run.smithi064.stderr:0 2024-09-21T22:07:20.468 INFO:teuthology.orchestra.run.smithi064.stderr:1 2024-09-21T22:07:20.468 INFO:teuthology.orchestra.run.smithi064.stderr:2 2024-09-21T22:07:20.468 INFO:teuthology.orchestra.run.smithi064.stderr:abort 2024-09-21T22:07:20.468 INFO:teuthology.orchestra.run.smithi064.stderr:assert 2024-09-21T22:07:20.468 INFO:teuthology.orchestra.run.smithi064.stderr:config diff 2024-09-21T22:07:20.468 INFO:teuthology.orchestra.run.smithi064.stderr:config diff get 2024-09-21T22:07:20.468 INFO:teuthology.orchestra.run.smithi064.stderr:config get 2024-09-21T22:07:20.468 INFO:teuthology.orchestra.run.smithi064.stderr:config help [] 2024-09-21T22:07:20.468 INFO:teuthology.orchestra.run.smithi064.stderr:config set ... 2024-09-21T22:07:20.468 INFO:teuthology.orchestra.run.smithi064.stderr:admin_socket: invalid command 2024-09-21T22:07:20.471 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-21T22:07:20.471 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-5, ['dump_ops_in_flight'] 2024-09-21T22:07:20.483 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:07:20.483 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:07:20.483 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-09-21T22:07:20.483 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:07:20.498 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T22:07:20.539 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:20.536+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:20.606 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:20.603+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:20.662 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:07:20.663 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:07:20.663 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-21T22:07:20.663 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-21T22:07:20.663 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:07:20.677 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T22:07:20.707 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:20.703+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:20.758 INFO:tasks.workunit.client.0.smithi157.stderr:+ DEV= 2024-09-21T22:07:20.759 INFO:tasks.workunit.client.0.smithi157.stderr:++ _sudo rbd-nbd --device /dev/nbd0 map rbd/testrbdnbd21553 2024-09-21T22:07:20.759 INFO:tasks.workunit.client.0.smithi157.stderr:++ local cmd 2024-09-21T22:07:20.759 INFO:tasks.workunit.client.0.smithi157.stderr:+++ id -u 2024-09-21T22:07:20.761 INFO:tasks.workunit.client.0.smithi157.stderr:++ '[' 1000 -eq 0 ']' 2024-09-21T22:07:20.762 INFO:tasks.workunit.client.0.smithi157.stderr:+++ which rbd-nbd 2024-09-21T22:07:20.764 INFO:tasks.workunit.client.0.smithi157.stderr:++ cmd=/usr/bin/rbd-nbd 2024-09-21T22:07:20.764 INFO:tasks.workunit.client.0.smithi157.stderr:++ shift 2024-09-21T22:07:20.764 INFO:tasks.workunit.client.0.smithi157.stderr:++ sudo -nE /usr/bin/rbd-nbd --device /dev/nbd0 map rbd/testrbdnbd21553 2024-09-21T22:07:20.798 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:07:20.798 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-21T22:07:20.798 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-21T22:07:20.798 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-21T22:07:20.798 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.798 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4887.0:1 3.3 3:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-21T22:07:20.798 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:17.151514+0000", 2024-09-21T22:07:20.799 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.6456674529999997, 2024-09-21T22:07:20.799 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.01157512, 2024-09-21T22:07:20.799 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:20.799 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:20.799 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:20.799 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:20.799 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4887", 2024-09-21T22:07:20.799 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/3431621434", 2024-09-21T22:07:20.799 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1 2024-09-21T22:07:20.799 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.799 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:20.799 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.799 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:20.800 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.151514+0000", 2024-09-21T22:07:20.800 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:20.800 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.800 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.800 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:20.800 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.151514+0000", 2024-09-21T22:07:20.800 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:20.800 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.800 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.800 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:20.800 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.151515+0000", 2024-09-21T22:07:20.800 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.469e-06 2024-09-21T22:07:20.800 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.801 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.801 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:20.801 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.151522+0000", 2024-09-21T22:07:20.801 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1330000000000001e-06 2024-09-21T22:07:20.801 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.801 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.801 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:20.801 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.151536+0000", 2024-09-21T22:07:20.801 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3563e-05 2024-09-21T22:07:20.801 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.801 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.801 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:20.801 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.151574+0000", 2024-09-21T22:07:20.802 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8467999999999998e-05 2024-09-21T22:07:20.802 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.802 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.802 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:20.802 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.151665+0000", 2024-09-21T22:07:20.802 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0668000000000007e-05 2024-09-21T22:07:20.802 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.802 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.802 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-21T22:07:20.802 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.151676+0000", 2024-09-21T22:07:20.802 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1443999999999999e-05 2024-09-21T22:07:20.802 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.803 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.803 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:20.803 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.161694+0000", 2024-09-21T22:07:20.803 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010017402999999999 2024-09-21T22:07:20.803 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.803 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.803 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:20.803 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.161805+0000", 2024-09-21T22:07:20.803 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011122500000000001 2024-09-21T22:07:20.803 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.803 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.803 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:20.803 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.161937+0000", 2024-09-21T22:07:20.803 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013177299999999999 2024-09-21T22:07:20.804 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.804 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.804 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:20.804 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.162555+0000", 2024-09-21T22:07:20.804 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00061786199999999995 2024-09-21T22:07:20.804 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.804 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.804 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:20.804 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.163021+0000", 2024-09-21T22:07:20.804 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000465749 2024-09-21T22:07:20.804 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.804 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.804 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:20.804 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.163059+0000", 2024-09-21T22:07:20.805 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8238000000000002e-05 2024-09-21T22:07:20.805 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.805 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.805 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:20.805 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.163089+0000", 2024-09-21T22:07:20.805 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0125e-05 2024-09-21T22:07:20.805 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:20.805 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:20.805 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:20.805 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.805 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.805 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4887.0:1 3.3 e59/58 ondisk, result = 0)", 2024-09-21T22:07:20.805 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:17.162939+0000", 2024-09-21T22:07:20.805 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.6342420620000002, 2024-09-21T22:07:20.806 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015730900000000001, 2024-09-21T22:07:20.806 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:20.806 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:20.806 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:07:20.806 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:20.806 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.806 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:20.806 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.162939+0000", 2024-09-21T22:07:20.806 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:20.806 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.806 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.806 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:20.806 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.162939+0000", 2024-09-21T22:07:20.807 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:20.807 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.807 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.807 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:20.807 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.162940+0000", 2024-09-21T22:07:20.807 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4499999999999996e-07 2024-09-21T22:07:20.807 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.807 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.807 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:20.807 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.162945+0000", 2024-09-21T22:07:20.807 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.758e-06 2024-09-21T22:07:20.807 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.807 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.808 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:20.808 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.162957+0000", 2024-09-21T22:07:20.808 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2259000000000001e-05 2024-09-21T22:07:20.808 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.808 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.808 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:20.808 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.162966+0000", 2024-09-21T22:07:20.808 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3270000000000007e-06 2024-09-21T22:07:20.808 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.808 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.808 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:20.808 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.163010+0000", 2024-09-21T22:07:20.808 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.422e-05 2024-09-21T22:07:20.808 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.809 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.809 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:20.809 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.163016+0000", 2024-09-21T22:07:20.809 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8579999999999996e-06 2024-09-21T22:07:20.809 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:20.809 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:20.809 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:20.809 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.163096+0000", 2024-09-21T22:07:20.809 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0142000000000006e-05 2024-09-21T22:07:20.809 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:20.809 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:20.809 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:20.809 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:20.809 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:20.810 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:07:20.811 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:20.808+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:20.812 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T22:07:20.879 INFO:tasks.workunit.client.0.smithi157.stderr:+ DEV=/dev/nbd0 2024-09-21T22:07:20.879 INFO:tasks.workunit.client.0.smithi157.stderr:+ '[' /dev/nbd0 = /dev/nbd0 ']' 2024-09-21T22:07:20.879 INFO:tasks.workunit.client.0.smithi157.stderr:+ rbd device --device-type nbd list 2024-09-21T22:07:20.879 INFO:tasks.workunit.client.0.smithi157.stderr:+ grep testrbdnbd21553 2024-09-21T22:07:20.912 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:20.911+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:20.944 INFO:tasks.workunit.client.0.smithi157.stdout:21953 rbd testrbdnbd21553 - /dev/nbd0 589208db-ca4a-43bb-83a6-52dea926ed7e 2024-09-21T22:07:20.944 INFO:tasks.workunit.client.0.smithi157.stderr:+ get_pid rbd 2024-09-21T22:07:20.945 INFO:tasks.workunit.client.0.smithi157.stderr:+ local pool=rbd 2024-09-21T22:07:20.945 INFO:tasks.workunit.client.0.smithi157.stderr:+ local ns= 2024-09-21T22:07:20.945 INFO:tasks.workunit.client.0.smithi157.stderr:++ rbd device --device-type nbd --format xml list 2024-09-21T22:07:20.945 INFO:tasks.workunit.client.0.smithi157.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd21553'\''][device='\''/dev/nbd0'\'']/id' 2024-09-21T22:07:20.949 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:07:20.949 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:07:20.950 INFO:teuthology.orchestra.run.smithi039.stdout: "num_ops": 0 2024-09-21T22:07:20.950 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:07:20.964 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T22:07:21.013 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:21.012+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:21.017 INFO:tasks.workunit.client.0.smithi157.stderr:+ PID=21953 2024-09-21T22:07:21.018 INFO:tasks.workunit.client.0.smithi157.stderr:+ test -n 21953 2024-09-21T22:07:21.018 INFO:tasks.workunit.client.0.smithi157.stderr:+ ps -p 21953 -C rbd-nbd 2024-09-21T22:07:21.024 INFO:tasks.workunit.client.0.smithi157.stdout: PID TTY TIME CMD 2024-09-21T22:07:21.024 INFO:tasks.workunit.client.0.smithi157.stdout: 21953 ? 00:00:00 rbd-nbd 2024-09-21T22:07:21.025 INFO:tasks.workunit.client.0.smithi157.stderr:++ dd if=/tmp/tmp.GqKuv7rsZi/data bs=1M 2024-09-21T22:07:21.025 INFO:tasks.workunit.client.0.smithi157.stderr:++ md5sum 2024-09-21T22:07:21.113 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:21.111+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:21.125 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:07:21.126 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:07:21.126 INFO:teuthology.orchestra.run.smithi039.stdout: "complaint_time": 30, 2024-09-21T22:07:21.126 INFO:teuthology.orchestra.run.smithi039.stdout: "num_blocked_ops": 0 2024-09-21T22:07:21.126 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:07:21.140 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T22:07:21.147 INFO:tasks.workunit.client.0.smithi157.stderr:64+0 records in 2024-09-21T22:07:21.148 INFO:tasks.workunit.client.0.smithi157.stderr:64+0 records out 2024-09-21T22:07:21.148 INFO:tasks.workunit.client.0.smithi157.stderr:67108864 bytes (67 MB, 64 MiB) copied, 0.121623 s, 552 MB/s 2024-09-21T22:07:21.149 INFO:tasks.workunit.client.0.smithi157.stderr:++ _sudo dd if=/dev/nbd0 bs=1M 2024-09-21T22:07:21.149 INFO:tasks.workunit.client.0.smithi157.stderr:++ local cmd 2024-09-21T22:07:21.149 INFO:tasks.workunit.client.0.smithi157.stderr:++ md5sum 2024-09-21T22:07:21.149 INFO:tasks.workunit.client.0.smithi157.stderr:+++ id -u 2024-09-21T22:07:21.151 INFO:tasks.workunit.client.0.smithi157.stderr:++ '[' 1000 -eq 0 ']' 2024-09-21T22:07:21.151 INFO:tasks.workunit.client.0.smithi157.stderr:+++ which dd 2024-09-21T22:07:21.153 INFO:tasks.workunit.client.0.smithi157.stderr:++ cmd=/usr/bin/dd 2024-09-21T22:07:21.153 INFO:tasks.workunit.client.0.smithi157.stderr:++ shift 2024-09-21T22:07:21.153 INFO:tasks.workunit.client.0.smithi157.stderr:++ sudo -nE /usr/bin/dd if=/dev/nbd0 bs=1M 2024-09-21T22:07:21.214 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:21.212+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:21.315 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:21.311+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:21.318 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:07:21.318 INFO:teuthology.orchestra.run.smithi039.stdout: "size": 20, 2024-09-21T22:07:21.318 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 600, 2024-09-21T22:07:21.319 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [ 2024-09-21T22:07:21.319 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.319 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:38 1.0 e14/13 1:30306672:devicehealth::main.db.0000000000000000:head v 14'24, mlcod=14'22)", 2024-09-21T22:07:21.319 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.180244+0000", 2024-09-21T22:07:21.319 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 79.134163448999999, 2024-09-21T22:07:21.319 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0041712219999999996, 2024-09-21T22:07:21.319 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:21.319 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:21.319 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:21.319 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:21.319 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.319 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:21.319 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.180244+0000", 2024-09-21T22:07:21.320 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:21.320 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.320 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.320 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:21.320 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.180244+0000", 2024-09-21T22:07:21.320 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:21.320 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.320 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.320 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:21.320 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.180246+0000", 2024-09-21T22:07:21.320 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6959999999999999e-06 2024-09-21T22:07:21.320 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.320 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.320 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:21.320 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.180508+0000", 2024-09-21T22:07:21.321 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00026275199999999999 2024-09-21T22:07:21.321 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.321 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.321 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:21.321 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.180538+0000", 2024-09-21T22:07:21.321 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9142999999999999e-05 2024-09-21T22:07:21.321 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.321 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.321 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:21.321 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.180552+0000", 2024-09-21T22:07:21.321 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4466e-05 2024-09-21T22:07:21.321 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.321 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.321 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:21.322 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.180677+0000", 2024-09-21T22:07:21.322 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000125413 2024-09-21T22:07:21.322 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.322 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.322 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:21.322 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.180739+0000", 2024-09-21T22:07:21.322 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.1122000000000002e-05 2024-09-21T22:07:21.322 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.322 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.322 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:21.322 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.184291+0000", 2024-09-21T22:07:21.322 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0035525510000000001 2024-09-21T22:07:21.322 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.322 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.323 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:21.323 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.184415+0000", 2024-09-21T22:07:21.323 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000124079 2024-09-21T22:07:21.323 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:21.323 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:21.323 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:21.323 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.323 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.323 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:57 1.0 e14/13 1:30306672:devicehealth::main.db.0000000000000000:head v 14'35, mlcod=14'33)", 2024-09-21T22:07:21.323 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.221549+0000", 2024-09-21T22:07:21.323 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 79.092858712999998, 2024-09-21T22:07:21.323 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0041720860000000002, 2024-09-21T22:07:21.323 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:21.323 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:21.324 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:21.324 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:21.324 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.324 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:21.324 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.221549+0000", 2024-09-21T22:07:21.324 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:21.324 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.324 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.324 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:21.324 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.221549+0000", 2024-09-21T22:07:21.324 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:21.324 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.324 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.324 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:21.325 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.221550+0000", 2024-09-21T22:07:21.325 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.717e-06 2024-09-21T22:07:21.325 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.325 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.325 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:21.325 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.221673+0000", 2024-09-21T22:07:21.325 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000122612 2024-09-21T22:07:21.325 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.325 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.325 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:21.325 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.221703+0000", 2024-09-21T22:07:21.325 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9539999999999998e-05 2024-09-21T22:07:21.325 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.325 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.326 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:21.326 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.221716+0000", 2024-09-21T22:07:21.326 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3149999999999999e-05 2024-09-21T22:07:21.326 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.326 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.326 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:21.326 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.221848+0000", 2024-09-21T22:07:21.326 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013201900000000001 2024-09-21T22:07:21.326 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.326 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.326 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:21.326 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.221908+0000", 2024-09-21T22:07:21.326 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.0452999999999999e-05 2024-09-21T22:07:21.326 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.327 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.327 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:21.327 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.225573+0000", 2024-09-21T22:07:21.327 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.003664676 2024-09-21T22:07:21.327 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.327 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.327 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:21.327 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.225721+0000", 2024-09-21T22:07:21.327 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00014791899999999999 2024-09-21T22:07:21.327 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:21.327 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:21.327 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:21.327 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.327 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.328 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:71 1.0 e14/13 1:30306672:devicehealth::main.db.0000000000000000:head v 14'45, mlcod=14'43)", 2024-09-21T22:07:21.328 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.256068+0000", 2024-09-21T22:07:21.328 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 79.058338961999993, 2024-09-21T22:07:21.328 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0041712579999999997, 2024-09-21T22:07:21.328 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:21.328 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:21.328 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:21.328 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:21.328 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.328 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:21.328 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.256068+0000", 2024-09-21T22:07:21.328 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:21.328 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.328 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.328 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:21.329 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.256068+0000", 2024-09-21T22:07:21.329 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:21.329 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.329 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.329 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:21.329 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.256070+0000", 2024-09-21T22:07:21.329 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.61e-06 2024-09-21T22:07:21.329 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.329 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.329 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:21.329 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.256129+0000", 2024-09-21T22:07:21.329 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.8702999999999997e-05 2024-09-21T22:07:21.329 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.329 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.330 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:21.330 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.256158+0000", 2024-09-21T22:07:21.330 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9533e-05 2024-09-21T22:07:21.330 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.330 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.330 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:21.330 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.256173+0000", 2024-09-21T22:07:21.330 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4267000000000001e-05 2024-09-21T22:07:21.330 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.330 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.330 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:21.330 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.256304+0000", 2024-09-21T22:07:21.330 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000131352 2024-09-21T22:07:21.331 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.331 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.331 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:21.331 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.256357+0000", 2024-09-21T22:07:21.331 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.3492999999999998e-05 2024-09-21T22:07:21.331 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.331 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.331 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:21.331 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.260147+0000", 2024-09-21T22:07:21.331 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0037900410000000001 2024-09-21T22:07:21.331 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.331 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.331 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:21.331 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.260240+0000", 2024-09-21T22:07:21.331 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.2258999999999999e-05 2024-09-21T22:07:21.332 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:21.332 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:21.332 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:21.332 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.332 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.332 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:72 1.0 e14/13 1:30306672:devicehealth::main.db.0000000000000000:head v 14'46, mlcod=14'43)", 2024-09-21T22:07:21.332 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.258358+0000", 2024-09-21T22:07:21.332 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 79.056048969000003, 2024-09-21T22:07:21.332 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0041492919999999997, 2024-09-21T22:07:21.332 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:21.332 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:21.332 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:21.332 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:21.332 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.333 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:21.333 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.258358+0000", 2024-09-21T22:07:21.333 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:21.333 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.333 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.333 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:21.333 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.258358+0000", 2024-09-21T22:07:21.333 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:21.333 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.333 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.333 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:21.333 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.258360+0000", 2024-09-21T22:07:21.333 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.623e-06 2024-09-21T22:07:21.333 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.333 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.334 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:21.334 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.258415+0000", 2024-09-21T22:07:21.334 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.4756000000000002e-05 2024-09-21T22:07:21.334 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.334 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.334 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:21.334 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.258444+0000", 2024-09-21T22:07:21.334 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9164000000000001e-05 2024-09-21T22:07:21.334 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.334 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.334 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:21.334 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.258459+0000", 2024-09-21T22:07:21.334 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4996e-05 2024-09-21T22:07:21.335 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.335 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.335 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:21.335 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.258637+0000", 2024-09-21T22:07:21.335 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00017796500000000001 2024-09-21T22:07:21.335 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.335 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.335 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:21.335 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.258695+0000", 2024-09-21T22:07:21.335 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.8207e-05 2024-09-21T22:07:21.335 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.335 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.335 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:21.335 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.262421+0000", 2024-09-21T22:07:21.335 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0037261550000000001 2024-09-21T22:07:21.336 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.336 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.336 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:21.336 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.262508+0000", 2024-09-21T22:07:21.336 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.6426000000000002e-05 2024-09-21T22:07:21.336 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:21.336 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:21.336 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:21.336 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.336 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.336 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:114 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'76, mlcod=14'73)", 2024-09-21T22:07:21.336 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.490030+0000", 2024-09-21T22:07:21.336 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 78.824377084999995, 2024-09-21T22:07:21.336 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.004227873, 2024-09-21T22:07:21.336 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:21.337 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:21.337 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:21.337 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:21.337 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.337 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:21.337 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.490030+0000", 2024-09-21T22:07:21.337 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:21.337 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.337 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.337 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:21.337 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.490030+0000", 2024-09-21T22:07:21.337 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:21.337 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.337 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.337 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:21.337 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.490031+0000", 2024-09-21T22:07:21.338 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.3300000000000001e-07 2024-09-21T22:07:21.338 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.338 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.338 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:21.338 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.490042+0000", 2024-09-21T22:07:21.338 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.1095e-05 2024-09-21T22:07:21.338 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.338 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.338 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:21.338 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.490053+0000", 2024-09-21T22:07:21.338 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0501e-05 2024-09-21T22:07:21.338 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.338 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.339 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:21.339 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.490060+0000", 2024-09-21T22:07:21.339 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.9829999999999999e-06 2024-09-21T22:07:21.339 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.339 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.339 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:21.339 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.490107+0000", 2024-09-21T22:07:21.339 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.7278999999999999e-05 2024-09-21T22:07:21.339 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.339 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.339 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:21.339 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.490124+0000", 2024-09-21T22:07:21.339 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7190999999999999e-05 2024-09-21T22:07:21.339 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.340 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.340 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:21.340 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.494229+0000", 2024-09-21T22:07:21.340 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0041050440000000004 2024-09-21T22:07:21.340 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.340 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.340 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:21.340 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.494258+0000", 2024-09-21T22:07:21.340 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9047000000000002e-05 2024-09-21T22:07:21.340 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:21.340 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:21.340 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:21.340 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.340 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.341 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:136 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'94, mlcod=14'89)", 2024-09-21T22:07:21.341 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.903810+0000", 2024-09-21T22:07:21.341 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 78.410597490000001, 2024-09-21T22:07:21.341 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0062746829999999996, 2024-09-21T22:07:21.341 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:21.341 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:21.341 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:21.341 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:21.341 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.341 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:21.341 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.903810+0000", 2024-09-21T22:07:21.341 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:21.341 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.341 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.342 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:21.342 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.903810+0000", 2024-09-21T22:07:21.342 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:21.342 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.342 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.342 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:21.342 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.903810+0000", 2024-09-21T22:07:21.342 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.03e-07 2024-09-21T22:07:21.342 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.342 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.342 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:21.342 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.903814+0000", 2024-09-21T22:07:21.342 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.676e-06 2024-09-21T22:07:21.343 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.343 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.343 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:21.343 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.903822+0000", 2024-09-21T22:07:21.343 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.2879999999999993e-06 2024-09-21T22:07:21.343 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.343 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.343 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:21.343 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.903829+0000", 2024-09-21T22:07:21.343 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.2920000000000001e-06 2024-09-21T22:07:21.343 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.343 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.343 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:21.343 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.903879+0000", 2024-09-21T22:07:21.344 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.0253999999999997e-05 2024-09-21T22:07:21.344 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.344 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.344 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:21.344 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.903897+0000", 2024-09-21T22:07:21.344 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8468999999999999e-05 2024-09-21T22:07:21.344 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.344 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.344 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:21.344 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.910044+0000", 2024-09-21T22:07:21.344 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0061468950000000003 2024-09-21T22:07:21.344 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.344 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.344 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:21.345 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.910085+0000", 2024-09-21T22:07:21.345 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.0405999999999999e-05 2024-09-21T22:07:21.345 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:21.345 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:21.345 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:21.345 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.345 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.345 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:137 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'95, mlcod=14'90)", 2024-09-21T22:07:21.345 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.904439+0000", 2024-09-21T22:07:21.345 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 78.409968094000007, 2024-09-21T22:07:21.345 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0072625019999999997, 2024-09-21T22:07:21.345 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:21.345 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:21.345 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:21.345 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:21.346 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.346 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:21.346 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.904439+0000", 2024-09-21T22:07:21.346 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:21.346 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.346 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.346 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:21.346 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.904439+0000", 2024-09-21T22:07:21.346 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:21.346 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.346 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.346 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:21.346 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.904440+0000", 2024-09-21T22:07:21.347 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8400000000000005e-07 2024-09-21T22:07:21.347 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.347 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.347 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:21.347 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.904498+0000", 2024-09-21T22:07:21.347 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.8320999999999999e-05 2024-09-21T22:07:21.347 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.347 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.347 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:21.347 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.904507+0000", 2024-09-21T22:07:21.347 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.7790000000000001e-06 2024-09-21T22:07:21.347 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.347 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.347 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:21.348 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.904511+0000", 2024-09-21T22:07:21.348 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.9700000000000001e-06 2024-09-21T22:07:21.348 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.348 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.348 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:21.348 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.904551+0000", 2024-09-21T22:07:21.348 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.9910000000000002e-05 2024-09-21T22:07:21.348 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.348 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.348 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:21.348 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.904567+0000", 2024-09-21T22:07:21.348 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5903999999999999e-05 2024-09-21T22:07:21.348 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.348 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.348 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:21.349 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.911666+0000", 2024-09-21T22:07:21.349 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0070996929999999998 2024-09-21T22:07:21.349 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.349 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.349 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:21.349 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.911702+0000", 2024-09-21T22:07:21.349 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.5441000000000002e-05 2024-09-21T22:07:21.349 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:21.349 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:21.349 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:21.349 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.349 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.349 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:138 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'96, mlcod=14'90)", 2024-09-21T22:07:21.350 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.905402+0000", 2024-09-21T22:07:21.350 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 78.409005117000007, 2024-09-21T22:07:21.350 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0063366890000000004, 2024-09-21T22:07:21.350 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:21.350 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:21.350 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:21.350 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:21.350 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.350 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:21.350 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905402+0000", 2024-09-21T22:07:21.350 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:21.350 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.350 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.351 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:21.351 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905402+0000", 2024-09-21T22:07:21.351 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:21.351 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.351 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.351 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:21.351 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905403+0000", 2024-09-21T22:07:21.351 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.9200000000000002e-07 2024-09-21T22:07:21.351 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.351 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.351 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:21.351 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905406+0000", 2024-09-21T22:07:21.351 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.326e-06 2024-09-21T22:07:21.351 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.352 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.352 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:21.352 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905413+0000", 2024-09-21T22:07:21.352 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.3320000000000003e-06 2024-09-21T22:07:21.352 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.352 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.352 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:21.352 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905417+0000", 2024-09-21T22:07:21.352 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.7139999999999999e-06 2024-09-21T22:07:21.352 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.352 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.352 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:21.352 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.910115+0000", 2024-09-21T22:07:21.353 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0046983939999999998 2024-09-21T22:07:21.353 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.353 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.353 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:21.353 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.910140+0000", 2024-09-21T22:07:21.353 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4689999999999999e-05 2024-09-21T22:07:21.353 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.353 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.353 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:21.353 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.911706+0000", 2024-09-21T22:07:21.353 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0015658479999999999 2024-09-21T22:07:21.353 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.353 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.353 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:21.354 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.911739+0000", 2024-09-21T22:07:21.354 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.2993000000000001e-05 2024-09-21T22:07:21.354 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:21.354 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:21.354 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:21.354 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.354 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.354 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:139 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'97, mlcod=14'90)", 2024-09-21T22:07:21.354 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.905983+0000", 2024-09-21T22:07:21.354 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 78.408424198999995, 2024-09-21T22:07:21.354 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0057906030000000001, 2024-09-21T22:07:21.354 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:21.354 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:21.354 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:21.354 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:21.355 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.355 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:21.355 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905983+0000", 2024-09-21T22:07:21.355 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:21.355 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.355 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.355 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:21.355 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905983+0000", 2024-09-21T22:07:21.355 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:21.355 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.355 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.355 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:21.355 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905984+0000", 2024-09-21T22:07:21.356 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.15e-07 2024-09-21T22:07:21.356 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.356 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.356 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:21.356 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905986+0000", 2024-09-21T22:07:21.356 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.852e-06 2024-09-21T22:07:21.356 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.356 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.356 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:21.356 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905993+0000", 2024-09-21T22:07:21.356 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.652e-06 2024-09-21T22:07:21.356 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.356 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.356 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:21.357 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.905996+0000", 2024-09-21T22:07:21.357 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.3519999999999998e-06 2024-09-21T22:07:21.357 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.357 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.357 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:21.357 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.910519+0000", 2024-09-21T22:07:21.357 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0045230820000000003 2024-09-21T22:07:21.357 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.357 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.357 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:21.357 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.910548+0000", 2024-09-21T22:07:21.357 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8399999999999999e-05 2024-09-21T22:07:21.357 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.357 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.358 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:21.358 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.911742+0000", 2024-09-21T22:07:21.358 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011939909999999999 2024-09-21T22:07:21.358 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.358 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.358 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:21.358 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.911774+0000", 2024-09-21T22:07:21.358 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.1958999999999997e-05 2024-09-21T22:07:21.358 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:21.358 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:21.358 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:21.358 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.358 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.359 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4195.0:140 1.0 e14/13 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 14'98, mlcod=14'90)", 2024-09-21T22:07:21.359 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:06:02.906441+0000", 2024-09-21T22:07:21.359 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 78.407966070000001, 2024-09-21T22:07:21.359 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0057052989999999996, 2024-09-21T22:07:21.359 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:21.359 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:21.359 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:21.359 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:21.359 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.359 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:21.359 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.906441+0000", 2024-09-21T22:07:21.359 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:21.359 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.359 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.360 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:21.360 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.906441+0000", 2024-09-21T22:07:21.360 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:21.360 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.360 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.360 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:21.360 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.906442+0000", 2024-09-21T22:07:21.360 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.7e-07 2024-09-21T22:07:21.360 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.360 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.360 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:21.360 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.906511+0000", 2024-09-21T22:07:21.360 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.8857999999999997e-05 2024-09-21T22:07:21.361 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.361 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.361 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:21.361 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.906518+0000", 2024-09-21T22:07:21.361 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.1690000000000003e-06 2024-09-21T22:07:21.361 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.361 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.361 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:21.361 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.906521+0000", 2024-09-21T22:07:21.361 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.4829999999999999e-06 2024-09-21T22:07:21.361 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.361 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.361 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:21.361 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.910939+0000", 2024-09-21T22:07:21.362 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0044176809999999997 2024-09-21T22:07:21.362 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.362 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.362 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:21.362 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.910959+0000", 2024-09-21T22:07:21.362 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9898000000000002e-05 2024-09-21T22:07:21.362 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.362 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.362 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:21.362 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.912113+0000", 2024-09-21T22:07:21.362 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001153811 2024-09-21T22:07:21.362 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.362 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.362 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:21.363 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:06:02.912147+0000", 2024-09-21T22:07:21.363 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.4029000000000002e-05 2024-09-21T22:07:21.363 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:21.363 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:21.363 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:21.363 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.363 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.363 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.1 54/53 [PushOp(2:837ddd83:::rbd_data.126483d75f2d.000000000000000d:head, version: 52'1, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:837ddd83:::rbd_data.126483d75f2d.000000000000000d:head@52'1, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T22:07:21.363 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:11.071135+0000", 2024-09-21T22:07:21.363 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 10.243271958999999, 2024-09-21T22:07:21.363 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.025278432, 2024-09-21T22:07:21.363 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:21.363 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:21.363 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:21.364 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:21.364 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.364 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:21.364 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.071135+0000", 2024-09-21T22:07:21.364 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:21.364 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.364 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.364 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:21.364 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.071135+0000", 2024-09-21T22:07:21.364 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:21.364 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.364 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.364 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:21.365 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.071137+0000", 2024-09-21T22:07:21.365 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.0430000000000002e-06 2024-09-21T22:07:21.365 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.365 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.365 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:21.365 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.071783+0000", 2024-09-21T22:07:21.365 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00064543399999999996 2024-09-21T22:07:21.365 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.365 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.365 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:21.365 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.071820+0000", 2024-09-21T22:07:21.365 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.7588e-05 2024-09-21T22:07:21.365 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.365 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.365 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:21.366 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.071846+0000", 2024-09-21T22:07:21.366 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5673999999999998e-05 2024-09-21T22:07:21.366 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.366 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.366 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:21.366 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.071967+0000", 2024-09-21T22:07:21.366 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00012071899999999999 2024-09-21T22:07:21.366 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.366 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.366 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:21.366 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.071983+0000", 2024-09-21T22:07:21.366 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6013999999999999e-05 2024-09-21T22:07:21.367 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.367 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.367 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:21.367 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.096414+0000", 2024-09-21T22:07:21.367 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.024430960000000002 2024-09-21T22:07:21.367 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:21.367 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:21.367 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:21.367 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.367 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.367 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.1 54/53 [PushOp(2:837ddd83:::rbd_data.126483d75f2d.000000000000000d:head, version: 52'1, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:837ddd83:::rbd_data.126483d75f2d.000000000000000d:head@52'1, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:07:21.367 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:11.099645+0000", 2024-09-21T22:07:21.367 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 10.214762398, 2024-09-21T22:07:21.367 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.026929337000000001, 2024-09-21T22:07:21.368 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:21.368 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:21.368 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:21.368 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:21.368 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.368 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:21.368 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.099645+0000", 2024-09-21T22:07:21.368 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:21.368 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.368 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.368 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:21.368 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.099645+0000", 2024-09-21T22:07:21.368 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:21.369 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.369 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.369 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:21.369 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.099646+0000", 2024-09-21T22:07:21.369 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.221e-06 2024-09-21T22:07:21.369 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.369 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.369 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:21.369 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.100121+0000", 2024-09-21T22:07:21.369 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00047480500000000002 2024-09-21T22:07:21.369 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.369 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.369 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:21.369 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.100144+0000", 2024-09-21T22:07:21.370 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.2812000000000001e-05 2024-09-21T22:07:21.370 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.370 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.370 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:21.370 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.100178+0000", 2024-09-21T22:07:21.370 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.4379999999999999e-05 2024-09-21T22:07:21.370 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.370 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.370 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:21.370 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.100313+0000", 2024-09-21T22:07:21.370 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013434499999999999 2024-09-21T22:07:21.370 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.370 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.371 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:21.371 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.100323+0000", 2024-09-21T22:07:21.371 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0930999999999999e-05 2024-09-21T22:07:21.371 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.371 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.371 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:21.371 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.126574+0000", 2024-09-21T22:07:21.371 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.026250842999999999 2024-09-21T22:07:21.371 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:21.371 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:21.371 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:21.371 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.371 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.371 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.1 54/53 [PushOp(2:837ddd83:::rbd_data.126483d75f2d.000000000000000d:head, version: 52'1, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:837ddd83:::rbd_data.126483d75f2d.000000000000000d:head@52'1, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:07:21.372 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:11.129410+0000", 2024-09-21T22:07:21.372 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 10.184997214999999, 2024-09-21T22:07:21.372 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.027274217, 2024-09-21T22:07:21.372 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:21.372 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:21.372 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:21.372 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:21.372 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.372 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:21.372 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.129410+0000", 2024-09-21T22:07:21.372 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:21.372 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.372 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.372 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:21.372 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.129410+0000", 2024-09-21T22:07:21.373 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:21.373 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.373 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.373 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:21.373 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.129412+0000", 2024-09-21T22:07:21.373 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6160000000000001e-06 2024-09-21T22:07:21.373 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.373 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.373 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:21.373 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.130407+0000", 2024-09-21T22:07:21.373 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00099554299999999999 2024-09-21T22:07:21.373 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.373 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.374 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:21.374 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.130454+0000", 2024-09-21T22:07:21.374 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.6545000000000002e-05 2024-09-21T22:07:21.374 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.374 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.374 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:21.374 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.130508+0000", 2024-09-21T22:07:21.374 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.3940999999999997e-05 2024-09-21T22:07:21.374 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.374 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.374 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:21.374 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.130658+0000", 2024-09-21T22:07:21.374 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000149732 2024-09-21T22:07:21.374 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.375 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.375 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:21.375 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.130673+0000", 2024-09-21T22:07:21.375 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5608e-05 2024-09-21T22:07:21.375 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.375 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.375 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:21.375 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.156684+0000", 2024-09-21T22:07:21.375 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.026011231999999999 2024-09-21T22:07:21.375 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:21.375 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:21.375 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:21.375 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.375 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.375 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.1 54/53 [PushOp(2:837ddd83:::rbd_data.126483d75f2d.000000000000000d:head, version: 52'1, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:837ddd83:::rbd_data.126483d75f2d.000000000000000d:head@52'1, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 4194304, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:07:21.376 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:11.159382+0000", 2024-09-21T22:07:21.376 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 10.155025826999999, 2024-09-21T22:07:21.376 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.026585683999999998, 2024-09-21T22:07:21.376 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:21.376 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:21.376 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:21.376 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:21.376 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.376 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:21.376 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.159382+0000", 2024-09-21T22:07:21.376 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:21.376 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.376 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.376 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:21.377 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.159382+0000", 2024-09-21T22:07:21.377 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:21.377 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.377 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.377 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:21.377 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.159383+0000", 2024-09-21T22:07:21.377 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.544e-06 2024-09-21T22:07:21.377 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.377 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.377 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:21.377 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.160369+0000", 2024-09-21T22:07:21.377 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00098620700000000001 2024-09-21T22:07:21.377 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.377 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.378 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:21.378 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.160415+0000", 2024-09-21T22:07:21.378 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.6074e-05 2024-09-21T22:07:21.378 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.378 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.378 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:21.378 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.160463+0000", 2024-09-21T22:07:21.378 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8087999999999998e-05 2024-09-21T22:07:21.378 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.378 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.378 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:21.378 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.160577+0000", 2024-09-21T22:07:21.378 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000113893 2024-09-21T22:07:21.378 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.379 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.379 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:21.379 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.160592+0000", 2024-09-21T22:07:21.379 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.465e-05 2024-09-21T22:07:21.379 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.379 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.379 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:21.379 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.185967+0000", 2024-09-21T22:07:21.379 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.025375228 2024-09-21T22:07:21.379 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:21.379 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:21.379 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:21.379 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.379 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.380 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.1 54/53 [PushOp(2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head, version: 52'2, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head@52'2, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T22:07:21.380 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:11.189894+0000", 2024-09-21T22:07:21.380 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 10.124513862000001, 2024-09-21T22:07:21.380 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.025252653, 2024-09-21T22:07:21.380 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:21.380 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:21.380 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:21.380 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:21.380 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.380 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:21.380 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.189894+0000", 2024-09-21T22:07:21.380 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:21.380 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.380 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.381 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:21.381 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.189894+0000", 2024-09-21T22:07:21.381 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:21.381 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.381 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.381 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:21.381 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.189895+0000", 2024-09-21T22:07:21.381 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.916e-06 2024-09-21T22:07:21.381 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.381 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.381 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:21.381 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.190890+0000", 2024-09-21T22:07:21.381 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00099456900000000005 2024-09-21T22:07:21.381 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.382 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.382 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:21.382 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.190939+0000", 2024-09-21T22:07:21.382 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.9132000000000003e-05 2024-09-21T22:07:21.382 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.382 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.382 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:21.382 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.190989+0000", 2024-09-21T22:07:21.382 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.9913999999999997e-05 2024-09-21T22:07:21.382 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.382 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.382 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:21.382 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.191122+0000", 2024-09-21T22:07:21.383 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013264 2024-09-21T22:07:21.383 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.383 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.383 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:21.383 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.191137+0000", 2024-09-21T22:07:21.383 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4871999999999999e-05 2024-09-21T22:07:21.383 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.383 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.383 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:21.383 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.215146+0000", 2024-09-21T22:07:21.383 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.024009610000000001 2024-09-21T22:07:21.383 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:21.383 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:21.383 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:21.383 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.384 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.384 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.1 54/53 [PushOp(2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head, version: 52'2, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head@52'2, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:07:21.384 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:11.217669+0000", 2024-09-21T22:07:21.384 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 10.096738421, 2024-09-21T22:07:21.384 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.024843860999999998, 2024-09-21T22:07:21.384 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:21.384 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:21.384 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:21.384 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:21.384 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.384 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:21.384 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.217669+0000", 2024-09-21T22:07:21.384 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:21.385 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.385 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.385 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:21.385 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.217669+0000", 2024-09-21T22:07:21.385 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:21.385 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.385 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.385 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:21.385 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.217671+0000", 2024-09-21T22:07:21.385 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7129999999999999e-06 2024-09-21T22:07:21.385 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.385 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.385 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:21.385 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.218680+0000", 2024-09-21T22:07:21.386 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0010089630000000001 2024-09-21T22:07:21.386 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.386 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.386 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:21.386 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.218726+0000", 2024-09-21T22:07:21.386 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.6307999999999998e-05 2024-09-21T22:07:21.386 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.386 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.386 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:21.386 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.218779+0000", 2024-09-21T22:07:21.386 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.3344999999999998e-05 2024-09-21T22:07:21.386 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.386 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.387 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:21.387 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.218907+0000", 2024-09-21T22:07:21.387 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00012746799999999999 2024-09-21T22:07:21.387 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.387 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.387 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:21.387 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.218921+0000", 2024-09-21T22:07:21.387 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4581e-05 2024-09-21T22:07:21.387 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.387 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.387 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:21.387 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.242513+0000", 2024-09-21T22:07:21.387 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.023591483 2024-09-21T22:07:21.387 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:21.388 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:21.388 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:21.388 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.388 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.388 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.1 54/53 [PushOp(2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head, version: 52'2, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head@52'2, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:07:21.388 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:11.245267+0000", 2024-09-21T22:07:21.388 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 10.069140639, 2024-09-21T22:07:21.388 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.025143901, 2024-09-21T22:07:21.388 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:21.388 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:21.388 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:21.388 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:21.388 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.388 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:21.389 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.245267+0000", 2024-09-21T22:07:21.389 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:21.389 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.389 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.389 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:21.389 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.245267+0000", 2024-09-21T22:07:21.389 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:21.389 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.389 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.389 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:21.389 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.245269+0000", 2024-09-21T22:07:21.389 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5299999999999999e-06 2024-09-21T22:07:21.389 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.389 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.389 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:21.390 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.246497+0000", 2024-09-21T22:07:21.390 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00122766 2024-09-21T22:07:21.390 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.390 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.390 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:21.390 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.246545+0000", 2024-09-21T22:07:21.390 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.7757999999999999e-05 2024-09-21T22:07:21.390 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.390 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.390 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:21.390 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.246579+0000", 2024-09-21T22:07:21.390 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.4186000000000001e-05 2024-09-21T22:07:21.390 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.390 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.391 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:21.391 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.246712+0000", 2024-09-21T22:07:21.391 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013312099999999999 2024-09-21T22:07:21.391 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.391 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.391 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:21.391 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.246722+0000", 2024-09-21T22:07:21.391 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.7869999999999996e-06 2024-09-21T22:07:21.391 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.391 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.391 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:21.391 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.270411+0000", 2024-09-21T22:07:21.391 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.023688859 2024-09-21T22:07:21.391 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:21.392 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:21.392 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:21.392 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.392 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.392 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.1 54/53 [PushOp(2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head, version: 52'2, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head@52'2, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 4194304, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:07:21.392 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:11.273163+0000", 2024-09-21T22:07:21.392 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 10.04124395, 2024-09-21T22:07:21.392 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.025286874000000001, 2024-09-21T22:07:21.392 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:21.392 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:21.392 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:21.392 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:21.392 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.392 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:21.393 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.273163+0000", 2024-09-21T22:07:21.393 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:21.393 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.393 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.393 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:21.393 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.273163+0000", 2024-09-21T22:07:21.393 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:21.393 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.393 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.393 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:21.393 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.273166+0000", 2024-09-21T22:07:21.393 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5969999999999999e-06 2024-09-21T22:07:21.393 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.393 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.393 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:21.394 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.274383+0000", 2024-09-21T22:07:21.394 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001217274 2024-09-21T22:07:21.394 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.394 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.394 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:21.394 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.274430+0000", 2024-09-21T22:07:21.394 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.6848000000000003e-05 2024-09-21T22:07:21.394 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.394 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.394 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:21.394 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.274465+0000", 2024-09-21T22:07:21.394 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.5231999999999997e-05 2024-09-21T22:07:21.394 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.394 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.395 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:21.395 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.274561+0000", 2024-09-21T22:07:21.395 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.5962e-05 2024-09-21T22:07:21.395 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.395 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.395 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:21.395 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.274571+0000", 2024-09-21T22:07:21.395 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.5449999999999992e-06 2024-09-21T22:07:21.395 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.395 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.395 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:21.395 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.298450+0000", 2024-09-21T22:07:21.395 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.023879416000000001 2024-09-21T22:07:21.396 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:21.396 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:21.396 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:21.396 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.396 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.396 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(1.0 55/54 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 15'116, data_included: [0~262688], data_size: 262688, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@15'116, size: 262688, copy_subset: [(0, 262688)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 262688, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T22:07:21.396 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.155955+0000", 2024-09-21T22:07:21.396 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 9.1584520489999992, 2024-09-21T22:07:21.396 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.004903368, 2024-09-21T22:07:21.396 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:21.396 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:21.396 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:21.396 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:21.396 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.397 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:21.397 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.155955+0000", 2024-09-21T22:07:21.397 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:21.397 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.397 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.397 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:21.397 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.155955+0000", 2024-09-21T22:07:21.397 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:21.397 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.397 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.397 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:21.397 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.155957+0000", 2024-09-21T22:07:21.397 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5829999999999999e-06 2024-09-21T22:07:21.397 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.398 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.398 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:21.398 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.156375+0000", 2024-09-21T22:07:21.398 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00041813700000000001 2024-09-21T22:07:21.398 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.398 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.398 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:21.398 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.156420+0000", 2024-09-21T22:07:21.398 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.4734999999999998e-05 2024-09-21T22:07:21.398 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.398 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.398 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:21.398 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.156452+0000", 2024-09-21T22:07:21.399 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.1985e-05 2024-09-21T22:07:21.399 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.399 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.399 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:21.399 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.156629+0000", 2024-09-21T22:07:21.399 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000177107 2024-09-21T22:07:21.399 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.399 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.399 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:21.399 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.156657+0000", 2024-09-21T22:07:21.399 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.7871000000000001e-05 2024-09-21T22:07:21.399 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.399 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.399 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:21.400 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.160859+0000", 2024-09-21T22:07:21.400 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0042019500000000003 2024-09-21T22:07:21.400 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:21.400 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:21.400 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:21.400 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.400 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.400 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(1.0 55/54 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 15'117, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@15'117, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T22:07:21.400 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.162929+0000", 2024-09-21T22:07:21.400 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 9.1514788249999999, 2024-09-21T22:07:21.400 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0056463110000000002, 2024-09-21T22:07:21.400 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:21.400 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:21.400 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:21.401 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:21.401 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.401 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:21.401 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.162929+0000", 2024-09-21T22:07:21.401 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:21.401 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.401 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.401 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:21.401 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.162929+0000", 2024-09-21T22:07:21.401 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:21.401 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.401 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.401 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:21.401 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.162930+0000", 2024-09-21T22:07:21.402 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.77e-06 2024-09-21T22:07:21.402 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.402 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.402 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:21.402 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.163405+0000", 2024-09-21T22:07:21.402 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00047464499999999999 2024-09-21T22:07:21.402 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.402 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.402 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:21.402 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.163453+0000", 2024-09-21T22:07:21.402 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8211999999999999e-05 2024-09-21T22:07:21.402 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.402 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.402 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:21.403 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.163485+0000", 2024-09-21T22:07:21.403 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.2011000000000002e-05 2024-09-21T22:07:21.403 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.403 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.403 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:21.403 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.163662+0000", 2024-09-21T22:07:21.403 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000177059 2024-09-21T22:07:21.403 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.403 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.403 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:21.403 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.163688+0000", 2024-09-21T22:07:21.403 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5831000000000001e-05 2024-09-21T22:07:21.403 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.403 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.404 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:21.404 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.168575+0000", 2024-09-21T22:07:21.404 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0048867829999999996 2024-09-21T22:07:21.404 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:21.404 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:21.404 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:21.404 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:21.404 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:21.404 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:07:21.405 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T22:07:21.416 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:21.412+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:21.517 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:21.514+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:21.579 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:07:21.579 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [ 2024-09-21T22:07:21.579 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.579 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4840.0:328 2.3 2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head [sparse-read 397312~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-21T22:07:21.579 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:21.577435+0000", 2024-09-21T22:07:21.579 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.00064121100000000004, 2024-09-21T22:07:21.579 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00069074900000000005, 2024-09-21T22:07:21.579 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:21.580 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:21.580 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:21.580 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-21T22:07:21.580 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4840", 2024-09-21T22:07:21.580 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:21.580 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 328 2024-09-21T22:07:21.580 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.580 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:21.580 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.580 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:21.580 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.577435+0000", 2024-09-21T22:07:21.580 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:21.580 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.580 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.581 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:21.581 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.577435+0000", 2024-09-21T22:07:21.581 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:21.581 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.581 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.581 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:21.581 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.577437+0000", 2024-09-21T22:07:21.581 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1440000000000001e-06 2024-09-21T22:07:21.581 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.581 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.581 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:21.581 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.577447+0000", 2024-09-21T22:07:21.581 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.6700000000000006e-06 2024-09-21T22:07:21.582 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.582 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.582 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:21.582 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.577478+0000", 2024-09-21T22:07:21.582 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.1182000000000001e-05 2024-09-21T22:07:21.582 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.582 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.582 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:21.582 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.577492+0000", 2024-09-21T22:07:21.582 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4559000000000001e-05 2024-09-21T22:07:21.582 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.582 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.582 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:21.582 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.577613+0000", 2024-09-21T22:07:21.582 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000120147 2024-09-21T22:07:21.583 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.583 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.583 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:21.583 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.577733+0000", 2024-09-21T22:07:21.583 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000120213 2024-09-21T22:07:21.583 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:21.583 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:21.583 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:21.583 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:21.583 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T22:07:21.583 INFO:teuthology.orchestra.run.smithi039.stdout: "num_ops": 1 2024-09-21T22:07:21.583 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:07:21.595 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T22:07:21.640 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:21.638+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:21.719 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:21.714+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:21.756 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:07:21.752+0000 7f2078a4a7c0 -1 Falling back to public interface 2024-09-21T22:07:21.769 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:07:21.769 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:07:21.769 INFO:teuthology.orchestra.run.smithi039.stdout: "complaint_time": 30, 2024-09-21T22:07:21.769 INFO:teuthology.orchestra.run.smithi039.stdout: "num_blocked_ops": 0 2024-09-21T22:07:21.769 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:07:21.783 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T22:07:21.807 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T22:07:21.820 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:21.816+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:21.921 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:21.920+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:21.958 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:07:21.958 INFO:teuthology.orchestra.run.smithi039.stdout: "size": 20, 2024-09-21T22:07:21.958 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 600, 2024-09-21T22:07:21.958 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [ 2024-09-21T22:07:21.959 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.959 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.3 55/54 [PushOp(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head, version: 52'2, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head@52'2, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T22:07:21.959 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.084379+0000", 2024-09-21T22:07:21.959 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 9.8698485229999999, 2024-09-21T22:07:21.959 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.051020674000000002, 2024-09-21T22:07:21.959 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:21.959 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:21.959 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:21.959 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:21.959 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.959 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:21.959 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.084379+0000", 2024-09-21T22:07:21.959 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:21.959 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.960 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.960 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:21.960 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.084379+0000", 2024-09-21T22:07:21.960 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:21.960 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.960 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.960 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:21.960 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.084381+0000", 2024-09-21T22:07:21.960 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.73e-06 2024-09-21T22:07:21.960 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.960 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.960 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:21.960 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.085494+0000", 2024-09-21T22:07:21.960 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011131559999999999 2024-09-21T22:07:21.961 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.961 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.961 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:21.961 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.085542+0000", 2024-09-21T22:07:21.961 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8334999999999997e-05 2024-09-21T22:07:21.961 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.961 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.961 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:21.961 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.085591+0000", 2024-09-21T22:07:21.961 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.9165000000000001e-05 2024-09-21T22:07:21.961 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.961 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.961 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:21.961 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.085777+0000", 2024-09-21T22:07:21.962 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000186152 2024-09-21T22:07:21.962 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.962 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.962 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:21.962 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.085796+0000", 2024-09-21T22:07:21.962 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8612999999999999e-05 2024-09-21T22:07:21.962 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.962 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.962 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:21.962 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.135399+0000", 2024-09-21T22:07:21.963 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.049603522999999997 2024-09-21T22:07:21.963 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:21.963 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:21.963 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:21.963 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.963 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.963 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.3 55/54 [PushOp(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head, version: 52'2, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head@52'2, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:07:21.963 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.139554+0000", 2024-09-21T22:07:21.963 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 9.8146734200000001, 2024-09-21T22:07:21.963 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.056215389999999997, 2024-09-21T22:07:21.963 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:21.963 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:21.963 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:21.963 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:21.964 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.964 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:21.964 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.139554+0000", 2024-09-21T22:07:21.964 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:21.964 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.964 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.964 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:21.964 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.139554+0000", 2024-09-21T22:07:21.964 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:21.964 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.964 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.964 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:21.964 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.139556+0000", 2024-09-21T22:07:21.964 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7999999999999999e-06 2024-09-21T22:07:21.965 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.965 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.965 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:21.965 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.140671+0000", 2024-09-21T22:07:21.965 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001115612 2024-09-21T22:07:21.965 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.965 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.965 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:21.965 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.140720+0000", 2024-09-21T22:07:21.965 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8631999999999998e-05 2024-09-21T22:07:21.965 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.965 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.965 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:21.966 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.140752+0000", 2024-09-21T22:07:21.966 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.2199000000000001e-05 2024-09-21T22:07:21.966 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.966 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.966 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:21.966 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.140900+0000", 2024-09-21T22:07:21.966 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00014786000000000001 2024-09-21T22:07:21.966 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.966 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.966 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:21.966 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.140917+0000", 2024-09-21T22:07:21.966 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6753999999999999e-05 2024-09-21T22:07:21.967 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.967 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.967 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:21.967 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.195769+0000", 2024-09-21T22:07:21.967 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.054852533000000002 2024-09-21T22:07:21.967 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:21.967 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:21.967 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:21.967 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.967 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.967 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.3 55/54 [PushOp(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head, version: 52'2, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head@52'2, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:07:21.967 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.199187+0000", 2024-09-21T22:07:21.967 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 9.7550398759999997, 2024-09-21T22:07:21.967 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.051543975999999998, 2024-09-21T22:07:21.968 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:21.968 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:21.968 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:21.968 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:21.968 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.968 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:21.968 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.199187+0000", 2024-09-21T22:07:21.968 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:21.968 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.968 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.968 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:21.968 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.199187+0000", 2024-09-21T22:07:21.968 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:21.969 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.969 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.969 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:21.969 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.199189+0000", 2024-09-21T22:07:21.969 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7969999999999999e-06 2024-09-21T22:07:21.969 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.969 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.969 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:21.969 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.200355+0000", 2024-09-21T22:07:21.969 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011660139999999999 2024-09-21T22:07:21.969 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.970 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.970 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:21.970 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.200405+0000", 2024-09-21T22:07:21.970 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.0124000000000003e-05 2024-09-21T22:07:21.970 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.970 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.970 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:21.970 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.200457+0000", 2024-09-21T22:07:21.970 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.1177999999999997e-05 2024-09-21T22:07:21.970 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.970 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.970 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:21.971 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.200635+0000", 2024-09-21T22:07:21.971 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00017841299999999999 2024-09-21T22:07:21.971 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.971 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.971 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:21.971 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.200653+0000", 2024-09-21T22:07:21.971 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7944999999999998e-05 2024-09-21T22:07:21.971 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.971 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.971 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:21.971 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.250731+0000", 2024-09-21T22:07:21.971 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.050078505000000002 2024-09-21T22:07:21.971 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:21.971 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:21.972 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:21.972 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.972 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.972 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.3 55/54 [PushOp(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head, version: 52'2, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head@52'2, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 4194304, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:07:21.972 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.253966+0000", 2024-09-21T22:07:21.972 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 9.7002615460000001, 2024-09-21T22:07:21.972 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.041100606999999997, 2024-09-21T22:07:21.972 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:21.972 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:21.972 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:21.972 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:21.972 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.972 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:21.972 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.253966+0000", 2024-09-21T22:07:21.973 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:21.973 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.973 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.973 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:21.973 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.253966+0000", 2024-09-21T22:07:21.973 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:21.973 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.973 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.973 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:21.973 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.253967+0000", 2024-09-21T22:07:21.973 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.423e-06 2024-09-21T22:07:21.973 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.973 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.973 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:21.974 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.255117+0000", 2024-09-21T22:07:21.974 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011496709999999999 2024-09-21T22:07:21.974 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.974 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.974 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:21.974 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.255167+0000", 2024-09-21T22:07:21.974 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.0315000000000002e-05 2024-09-21T22:07:21.974 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.974 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.974 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:21.974 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.255218+0000", 2024-09-21T22:07:21.974 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.0390999999999999e-05 2024-09-21T22:07:21.974 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.975 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.975 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:21.975 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.255391+0000", 2024-09-21T22:07:21.975 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00017332799999999999 2024-09-21T22:07:21.975 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.975 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.975 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:21.975 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.255405+0000", 2024-09-21T22:07:21.975 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3723999999999999e-05 2024-09-21T22:07:21.975 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.975 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.975 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:21.975 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.295066+0000", 2024-09-21T22:07:21.975 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.039661755 2024-09-21T22:07:21.975 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:21.976 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:21.976 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:21.976 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.976 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.976 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.3 55/54 [PushOp(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head, version: 52'3, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head@52'3, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T22:07:21.976 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.299195+0000", 2024-09-21T22:07:21.976 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 9.6550327809999992, 2024-09-21T22:07:21.976 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.040955077999999999, 2024-09-21T22:07:21.976 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:21.976 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:21.976 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:21.976 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:21.976 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.976 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:21.977 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.299195+0000", 2024-09-21T22:07:21.977 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:21.977 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.977 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.977 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:21.977 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.299195+0000", 2024-09-21T22:07:21.977 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:21.977 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.977 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.977 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:21.977 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.299196+0000", 2024-09-21T22:07:21.977 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.604e-06 2024-09-21T22:07:21.977 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.978 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.978 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:21.978 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.300356+0000", 2024-09-21T22:07:21.978 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011602699999999999 2024-09-21T22:07:21.978 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.978 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.978 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:21.978 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.300406+0000", 2024-09-21T22:07:21.978 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.9144999999999998e-05 2024-09-21T22:07:21.978 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.978 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.978 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:21.978 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.300435+0000", 2024-09-21T22:07:21.978 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9886000000000002e-05 2024-09-21T22:07:21.979 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.979 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.979 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:21.979 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.300590+0000", 2024-09-21T22:07:21.979 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00015431200000000001 2024-09-21T22:07:21.979 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.979 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.979 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:21.979 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.300604+0000", 2024-09-21T22:07:21.979 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4559000000000001e-05 2024-09-21T22:07:21.979 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.979 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.979 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:21.979 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.340150+0000", 2024-09-21T22:07:21.979 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.039545301999999997 2024-09-21T22:07:21.980 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:21.980 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:21.980 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:21.980 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.980 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.981 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.3 55/54 [PushOp(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head, version: 52'3, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head@52'3, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:07:21.981 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.343754+0000", 2024-09-21T22:07:21.981 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 9.6104731650000002, 2024-09-21T22:07:21.981 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.063560607000000005, 2024-09-21T22:07:21.981 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:21.981 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:21.981 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:21.981 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:21.981 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.981 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:21.981 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.343754+0000", 2024-09-21T22:07:21.982 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:21.982 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.982 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.982 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:21.982 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.343754+0000", 2024-09-21T22:07:21.982 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:21.982 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.982 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.983 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:21.983 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.343756+0000", 2024-09-21T22:07:21.983 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7069999999999999e-06 2024-09-21T22:07:21.983 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.983 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.983 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:21.983 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.344864+0000", 2024-09-21T22:07:21.983 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011085159999999999 2024-09-21T22:07:21.983 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.983 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.983 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:21.983 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.344914+0000", 2024-09-21T22:07:21.984 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.9462000000000002e-05 2024-09-21T22:07:21.984 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.984 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.984 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:21.984 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.344961+0000", 2024-09-21T22:07:21.984 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.7633999999999998e-05 2024-09-21T22:07:21.984 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.984 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.984 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:21.984 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.345171+0000", 2024-09-21T22:07:21.984 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000209652 2024-09-21T22:07:21.984 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.984 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.984 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:21.985 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.345193+0000", 2024-09-21T22:07:21.985 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1985999999999999e-05 2024-09-21T22:07:21.985 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.985 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.985 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:21.985 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.407315+0000", 2024-09-21T22:07:21.985 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.06212165 2024-09-21T22:07:21.985 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:21.985 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:21.985 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:21.985 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.985 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.985 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.3 55/54 [PushOp(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head, version: 52'3, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head@52'3, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:07:21.986 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.410551+0000", 2024-09-21T22:07:21.986 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 9.5436760340000006, 2024-09-21T22:07:21.986 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.054517522999999998, 2024-09-21T22:07:21.986 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:21.986 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:21.986 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:21.986 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:21.986 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.986 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:21.986 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.410551+0000", 2024-09-21T22:07:21.986 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:21.986 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.987 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.987 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:21.987 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.410551+0000", 2024-09-21T22:07:21.987 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:21.987 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.987 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.987 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:21.987 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.410553+0000", 2024-09-21T22:07:21.987 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5200000000000001e-06 2024-09-21T22:07:21.987 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.988 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.988 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:21.988 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.411747+0000", 2024-09-21T22:07:21.988 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011946089999999999 2024-09-21T22:07:21.988 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.988 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.988 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:21.988 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.411796+0000", 2024-09-21T22:07:21.988 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8717999999999997e-05 2024-09-21T22:07:21.988 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.988 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.988 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:21.988 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.411844+0000", 2024-09-21T22:07:21.988 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8291999999999998e-05 2024-09-21T22:07:21.989 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.989 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.989 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:21.989 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.412066+0000", 2024-09-21T22:07:21.989 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00022115899999999999 2024-09-21T22:07:21.989 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.989 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.989 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:21.989 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.412081+0000", 2024-09-21T22:07:21.989 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5654e-05 2024-09-21T22:07:21.989 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.989 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.989 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:21.989 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.465069+0000", 2024-09-21T22:07:21.990 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.052987570999999997 2024-09-21T22:07:21.990 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:21.990 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:21.990 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:21.990 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.990 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.990 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.3 55/54 [PushOp(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head, version: 52'3, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head@52'3, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 4194304, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:07:21.990 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.468339+0000", 2024-09-21T22:07:21.990 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 9.4858883699999996, 2024-09-21T22:07:21.990 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.064010024999999998, 2024-09-21T22:07:21.990 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:21.990 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:21.990 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:21.991 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:21.991 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.991 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:21.991 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.468339+0000", 2024-09-21T22:07:21.991 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:21.991 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.991 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.991 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:21.991 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.468339+0000", 2024-09-21T22:07:21.991 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:21.991 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.991 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.991 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:21.991 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.468340+0000", 2024-09-21T22:07:21.991 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4330000000000001e-06 2024-09-21T22:07:21.992 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.992 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.992 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:21.992 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.469484+0000", 2024-09-21T22:07:21.992 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001143535 2024-09-21T22:07:21.992 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.992 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.992 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:21.992 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.469533+0000", 2024-09-21T22:07:21.992 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8980999999999997e-05 2024-09-21T22:07:21.992 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.992 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.992 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:21.992 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.469581+0000", 2024-09-21T22:07:21.992 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8551999999999999e-05 2024-09-21T22:07:21.993 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.993 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.993 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:21.993 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.469804+0000", 2024-09-21T22:07:21.993 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00022287699999999999 2024-09-21T22:07:21.993 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.993 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.993 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:21.993 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.469829+0000", 2024-09-21T22:07:21.993 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4766e-05 2024-09-21T22:07:21.993 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.993 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.993 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:21.993 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.532349+0000", 2024-09-21T22:07:21.994 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.062519880999999999 2024-09-21T22:07:21.994 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:21.994 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:21.994 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:21.994 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.994 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.994 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4828.0:12 2.2 2:53d5e6d0:::rbd_header.126483d75f2d:head [watch watch cookie 140505962875408] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-21T22:07:21.994 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:20.109949+0000", 2024-09-21T22:07:21.994 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 1.8442779730000001, 2024-09-21T22:07:21.994 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.002473294, 2024-09-21T22:07:21.994 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:21.994 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:21.994 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:21.994 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-21T22:07:21.994 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4828", 2024-09-21T22:07:21.995 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.157:0/2626096239", 2024-09-21T22:07:21.995 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 12 2024-09-21T22:07:21.995 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.995 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:21.995 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.995 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:21.995 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:20.109949+0000", 2024-09-21T22:07:21.995 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:21.995 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.995 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.995 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:21.995 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:20.109949+0000", 2024-09-21T22:07:21.996 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:21.996 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.996 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.996 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:21.996 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:20.109951+0000", 2024-09-21T22:07:21.996 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9870000000000002e-06 2024-09-21T22:07:21.996 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.996 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.996 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:21.996 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:20.109961+0000", 2024-09-21T22:07:21.996 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.2629999999999992e-06 2024-09-21T22:07:21.996 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.996 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.996 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:21.996 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:20.109993+0000", 2024-09-21T22:07:21.997 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.1893000000000001e-05 2024-09-21T22:07:21.997 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.997 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.997 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:21.997 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:20.110007+0000", 2024-09-21T22:07:21.997 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4962e-05 2024-09-21T22:07:21.997 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.997 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.997 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:21.997 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:20.110131+0000", 2024-09-21T22:07:21.997 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000123423 2024-09-21T22:07:21.997 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.998 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.998 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:21.998 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:20.110272+0000", 2024-09-21T22:07:21.998 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00014100199999999999 2024-09-21T22:07:21.998 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.998 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.998 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-21T22:07:21.998 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:20.112080+0000", 2024-09-21T22:07:21.998 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001808231 2024-09-21T22:07:21.998 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.998 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.998 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:21.998 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:20.112183+0000", 2024-09-21T22:07:21.999 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00010296700000000001 2024-09-21T22:07:21.999 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.999 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.999 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:21.999 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:20.112423+0000", 2024-09-21T22:07:21.999 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000239566 2024-09-21T22:07:21.999 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:21.999 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:21.999 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:21.999 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:21.999 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:21.999 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4840.0:3 2.2 2:53d5e6d0:::rbd_header.126483d75f2d:head [call rbd.get_size in=11b,call,call rbd.get_features in=15b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-21T22:07:21.999 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:20.832168+0000", 2024-09-21T22:07:22.000 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 1.122059387, 2024-09-21T22:07:22.000 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0044135839999999999, 2024-09-21T22:07:22.000 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:22.000 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:22.000 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:22.000 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-21T22:07:22.000 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4840", 2024-09-21T22:07:22.000 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:22.000 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 3 2024-09-21T22:07:22.000 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.000 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:22.000 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.000 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:22.000 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:20.832168+0000", 2024-09-21T22:07:22.000 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:22.001 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.001 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.001 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:22.001 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:20.832168+0000", 2024-09-21T22:07:22.001 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:22.001 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.001 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.001 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:22.001 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:20.832171+0000", 2024-09-21T22:07:22.001 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.92e-06 2024-09-21T22:07:22.001 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.001 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.001 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:22.001 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:20.832184+0000", 2024-09-21T22:07:22.002 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3533000000000001e-05 2024-09-21T22:07:22.002 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.002 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.002 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:22.002 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:20.832209+0000", 2024-09-21T22:07:22.002 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4468999999999999e-05 2024-09-21T22:07:22.002 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.002 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.002 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:22.002 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:20.832231+0000", 2024-09-21T22:07:22.002 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1656000000000001e-05 2024-09-21T22:07:22.002 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.002 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.002 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:22.003 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:20.832353+0000", 2024-09-21T22:07:22.003 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00012282699999999999 2024-09-21T22:07:22.003 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.003 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.003 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op must wait for map", 2024-09-21T22:07:22.003 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:20.832362+0000", 2024-09-21T22:07:22.003 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.5620000000000004e-06 2024-09-21T22:07:22.003 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.003 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.003 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:22.003 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:20.836243+0000", 2024-09-21T22:07:22.003 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.003880957 2024-09-21T22:07:22.003 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.003 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.003 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:22.004 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:20.836340+0000", 2024-09-21T22:07:22.004 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.7608000000000005e-05 2024-09-21T22:07:22.004 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.004 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.004 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:22.004 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:20.836582+0000", 2024-09-21T22:07:22.004 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00024105200000000001 2024-09-21T22:07:22.004 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:22.004 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:22.004 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:22.004 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.004 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.004 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4840.0:120 2.6 2:633e2f34:::rbd_data.126483d75f2d.0000000000000001:head [sparse-read 3411968~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-21T22:07:22.004 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:21.237843+0000", 2024-09-21T22:07:22.005 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.71638441900000005, 2024-09-21T22:07:22.005 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0023828040000000001, 2024-09-21T22:07:22.005 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:22.005 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:22.005 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:22.005 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-21T22:07:22.005 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4840", 2024-09-21T22:07:22.005 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:22.005 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 120 2024-09-21T22:07:22.005 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.005 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:22.005 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.005 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:22.005 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.237843+0000", 2024-09-21T22:07:22.005 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:22.006 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.006 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.006 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:22.006 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.237843+0000", 2024-09-21T22:07:22.006 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:22.006 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.006 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.006 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:22.006 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.237844+0000", 2024-09-21T22:07:22.006 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.54e-07 2024-09-21T22:07:22.006 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.006 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.006 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:22.006 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.237848+0000", 2024-09-21T22:07:22.007 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.8999999999999999e-06 2024-09-21T22:07:22.007 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.007 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.007 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:22.007 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.237861+0000", 2024-09-21T22:07:22.007 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3385e-05 2024-09-21T22:07:22.007 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.007 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.007 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:22.007 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.237869+0000", 2024-09-21T22:07:22.007 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.1140000000000006e-06 2024-09-21T22:07:22.007 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.007 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.007 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:22.008 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.239242+0000", 2024-09-21T22:07:22.008 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0013731699999999999 2024-09-21T22:07:22.008 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.008 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.008 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:22.008 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.239337+0000", 2024-09-21T22:07:22.008 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.5224999999999997e-05 2024-09-21T22:07:22.008 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.008 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.008 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:22.008 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.240226+0000", 2024-09-21T22:07:22.009 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00088825600000000003 2024-09-21T22:07:22.009 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:22.009 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:22.009 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:22.009 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.009 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.009 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4840.0:137 2.4 2:36553e04:::rbd_data.126483d75f2d.0000000000000002:head [sparse-read 1314816~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-21T22:07:22.009 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:21.264595+0000", 2024-09-21T22:07:22.009 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.68963252600000002, 2024-09-21T22:07:22.009 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0024356289999999999, 2024-09-21T22:07:22.009 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:22.009 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:22.009 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:22.009 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-21T22:07:22.010 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4840", 2024-09-21T22:07:22.010 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:22.010 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 137 2024-09-21T22:07:22.010 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.010 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:22.010 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.010 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:22.010 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.264595+0000", 2024-09-21T22:07:22.010 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:22.010 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.010 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.010 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:22.010 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.264595+0000", 2024-09-21T22:07:22.010 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:22.011 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.011 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.011 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:22.011 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.264596+0000", 2024-09-21T22:07:22.011 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.22e-06 2024-09-21T22:07:22.011 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.011 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.011 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:22.011 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.264602+0000", 2024-09-21T22:07:22.011 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.3459999999999999e-06 2024-09-21T22:07:22.011 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.011 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.011 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:22.011 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.264639+0000", 2024-09-21T22:07:22.012 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.6599e-05 2024-09-21T22:07:22.012 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.012 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.012 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:22.012 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.264655+0000", 2024-09-21T22:07:22.012 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6223e-05 2024-09-21T22:07:22.012 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.012 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.012 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:22.012 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.265795+0000", 2024-09-21T22:07:22.012 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001140088 2024-09-21T22:07:22.012 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.012 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.012 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:22.012 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.265985+0000", 2024-09-21T22:07:22.013 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00019020199999999999 2024-09-21T22:07:22.013 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.013 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.013 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:22.013 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.267030+0000", 2024-09-21T22:07:22.013 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0010449509999999999 2024-09-21T22:07:22.013 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:22.013 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:22.013 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:22.013 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.013 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.013 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4840.0:145 2.4 2:36553e04:::rbd_data.126483d75f2d.0000000000000002:head [sparse-read 2363392~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-21T22:07:22.013 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:21.279004+0000", 2024-09-21T22:07:22.013 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.67522332900000004, 2024-09-21T22:07:22.014 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0025832339999999998, 2024-09-21T22:07:22.014 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:22.014 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:22.014 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:22.014 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-21T22:07:22.014 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4840", 2024-09-21T22:07:22.014 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:22.014 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 145 2024-09-21T22:07:22.014 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.014 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:22.014 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.014 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:22.014 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.279004+0000", 2024-09-21T22:07:22.014 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:22.015 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.015 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.015 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:22.015 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.279004+0000", 2024-09-21T22:07:22.015 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:22.015 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.015 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.015 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:22.015 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.279005+0000", 2024-09-21T22:07:22.015 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.373e-06 2024-09-21T22:07:22.015 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.015 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.015 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:22.015 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.279013+0000", 2024-09-21T22:07:22.015 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.4000000000000003e-06 2024-09-21T22:07:22.016 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.016 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.016 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:22.016 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.279033+0000", 2024-09-21T22:07:22.016 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9913000000000001e-05 2024-09-21T22:07:22.016 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.016 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.016 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:22.016 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.279045+0000", 2024-09-21T22:07:22.016 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2156e-05 2024-09-21T22:07:22.016 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.016 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.016 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:22.017 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.280185+0000", 2024-09-21T22:07:22.017 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011401510000000001 2024-09-21T22:07:22.017 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.017 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.017 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:22.017 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.280358+0000", 2024-09-21T22:07:22.017 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000173365 2024-09-21T22:07:22.017 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.017 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.017 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:22.017 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.281587+0000", 2024-09-21T22:07:22.017 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0012288760000000001 2024-09-21T22:07:22.017 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:22.017 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:22.018 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:22.018 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.018 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.018 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4840.0:186 2.4 2:30a04224:::rbd_data.126483d75f2d.0000000000000003:head [sparse-read 3411968~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-21T22:07:22.018 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:21.347607+0000", 2024-09-21T22:07:22.018 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.60662025200000003, 2024-09-21T22:07:22.018 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0024242529999999999, 2024-09-21T22:07:22.018 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:22.018 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:22.018 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:22.018 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-21T22:07:22.018 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4840", 2024-09-21T22:07:22.018 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:22.018 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 186 2024-09-21T22:07:22.019 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.019 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:22.019 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.019 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:22.019 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.347607+0000", 2024-09-21T22:07:22.019 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:22.019 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.019 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.019 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:22.019 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.347607+0000", 2024-09-21T22:07:22.019 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:22.019 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.019 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.019 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:22.020 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.347608+0000", 2024-09-21T22:07:22.020 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.1829999999999999e-06 2024-09-21T22:07:22.020 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.020 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.020 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:22.020 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.347615+0000", 2024-09-21T22:07:22.020 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.9500000000000004e-06 2024-09-21T22:07:22.020 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.020 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.020 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:22.020 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.347638+0000", 2024-09-21T22:07:22.020 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3156e-05 2024-09-21T22:07:22.020 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.020 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.021 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:22.021 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.347656+0000", 2024-09-21T22:07:22.021 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7543000000000001e-05 2024-09-21T22:07:22.021 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.021 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.021 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:22.021 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.348804+0000", 2024-09-21T22:07:22.021 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011485180000000001 2024-09-21T22:07:22.021 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.021 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.021 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:22.021 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.348972+0000", 2024-09-21T22:07:22.021 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00016740799999999999 2024-09-21T22:07:22.021 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.022 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.022 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:22.022 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.350031+0000", 2024-09-21T22:07:22.022 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0010594949999999999 2024-09-21T22:07:22.022 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:22.022 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:22.022 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:22.022 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.022 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.022 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4840.0:327 2.3 2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head [sparse-read 266240~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-21T22:07:22.022 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:21.574545+0000", 2024-09-21T22:07:22.022 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.37968253299999999, 2024-09-21T22:07:22.022 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0028402670000000001, 2024-09-21T22:07:22.023 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:22.023 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:22.023 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:22.023 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-21T22:07:22.023 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4840", 2024-09-21T22:07:22.023 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:22.023 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 327 2024-09-21T22:07:22.023 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.023 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:22.023 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.023 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:22.024 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.574545+0000", 2024-09-21T22:07:22.024 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:22.024 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.024 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.024 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:22.024 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.574545+0000", 2024-09-21T22:07:22.024 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:22.024 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.024 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.024 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:22.024 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.574546+0000", 2024-09-21T22:07:22.024 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.8000000000000004e-07 2024-09-21T22:07:22.024 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.025 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.025 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:22.025 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.574551+0000", 2024-09-21T22:07:22.025 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.383e-06 2024-09-21T22:07:22.025 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.025 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.025 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:22.025 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.574565+0000", 2024-09-21T22:07:22.025 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4256999999999999e-05 2024-09-21T22:07:22.025 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.025 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.025 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:22.025 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.574574+0000", 2024-09-21T22:07:22.025 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.1560000000000001e-06 2024-09-21T22:07:22.026 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.026 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.026 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:22.026 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.576390+0000", 2024-09-21T22:07:22.026 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001815494 2024-09-21T22:07:22.026 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.026 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.026 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:22.026 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.576510+0000", 2024-09-21T22:07:22.026 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000120167 2024-09-21T22:07:22.026 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.026 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.026 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:22.026 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.577385+0000", 2024-09-21T22:07:22.026 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00087493 2024-09-21T22:07:22.027 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:22.027 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:22.027 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:22.027 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.027 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.027 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4840.0:335 2.3 2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head [sparse-read 1314816~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-21T22:07:22.027 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:21.589481+0000", 2024-09-21T22:07:22.027 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.36474657500000002, 2024-09-21T22:07:22.027 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0026417459999999999, 2024-09-21T22:07:22.027 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:22.027 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:22.027 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:22.027 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-21T22:07:22.028 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4840", 2024-09-21T22:07:22.028 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:22.028 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 335 2024-09-21T22:07:22.028 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.028 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:22.028 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.028 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:22.028 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.589481+0000", 2024-09-21T22:07:22.028 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:22.028 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.028 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.028 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:22.028 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.589481+0000", 2024-09-21T22:07:22.028 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:22.029 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.029 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.029 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:22.029 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.589482+0000", 2024-09-21T22:07:22.029 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.5000000000000001e-07 2024-09-21T22:07:22.029 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.029 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.029 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:22.029 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.589486+0000", 2024-09-21T22:07:22.029 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.3359999999999997e-06 2024-09-21T22:07:22.029 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.029 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.029 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:22.029 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.589500+0000", 2024-09-21T22:07:22.030 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4116e-05 2024-09-21T22:07:22.030 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.030 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.030 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:22.030 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.589509+0000", 2024-09-21T22:07:22.030 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.5599999999999994e-06 2024-09-21T22:07:22.030 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.030 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.030 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:22.030 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.591073+0000", 2024-09-21T22:07:22.030 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0015648719999999999 2024-09-21T22:07:22.030 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.030 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.030 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:22.031 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.591205+0000", 2024-09-21T22:07:22.031 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013118299999999999 2024-09-21T22:07:22.031 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.031 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.031 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:22.031 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.592123+0000", 2024-09-21T22:07:22.031 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000917829 2024-09-21T22:07:22.031 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:22.031 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:22.031 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:22.031 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.031 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.031 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4840.0:393 2.3 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head [sparse-read 266240~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-21T22:07:22.031 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:21.682145+0000", 2024-09-21T22:07:22.032 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.27208248200000001, 2024-09-21T22:07:22.032 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0028951290000000002, 2024-09-21T22:07:22.032 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:22.032 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:22.032 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:22.032 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-21T22:07:22.032 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4840", 2024-09-21T22:07:22.032 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:22.032 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 393 2024-09-21T22:07:22.032 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.032 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:22.032 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.032 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:22.032 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.682145+0000", 2024-09-21T22:07:22.033 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:22.033 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.033 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.033 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:22.033 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.682145+0000", 2024-09-21T22:07:22.033 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:22.033 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.033 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.033 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:22.033 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.682145+0000", 2024-09-21T22:07:22.033 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.6800000000000001e-07 2024-09-21T22:07:22.033 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.033 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.033 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:22.033 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.682151+0000", 2024-09-21T22:07:22.034 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.2190000000000003e-06 2024-09-21T22:07:22.034 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.034 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.034 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:22.034 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.682157+0000", 2024-09-21T22:07:22.034 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.9299999999999997e-06 2024-09-21T22:07:22.034 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.034 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.034 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:22.034 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.682162+0000", 2024-09-21T22:07:22.034 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.561e-06 2024-09-21T22:07:22.034 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.034 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.034 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:22.035 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.684128+0000", 2024-09-21T22:07:22.035 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0019663580000000001 2024-09-21T22:07:22.035 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.035 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.035 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:22.035 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.684263+0000", 2024-09-21T22:07:22.035 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013478200000000001 2024-09-21T22:07:22.035 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.035 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.035 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:22.035 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.685040+0000", 2024-09-21T22:07:22.035 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00077681099999999997 2024-09-21T22:07:22.035 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:22.035 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:22.036 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:22.036 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.036 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.036 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4840.0:401 2.3 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head [sparse-read 1314816~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-21T22:07:22.036 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:21.696985+0000", 2024-09-21T22:07:22.036 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.25724212200000002, 2024-09-21T22:07:22.036 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0028023810000000001, 2024-09-21T22:07:22.036 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:22.036 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:22.036 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:22.036 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-21T22:07:22.036 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4840", 2024-09-21T22:07:22.036 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:22.036 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 401 2024-09-21T22:07:22.036 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.037 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:22.037 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.037 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:22.037 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.696985+0000", 2024-09-21T22:07:22.037 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:22.037 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.037 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.037 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:22.037 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.696985+0000", 2024-09-21T22:07:22.037 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:22.037 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.037 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.037 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:22.037 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.696987+0000", 2024-09-21T22:07:22.038 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.31e-06 2024-09-21T22:07:22.038 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.038 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.038 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:22.038 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.696993+0000", 2024-09-21T22:07:22.038 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.409e-06 2024-09-21T22:07:22.038 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.038 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.038 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:22.038 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.697051+0000", 2024-09-21T22:07:22.038 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.8338999999999998e-05 2024-09-21T22:07:22.038 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.038 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.039 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:22.039 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.697070+0000", 2024-09-21T22:07:22.039 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8802999999999999e-05 2024-09-21T22:07:22.039 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.039 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.039 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:22.039 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.698347+0000", 2024-09-21T22:07:22.039 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001277151 2024-09-21T22:07:22.039 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.039 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.039 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:22.039 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.698584+0000", 2024-09-21T22:07:22.039 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00023639900000000001 2024-09-21T22:07:22.039 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.039 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.040 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:22.040 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.699788+0000", 2024-09-21T22:07:22.040 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00120397 2024-09-21T22:07:22.040 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:22.040 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:22.040 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:22.040 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.040 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.040 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4840.0:409 2.3 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head [sparse-read 2363392~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-21T22:07:22.040 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:21.711408+0000", 2024-09-21T22:07:22.040 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.24281978700000001, 2024-09-21T22:07:22.040 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0026978330000000002, 2024-09-21T22:07:22.040 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:22.040 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:22.041 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:22.041 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-21T22:07:22.041 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4840", 2024-09-21T22:07:22.041 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:22.041 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 409 2024-09-21T22:07:22.041 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.041 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:22.041 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.041 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:22.041 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.711408+0000", 2024-09-21T22:07:22.041 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:22.041 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.041 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.041 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:22.042 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.711408+0000", 2024-09-21T22:07:22.042 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:22.042 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.042 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.042 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:22.042 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.711408+0000", 2024-09-21T22:07:22.042 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.01e-07 2024-09-21T22:07:22.042 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.042 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.042 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:22.042 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.711410+0000", 2024-09-21T22:07:22.042 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4710000000000001e-06 2024-09-21T22:07:22.042 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.043 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.043 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:22.043 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.711416+0000", 2024-09-21T22:07:22.043 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.6620000000000002e-06 2024-09-21T22:07:22.043 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.043 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.043 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:22.043 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.711420+0000", 2024-09-21T22:07:22.043 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.0949999999999998e-06 2024-09-21T22:07:22.043 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.043 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.043 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:22.043 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.713106+0000", 2024-09-21T22:07:22.043 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0016855780000000001 2024-09-21T22:07:22.043 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.044 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.044 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:22.044 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.713242+0000", 2024-09-21T22:07:22.044 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000136229 2024-09-21T22:07:22.044 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.044 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.044 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:22.044 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.714105+0000", 2024-09-21T22:07:22.044 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00086339700000000001 2024-09-21T22:07:22.044 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:22.044 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:22.044 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:22.044 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.045 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.045 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4840.0:417 2.3 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head [sparse-read 3411968~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-21T22:07:22.045 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:21.727425+0000", 2024-09-21T22:07:22.045 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.22680235400000001, 2024-09-21T22:07:22.045 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0025878110000000002, 2024-09-21T22:07:22.045 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:22.045 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:22.045 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:22.045 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-21T22:07:22.045 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4840", 2024-09-21T22:07:22.045 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:22.045 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 417 2024-09-21T22:07:22.045 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.045 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:22.046 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.046 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:22.046 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.727425+0000", 2024-09-21T22:07:22.046 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:22.046 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.046 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.046 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:22.046 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.727425+0000", 2024-09-21T22:07:22.046 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:22.046 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.046 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.046 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:22.046 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.727426+0000", 2024-09-21T22:07:22.046 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.44e-06 2024-09-21T22:07:22.047 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.047 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.047 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:22.047 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.727435+0000", 2024-09-21T22:07:22.047 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.5029999999999997e-06 2024-09-21T22:07:22.047 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.047 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.047 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:22.047 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.727451+0000", 2024-09-21T22:07:22.047 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.645e-05 2024-09-21T22:07:22.047 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.047 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.047 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:22.047 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.727468+0000", 2024-09-21T22:07:22.047 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6246e-05 2024-09-21T22:07:22.048 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.048 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.048 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:22.048 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.729044+0000", 2024-09-21T22:07:22.048 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001576832 2024-09-21T22:07:22.048 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.048 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.048 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:22.048 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.729153+0000", 2024-09-21T22:07:22.048 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000108834 2024-09-21T22:07:22.048 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:22.048 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:22.048 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:22.048 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.730013+0000", 2024-09-21T22:07:22.048 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00085950600000000001 2024-09-21T22:07:22.049 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:22.049 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:22.049 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:22.049 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:22.049 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:22.049 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:07:22.050 INFO:tasks.workunit.client.0.smithi157.stderr:64+0 records in 2024-09-21T22:07:22.050 INFO:tasks.workunit.client.0.smithi157.stderr:64+0 records out 2024-09-21T22:07:22.051 INFO:tasks.workunit.client.0.smithi157.stderr:67108864 bytes (67 MB, 64 MiB) copied, 0.836773 s, 80.2 MB/s 2024-09-21T22:07:22.051 INFO:tasks.workunit.client.0.smithi157.stderr:+ '[' '9989b342f29fb0180283bb48199b50b9 -' = '9989b342f29fb0180283bb48199b50b9 -' ']' 2024-09-21T22:07:22.051 INFO:tasks.workunit.client.0.smithi157.stderr:+ dd if=/dev/urandom of=/tmp/tmp.GqKuv7rsZi/data bs=1M count=64 2024-09-21T22:07:22.051 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T22:07:22.052 INFO:teuthology.orchestra.run.smithi039.stdout:osd.0: {} 2024-09-21T22:07:22.052 INFO:teuthology.orchestra.run.smithi039.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T22:07:22.052 INFO:teuthology.orchestra.run.smithi039.stdout:osd.1: {} 2024-09-21T22:07:22.053 INFO:teuthology.orchestra.run.smithi039.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T22:07:22.053 INFO:teuthology.orchestra.run.smithi039.stdout:osd.2: {} 2024-09-21T22:07:22.053 INFO:teuthology.orchestra.run.smithi039.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T22:07:22.053 INFO:teuthology.orchestra.run.smithi039.stdout:osd.3: {} 2024-09-21T22:07:22.054 INFO:teuthology.orchestra.run.smithi039.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T22:07:22.054 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:22.048+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:22.064 INFO:teuthology.orchestra.run.smithi039.stdout:osd.4: {} 2024-09-21T22:07:22.064 INFO:teuthology.orchestra.run.smithi039.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T22:07:22.064 INFO:teuthology.orchestra.run.smithi039.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-21T22:07:22.068 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T22:07:22.068 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T22:07:22.151 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:22.148+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:22.170 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:22.170 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:22.178 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-21T22:07:22.186 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T22:07:22.276 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:22.274+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:22.349 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:22.349 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:22.352 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:22.348+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:22.363 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T22:07:22.453 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:22.452+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:22.535 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:22.535 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:22.549 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T22:07:22.573 INFO:tasks.workunit.client.0.smithi157.stderr:64+0 records in 2024-09-21T22:07:22.573 INFO:tasks.workunit.client.0.smithi157.stderr:64+0 records out 2024-09-21T22:07:22.574 INFO:tasks.workunit.client.0.smithi157.stderr:67108864 bytes (67 MB, 64 MiB) copied, 0.559954 s, 120 MB/s 2024-09-21T22:07:22.574 INFO:tasks.workunit.client.0.smithi157.stderr:+ _sudo dd if=/tmp/tmp.GqKuv7rsZi/data of=/dev/nbd0 bs=1M oflag=direct 2024-09-21T22:07:22.574 INFO:tasks.workunit.client.0.smithi157.stderr:+ local cmd 2024-09-21T22:07:22.574 INFO:tasks.workunit.client.0.smithi157.stderr:++ id -u 2024-09-21T22:07:22.576 INFO:tasks.workunit.client.0.smithi157.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T22:07:22.577 INFO:tasks.workunit.client.0.smithi157.stderr:++ which dd 2024-09-21T22:07:22.579 INFO:tasks.workunit.client.0.smithi157.stderr:+ cmd=/usr/bin/dd 2024-09-21T22:07:22.579 INFO:tasks.workunit.client.0.smithi157.stderr:+ shift 2024-09-21T22:07:22.579 INFO:tasks.workunit.client.0.smithi157.stderr:+ sudo -nE /usr/bin/dd if=/tmp/tmp.GqKuv7rsZi/data of=/dev/nbd0 bs=1M oflag=direct 2024-09-21T22:07:22.594 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:22.592+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:22.654 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:22.652+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:22.696 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:22.696 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:22.712 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T22:07:22.755 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:22.750+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:22.856 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:22.854+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:22.884 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:22.884 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:22.898 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T22:07:22.957 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:22.952+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:23.041 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:23.041 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:23.055 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T22:07:23.099 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:23.096+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:23.159 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:23.154+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:23.190 INFO:teuthology.orchestra.run.smithi039.stderr:nodeep-scrub is unset 2024-09-21T22:07:23.227 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:23.227 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:23.241 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T22:07:23.290 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:23.288+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:23.362 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:23.358+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:23.367 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:23.367 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:23.381 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T22:07:23.466 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:23.464+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:23.553 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:23.553 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:23.564 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:23.562+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:23.569 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T22:07:23.665 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:23.662+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:23.713 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:23.713 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:23.728 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T22:07:23.766 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:23.764+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:23.867 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:23.864+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:23.910 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:23.911 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:23.926 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T22:07:23.968 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:23.964+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:24.053 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:24.053 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:24.067 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T22:07:24.069 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T22:07:24.069 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:24.064+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:24.170 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:24.164+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:24.189 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:24.189 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:24.205 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T22:07:24.209 INFO:teuthology.orchestra.run.smithi039.stdout:osd.0: {} 2024-09-21T22:07:24.209 INFO:teuthology.orchestra.run.smithi039.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T22:07:24.227 INFO:teuthology.orchestra.run.smithi039.stdout:osd.1: {} 2024-09-21T22:07:24.227 INFO:teuthology.orchestra.run.smithi039.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T22:07:24.245 INFO:teuthology.orchestra.run.smithi039.stdout:osd.2: {} 2024-09-21T22:07:24.246 INFO:teuthology.orchestra.run.smithi039.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T22:07:24.266 INFO:teuthology.orchestra.run.smithi039.stdout:osd.3: {} 2024-09-21T22:07:24.267 INFO:teuthology.orchestra.run.smithi039.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T22:07:24.271 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:24.266+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:24.287 INFO:teuthology.orchestra.run.smithi039.stdout:osd.4: {} 2024-09-21T22:07:24.288 INFO:teuthology.orchestra.run.smithi039.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T22:07:24.288 INFO:teuthology.orchestra.run.smithi039.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-21T22:07:24.291 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T22:07:24.292 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T22:07:24.324 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:07:24.324 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:07:24.325 INFO:teuthology.orchestra.run.smithi039.stdout: "complaint_time": 30, 2024-09-21T22:07:24.325 INFO:teuthology.orchestra.run.smithi039.stdout: "num_blocked_ops": 0 2024-09-21T22:07:24.325 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:07:24.339 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T22:07:24.372 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:24.368+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:24.473 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:24.470+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:24.529 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:07:24.529 INFO:teuthology.orchestra.run.smithi039.stdout: "size": 20, 2024-09-21T22:07:24.529 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 600, 2024-09-21T22:07:24.529 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [ 2024-09-21T22:07:24.529 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.529 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.3 55/54 [PushOp(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head, version: 52'2, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head@52'2, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T22:07:24.529 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.084379+0000", 2024-09-21T22:07:24.529 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 12.439647081, 2024-09-21T22:07:24.529 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.051020674000000002, 2024-09-21T22:07:24.530 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:24.530 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:24.530 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:24.530 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:24.530 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.530 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:24.530 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.084379+0000", 2024-09-21T22:07:24.530 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:24.530 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.530 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.530 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:24.530 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.084379+0000", 2024-09-21T22:07:24.530 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:24.531 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.531 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.531 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:24.531 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.084381+0000", 2024-09-21T22:07:24.531 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.73e-06 2024-09-21T22:07:24.531 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.531 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.531 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:24.531 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.085494+0000", 2024-09-21T22:07:24.531 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011131559999999999 2024-09-21T22:07:24.531 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.531 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.531 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:24.531 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.085542+0000", 2024-09-21T22:07:24.531 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8334999999999997e-05 2024-09-21T22:07:24.532 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.532 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.532 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:24.532 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.085591+0000", 2024-09-21T22:07:24.532 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.9165000000000001e-05 2024-09-21T22:07:24.532 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.532 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.532 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:24.532 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.085777+0000", 2024-09-21T22:07:24.532 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000186152 2024-09-21T22:07:24.532 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.532 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.532 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:24.532 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.085796+0000", 2024-09-21T22:07:24.533 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8612999999999999e-05 2024-09-21T22:07:24.533 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.533 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.533 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:24.533 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.135399+0000", 2024-09-21T22:07:24.533 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.049603522999999997 2024-09-21T22:07:24.533 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:24.533 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:24.533 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:24.533 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.533 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.533 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.3 55/54 [PushOp(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head, version: 52'2, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head@52'2, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:07:24.533 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.139554+0000", 2024-09-21T22:07:24.533 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 12.384471978000001, 2024-09-21T22:07:24.533 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.056215389999999997, 2024-09-21T22:07:24.534 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:24.534 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:24.534 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:24.534 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:24.534 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.534 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:24.534 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.139554+0000", 2024-09-21T22:07:24.534 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:24.534 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.534 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.534 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:24.534 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.139554+0000", 2024-09-21T22:07:24.534 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:24.534 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.534 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.535 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:24.535 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.139556+0000", 2024-09-21T22:07:24.535 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7999999999999999e-06 2024-09-21T22:07:24.535 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.535 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.535 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:24.535 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.140671+0000", 2024-09-21T22:07:24.535 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001115612 2024-09-21T22:07:24.535 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.535 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.535 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:24.535 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.140720+0000", 2024-09-21T22:07:24.535 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8631999999999998e-05 2024-09-21T22:07:24.535 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.536 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.536 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:24.536 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.140752+0000", 2024-09-21T22:07:24.536 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.2199000000000001e-05 2024-09-21T22:07:24.536 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.536 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.536 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:24.536 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.140900+0000", 2024-09-21T22:07:24.536 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00014786000000000001 2024-09-21T22:07:24.536 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.536 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.536 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:24.536 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.140917+0000", 2024-09-21T22:07:24.536 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6753999999999999e-05 2024-09-21T22:07:24.536 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.536 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.537 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:24.537 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.195769+0000", 2024-09-21T22:07:24.537 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.054852533000000002 2024-09-21T22:07:24.537 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:24.537 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:24.537 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:24.537 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.537 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.537 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.3 55/54 [PushOp(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head, version: 52'2, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head@52'2, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:07:24.537 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.199187+0000", 2024-09-21T22:07:24.537 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 12.324838434, 2024-09-21T22:07:24.537 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.051543975999999998, 2024-09-21T22:07:24.537 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:24.537 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:24.538 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:24.538 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:24.538 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.538 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:24.538 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.199187+0000", 2024-09-21T22:07:24.538 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:24.538 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.538 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.538 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:24.538 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.199187+0000", 2024-09-21T22:07:24.538 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:24.538 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.538 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.538 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:24.538 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.199189+0000", 2024-09-21T22:07:24.539 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7969999999999999e-06 2024-09-21T22:07:24.539 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.539 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.539 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:24.539 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.200355+0000", 2024-09-21T22:07:24.539 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011660139999999999 2024-09-21T22:07:24.539 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.539 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.539 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:24.539 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.200405+0000", 2024-09-21T22:07:24.539 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.0124000000000003e-05 2024-09-21T22:07:24.539 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.539 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.539 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:24.540 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.200457+0000", 2024-09-21T22:07:24.540 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.1177999999999997e-05 2024-09-21T22:07:24.540 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.540 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.540 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:24.540 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.200635+0000", 2024-09-21T22:07:24.540 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00017841299999999999 2024-09-21T22:07:24.540 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.540 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.540 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:24.540 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.200653+0000", 2024-09-21T22:07:24.540 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7944999999999998e-05 2024-09-21T22:07:24.540 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.540 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.541 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:24.541 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.250731+0000", 2024-09-21T22:07:24.541 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.050078505000000002 2024-09-21T22:07:24.541 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:24.541 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:24.541 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:24.541 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.541 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.541 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.3 55/54 [PushOp(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head, version: 52'2, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head@52'2, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 4194304, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:07:24.541 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.253966+0000", 2024-09-21T22:07:24.541 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 12.270060104000001, 2024-09-21T22:07:24.541 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.041100606999999997, 2024-09-21T22:07:24.541 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:24.541 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:24.541 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:24.542 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:24.542 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.542 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:24.542 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.253966+0000", 2024-09-21T22:07:24.542 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:24.542 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.542 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.542 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:24.542 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.253966+0000", 2024-09-21T22:07:24.542 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:24.542 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.542 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.542 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:24.542 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.253967+0000", 2024-09-21T22:07:24.542 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.423e-06 2024-09-21T22:07:24.543 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.543 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.543 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:24.543 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.255117+0000", 2024-09-21T22:07:24.543 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011496709999999999 2024-09-21T22:07:24.543 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.543 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.543 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:24.543 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.255167+0000", 2024-09-21T22:07:24.543 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.0315000000000002e-05 2024-09-21T22:07:24.543 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.543 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.543 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:24.543 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.255218+0000", 2024-09-21T22:07:24.543 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.0390999999999999e-05 2024-09-21T22:07:24.544 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.544 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.544 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:24.544 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.255391+0000", 2024-09-21T22:07:24.544 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00017332799999999999 2024-09-21T22:07:24.544 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.544 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.544 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:24.544 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.255405+0000", 2024-09-21T22:07:24.544 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3723999999999999e-05 2024-09-21T22:07:24.544 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.544 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.544 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:24.544 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.295066+0000", 2024-09-21T22:07:24.544 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.039661755 2024-09-21T22:07:24.545 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:24.545 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:24.545 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:24.545 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.545 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.545 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.3 55/54 [PushOp(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head, version: 52'3, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head@52'3, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T22:07:24.545 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.299195+0000", 2024-09-21T22:07:24.545 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 12.224831339, 2024-09-21T22:07:24.545 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.040955077999999999, 2024-09-21T22:07:24.545 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:24.545 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:24.545 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:24.545 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:24.545 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.545 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:24.546 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.299195+0000", 2024-09-21T22:07:24.546 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:24.546 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.546 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.546 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:24.546 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.299195+0000", 2024-09-21T22:07:24.546 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:24.546 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.546 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.546 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:24.546 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.299196+0000", 2024-09-21T22:07:24.546 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.604e-06 2024-09-21T22:07:24.546 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.547 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.547 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:24.547 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.300356+0000", 2024-09-21T22:07:24.547 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011602699999999999 2024-09-21T22:07:24.547 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.547 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.547 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:24.547 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.300406+0000", 2024-09-21T22:07:24.547 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.9144999999999998e-05 2024-09-21T22:07:24.547 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.547 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.547 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:24.547 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.300435+0000", 2024-09-21T22:07:24.547 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9886000000000002e-05 2024-09-21T22:07:24.547 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.548 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.548 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:24.548 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.300590+0000", 2024-09-21T22:07:24.548 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00015431200000000001 2024-09-21T22:07:24.548 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.548 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.548 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:24.548 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.300604+0000", 2024-09-21T22:07:24.548 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4559000000000001e-05 2024-09-21T22:07:24.548 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.548 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.548 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:24.548 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.340150+0000", 2024-09-21T22:07:24.548 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.039545301999999997 2024-09-21T22:07:24.548 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:24.548 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:24.549 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:24.549 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.549 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.549 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.3 55/54 [PushOp(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head, version: 52'3, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head@52'3, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:07:24.549 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.343754+0000", 2024-09-21T22:07:24.549 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 12.180271723000001, 2024-09-21T22:07:24.549 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.063560607000000005, 2024-09-21T22:07:24.549 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:24.549 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:24.549 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:24.549 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:24.549 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.549 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:24.549 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.343754+0000", 2024-09-21T22:07:24.550 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:24.550 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.550 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.550 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:24.550 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.343754+0000", 2024-09-21T22:07:24.550 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:24.550 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.550 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.550 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:24.550 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.343756+0000", 2024-09-21T22:07:24.550 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7069999999999999e-06 2024-09-21T22:07:24.550 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.550 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.550 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:24.551 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.344864+0000", 2024-09-21T22:07:24.551 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011085159999999999 2024-09-21T22:07:24.551 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.551 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.551 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:24.551 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.344914+0000", 2024-09-21T22:07:24.551 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.9462000000000002e-05 2024-09-21T22:07:24.551 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.551 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.551 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:24.551 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.344961+0000", 2024-09-21T22:07:24.551 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.7633999999999998e-05 2024-09-21T22:07:24.551 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.551 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.551 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:24.552 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.345171+0000", 2024-09-21T22:07:24.552 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000209652 2024-09-21T22:07:24.552 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.552 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.552 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:24.552 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.345193+0000", 2024-09-21T22:07:24.552 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1985999999999999e-05 2024-09-21T22:07:24.552 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.552 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.552 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:24.552 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.407315+0000", 2024-09-21T22:07:24.552 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.06212165 2024-09-21T22:07:24.552 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:24.552 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:24.552 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:24.553 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.553 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.553 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.3 55/54 [PushOp(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head, version: 52'3, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head@52'3, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:07:24.553 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.410551+0000", 2024-09-21T22:07:24.553 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 12.113474591999999, 2024-09-21T22:07:24.553 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.054517522999999998, 2024-09-21T22:07:24.553 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:24.553 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:24.553 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:24.553 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:24.553 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.553 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:24.553 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.410551+0000", 2024-09-21T22:07:24.553 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:24.553 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.554 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.554 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:24.554 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.410551+0000", 2024-09-21T22:07:24.554 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:24.554 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.554 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.554 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:24.554 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.410553+0000", 2024-09-21T22:07:24.554 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5200000000000001e-06 2024-09-21T22:07:24.554 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.554 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.554 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:24.554 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.411747+0000", 2024-09-21T22:07:24.554 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011946089999999999 2024-09-21T22:07:24.555 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.555 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.555 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:24.555 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.411796+0000", 2024-09-21T22:07:24.555 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8717999999999997e-05 2024-09-21T22:07:24.555 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.555 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.555 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:24.555 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.411844+0000", 2024-09-21T22:07:24.555 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8291999999999998e-05 2024-09-21T22:07:24.555 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.555 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.555 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:24.555 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.412066+0000", 2024-09-21T22:07:24.555 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00022115899999999999 2024-09-21T22:07:24.556 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.556 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.556 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:24.556 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.412081+0000", 2024-09-21T22:07:24.556 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5654e-05 2024-09-21T22:07:24.556 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.556 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.556 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:24.556 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.465069+0000", 2024-09-21T22:07:24.556 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.052987570999999997 2024-09-21T22:07:24.556 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:24.556 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:24.556 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:24.556 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.556 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.556 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.3 55/54 [PushOp(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head, version: 52'3, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head@52'3, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 4194304, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:07:24.557 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.468339+0000", 2024-09-21T22:07:24.557 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 12.055686928, 2024-09-21T22:07:24.557 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.064010024999999998, 2024-09-21T22:07:24.557 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:24.557 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:24.557 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:24.557 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:24.557 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.557 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:24.557 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.468339+0000", 2024-09-21T22:07:24.557 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:24.557 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.557 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.557 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:24.558 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.468339+0000", 2024-09-21T22:07:24.558 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:24.558 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.558 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.558 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:24.558 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.468340+0000", 2024-09-21T22:07:24.558 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4330000000000001e-06 2024-09-21T22:07:24.558 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.558 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.558 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:24.558 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.469484+0000", 2024-09-21T22:07:24.558 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001143535 2024-09-21T22:07:24.558 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.559 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.559 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:24.559 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.469533+0000", 2024-09-21T22:07:24.559 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8980999999999997e-05 2024-09-21T22:07:24.559 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.559 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.559 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:24.559 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.469581+0000", 2024-09-21T22:07:24.559 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8551999999999999e-05 2024-09-21T22:07:24.559 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.559 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.559 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:24.559 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.469804+0000", 2024-09-21T22:07:24.559 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00022287699999999999 2024-09-21T22:07:24.560 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.560 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.560 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:24.560 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.469829+0000", 2024-09-21T22:07:24.560 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4766e-05 2024-09-21T22:07:24.560 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.560 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.560 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:24.560 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.532349+0000", 2024-09-21T22:07:24.560 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.062519880999999999 2024-09-21T22:07:24.560 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:24.560 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:24.560 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:24.560 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.560 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.561 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4840.0:520 2.1 2:837ddd83:::rbd_data.126483d75f2d.000000000000000d:head [sparse-read 3936256~131072] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-21T22:07:24.561 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:21.884577+0000", 2024-09-21T22:07:24.561 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 2.6394489339999998, 2024-09-21T22:07:24.561 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0054559689999999997, 2024-09-21T22:07:24.561 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:24.561 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:24.561 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:24.561 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-21T22:07:24.561 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4840", 2024-09-21T22:07:24.561 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:24.561 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 520 2024-09-21T22:07:24.561 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.561 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:24.561 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.562 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:24.562 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.884577+0000", 2024-09-21T22:07:24.562 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:24.562 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.562 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.562 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:24.562 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.884577+0000", 2024-09-21T22:07:24.562 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:24.562 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.562 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.562 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:24.562 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.884578+0000", 2024-09-21T22:07:24.562 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.7199999999999997e-07 2024-09-21T22:07:24.562 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.563 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.563 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:24.563 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.884583+0000", 2024-09-21T22:07:24.563 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.0939999999999998e-06 2024-09-21T22:07:24.563 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.563 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.563 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:24.563 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.884599+0000", 2024-09-21T22:07:24.563 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5603999999999998e-05 2024-09-21T22:07:24.563 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.563 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.563 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:24.563 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.884608+0000", 2024-09-21T22:07:24.564 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.8209999999999997e-06 2024-09-21T22:07:24.564 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.564 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.564 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:24.564 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.884673+0000", 2024-09-21T22:07:24.564 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.4227999999999996e-05 2024-09-21T22:07:24.564 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.564 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.564 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:24.564 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.884757+0000", 2024-09-21T22:07:24.564 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.4184000000000006e-05 2024-09-21T22:07:24.564 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.564 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.564 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:24.564 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.890033+0000", 2024-09-21T22:07:24.565 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0052760660000000003 2024-09-21T22:07:24.565 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:24.565 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:24.565 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:24.565 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.565 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.565 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4840.0:521 2.1 2:837ddd83:::rbd_data.126483d75f2d.000000000000000d:head [sparse-read 4067328~126976] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-21T22:07:24.565 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:21.885652+0000", 2024-09-21T22:07:24.565 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 2.6383740590000002, 2024-09-21T22:07:24.565 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0051672310000000004, 2024-09-21T22:07:24.565 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:24.565 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:24.565 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:24.565 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-21T22:07:24.566 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4840", 2024-09-21T22:07:24.566 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:24.566 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 521 2024-09-21T22:07:24.566 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.566 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:24.566 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.566 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:24.566 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.885652+0000", 2024-09-21T22:07:24.566 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:24.566 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.566 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.566 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:24.566 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.885652+0000", 2024-09-21T22:07:24.566 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:24.566 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.567 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.567 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:24.567 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.885653+0000", 2024-09-21T22:07:24.567 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.5300000000000002e-07 2024-09-21T22:07:24.567 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.567 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.567 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:24.567 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.885658+0000", 2024-09-21T22:07:24.567 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.0760000000000002e-06 2024-09-21T22:07:24.567 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.567 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.567 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:24.567 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.885673+0000", 2024-09-21T22:07:24.567 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5613000000000001e-05 2024-09-21T22:07:24.568 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.568 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.568 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:24.568 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.885681+0000", 2024-09-21T22:07:24.568 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.7349999999999996e-06 2024-09-21T22:07:24.568 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.568 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.568 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:24.568 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.890102+0000", 2024-09-21T22:07:24.568 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0044207200000000004 2024-09-21T22:07:24.568 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.568 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.568 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:24.568 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.890202+0000", 2024-09-21T22:07:24.568 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00010026000000000001 2024-09-21T22:07:24.569 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.569 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.569 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:24.569 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.890819+0000", 2024-09-21T22:07:24.569 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000616874 2024-09-21T22:07:24.569 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:24.569 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:24.569 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:24.569 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.569 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.569 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4840.0:522 2.1 2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head [read 0~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e62)", 2024-09-21T22:07:24.569 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:21.885698+0000", 2024-09-21T22:07:24.569 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 2.6383274220000001, 2024-09-21T22:07:24.569 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0053696739999999996, 2024-09-21T22:07:24.570 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:24.570 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:24.570 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:24.570 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-21T22:07:24.570 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4840", 2024-09-21T22:07:24.570 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:24.570 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 522 2024-09-21T22:07:24.570 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.570 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:24.570 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.570 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:24.570 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.885698+0000", 2024-09-21T22:07:24.570 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:24.570 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.571 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.571 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:24.571 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.885698+0000", 2024-09-21T22:07:24.571 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:24.571 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.571 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.571 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:24.571 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.885699+0000", 2024-09-21T22:07:24.571 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.1799999999999995e-07 2024-09-21T22:07:24.571 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.571 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.571 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:24.571 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.885703+0000", 2024-09-21T22:07:24.571 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.4460000000000001e-06 2024-09-21T22:07:24.572 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.572 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.572 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:24.572 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.885723+0000", 2024-09-21T22:07:24.572 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.0446999999999999e-05 2024-09-21T22:07:24.572 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.572 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.572 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:24.572 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.885733+0000", 2024-09-21T22:07:24.572 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0261999999999999e-05 2024-09-21T22:07:24.572 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.572 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.572 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:24.572 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.890859+0000", 2024-09-21T22:07:24.573 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0051257009999999999 2024-09-21T22:07:24.573 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.573 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.573 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:24.573 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.890946+0000", 2024-09-21T22:07:24.573 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.6960999999999996e-05 2024-09-21T22:07:24.573 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.573 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.573 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:24.573 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:21.891068+0000", 2024-09-21T22:07:24.573 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00012223900000000001 2024-09-21T22:07:24.573 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:24.573 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:24.573 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:24.573 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.574 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.574 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:660 2.0 e63/16 2:13c3ab2f:::rbd_data.126483d75f2d.0000000000000007:head v 63'25, mlcod=63'23)", 2024-09-21T22:07:24.574 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.237326+0000", 2024-09-21T22:07:24.574 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.28670005500000001, 2024-09-21T22:07:24.574 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.023251076999999998, 2024-09-21T22:07:24.574 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:24.574 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:24.574 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:24.574 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:24.574 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.574 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:24.574 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.237326+0000", 2024-09-21T22:07:24.574 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:24.574 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.574 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.575 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:24.575 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.237326+0000", 2024-09-21T22:07:24.575 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:24.575 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.575 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.575 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:24.575 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.237327+0000", 2024-09-21T22:07:24.575 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.556e-06 2024-09-21T22:07:24.575 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.575 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.575 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:24.575 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.238306+0000", 2024-09-21T22:07:24.575 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00097845599999999994 2024-09-21T22:07:24.575 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.576 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.576 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:24.576 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.238327+0000", 2024-09-21T22:07:24.576 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.0937999999999998e-05 2024-09-21T22:07:24.576 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.576 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.576 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:24.576 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.238336+0000", 2024-09-21T22:07:24.576 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.6139999999999998e-06 2024-09-21T22:07:24.576 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.576 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.576 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:24.576 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.238395+0000", 2024-09-21T22:07:24.576 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.8619000000000002e-05 2024-09-21T22:07:24.576 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.577 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.577 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:24.577 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.238420+0000", 2024-09-21T22:07:24.577 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5401e-05 2024-09-21T22:07:24.577 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.577 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.577 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:24.577 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.260531+0000", 2024-09-21T22:07:24.577 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.022110696999999999 2024-09-21T22:07:24.577 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.577 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.577 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:24.577 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.260577+0000", 2024-09-21T22:07:24.577 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.5796e-05 2024-09-21T22:07:24.577 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:24.578 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:24.578 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:24.578 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.578 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.578 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:662 2.0 e63/16 2:13c3ab2f:::rbd_data.126483d75f2d.0000000000000007:head v 63'27, mlcod=63'25)", 2024-09-21T22:07:24.578 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.283744+0000", 2024-09-21T22:07:24.578 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.24028228700000001, 2024-09-21T22:07:24.578 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.023024228000000001, 2024-09-21T22:07:24.578 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:24.578 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:24.578 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:24.578 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:24.578 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.578 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:24.578 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.283744+0000", 2024-09-21T22:07:24.579 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:24.579 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.579 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.579 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:24.579 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.283744+0000", 2024-09-21T22:07:24.579 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:24.579 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.579 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.579 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:24.579 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.283745+0000", 2024-09-21T22:07:24.579 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.094e-06 2024-09-21T22:07:24.579 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.579 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.579 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:24.580 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.284666+0000", 2024-09-21T22:07:24.580 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00092117100000000003 2024-09-21T22:07:24.580 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.580 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.580 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:24.580 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.284681+0000", 2024-09-21T22:07:24.580 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5282e-05 2024-09-21T22:07:24.580 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.580 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.580 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:24.580 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.284691+0000", 2024-09-21T22:07:24.580 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.028e-05 2024-09-21T22:07:24.580 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.580 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.580 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:24.581 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.284746+0000", 2024-09-21T22:07:24.581 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.5010999999999999e-05 2024-09-21T22:07:24.581 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.581 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.581 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:24.581 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.284765+0000", 2024-09-21T22:07:24.581 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8614000000000002e-05 2024-09-21T22:07:24.581 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.581 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.581 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:24.581 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.306702+0000", 2024-09-21T22:07:24.581 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.021936619000000001 2024-09-21T22:07:24.581 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.581 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.581 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:24.582 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.306768+0000", 2024-09-21T22:07:24.582 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.6156999999999997e-05 2024-09-21T22:07:24.582 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:24.582 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:24.582 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:24.582 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.582 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.582 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:663 2.0 e63/16 2:13c3ab2f:::rbd_data.126483d75f2d.0000000000000007:head v 63'28, mlcod=63'26)", 2024-09-21T22:07:24.582 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.321198+0000", 2024-09-21T22:07:24.582 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.202828025, 2024-09-21T22:07:24.582 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0046467139999999997, 2024-09-21T22:07:24.582 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:24.582 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:24.582 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:24.582 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:24.583 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.583 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:24.583 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.321198+0000", 2024-09-21T22:07:24.583 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:24.583 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.583 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.583 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:24.583 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.321198+0000", 2024-09-21T22:07:24.583 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:24.583 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.583 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.583 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:24.583 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.321206+0000", 2024-09-21T22:07:24.583 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.8560000000000007e-06 2024-09-21T22:07:24.583 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.584 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.584 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:24.584 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.321326+0000", 2024-09-21T22:07:24.584 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000120091 2024-09-21T22:07:24.584 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.584 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.584 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:24.584 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.321355+0000", 2024-09-21T22:07:24.584 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9465e-05 2024-09-21T22:07:24.584 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.584 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.584 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:24.584 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.321371+0000", 2024-09-21T22:07:24.584 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5800000000000001e-05 2024-09-21T22:07:24.584 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.585 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.585 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:24.585 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.321448+0000", 2024-09-21T22:07:24.585 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.6643999999999993e-05 2024-09-21T22:07:24.585 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.585 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.585 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:24.585 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.321473+0000", 2024-09-21T22:07:24.585 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5440999999999999e-05 2024-09-21T22:07:24.585 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.585 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.585 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:24.585 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.325782+0000", 2024-09-21T22:07:24.585 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0043090050000000003 2024-09-21T22:07:24.585 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.586 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.586 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:24.586 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.325845+0000", 2024-09-21T22:07:24.586 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.2411999999999998e-05 2024-09-21T22:07:24.586 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:24.586 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:24.586 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:24.586 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.586 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.586 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:664 2.0 e63/16 2:13c3ab2f:::rbd_data.126483d75f2d.0000000000000007:head v 63'29, mlcod=63'27)", 2024-09-21T22:07:24.586 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.328877+0000", 2024-09-21T22:07:24.586 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.19514843200000001, 2024-09-21T22:07:24.586 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.023841243000000002, 2024-09-21T22:07:24.586 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:24.586 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:24.587 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:24.587 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:24.587 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.587 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:24.587 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.328877+0000", 2024-09-21T22:07:24.587 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:24.587 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.587 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.587 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:24.587 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.328877+0000", 2024-09-21T22:07:24.587 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:24.587 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.587 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.587 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:24.587 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.328879+0000", 2024-09-21T22:07:24.588 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.345e-06 2024-09-21T22:07:24.588 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.588 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.588 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:24.588 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.329896+0000", 2024-09-21T22:07:24.588 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0010169860000000001 2024-09-21T22:07:24.588 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.588 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.588 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:24.588 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.329930+0000", 2024-09-21T22:07:24.588 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.4001999999999997e-05 2024-09-21T22:07:24.588 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.588 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.588 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:24.589 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.329946+0000", 2024-09-21T22:07:24.589 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6297e-05 2024-09-21T22:07:24.589 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.589 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.589 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:24.589 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.330035+0000", 2024-09-21T22:07:24.589 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.9398e-05 2024-09-21T22:07:24.589 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.589 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.589 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:24.589 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.330063+0000", 2024-09-21T22:07:24.589 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.7013e-05 2024-09-21T22:07:24.589 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.590 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.590 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:24.590 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.352655+0000", 2024-09-21T22:07:24.590 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.022592325 2024-09-21T22:07:24.590 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.590 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.590 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:24.590 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.352719+0000", 2024-09-21T22:07:24.590 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.3876999999999998e-05 2024-09-21T22:07:24.590 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:24.590 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:24.590 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:24.590 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.590 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.591 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:666 2.0 e63/16 2:13c3ab2f:::rbd_data.126483d75f2d.0000000000000007:head v 63'31, mlcod=63'29)", 2024-09-21T22:07:24.591 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.371108+0000", 2024-09-21T22:07:24.591 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.15291808400000001, 2024-09-21T22:07:24.591 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.023120518, 2024-09-21T22:07:24.591 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:24.591 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:24.591 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:24.591 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:24.591 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.591 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:24.591 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.371108+0000", 2024-09-21T22:07:24.591 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:24.591 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.591 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.591 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:24.592 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.371108+0000", 2024-09-21T22:07:24.592 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:24.592 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.592 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.592 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:24.592 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.371110+0000", 2024-09-21T22:07:24.592 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.807e-06 2024-09-21T22:07:24.592 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.592 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.592 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:24.592 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.371834+0000", 2024-09-21T22:07:24.592 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00072392799999999998 2024-09-21T22:07:24.592 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.592 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.592 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:24.593 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.371861+0000", 2024-09-21T22:07:24.593 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.7650000000000001e-05 2024-09-21T22:07:24.593 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.593 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.593 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:24.593 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.371877+0000", 2024-09-21T22:07:24.593 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5438999999999999e-05 2024-09-21T22:07:24.593 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.593 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.593 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:24.593 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.371940+0000", 2024-09-21T22:07:24.593 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.3002000000000004e-05 2024-09-21T22:07:24.593 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.593 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.593 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:24.594 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.371960+0000", 2024-09-21T22:07:24.594 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.0021e-05 2024-09-21T22:07:24.594 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.594 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.594 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:24.594 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.394072+0000", 2024-09-21T22:07:24.594 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.022112634999999999 2024-09-21T22:07:24.594 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.594 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.594 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:24.594 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.394228+0000", 2024-09-21T22:07:24.594 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00015603599999999999 2024-09-21T22:07:24.594 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:24.594 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:24.595 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:24.595 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.595 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.595 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4840.0:667 2.3 2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head [write 0~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:07:24.595 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.401864+0000", 2024-09-21T22:07:24.595 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.12216152, 2024-09-21T22:07:24.595 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0067272790000000001, 2024-09-21T22:07:24.595 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:24.595 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:24.595 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:24.595 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-21T22:07:24.595 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4840", 2024-09-21T22:07:24.595 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:24.595 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 667 2024-09-21T22:07:24.596 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.596 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:24.596 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.596 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:24.596 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.401864+0000", 2024-09-21T22:07:24.596 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:24.596 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.596 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.596 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:24.596 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.401864+0000", 2024-09-21T22:07:24.596 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:24.596 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.596 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.596 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:24.596 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.401867+0000", 2024-09-21T22:07:24.597 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.2539999999999999e-06 2024-09-21T22:07:24.597 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.597 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.597 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:24.597 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.402043+0000", 2024-09-21T22:07:24.597 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000176622 2024-09-21T22:07:24.597 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.597 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.597 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:24.597 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.402070+0000", 2024-09-21T22:07:24.597 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.6347999999999999e-05 2024-09-21T22:07:24.597 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.597 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.597 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:24.597 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.402083+0000", 2024-09-21T22:07:24.598 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3611000000000001e-05 2024-09-21T22:07:24.598 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.598 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.598 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:24.598 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.402156+0000", 2024-09-21T22:07:24.598 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.3109999999999996e-05 2024-09-21T22:07:24.598 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.598 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.598 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:24.598 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.402229+0000", 2024-09-21T22:07:24.598 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.2835000000000003e-05 2024-09-21T22:07:24.598 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.598 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.598 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-21T22:07:24.598 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.402327+0000", 2024-09-21T22:07:24.599 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.8047999999999994e-05 2024-09-21T22:07:24.599 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.599 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.599 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-21T22:07:24.599 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.408258+0000", 2024-09-21T22:07:24.599 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0059308690000000001 2024-09-21T22:07:24.599 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.599 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.599 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:24.599 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.408450+0000", 2024-09-21T22:07:24.599 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00019169599999999999 2024-09-21T22:07:24.599 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.599 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.599 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:24.599 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.408505+0000", 2024-09-21T22:07:24.600 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.5364000000000001e-05 2024-09-21T22:07:24.600 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.600 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.600 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:24.600 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.408592+0000", 2024-09-21T22:07:24.600 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.6521999999999996e-05 2024-09-21T22:07:24.600 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:24.600 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:24.600 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:24.600 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.600 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.600 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4840.0:668 2.3 2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:07:24.600 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.409114+0000", 2024-09-21T22:07:24.600 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.11491172199999999, 2024-09-21T22:07:24.601 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.048703758999999999, 2024-09-21T22:07:24.601 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:24.601 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:24.601 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:24.601 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-21T22:07:24.601 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4840", 2024-09-21T22:07:24.601 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:24.601 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 668 2024-09-21T22:07:24.601 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.601 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:24.601 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.601 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:24.601 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.409114+0000", 2024-09-21T22:07:24.601 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:24.601 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.601 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.602 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:24.602 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.409114+0000", 2024-09-21T22:07:24.602 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:24.602 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.602 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.602 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:24.602 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.409115+0000", 2024-09-21T22:07:24.602 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.114e-06 2024-09-21T22:07:24.602 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.602 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.602 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:24.602 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.410308+0000", 2024-09-21T22:07:24.602 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011930949999999999 2024-09-21T22:07:24.602 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.603 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.603 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:24.603 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.410325+0000", 2024-09-21T22:07:24.603 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7088e-05 2024-09-21T22:07:24.603 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.603 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.603 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:24.603 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.410334+0000", 2024-09-21T22:07:24.603 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.2549999999999997e-06 2024-09-21T22:07:24.603 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.603 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.603 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:24.603 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.410393+0000", 2024-09-21T22:07:24.603 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.9327999999999998e-05 2024-09-21T22:07:24.603 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.604 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.604 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:24.604 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.410482+0000", 2024-09-21T22:07:24.604 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.9019999999999998e-05 2024-09-21T22:07:24.604 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.604 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.604 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-21T22:07:24.604 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.410617+0000", 2024-09-21T22:07:24.604 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000135232 2024-09-21T22:07:24.604 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.604 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.604 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:24.604 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.457051+0000", 2024-09-21T22:07:24.604 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.04643415 2024-09-21T22:07:24.604 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.605 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.605 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-21T22:07:24.605 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.457675+0000", 2024-09-21T22:07:24.605 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00062332700000000002 2024-09-21T22:07:24.605 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.605 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.605 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:24.605 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.457735+0000", 2024-09-21T22:07:24.605 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.029e-05 2024-09-21T22:07:24.605 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.605 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.605 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:24.605 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.457818+0000", 2024-09-21T22:07:24.605 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.2860000000000003e-05 2024-09-21T22:07:24.605 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:24.606 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:24.606 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:24.606 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.606 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.606 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop_reply(client.4840.0:668 2.3 e63/54 ondisk, result = 0)", 2024-09-21T22:07:24.606 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.442476+0000", 2024-09-21T22:07:24.606 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.081549430000000006, 2024-09-21T22:07:24.606 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.014582236, 2024-09-21T22:07:24.606 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:24.606 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:24.606 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:24.606 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:24.606 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.606 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:24.607 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.442476+0000", 2024-09-21T22:07:24.607 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:24.607 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.607 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.607 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:24.607 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.442476+0000", 2024-09-21T22:07:24.607 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:24.607 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.607 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.607 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:24.607 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.442478+0000", 2024-09-21T22:07:24.607 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6920000000000001e-06 2024-09-21T22:07:24.607 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.607 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.607 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:24.608 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.442490+0000", 2024-09-21T22:07:24.608 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.1807999999999999e-05 2024-09-21T22:07:24.608 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.608 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.608 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:24.608 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.442510+0000", 2024-09-21T22:07:24.608 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.0356999999999998e-05 2024-09-21T22:07:24.608 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.608 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.608 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:24.608 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.442522+0000", 2024-09-21T22:07:24.608 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2065000000000001e-05 2024-09-21T22:07:24.608 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.608 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.608 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:24.609 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.457034+0000", 2024-09-21T22:07:24.609 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.014511896 2024-09-21T22:07:24.609 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.609 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.609 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:24.609 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.457045+0000", 2024-09-21T22:07:24.609 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0701e-05 2024-09-21T22:07:24.609 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.609 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.609 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:24.609 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.457059+0000", 2024-09-21T22:07:24.609 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3716999999999999e-05 2024-09-21T22:07:24.609 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:24.609 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:24.609 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:24.610 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.610 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.610 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4840.0:669 2.3 2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head [write 1048576~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:07:24.610 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.459305+0000", 2024-09-21T22:07:24.610 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.064720544000000005, 2024-09-21T22:07:24.610 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0071712740000000001, 2024-09-21T22:07:24.610 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:24.610 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:24.610 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:24.610 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-21T22:07:24.610 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4840", 2024-09-21T22:07:24.610 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:24.610 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 669 2024-09-21T22:07:24.610 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.611 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:24.611 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.611 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:24.611 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.459305+0000", 2024-09-21T22:07:24.611 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:24.611 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.611 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.611 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:24.611 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.459305+0000", 2024-09-21T22:07:24.611 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:24.611 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.611 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.611 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:24.611 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.459308+0000", 2024-09-21T22:07:24.611 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1579999999999999e-06 2024-09-21T22:07:24.612 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.612 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.612 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:24.612 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.459469+0000", 2024-09-21T22:07:24.612 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00016120699999999999 2024-09-21T22:07:24.612 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.612 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.612 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:24.612 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.459492+0000", 2024-09-21T22:07:24.612 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3226000000000001e-05 2024-09-21T22:07:24.612 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.612 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.612 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:24.612 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.459504+0000", 2024-09-21T22:07:24.613 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.1858999999999999e-05 2024-09-21T22:07:24.613 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.613 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.613 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:24.613 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.459587+0000", 2024-09-21T22:07:24.613 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.3448000000000005e-05 2024-09-21T22:07:24.613 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.613 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.613 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:24.613 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.459676+0000", 2024-09-21T22:07:24.613 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.9124999999999998e-05 2024-09-21T22:07:24.613 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.613 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.613 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-21T22:07:24.613 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.459799+0000", 2024-09-21T22:07:24.614 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000122645 2024-09-21T22:07:24.614 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.614 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.614 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-21T22:07:24.614 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.465060+0000", 2024-09-21T22:07:24.614 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0052607180000000002 2024-09-21T22:07:24.614 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.614 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.614 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:24.614 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.466356+0000", 2024-09-21T22:07:24.614 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001296756 2024-09-21T22:07:24.614 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.614 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.614 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:24.614 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.466404+0000", 2024-09-21T22:07:24.615 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.7568999999999998e-05 2024-09-21T22:07:24.615 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:24.615 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:24.615 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:24.615 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.466477+0000", 2024-09-21T22:07:24.615 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.2563000000000003e-05 2024-09-21T22:07:24.615 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:24.615 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:24.615 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:24.615 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:24.615 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:24.615 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:07:24.617 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T22:07:24.617 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:24.612+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:24.718 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:24.714+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:24.787 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:07:24.787 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:07:24.787 INFO:teuthology.orchestra.run.smithi039.stdout: "num_ops": 0 2024-09-21T22:07:24.787 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:07:24.801 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T22:07:24.819 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:24.816+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:24.919 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:24.916+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:24.971 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:07:24.972 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:07:24.972 INFO:teuthology.orchestra.run.smithi039.stdout: "complaint_time": 30, 2024-09-21T22:07:24.972 INFO:teuthology.orchestra.run.smithi039.stdout: "num_blocked_ops": 0 2024-09-21T22:07:24.972 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:07:24.986 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T22:07:25.021 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:25.016+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:25.115 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:07:25.115 INFO:teuthology.orchestra.run.smithi039.stdout: "size": 20, 2024-09-21T22:07:25.115 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 600, 2024-09-21T22:07:25.115 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [ 2024-09-21T22:07:25.115 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.115 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPull(2.3 e55/54 cost 1048576)", 2024-09-21T22:07:25.115 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.081116+0000", 2024-09-21T22:07:25.115 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 13.030249453, 2024-09-21T22:07:25.115 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0030214159999999999, 2024-09-21T22:07:25.115 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:25.115 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:25.116 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-21T22:07:25.116 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:25.116 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.116 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:25.116 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.081116+0000", 2024-09-21T22:07:25.116 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:25.116 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.116 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.116 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:25.116 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.081116+0000", 2024-09-21T22:07:25.116 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:25.116 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.116 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.116 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:25.117 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.081118+0000", 2024-09-21T22:07:25.117 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.545e-06 2024-09-21T22:07:25.117 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.117 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.117 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:25.117 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.081133+0000", 2024-09-21T22:07:25.117 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5642e-05 2024-09-21T22:07:25.117 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.117 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.117 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:25.117 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.081155+0000", 2024-09-21T22:07:25.117 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1858e-05 2024-09-21T22:07:25.117 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.117 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.117 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:25.118 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.081171+0000", 2024-09-21T22:07:25.118 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5804e-05 2024-09-21T22:07:25.118 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.118 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.118 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:25.118 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.081291+0000", 2024-09-21T22:07:25.118 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000120553 2024-09-21T22:07:25.118 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.118 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.118 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:25.118 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.084137+0000", 2024-09-21T22:07:25.118 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.002846014 2024-09-21T22:07:25.118 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:25.118 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:25.119 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:25.119 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.119 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.119 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPull(2.3 e55/54 cost 1048576)", 2024-09-21T22:07:25.119 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.136357+0000", 2024-09-21T22:07:25.119 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 12.975008387000001, 2024-09-21T22:07:25.119 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.002843904, 2024-09-21T22:07:25.119 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:25.119 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:25.119 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-21T22:07:25.119 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:25.119 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.119 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:25.119 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.136357+0000", 2024-09-21T22:07:25.120 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:25.120 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.120 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.120 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:25.120 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.136357+0000", 2024-09-21T22:07:25.120 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:25.120 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.120 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.120 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:25.120 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.136359+0000", 2024-09-21T22:07:25.120 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.0669999999999999e-06 2024-09-21T22:07:25.120 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.120 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.120 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:25.121 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.136381+0000", 2024-09-21T22:07:25.121 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.2104999999999999e-05 2024-09-21T22:07:25.121 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.121 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.121 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:25.121 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.136412+0000", 2024-09-21T22:07:25.121 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.0602999999999998e-05 2024-09-21T22:07:25.121 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.121 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.121 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:25.121 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.136427+0000", 2024-09-21T22:07:25.121 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4832e-05 2024-09-21T22:07:25.121 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.121 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.122 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:25.122 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.136569+0000", 2024-09-21T22:07:25.122 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000142702 2024-09-21T22:07:25.122 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.122 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.122 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:25.122 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.139201+0000", 2024-09-21T22:07:25.122 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.002631595 2024-09-21T22:07:25.122 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:25.122 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:25.122 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:25.122 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.122 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.123 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPull(2.3 e55/54 cost 1048576)", 2024-09-21T22:07:25.123 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.296534+0000", 2024-09-21T22:07:25.123 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 12.814831055999999, 2024-09-21T22:07:25.123 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0023727290000000001, 2024-09-21T22:07:25.123 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:25.123 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:25.123 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-21T22:07:25.123 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:25.123 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.123 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:25.123 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.296534+0000", 2024-09-21T22:07:25.123 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:25.123 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.123 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.124 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:25.124 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.296534+0000", 2024-09-21T22:07:25.124 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:25.124 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.124 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.124 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:25.124 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.296537+0000", 2024-09-21T22:07:25.124 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1299999999999999e-06 2024-09-21T22:07:25.124 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.124 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.124 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:25.124 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.296567+0000", 2024-09-21T22:07:25.124 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.0812000000000002e-05 2024-09-21T22:07:25.124 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.125 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.125 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:25.125 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.296598+0000", 2024-09-21T22:07:25.125 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.0803000000000003e-05 2024-09-21T22:07:25.125 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.125 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.125 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:25.125 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.296625+0000", 2024-09-21T22:07:25.125 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.6452e-05 2024-09-21T22:07:25.125 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.125 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.125 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:25.125 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.296717+0000", 2024-09-21T22:07:25.125 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.1953000000000005e-05 2024-09-21T22:07:25.125 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.126 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.126 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:25.126 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.298907+0000", 2024-09-21T22:07:25.126 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0021905789999999998 2024-09-21T22:07:25.126 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:25.126 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:25.126 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:25.126 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.126 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.126 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPull(2.3 e55/54 cost 1048576)", 2024-09-21T22:07:25.126 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.341173+0000", 2024-09-21T22:07:25.126 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 12.770192058999999, 2024-09-21T22:07:25.126 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.002347933, 2024-09-21T22:07:25.127 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:25.127 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:25.127 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-21T22:07:25.127 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:25.127 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.127 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:25.127 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.341173+0000", 2024-09-21T22:07:25.127 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:25.127 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.127 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.127 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:25.127 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.341173+0000", 2024-09-21T22:07:25.127 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:25.127 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.127 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.128 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:25.128 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.341175+0000", 2024-09-21T22:07:25.128 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4479999999999999e-06 2024-09-21T22:07:25.128 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.128 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.128 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:25.128 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.341187+0000", 2024-09-21T22:07:25.128 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.221e-05 2024-09-21T22:07:25.128 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.128 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.128 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:25.128 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.341203+0000", 2024-09-21T22:07:25.128 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6099e-05 2024-09-21T22:07:25.128 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.128 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.129 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:25.129 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.341214+0000", 2024-09-21T22:07:25.129 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0461e-05 2024-09-21T22:07:25.129 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.129 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.129 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:25.129 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.341307+0000", 2024-09-21T22:07:25.129 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.3772999999999998e-05 2024-09-21T22:07:25.129 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.129 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.129 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:25.129 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.343521+0000", 2024-09-21T22:07:25.129 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.002213942 2024-09-21T22:07:25.129 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:25.130 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:25.130 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:25.130 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.130 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.130 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:667 2.3 e63/54 2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head v 63'4, mlcod=0'0)", 2024-09-21T22:07:25.130 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.402454+0000", 2024-09-21T22:07:25.130 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.70891186900000003, 2024-09-21T22:07:25.130 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0058375959999999996, 2024-09-21T22:07:25.130 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:25.130 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:25.130 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:25.130 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:25.130 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.130 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:25.131 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.402454+0000", 2024-09-21T22:07:25.131 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:25.131 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.131 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.131 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:25.131 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.402454+0000", 2024-09-21T22:07:25.131 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:25.131 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.131 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.131 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:25.131 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.402456+0000", 2024-09-21T22:07:25.131 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1179999999999999e-06 2024-09-21T22:07:25.131 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.131 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.132 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:25.132 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.402618+0000", 2024-09-21T22:07:25.132 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00016243 2024-09-21T22:07:25.132 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.132 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.132 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:25.132 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.402643+0000", 2024-09-21T22:07:25.132 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4638999999999999e-05 2024-09-21T22:07:25.132 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.132 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.132 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:25.132 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.402658+0000", 2024-09-21T22:07:25.132 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5211000000000001e-05 2024-09-21T22:07:25.132 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.132 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.133 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:25.133 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.402756+0000", 2024-09-21T22:07:25.133 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.7880000000000005e-05 2024-09-21T22:07:25.133 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.133 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.133 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:25.133 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.402779+0000", 2024-09-21T22:07:25.133 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3524999999999999e-05 2024-09-21T22:07:25.133 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.133 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.133 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:25.133 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.408233+0000", 2024-09-21T22:07:25.133 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0054534700000000002 2024-09-21T22:07:25.133 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.134 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.134 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:25.134 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.408291+0000", 2024-09-21T22:07:25.134 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.8322999999999997e-05 2024-09-21T22:07:25.134 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:25.134 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:25.134 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:25.134 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.134 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.134 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:668 2.3 e63/54 2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head v 63'5, mlcod=52'3)", 2024-09-21T22:07:25.134 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.410724+0000", 2024-09-21T22:07:25.134 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.70064145200000005, 2024-09-21T22:07:25.134 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.031657809000000002, 2024-09-21T22:07:25.135 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:25.135 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:25.135 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:25.135 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:25.135 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.135 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:25.135 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.410724+0000", 2024-09-21T22:07:25.135 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:25.135 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.135 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.135 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:25.135 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.410724+0000", 2024-09-21T22:07:25.135 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:25.135 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.136 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.136 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:25.136 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.410725+0000", 2024-09-21T22:07:25.136 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.4300000000000001e-07 2024-09-21T22:07:25.136 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.136 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.136 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:25.136 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.411436+0000", 2024-09-21T22:07:25.136 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00071118600000000004 2024-09-21T22:07:25.136 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.136 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.136 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:25.136 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.411455+0000", 2024-09-21T22:07:25.136 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8357999999999999e-05 2024-09-21T22:07:25.136 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.137 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.137 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:25.137 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.411470+0000", 2024-09-21T22:07:25.137 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5753999999999999e-05 2024-09-21T22:07:25.137 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.137 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.137 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:25.137 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.411537+0000", 2024-09-21T22:07:25.137 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.6766000000000004e-05 2024-09-21T22:07:25.137 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.137 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.137 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:25.137 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.411571+0000", 2024-09-21T22:07:25.137 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.3555e-05 2024-09-21T22:07:25.138 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.138 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.138 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:25.138 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.442317+0000", 2024-09-21T22:07:25.138 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.030746393 2024-09-21T22:07:25.138 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.138 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.138 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:25.138 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.442382+0000", 2024-09-21T22:07:25.138 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.4853999999999999e-05 2024-09-21T22:07:25.138 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:25.138 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:25.138 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:25.138 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.139 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.139 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:669 2.3 e63/54 2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head v 63'6, mlcod=63'4)", 2024-09-21T22:07:25.139 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.459924+0000", 2024-09-21T22:07:25.139 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.65144189399999997, 2024-09-21T22:07:25.139 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0062657900000000002, 2024-09-21T22:07:25.139 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:25.139 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:25.139 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:25.139 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:25.139 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.139 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:25.139 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.459924+0000", 2024-09-21T22:07:25.139 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:25.139 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.140 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.140 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:25.140 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.459924+0000", 2024-09-21T22:07:25.140 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:25.140 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.140 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.140 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:25.140 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.459925+0000", 2024-09-21T22:07:25.140 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8169999999999999e-06 2024-09-21T22:07:25.140 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.140 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.140 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:25.140 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.459983+0000", 2024-09-21T22:07:25.140 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.7567e-05 2024-09-21T22:07:25.141 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.141 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.141 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:25.141 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.460011+0000", 2024-09-21T22:07:25.141 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8455000000000001e-05 2024-09-21T22:07:25.141 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.141 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.141 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:25.141 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.460021+0000", 2024-09-21T22:07:25.141 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.9410000000000002e-06 2024-09-21T22:07:25.141 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.141 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.141 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:25.141 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.460093+0000", 2024-09-21T22:07:25.141 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.1231999999999998e-05 2024-09-21T22:07:25.142 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.142 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.142 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:25.142 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.460124+0000", 2024-09-21T22:07:25.142 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.1335999999999999e-05 2024-09-21T22:07:25.142 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.142 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.142 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:25.142 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.466130+0000", 2024-09-21T22:07:25.142 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0060055519999999999 2024-09-21T22:07:25.142 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.142 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.142 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:25.142 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.466189+0000", 2024-09-21T22:07:25.143 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.9889999999999997e-05 2024-09-21T22:07:25.143 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:25.143 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:25.143 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:25.143 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.143 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.143 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:670 2.3 e63/54 2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head v 63'7, mlcod=63'5)", 2024-09-21T22:07:25.143 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.468764+0000", 2024-09-21T22:07:25.143 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.64260191300000002, 2024-09-21T22:07:25.143 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.037963433999999997, 2024-09-21T22:07:25.143 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:25.143 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:25.143 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:25.143 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:25.144 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.144 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:25.144 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.468764+0000", 2024-09-21T22:07:25.144 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:25.144 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.144 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.144 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:25.144 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.468764+0000", 2024-09-21T22:07:25.144 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:25.144 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.144 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.144 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:25.144 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.468765+0000", 2024-09-21T22:07:25.144 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.9699999999999994e-07 2024-09-21T22:07:25.145 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.145 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.145 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:25.145 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.469295+0000", 2024-09-21T22:07:25.145 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00053003600000000001 2024-09-21T22:07:25.145 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.145 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.145 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:25.145 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.469311+0000", 2024-09-21T22:07:25.145 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6841999999999999e-05 2024-09-21T22:07:25.145 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.145 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.145 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:25.145 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.469324+0000", 2024-09-21T22:07:25.146 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2373e-05 2024-09-21T22:07:25.146 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.146 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.146 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:25.146 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.469391+0000", 2024-09-21T22:07:25.146 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.6880999999999999e-05 2024-09-21T22:07:25.146 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.146 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.146 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:25.146 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.469419+0000", 2024-09-21T22:07:25.146 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8742999999999999e-05 2024-09-21T22:07:25.146 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.146 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.147 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:25.147 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.506662+0000", 2024-09-21T22:07:25.147 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.037242994000000001 2024-09-21T22:07:25.147 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.147 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.147 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:25.147 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.506727+0000", 2024-09-21T22:07:25.147 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.4567999999999996e-05 2024-09-21T22:07:25.147 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:25.147 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:25.147 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:25.147 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.147 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.147 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:671 2.3 e63/54 2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head v 63'8, mlcod=63'6)", 2024-09-21T22:07:25.148 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.519938+0000", 2024-09-21T22:07:25.148 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.59142737400000001, 2024-09-21T22:07:25.148 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0058323419999999999, 2024-09-21T22:07:25.148 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:25.148 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:25.148 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:25.148 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:25.148 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.148 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:25.148 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.519938+0000", 2024-09-21T22:07:25.148 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:25.148 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.148 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.148 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:25.148 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.519938+0000", 2024-09-21T22:07:25.149 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:25.149 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.149 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.149 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:25.149 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.519940+0000", 2024-09-21T22:07:25.149 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.666e-06 2024-09-21T22:07:25.149 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.149 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.149 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:25.149 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.520031+0000", 2024-09-21T22:07:25.149 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.1051999999999995e-05 2024-09-21T22:07:25.149 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.149 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.149 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:25.150 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.520056+0000", 2024-09-21T22:07:25.150 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5459000000000001e-05 2024-09-21T22:07:25.150 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.150 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.150 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:25.150 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.520072+0000", 2024-09-21T22:07:25.150 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5801e-05 2024-09-21T22:07:25.150 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.150 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.150 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:25.150 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.520178+0000", 2024-09-21T22:07:25.150 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000105388 2024-09-21T22:07:25.150 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.150 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.151 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:25.151 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.520204+0000", 2024-09-21T22:07:25.151 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.6208e-05 2024-09-21T22:07:25.151 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.151 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.151 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:25.151 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.525735+0000", 2024-09-21T22:07:25.151 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0055308520000000002 2024-09-21T22:07:25.151 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.151 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.151 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:25.151 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.525770+0000", 2024-09-21T22:07:25.151 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.5916e-05 2024-09-21T22:07:25.151 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:25.152 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:25.152 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:25.152 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.152 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.152 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:672 2.3 e63/54 2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head v 63'9, mlcod=63'7)", 2024-09-21T22:07:25.152 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.528432+0000", 2024-09-21T22:07:25.152 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.58293399800000001, 2024-09-21T22:07:25.152 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.029763412999999999, 2024-09-21T22:07:25.152 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:25.152 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:25.152 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:25.152 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:25.152 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.152 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:25.153 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.528432+0000", 2024-09-21T22:07:25.153 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:25.153 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.153 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.153 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:25.153 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.528432+0000", 2024-09-21T22:07:25.153 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:25.153 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.153 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.153 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:25.153 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.528433+0000", 2024-09-21T22:07:25.153 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7179999999999999e-06 2024-09-21T22:07:25.153 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.153 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.154 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:25.154 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.528926+0000", 2024-09-21T22:07:25.154 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00049238500000000002 2024-09-21T22:07:25.154 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.154 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.154 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:25.154 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.528955+0000", 2024-09-21T22:07:25.154 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9037e-05 2024-09-21T22:07:25.154 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.154 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.154 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:25.154 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.528971+0000", 2024-09-21T22:07:25.154 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6622000000000001e-05 2024-09-21T22:07:25.154 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.155 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.155 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:25.155 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.529047+0000", 2024-09-21T22:07:25.155 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.5246000000000004e-05 2024-09-21T22:07:25.155 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.155 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.155 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:25.155 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.529074+0000", 2024-09-21T22:07:25.155 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.7532e-05 2024-09-21T22:07:25.155 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.155 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.155 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:25.155 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.558149+0000", 2024-09-21T22:07:25.155 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.029075192999999999 2024-09-21T22:07:25.156 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.156 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.156 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:25.156 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.558195+0000", 2024-09-21T22:07:25.156 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.5680000000000003e-05 2024-09-21T22:07:25.156 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:25.156 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:25.156 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:25.156 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.156 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.156 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:673 2.3 e63/54 2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head v 63'10, mlcod=63'8)", 2024-09-21T22:07:25.156 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.576513+0000", 2024-09-21T22:07:25.156 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.53485256299999995, 2024-09-21T22:07:25.156 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.003674666, 2024-09-21T22:07:25.156 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:25.157 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:25.157 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:25.157 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:25.157 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.157 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:25.157 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.576513+0000", 2024-09-21T22:07:25.157 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:25.157 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.157 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.157 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:25.157 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.576513+0000", 2024-09-21T22:07:25.157 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:25.157 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.158 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.158 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:25.158 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.576514+0000", 2024-09-21T22:07:25.158 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.1299999999999998e-07 2024-09-21T22:07:25.158 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.158 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.158 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:25.158 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.576567+0000", 2024-09-21T22:07:25.158 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.2710000000000002e-05 2024-09-21T22:07:25.158 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.158 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.158 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:25.158 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.576579+0000", 2024-09-21T22:07:25.159 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2035000000000001e-05 2024-09-21T22:07:25.159 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.159 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.159 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:25.159 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.576585+0000", 2024-09-21T22:07:25.159 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.6050000000000003e-06 2024-09-21T22:07:25.159 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.159 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.159 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:25.159 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.576638+0000", 2024-09-21T22:07:25.159 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.2726999999999998e-05 2024-09-21T22:07:25.159 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.159 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.159 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:25.160 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.576656+0000", 2024-09-21T22:07:25.160 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8084000000000002e-05 2024-09-21T22:07:25.160 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.160 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.160 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:25.160 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.580156+0000", 2024-09-21T22:07:25.160 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0035002700000000002 2024-09-21T22:07:25.160 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.160 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.160 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:25.160 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.580188+0000", 2024-09-21T22:07:25.160 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.1322000000000003e-05 2024-09-21T22:07:25.160 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:25.160 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:25.161 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:25.161 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.161 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.161 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:674 2.3 e63/54 2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head v 63'11, mlcod=63'9)", 2024-09-21T22:07:25.161 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.584683+0000", 2024-09-21T22:07:25.161 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.52668212199999997, 2024-09-21T22:07:25.161 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.028028352999999999, 2024-09-21T22:07:25.161 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:25.161 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:25.161 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:25.161 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:25.161 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.161 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:25.161 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.584683+0000", 2024-09-21T22:07:25.161 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:25.162 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.162 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.162 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:25.162 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.584683+0000", 2024-09-21T22:07:25.162 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:25.162 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.162 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.162 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:25.162 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.584684+0000", 2024-09-21T22:07:25.162 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.4600000000000004e-07 2024-09-21T22:07:25.162 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.162 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.162 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:25.162 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.585041+0000", 2024-09-21T22:07:25.163 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000356752 2024-09-21T22:07:25.163 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.163 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.163 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:25.163 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.585054+0000", 2024-09-21T22:07:25.163 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3387000000000001e-05 2024-09-21T22:07:25.163 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.163 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.163 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:25.163 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.585065+0000", 2024-09-21T22:07:25.163 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.1124999999999999e-05 2024-09-21T22:07:25.163 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.163 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.163 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:25.164 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.585142+0000", 2024-09-21T22:07:25.164 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.6743999999999996e-05 2024-09-21T22:07:25.164 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.164 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.164 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:25.164 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.585170+0000", 2024-09-21T22:07:25.164 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.7569999999999999e-05 2024-09-21T22:07:25.164 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.164 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.164 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:25.164 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.612637+0000", 2024-09-21T22:07:25.164 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.027467803999999998 2024-09-21T22:07:25.164 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.164 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.165 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:25.165 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.612712+0000", 2024-09-21T22:07:25.165 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.4325000000000004e-05 2024-09-21T22:07:25.165 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:25.165 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:25.165 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:25.165 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.165 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.165 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:683 2.3 e63/54 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head v 63'12, mlcod=63'10)", 2024-09-21T22:07:25.165 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.888242+0000", 2024-09-21T22:07:25.165 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.223123504, 2024-09-21T22:07:25.165 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0085158290000000008, 2024-09-21T22:07:25.165 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:25.165 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:25.166 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:25.166 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:25.166 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.166 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:25.166 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.888242+0000", 2024-09-21T22:07:25.166 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:25.166 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.166 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.166 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:25.166 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.888242+0000", 2024-09-21T22:07:25.166 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:25.166 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.166 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.166 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:25.167 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.888244+0000", 2024-09-21T22:07:25.167 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.553e-06 2024-09-21T22:07:25.167 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.167 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.167 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:25.167 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.888295+0000", 2024-09-21T22:07:25.167 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.1916000000000002e-05 2024-09-21T22:07:25.167 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.167 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.167 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:25.167 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.888319+0000", 2024-09-21T22:07:25.167 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3139999999999999e-05 2024-09-21T22:07:25.167 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.167 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.168 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:25.168 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.888332+0000", 2024-09-21T22:07:25.168 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2993000000000001e-05 2024-09-21T22:07:25.168 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.168 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.168 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:25.168 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.888441+0000", 2024-09-21T22:07:25.168 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000109039 2024-09-21T22:07:25.168 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.168 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.168 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:25.168 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.888481+0000", 2024-09-21T22:07:25.168 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.9950000000000002e-05 2024-09-21T22:07:25.168 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.168 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.169 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:25.169 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.896688+0000", 2024-09-21T22:07:25.169 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0082078729999999992 2024-09-21T22:07:25.169 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.169 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.169 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:25.169 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.896758+0000", 2024-09-21T22:07:25.169 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.9364999999999997e-05 2024-09-21T22:07:25.169 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:25.169 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:25.169 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:25.169 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.169 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.169 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:684 2.3 e63/54 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head v 63'13, mlcod=63'11)", 2024-09-21T22:07:25.169 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.899243+0000", 2024-09-21T22:07:25.170 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.21212272700000001, 2024-09-21T22:07:25.170 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.033436192000000003, 2024-09-21T22:07:25.170 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:25.170 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:25.170 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:25.170 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:25.170 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.170 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:25.170 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.899243+0000", 2024-09-21T22:07:25.170 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:25.170 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.170 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.170 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:25.170 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.899243+0000", 2024-09-21T22:07:25.170 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:25.171 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.171 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.171 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:25.171 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.899244+0000", 2024-09-21T22:07:25.171 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.705e-06 2024-09-21T22:07:25.171 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.171 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.171 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:25.171 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.899604+0000", 2024-09-21T22:07:25.171 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00035930599999999999 2024-09-21T22:07:25.171 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.171 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.171 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:25.171 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.899622+0000", 2024-09-21T22:07:25.172 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8487000000000001e-05 2024-09-21T22:07:25.172 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.172 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.172 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:25.172 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.899634+0000", 2024-09-21T22:07:25.172 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2116999999999999e-05 2024-09-21T22:07:25.172 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.172 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.172 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:25.172 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.899721+0000", 2024-09-21T22:07:25.172 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.6718999999999994e-05 2024-09-21T22:07:25.172 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.172 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.173 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:25.173 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.899755+0000", 2024-09-21T22:07:25.173 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.4292000000000003e-05 2024-09-21T22:07:25.173 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.173 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.173 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:25.173 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.932545+0000", 2024-09-21T22:07:25.173 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.03278959 2024-09-21T22:07:25.173 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.173 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.173 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:25.173 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.932679+0000", 2024-09-21T22:07:25.173 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013397600000000001 2024-09-21T22:07:25.173 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:25.174 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:25.174 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:25.174 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.174 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.174 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:685 2.3 e63/54 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head v 63'14, mlcod=63'12)", 2024-09-21T22:07:25.174 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.935246+0000", 2024-09-21T22:07:25.174 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.17611989, 2024-09-21T22:07:25.174 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0073590590000000003, 2024-09-21T22:07:25.174 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:25.174 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:25.174 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:25.174 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:25.174 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.174 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:25.175 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.935246+0000", 2024-09-21T22:07:25.175 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:25.175 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.175 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.175 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:25.175 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.935246+0000", 2024-09-21T22:07:25.175 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:25.175 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.175 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.175 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:25.175 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.935247+0000", 2024-09-21T22:07:25.175 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4470000000000001e-06 2024-09-21T22:07:25.175 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.175 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.176 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:25.176 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.935295+0000", 2024-09-21T22:07:25.176 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.7963000000000001e-05 2024-09-21T22:07:25.176 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.176 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.176 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:25.176 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.935313+0000", 2024-09-21T22:07:25.176 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8317000000000001e-05 2024-09-21T22:07:25.176 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.176 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.176 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:25.176 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.935352+0000", 2024-09-21T22:07:25.176 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.8552e-05 2024-09-21T22:07:25.176 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.177 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.177 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:25.177 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.935438+0000", 2024-09-21T22:07:25.177 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.6405999999999999e-05 2024-09-21T22:07:25.177 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.177 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.177 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:25.177 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.935471+0000", 2024-09-21T22:07:25.177 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.3142999999999998e-05 2024-09-21T22:07:25.177 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.177 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.177 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:25.177 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.942552+0000", 2024-09-21T22:07:25.177 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0070809510000000003 2024-09-21T22:07:25.178 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.178 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.178 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:25.178 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.942605+0000", 2024-09-21T22:07:25.178 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.2280000000000001e-05 2024-09-21T22:07:25.178 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:25.178 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:25.178 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:25.178 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.178 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.178 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:686 2.3 e63/54 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head v 63'15, mlcod=63'13)", 2024-09-21T22:07:25.178 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.945043+0000", 2024-09-21T22:07:25.178 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.166322577, 2024-09-21T22:07:25.178 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.04716302, 2024-09-21T22:07:25.179 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:25.179 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:25.179 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:25.179 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:25.179 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.179 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:25.179 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.945043+0000", 2024-09-21T22:07:25.179 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:25.179 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.179 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.179 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:25.179 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.945043+0000", 2024-09-21T22:07:25.179 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:25.180 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.180 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.180 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:25.180 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.945045+0000", 2024-09-21T22:07:25.180 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8700000000000001e-06 2024-09-21T22:07:25.180 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.180 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.180 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:25.180 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.945899+0000", 2024-09-21T22:07:25.180 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00085401900000000002 2024-09-21T22:07:25.180 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.180 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.180 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:25.180 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.945928+0000", 2024-09-21T22:07:25.181 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9193999999999999e-05 2024-09-21T22:07:25.181 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.181 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.181 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:25.181 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.945948+0000", 2024-09-21T22:07:25.181 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9876000000000001e-05 2024-09-21T22:07:25.181 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.181 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.181 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:25.181 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.946060+0000", 2024-09-21T22:07:25.181 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00011175099999999999 2024-09-21T22:07:25.181 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.181 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.181 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:25.182 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.946103+0000", 2024-09-21T22:07:25.182 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.3090999999999998e-05 2024-09-21T22:07:25.182 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.182 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.182 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:25.182 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.992151+0000", 2024-09-21T22:07:25.182 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.046047948999999998 2024-09-21T22:07:25.182 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.182 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.182 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:25.182 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.992206+0000", 2024-09-21T22:07:25.182 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.5269999999999998e-05 2024-09-21T22:07:25.182 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:25.182 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:25.183 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:25.183 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.183 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.183 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:687 2.3 e63/54 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head v 63'16, mlcod=63'14)", 2024-09-21T22:07:25.183 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.994473+0000", 2024-09-21T22:07:25.183 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.11689295399999999, 2024-09-21T22:07:25.183 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0054660990000000003, 2024-09-21T22:07:25.183 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:25.183 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:25.183 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:25.183 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:25.183 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.183 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:25.183 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.994473+0000", 2024-09-21T22:07:25.184 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:25.184 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.184 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.184 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:25.184 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.994473+0000", 2024-09-21T22:07:25.184 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:25.184 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.184 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.184 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:25.184 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.994474+0000", 2024-09-21T22:07:25.184 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.463e-06 2024-09-21T22:07:25.184 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.184 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.184 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:25.185 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.994546+0000", 2024-09-21T22:07:25.185 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.2260000000000003e-05 2024-09-21T22:07:25.185 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.185 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.185 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:25.185 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.994564+0000", 2024-09-21T22:07:25.185 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7983999999999999e-05 2024-09-21T22:07:25.185 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.185 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.185 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:25.185 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.994573+0000", 2024-09-21T22:07:25.185 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.4710000000000007e-06 2024-09-21T22:07:25.185 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.185 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.186 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:25.186 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.994634+0000", 2024-09-21T22:07:25.186 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.0872000000000003e-05 2024-09-21T22:07:25.186 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.186 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.186 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:25.186 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.994658+0000", 2024-09-21T22:07:25.186 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4573e-05 2024-09-21T22:07:25.186 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.186 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.186 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:25.186 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.999894+0000", 2024-09-21T22:07:25.187 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0052362390000000002 2024-09-21T22:07:25.187 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.187 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.187 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:25.187 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.999939+0000", 2024-09-21T22:07:25.187 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.4236999999999997e-05 2024-09-21T22:07:25.187 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:25.187 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:25.187 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:25.187 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.187 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.187 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:688 2.3 e63/54 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head v 63'17, mlcod=63'15)", 2024-09-21T22:07:25.187 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.001812+0000", 2024-09-21T22:07:25.188 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.109553873, 2024-09-21T22:07:25.188 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.028785011999999999, 2024-09-21T22:07:25.188 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:25.188 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:25.188 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:25.188 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:25.188 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.188 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:25.188 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.001812+0000", 2024-09-21T22:07:25.188 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:25.188 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.188 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.188 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:25.188 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.001812+0000", 2024-09-21T22:07:25.188 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:25.189 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.189 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.189 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:25.189 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.001813+0000", 2024-09-21T22:07:25.189 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2699999999999999e-06 2024-09-21T22:07:25.189 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.189 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.189 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:25.189 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.002252+0000", 2024-09-21T22:07:25.189 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00043878599999999998 2024-09-21T22:07:25.189 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.189 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.189 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:25.189 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.002271+0000", 2024-09-21T22:07:25.190 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9346000000000001e-05 2024-09-21T22:07:25.190 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.190 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.190 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:25.190 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.002282+0000", 2024-09-21T22:07:25.190 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0951000000000001e-05 2024-09-21T22:07:25.190 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.190 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.190 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:25.190 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.002398+0000", 2024-09-21T22:07:25.190 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000116412 2024-09-21T22:07:25.190 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.190 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.190 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:25.191 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.002425+0000", 2024-09-21T22:07:25.191 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.6291999999999998e-05 2024-09-21T22:07:25.191 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.191 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.191 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:25.191 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.030528+0000", 2024-09-21T22:07:25.191 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.028103390999999998 2024-09-21T22:07:25.191 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.191 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.191 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:25.191 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.030597+0000", 2024-09-21T22:07:25.191 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.8564000000000003e-05 2024-09-21T22:07:25.191 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:25.191 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:25.192 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:25.192 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.192 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.192 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:689 2.3 e63/54 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head v 63'18, mlcod=63'16)", 2024-09-21T22:07:25.192 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.032891+0000", 2024-09-21T22:07:25.192 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.078474291000000002, 2024-09-21T22:07:25.192 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.005061032, 2024-09-21T22:07:25.192 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:25.192 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:25.192 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:25.192 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:25.192 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.192 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:25.193 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.032891+0000", 2024-09-21T22:07:25.193 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:25.193 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.193 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.193 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:25.193 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.032891+0000", 2024-09-21T22:07:25.193 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:25.193 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.193 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.193 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:25.193 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.032893+0000", 2024-09-21T22:07:25.193 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3030000000000001e-06 2024-09-21T22:07:25.193 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.193 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.194 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:25.194 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.032952+0000", 2024-09-21T22:07:25.194 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.9070000000000002e-05 2024-09-21T22:07:25.194 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.194 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.194 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:25.194 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.032969+0000", 2024-09-21T22:07:25.194 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7839e-05 2024-09-21T22:07:25.194 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.194 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.194 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:25.194 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.032979+0000", 2024-09-21T22:07:25.194 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.5370000000000003e-06 2024-09-21T22:07:25.194 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.195 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.195 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:25.195 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.033041+0000", 2024-09-21T22:07:25.195 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.1562000000000005e-05 2024-09-21T22:07:25.195 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.195 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.195 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:25.195 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.033068+0000", 2024-09-21T22:07:25.195 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.7226999999999999e-05 2024-09-21T22:07:25.195 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.195 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.195 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:25.195 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.037908+0000", 2024-09-21T22:07:25.195 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0048401770000000002 2024-09-21T22:07:25.196 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.196 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.196 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:25.196 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.037952+0000", 2024-09-21T22:07:25.196 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.4317000000000003e-05 2024-09-21T22:07:25.196 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:25.196 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:25.196 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:25.196 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.196 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.196 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:690 2.3 e63/54 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head v 63'19, mlcod=63'17)", 2024-09-21T22:07:25.196 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.039813+0000", 2024-09-21T22:07:25.196 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.071552073999999993, 2024-09-21T22:07:25.196 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.035351891000000003, 2024-09-21T22:07:25.197 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:25.197 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:25.197 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:25.197 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:25.197 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.197 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:25.197 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.039813+0000", 2024-09-21T22:07:25.197 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:25.197 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.197 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.197 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:25.197 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.039813+0000", 2024-09-21T22:07:25.197 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:25.197 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.198 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.198 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:25.198 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.039814+0000", 2024-09-21T22:07:25.198 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.5300000000000002e-07 2024-09-21T22:07:25.198 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.198 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.198 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:25.198 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.040166+0000", 2024-09-21T22:07:25.198 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00035177 2024-09-21T22:07:25.198 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.198 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.198 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:25.198 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.040185+0000", 2024-09-21T22:07:25.198 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9000000000000001e-05 2024-09-21T22:07:25.198 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.199 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.199 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:25.199 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.040195+0000", 2024-09-21T22:07:25.199 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0254999999999999e-05 2024-09-21T22:07:25.199 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.199 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.199 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:25.199 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.040262+0000", 2024-09-21T22:07:25.199 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.6198000000000006e-05 2024-09-21T22:07:25.199 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.199 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.199 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:25.199 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.040288+0000", 2024-09-21T22:07:25.199 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5931999999999998e-05 2024-09-21T22:07:25.200 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.200 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.200 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:25.200 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.075106+0000", 2024-09-21T22:07:25.200 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.034818546999999998 2024-09-21T22:07:25.200 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:25.200 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:25.200 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:25.200 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.075165+0000", 2024-09-21T22:07:25.200 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.9236e-05 2024-09-21T22:07:25.200 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:25.200 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:25.200 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:25.200 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:25.200 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:25.201 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:07:25.202 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T22:07:25.203 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:25.196+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:25.215 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-21T22:07:25.302 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:25.298+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:25.358 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:07:25.359 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-21T22:07:25.359 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.359 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4840.0:700 2.5 2:a70ae99f:::rbd_data.126483d75f2d.000000000000000c:head [write 131072~917504 in=917504b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:07:25.359 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:25.352433+0000", 2024-09-21T22:07:25.359 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0068795310000000004, 2024-09-21T22:07:25.359 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0069305269999999997, 2024-09-21T22:07:25.359 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:25.359 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:25.359 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "waiting for sub ops", 2024-09-21T22:07:25.359 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:25.359 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4840", 2024-09-21T22:07:25.360 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:25.360 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 700 2024-09-21T22:07:25.360 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.360 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:25.360 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.360 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:25.360 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.352433+0000", 2024-09-21T22:07:25.360 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:25.360 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.360 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.360 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:25.360 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.352433+0000", 2024-09-21T22:07:25.360 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:25.361 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.361 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.361 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:25.361 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.352436+0000", 2024-09-21T22:07:25.361 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6429999999999999e-06 2024-09-21T22:07:25.361 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.361 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.361 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:25.361 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.353322+0000", 2024-09-21T22:07:25.361 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000886395 2024-09-21T22:07:25.361 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.361 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.361 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:25.361 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.353353+0000", 2024-09-21T22:07:25.362 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0685999999999997e-05 2024-09-21T22:07:25.362 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.362 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.362 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:25.362 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.353371+0000", 2024-09-21T22:07:25.362 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7873e-05 2024-09-21T22:07:25.362 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.362 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.362 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:25.362 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.353432+0000", 2024-09-21T22:07:25.362 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1365000000000006e-05 2024-09-21T22:07:25.362 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.362 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.362 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:25.362 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.353502+0000", 2024-09-21T22:07:25.363 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0018000000000006e-05 2024-09-21T22:07:25.363 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.363 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.363 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:25.363 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.353611+0000", 2024-09-21T22:07:25.363 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010871500000000001 2024-09-21T22:07:25.363 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:25.363 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:25.363 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:25.363 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:25.363 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-21T22:07:25.363 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 1 2024-09-21T22:07:25.363 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:07:25.373 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T22:07:25.403 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:25.398+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:25.473 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T22:07:25.504 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:07:25.504 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:07:25.504 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-21T22:07:25.504 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-21T22:07:25.504 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:07:25.505 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:25.502+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:25.519 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T22:07:25.585 INFO:teuthology.orchestra.run.smithi064.stderr:no valid command found; 10 closest matches: 2024-09-21T22:07:25.585 INFO:teuthology.orchestra.run.smithi064.stderr:0 2024-09-21T22:07:25.585 INFO:teuthology.orchestra.run.smithi064.stderr:1 2024-09-21T22:07:25.585 INFO:teuthology.orchestra.run.smithi064.stderr:2 2024-09-21T22:07:25.585 INFO:teuthology.orchestra.run.smithi064.stderr:abort 2024-09-21T22:07:25.585 INFO:teuthology.orchestra.run.smithi064.stderr:assert 2024-09-21T22:07:25.585 INFO:teuthology.orchestra.run.smithi064.stderr:bluestore allocator dump block 2024-09-21T22:07:25.585 INFO:teuthology.orchestra.run.smithi064.stderr:bluestore allocator fragmentation block 2024-09-21T22:07:25.585 INFO:teuthology.orchestra.run.smithi064.stderr:bluestore allocator fragmentation histogram block [] [] 2024-09-21T22:07:25.585 INFO:teuthology.orchestra.run.smithi064.stderr:bluestore allocator score block 2024-09-21T22:07:25.585 INFO:teuthology.orchestra.run.smithi064.stderr:config diff 2024-09-21T22:07:25.585 INFO:teuthology.orchestra.run.smithi064.stderr:admin_socket: invalid command 2024-09-21T22:07:25.587 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-21T22:07:25.587 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-5, ['dump_ops_in_flight'] 2024-09-21T22:07:25.634 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:25.632+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:25.681 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:07:25.681 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-21T22:07:25.681 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-21T22:07:25.681 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-21T22:07:25.681 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.681 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:07:25.681 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.036355+0000", 2024-09-21T22:07:25.681 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 83.643446940999993, 2024-09-21T22:07:25.681 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.062441934999999997, 2024-09-21T22:07:25.681 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:25.682 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:25.682 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:25.682 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:25.682 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:07:25.682 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:07:25.682 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1 2024-09-21T22:07:25.682 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.682 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:25.682 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.682 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:25.682 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036355+0000", 2024-09-21T22:07:25.682 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:25.682 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.683 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.683 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:25.683 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036355+0000", 2024-09-21T22:07:25.683 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:25.683 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.683 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.683 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:25.683 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036357+0000", 2024-09-21T22:07:25.683 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.446e-06 2024-09-21T22:07:25.683 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.683 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.683 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:25.683 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036373+0000", 2024-09-21T22:07:25.683 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5447e-05 2024-09-21T22:07:25.683 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.684 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.684 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:25.684 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036399+0000", 2024-09-21T22:07:25.684 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6423000000000001e-05 2024-09-21T22:07:25.684 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.684 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.684 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:25.684 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036420+0000", 2024-09-21T22:07:25.684 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1129999999999999e-05 2024-09-21T22:07:25.684 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.684 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.684 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:25.684 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036557+0000", 2024-09-21T22:07:25.684 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013686899999999999 2024-09-21T22:07:25.685 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.685 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.685 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-21T22:07:25.685 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036585+0000", 2024-09-21T22:07:25.685 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8362999999999999e-05 2024-09-21T22:07:25.685 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.685 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.685 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:25.685 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.095068+0000", 2024-09-21T22:07:25.685 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058482972000000001 2024-09-21T22:07:25.685 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.685 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.685 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:25.685 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.095516+0000", 2024-09-21T22:07:25.686 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00044802999999999998 2024-09-21T22:07:25.686 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.686 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.686 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:25.686 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.095869+0000", 2024-09-21T22:07:25.686 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000352104 2024-09-21T22:07:25.686 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.686 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.686 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:25.686 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.097055+0000", 2024-09-21T22:07:25.686 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011865599999999999 2024-09-21T22:07:25.686 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.686 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.686 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:25.687 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.098637+0000", 2024-09-21T22:07:25.687 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001581458 2024-09-21T22:07:25.687 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.687 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.687 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:25.687 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.098728+0000", 2024-09-21T22:07:25.687 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1043000000000002e-05 2024-09-21T22:07:25.687 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.687 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.687 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:25.687 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.098797+0000", 2024-09-21T22:07:25.687 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9090000000000004e-05 2024-09-21T22:07:25.687 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:25.687 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:25.687 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:25.688 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.688 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.688 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:07:25.688 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.468118+0000", 2024-09-21T22:07:25.688 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 83.211683594999997, 2024-09-21T22:07:25.688 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.027938877000000001, 2024-09-21T22:07:25.688 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:25.688 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:25.688 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:25.688 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:25.688 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:07:25.688 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:07:25.688 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 115 2024-09-21T22:07:25.688 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.689 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:25.689 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.689 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:25.689 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468118+0000", 2024-09-21T22:07:25.689 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:25.689 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.689 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.689 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:25.689 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468118+0000", 2024-09-21T22:07:25.689 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:25.689 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.689 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.689 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:25.689 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468120+0000", 2024-09-21T22:07:25.689 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7069999999999999e-06 2024-09-21T22:07:25.690 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.690 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.690 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:25.690 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468381+0000", 2024-09-21T22:07:25.690 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026135399999999999 2024-09-21T22:07:25.690 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.690 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.690 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:25.690 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468409+0000", 2024-09-21T22:07:25.690 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.781e-05 2024-09-21T22:07:25.690 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.690 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.690 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:25.690 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468433+0000", 2024-09-21T22:07:25.691 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4343999999999999e-05 2024-09-21T22:07:25.691 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.691 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.691 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:25.691 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.489785+0000", 2024-09-21T22:07:25.691 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021351221 2024-09-21T22:07:25.691 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.691 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.691 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:25.691 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.490018+0000", 2024-09-21T22:07:25.691 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00023317800000000001 2024-09-21T22:07:25.691 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.691 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.691 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:25.692 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.490412+0000", 2024-09-21T22:07:25.692 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00039417400000000001 2024-09-21T22:07:25.692 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.692 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.692 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:25.692 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.495694+0000", 2024-09-21T22:07:25.692 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.005282335 2024-09-21T22:07:25.692 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.692 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.692 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:25.692 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.495852+0000", 2024-09-21T22:07:25.692 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015777200000000001 2024-09-21T22:07:25.692 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.692 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.692 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:25.693 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.495938+0000", 2024-09-21T22:07:25.693 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6493e-05 2024-09-21T22:07:25.693 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.693 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.693 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:25.693 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.496057+0000", 2024-09-21T22:07:25.693 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000118489 2024-09-21T22:07:25.693 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:25.693 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:25.693 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:25.693 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.693 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.693 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:07:25.693 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.468474+0000", 2024-09-21T22:07:25.693 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 83.211327834000002, 2024-09-21T22:07:25.694 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.028015895999999998, 2024-09-21T22:07:25.694 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:25.694 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:25.694 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:25.694 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:25.694 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:07:25.694 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:07:25.694 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 116 2024-09-21T22:07:25.694 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.694 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:25.694 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.694 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:25.694 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468474+0000", 2024-09-21T22:07:25.695 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:25.695 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.695 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.695 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:25.695 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468474+0000", 2024-09-21T22:07:25.695 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:25.695 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.695 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.695 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:25.695 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468476+0000", 2024-09-21T22:07:25.695 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7060000000000001e-06 2024-09-21T22:07:25.695 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.695 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.695 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:25.696 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468488+0000", 2024-09-21T22:07:25.696 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.234e-05 2024-09-21T22:07:25.696 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.696 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.696 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:25.696 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468517+0000", 2024-09-21T22:07:25.696 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.898e-05 2024-09-21T22:07:25.696 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.696 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.696 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:25.696 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468536+0000", 2024-09-21T22:07:25.696 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8899999999999999e-05 2024-09-21T22:07:25.696 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.696 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.697 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:25.697 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.492376+0000", 2024-09-21T22:07:25.697 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023840659 2024-09-21T22:07:25.697 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.697 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.697 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:25.697 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.492621+0000", 2024-09-21T22:07:25.697 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000244794 2024-09-21T22:07:25.697 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.697 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.697 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:25.697 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.493008+0000", 2024-09-21T22:07:25.697 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00038629099999999998 2024-09-21T22:07:25.697 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.697 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.698 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:25.698 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.496192+0000", 2024-09-21T22:07:25.698 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0031847220000000001 2024-09-21T22:07:25.698 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.698 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.698 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:25.698 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.496233+0000", 2024-09-21T22:07:25.698 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0410000000000001e-05 2024-09-21T22:07:25.698 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.698 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.698 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:25.698 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.496374+0000", 2024-09-21T22:07:25.698 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000140872 2024-09-21T22:07:25.699 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.699 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.699 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:25.699 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.496490+0000", 2024-09-21T22:07:25.699 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000116222 2024-09-21T22:07:25.699 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:25.699 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:25.699 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:25.699 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.699 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.699 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:07:25.699 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.468568+0000", 2024-09-21T22:07:25.699 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 83.211233715000006, 2024-09-21T22:07:25.700 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.028181578999999998, 2024-09-21T22:07:25.700 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:25.700 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:25.700 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:25.700 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:25.700 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:07:25.700 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:07:25.700 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 117 2024-09-21T22:07:25.700 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.700 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:25.700 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.700 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:25.700 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468568+0000", 2024-09-21T22:07:25.700 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:25.701 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.701 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.701 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:25.701 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468568+0000", 2024-09-21T22:07:25.701 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:25.701 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.701 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.701 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:25.701 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468569+0000", 2024-09-21T22:07:25.701 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4139999999999999e-06 2024-09-21T22:07:25.701 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.701 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.701 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:25.701 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468583+0000", 2024-09-21T22:07:25.702 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3409e-05 2024-09-21T22:07:25.702 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.702 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.702 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:25.702 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468608+0000", 2024-09-21T22:07:25.702 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5559999999999999e-05 2024-09-21T22:07:25.702 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.702 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.702 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:25.702 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.468626+0000", 2024-09-21T22:07:25.702 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7863000000000002e-05 2024-09-21T22:07:25.702 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.702 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.702 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:25.702 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.494260+0000", 2024-09-21T22:07:25.703 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025633744999999999 2024-09-21T22:07:25.703 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.703 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.703 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:25.703 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.494519+0000", 2024-09-21T22:07:25.703 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025924800000000001 2024-09-21T22:07:25.703 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.703 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.703 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:25.703 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.494747+0000", 2024-09-21T22:07:25.703 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022792799999999999 2024-09-21T22:07:25.703 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.703 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.703 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:25.704 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.496519+0000", 2024-09-21T22:07:25.704 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017720030000000001 2024-09-21T22:07:25.704 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.704 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.704 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:25.704 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.496607+0000", 2024-09-21T22:07:25.704 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7873000000000004e-05 2024-09-21T22:07:25.704 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.704 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.704 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:25.704 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.496691+0000", 2024-09-21T22:07:25.704 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4009000000000002e-05 2024-09-21T22:07:25.704 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.704 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.704 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:25.705 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.496750+0000", 2024-09-21T22:07:25.705 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8526999999999997e-05 2024-09-21T22:07:25.705 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:25.705 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:25.705 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:25.705 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.705 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.705 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4840.0:662 2.0 2:13c3ab2f:::rbd_data.126483d75f2d.0000000000000007:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:07:25.705 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.283980+0000", 2024-09-21T22:07:25.705 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 1.395821379, 2024-09-21T22:07:25.705 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.036679130999999997, 2024-09-21T22:07:25.705 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:25.705 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:25.705 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:25.705 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:25.706 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4840", 2024-09-21T22:07:25.706 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:25.706 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 662 2024-09-21T22:07:25.706 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.706 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:25.706 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.706 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:25.706 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.283980+0000", 2024-09-21T22:07:25.706 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:25.706 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.706 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.706 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:25.706 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.283980+0000", 2024-09-21T22:07:25.706 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:25.707 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.707 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.707 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:25.707 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.283981+0000", 2024-09-21T22:07:25.707 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0920000000000001e-06 2024-09-21T22:07:25.707 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.707 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.707 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:25.707 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.284737+0000", 2024-09-21T22:07:25.707 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00075525500000000003 2024-09-21T22:07:25.707 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.707 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.707 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:25.707 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.284755+0000", 2024-09-21T22:07:25.708 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8428999999999999e-05 2024-09-21T22:07:25.708 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.708 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.708 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:25.708 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.284764+0000", 2024-09-21T22:07:25.708 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5839999999999995e-06 2024-09-21T22:07:25.708 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.708 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.708 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:25.708 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.284828+0000", 2024-09-21T22:07:25.708 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4709000000000007e-05 2024-09-21T22:07:25.708 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.708 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.708 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:25.708 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.284910+0000", 2024-09-21T22:07:25.709 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1478999999999997e-05 2024-09-21T22:07:25.709 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.709 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.709 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-09-21T22:07:25.709 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.285045+0000", 2024-09-21T22:07:25.709 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013465300000000001 2024-09-21T22:07:25.709 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.709 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.709 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:25.709 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.319249+0000", 2024-09-21T22:07:25.709 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.034204171999999998 2024-09-21T22:07:25.709 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.709 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.709 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:25.710 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.320488+0000", 2024-09-21T22:07:25.710 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001239789 2024-09-21T22:07:25.710 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.710 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.710 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:25.710 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.320595+0000", 2024-09-21T22:07:25.710 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010697299999999999 2024-09-21T22:07:25.710 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.710 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.710 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:25.710 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.320659+0000", 2024-09-21T22:07:25.710 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3996000000000002e-05 2024-09-21T22:07:25.710 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:25.710 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:25.711 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:25.711 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.711 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.711 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4840.0:664 2.0 2:13c3ab2f:::rbd_data.126483d75f2d.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:07:25.711 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.328703+0000", 2024-09-21T22:07:25.711 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 1.3510984960000001, 2024-09-21T22:07:25.711 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.032680873999999999, 2024-09-21T22:07:25.711 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:25.711 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:25.711 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:25.711 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:25.711 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4840", 2024-09-21T22:07:25.711 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:25.711 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 664 2024-09-21T22:07:25.711 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.712 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:25.712 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.712 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:25.712 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.328703+0000", 2024-09-21T22:07:25.712 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:25.712 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.712 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.712 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:25.712 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.328703+0000", 2024-09-21T22:07:25.712 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:25.712 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.712 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.712 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:25.712 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.328705+0000", 2024-09-21T22:07:25.713 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3459999999999999e-06 2024-09-21T22:07:25.713 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.713 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.713 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:25.713 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.329688+0000", 2024-09-21T22:07:25.713 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00098393799999999996 2024-09-21T22:07:25.713 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.713 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.713 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:25.713 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.329720+0000", 2024-09-21T22:07:25.713 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1779999999999997e-05 2024-09-21T22:07:25.713 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.713 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.713 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:25.713 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.329736+0000", 2024-09-21T22:07:25.714 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5396e-05 2024-09-21T22:07:25.714 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.714 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.714 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:25.714 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.329851+0000", 2024-09-21T22:07:25.714 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011579299999999999 2024-09-21T22:07:25.714 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.714 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.714 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:25.714 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.329982+0000", 2024-09-21T22:07:25.714 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000130176 2024-09-21T22:07:25.714 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.714 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.714 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-09-21T22:07:25.715 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.330150+0000", 2024-09-21T22:07:25.715 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016869699999999999 2024-09-21T22:07:25.715 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.715 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.715 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:25.715 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.360455+0000", 2024-09-21T22:07:25.715 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.030304714 2024-09-21T22:07:25.715 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.715 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.715 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:25.715 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.361223+0000", 2024-09-21T22:07:25.715 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00076771200000000004 2024-09-21T22:07:25.715 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.715 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.716 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:25.716 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.361318+0000", 2024-09-21T22:07:25.716 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5413000000000003e-05 2024-09-21T22:07:25.716 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.716 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.717 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:25.717 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.361384+0000", 2024-09-21T22:07:25.717 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5908999999999995e-05 2024-09-21T22:07:25.717 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:25.718 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:25.718 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:25.718 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.718 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.718 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4840.0:666 2.0 2:13c3ab2f:::rbd_data.126483d75f2d.0000000000000007:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:07:25.718 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.370777+0000", 2024-09-21T22:07:25.718 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 1.3090245190000001, 2024-09-21T22:07:25.718 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.031078489000000001, 2024-09-21T22:07:25.718 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:25.718 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:25.718 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:25.719 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:25.719 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4840", 2024-09-21T22:07:25.719 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:25.719 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 666 2024-09-21T22:07:25.719 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.719 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:25.719 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.719 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:25.719 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.370777+0000", 2024-09-21T22:07:25.719 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:25.719 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.719 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.719 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:25.719 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.370777+0000", 2024-09-21T22:07:25.720 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:25.720 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.720 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.720 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:25.720 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.370778+0000", 2024-09-21T22:07:25.720 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5799999999999998e-07 2024-09-21T22:07:25.720 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.720 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.720 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:25.720 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.371759+0000", 2024-09-21T22:07:25.720 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00098045500000000009 2024-09-21T22:07:25.720 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.720 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.720 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:25.721 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.371774+0000", 2024-09-21T22:07:25.721 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5227e-05 2024-09-21T22:07:25.721 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.721 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.721 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:25.721 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.371781+0000", 2024-09-21T22:07:25.721 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8809999999999998e-06 2024-09-21T22:07:25.721 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.721 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.721 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:25.721 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.371848+0000", 2024-09-21T22:07:25.721 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7772000000000001e-05 2024-09-21T22:07:25.721 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.721 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.721 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:25.722 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.371918+0000", 2024-09-21T22:07:25.722 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9709999999999995e-05 2024-09-21T22:07:25.722 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.722 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.722 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-09-21T22:07:25.722 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.372006+0000", 2024-09-21T22:07:25.722 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8246999999999997e-05 2024-09-21T22:07:25.722 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.722 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.722 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:25.722 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.401009+0000", 2024-09-21T22:07:25.722 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.02900266 2024-09-21T22:07:25.722 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.722 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.723 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:25.723 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.401750+0000", 2024-09-21T22:07:25.723 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000740903 2024-09-21T22:07:25.723 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.723 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.723 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:25.723 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.401812+0000", 2024-09-21T22:07:25.723 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2144000000000007e-05 2024-09-21T22:07:25.723 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.723 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.723 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:25.723 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.401856+0000", 2024-09-21T22:07:25.723 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3532000000000003e-05 2024-09-21T22:07:25.723 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:25.724 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:25.724 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:25.724 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.724 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.724 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4840.0:676 2.7 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:07:25.724 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.625676+0000", 2024-09-21T22:07:25.724 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 1.054125985, 2024-09-21T22:07:25.724 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.039450085000000003, 2024-09-21T22:07:25.724 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:25.724 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:25.724 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:25.724 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:25.724 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4840", 2024-09-21T22:07:25.724 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:25.724 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 676 2024-09-21T22:07:25.725 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.725 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:25.725 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.725 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:25.725 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.625676+0000", 2024-09-21T22:07:25.725 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:25.725 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.725 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.725 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:25.725 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.625676+0000", 2024-09-21T22:07:25.725 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:25.725 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.725 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.725 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:25.725 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.625677+0000", 2024-09-21T22:07:25.726 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5969999999999999e-06 2024-09-21T22:07:25.726 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.726 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.726 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:25.726 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.626810+0000", 2024-09-21T22:07:25.726 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001132267 2024-09-21T22:07:25.726 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.726 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.726 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:25.726 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.626842+0000", 2024-09-21T22:07:25.726 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2938999999999998e-05 2024-09-21T22:07:25.726 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.726 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.726 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:25.727 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.626857+0000", 2024-09-21T22:07:25.727 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4987000000000001e-05 2024-09-21T22:07:25.727 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.727 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.727 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:25.727 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.626930+0000", 2024-09-21T22:07:25.727 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2908000000000001e-05 2024-09-21T22:07:25.727 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.727 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.727 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:25.727 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.626990+0000", 2024-09-21T22:07:25.727 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9397000000000001e-05 2024-09-21T22:07:25.727 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.727 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.728 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-09-21T22:07:25.728 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.627082+0000", 2024-09-21T22:07:25.728 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1914000000000001e-05 2024-09-21T22:07:25.728 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.728 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.728 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:25.728 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.652887+0000", 2024-09-21T22:07:25.728 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025804911999999999 2024-09-21T22:07:25.728 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.728 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.728 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:25.728 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.665017+0000", 2024-09-21T22:07:25.728 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.01213008 2024-09-21T22:07:25.728 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.728 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.729 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:25.729 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.665075+0000", 2024-09-21T22:07:25.729 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8566000000000001e-05 2024-09-21T22:07:25.729 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.729 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.729 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:25.729 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.665126+0000", 2024-09-21T22:07:25.729 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0518e-05 2024-09-21T22:07:25.729 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:25.729 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:25.729 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:25.729 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.729 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.729 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4840.0:678 2.7 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:07:25.730 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.678289+0000", 2024-09-21T22:07:25.730 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 1.001512886, 2024-09-21T22:07:25.730 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.055089659999999999, 2024-09-21T22:07:25.730 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:25.730 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:25.730 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:25.730 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:25.730 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4840", 2024-09-21T22:07:25.730 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:25.730 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 678 2024-09-21T22:07:25.730 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.730 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:25.730 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.730 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:25.731 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.678289+0000", 2024-09-21T22:07:25.731 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:25.731 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.731 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.731 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:25.731 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.678289+0000", 2024-09-21T22:07:25.731 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:25.731 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.731 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.731 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:25.731 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.678291+0000", 2024-09-21T22:07:25.731 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7400000000000001e-06 2024-09-21T22:07:25.731 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.731 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.732 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:25.732 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.679129+0000", 2024-09-21T22:07:25.732 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00083801700000000004 2024-09-21T22:07:25.732 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.732 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.732 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:25.732 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.679161+0000", 2024-09-21T22:07:25.732 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2398999999999999e-05 2024-09-21T22:07:25.732 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.732 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.732 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:25.732 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.679176+0000", 2024-09-21T22:07:25.732 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5396999999999998e-05 2024-09-21T22:07:25.732 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.732 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.733 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:25.733 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.679256+0000", 2024-09-21T22:07:25.733 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9271999999999996e-05 2024-09-21T22:07:25.733 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.733 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.733 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:25.733 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.679333+0000", 2024-09-21T22:07:25.733 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7682999999999995e-05 2024-09-21T22:07:25.733 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.733 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.733 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-09-21T22:07:25.733 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.679457+0000", 2024-09-21T22:07:25.733 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012327900000000001 2024-09-21T22:07:25.733 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.733 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.734 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:25.734 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.713209+0000", 2024-09-21T22:07:25.734 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.033752058000000001 2024-09-21T22:07:25.734 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.734 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.734 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:25.734 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.733243+0000", 2024-09-21T22:07:25.734 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020034337999999999 2024-09-21T22:07:25.734 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.734 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.734 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:25.734 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.733320+0000", 2024-09-21T22:07:25.734 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7354999999999994e-05 2024-09-21T22:07:25.734 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.735 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.735 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:25.735 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.733378+0000", 2024-09-21T22:07:25.735 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8121999999999997e-05 2024-09-21T22:07:25.735 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:25.735 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:25.735 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:25.735 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.735 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.735 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4840.0:680 2.7 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:07:25.735 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.744506+0000", 2024-09-21T22:07:25.735 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.935296087, 2024-09-21T22:07:25.735 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.063865055000000004, 2024-09-21T22:07:25.735 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:25.736 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:25.736 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:25.736 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:25.736 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4840", 2024-09-21T22:07:25.736 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:25.736 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 680 2024-09-21T22:07:25.736 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.736 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:25.736 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.736 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:25.736 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.744506+0000", 2024-09-21T22:07:25.736 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:25.736 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.736 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.736 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:25.737 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.744506+0000", 2024-09-21T22:07:25.737 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:25.737 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.737 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.737 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:25.737 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.744508+0000", 2024-09-21T22:07:25.737 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.103e-06 2024-09-21T22:07:25.737 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.737 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.737 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:25.737 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.745371+0000", 2024-09-21T22:07:25.737 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00086322199999999997 2024-09-21T22:07:25.737 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.737 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.738 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:25.738 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.745401+0000", 2024-09-21T22:07:25.738 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9955999999999999e-05 2024-09-21T22:07:25.738 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.738 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.738 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:25.738 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.745416+0000", 2024-09-21T22:07:25.738 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4873e-05 2024-09-21T22:07:25.738 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.738 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.738 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:25.738 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.745509+0000", 2024-09-21T22:07:25.738 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3171000000000006e-05 2024-09-21T22:07:25.738 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.739 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.739 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:25.739 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.745632+0000", 2024-09-21T22:07:25.739 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012272699999999999 2024-09-21T22:07:25.739 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.739 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.739 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-09-21T22:07:25.739 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.745789+0000", 2024-09-21T22:07:25.739 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015781399999999999 2024-09-21T22:07:25.739 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.739 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.739 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:25.739 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.785494+0000", 2024-09-21T22:07:25.739 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.039704401 2024-09-21T22:07:25.739 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.740 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.740 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:25.740 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.808090+0000", 2024-09-21T22:07:25.740 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022596207 2024-09-21T22:07:25.740 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.740 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.740 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:25.740 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.808246+0000", 2024-09-21T22:07:25.740 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015567200000000001 2024-09-21T22:07:25.740 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.740 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.740 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:25.740 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.808371+0000", 2024-09-21T22:07:25.740 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012490900000000001 2024-09-21T22:07:25.741 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:25.741 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:25.741 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:25.741 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.741 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.741 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4840.0:682 2.7 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:07:25.741 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.822415+0000", 2024-09-21T22:07:25.741 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.85738652400000004, 2024-09-21T22:07:25.741 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.065044633000000004, 2024-09-21T22:07:25.741 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:25.741 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:25.741 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:25.741 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:25.741 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4840", 2024-09-21T22:07:25.742 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:25.742 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 682 2024-09-21T22:07:25.742 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.742 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:25.742 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.742 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:25.742 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.822415+0000", 2024-09-21T22:07:25.742 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:25.742 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.742 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.742 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:25.742 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.822415+0000", 2024-09-21T22:07:25.742 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:25.743 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.743 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.743 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:25.743 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.822417+0000", 2024-09-21T22:07:25.743 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8029999999999999e-06 2024-09-21T22:07:25.743 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.743 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.743 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:25.743 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.823308+0000", 2024-09-21T22:07:25.743 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00089146500000000003 2024-09-21T22:07:25.743 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.743 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.743 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:25.743 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.823339+0000", 2024-09-21T22:07:25.744 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0493000000000001e-05 2024-09-21T22:07:25.744 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.744 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.744 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:25.744 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.823354+0000", 2024-09-21T22:07:25.744 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5472999999999999e-05 2024-09-21T22:07:25.744 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.744 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.744 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:25.744 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.823476+0000", 2024-09-21T22:07:25.744 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000121209 2024-09-21T22:07:25.744 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.744 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.744 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:25.745 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.823650+0000", 2024-09-21T22:07:25.745 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017394599999999999 2024-09-21T22:07:25.745 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.745 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.745 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-09-21T22:07:25.745 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.823873+0000", 2024-09-21T22:07:25.745 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022314899999999999 2024-09-21T22:07:25.745 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.745 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.745 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:25.745 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.884396+0000", 2024-09-21T22:07:25.745 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.060523674999999999 2024-09-21T22:07:25.745 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.745 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.746 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:25.746 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.887238+0000", 2024-09-21T22:07:25.746 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028420450000000001 2024-09-21T22:07:25.746 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.746 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.746 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:25.746 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.887369+0000", 2024-09-21T22:07:25.746 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013010699999999999 2024-09-21T22:07:25.746 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.746 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.746 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:25.746 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.887460+0000", 2024-09-21T22:07:25.746 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1267999999999994e-05 2024-09-21T22:07:25.747 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:25.747 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:25.747 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:25.747 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.747 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.747 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4840.0:692 2.5 2:b1d55633:::rbd_data.126483d75f2d.000000000000000b:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:07:25.747 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:25.091048+0000", 2024-09-21T22:07:25.747 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.58875398999999995, 2024-09-21T22:07:25.747 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.060917092999999999, 2024-09-21T22:07:25.747 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:25.747 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:25.747 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:25.748 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:25.748 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4840", 2024-09-21T22:07:25.748 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:25.748 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 692 2024-09-21T22:07:25.748 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.748 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:25.748 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.748 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:25.748 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.091048+0000", 2024-09-21T22:07:25.748 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:25.748 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.748 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.748 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:25.748 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.091048+0000", 2024-09-21T22:07:25.748 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:25.749 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.749 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.749 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:25.749 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.091050+0000", 2024-09-21T22:07:25.749 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.03e-06 2024-09-21T22:07:25.749 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.749 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.749 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:25.749 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.091989+0000", 2024-09-21T22:07:25.749 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00093932100000000001 2024-09-21T22:07:25.749 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.749 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.749 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:25.749 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.092023+0000", 2024-09-21T22:07:25.750 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4310000000000002e-05 2024-09-21T22:07:25.750 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.750 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.750 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:25.750 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.092040+0000", 2024-09-21T22:07:25.750 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6426000000000002e-05 2024-09-21T22:07:25.750 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.750 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.750 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:25.750 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.092175+0000", 2024-09-21T22:07:25.750 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013500299999999999 2024-09-21T22:07:25.750 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.750 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.751 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:25.751 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.092307+0000", 2024-09-21T22:07:25.751 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000132562 2024-09-21T22:07:25.751 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.751 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.751 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:25.751 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.092499+0000", 2024-09-21T22:07:25.751 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000191972 2024-09-21T22:07:25.751 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.751 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.751 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:25.751 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.151065+0000", 2024-09-21T22:07:25.751 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058565514999999999 2024-09-21T22:07:25.751 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.751 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.752 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:25.752 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.151844+0000", 2024-09-21T22:07:25.752 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00077930800000000004 2024-09-21T22:07:25.752 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.752 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.752 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:25.752 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.151916+0000", 2024-09-21T22:07:25.752 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1921999999999993e-05 2024-09-21T22:07:25.752 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.752 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.752 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:25.752 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.151965+0000", 2024-09-21T22:07:25.752 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8724000000000003e-05 2024-09-21T22:07:25.753 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:25.753 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:25.753 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:25.753 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.753 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.753 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4840.0:692 2.5 e63/37 ondisk, result = 0)", 2024-09-21T22:07:25.753 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:25.118569+0000", 2024-09-21T22:07:25.753 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.56123262799999996, 2024-09-21T22:07:25.753 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.032507991, 2024-09-21T22:07:25.753 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:25.753 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:25.753 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:07:25.753 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:25.754 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.754 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:25.754 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.118569+0000", 2024-09-21T22:07:25.754 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:25.754 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.754 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.754 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:25.754 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.118569+0000", 2024-09-21T22:07:25.754 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:25.754 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.754 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.754 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:25.754 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.118571+0000", 2024-09-21T22:07:25.754 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5430000000000001e-06 2024-09-21T22:07:25.755 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.755 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.755 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:25.755 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.118579+0000", 2024-09-21T22:07:25.755 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5799999999999992e-06 2024-09-21T22:07:25.755 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.755 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.755 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:25.755 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.118607+0000", 2024-09-21T22:07:25.755 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7733999999999999e-05 2024-09-21T22:07:25.755 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.755 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.755 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:25.756 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.118622+0000", 2024-09-21T22:07:25.756 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5023e-05 2024-09-21T22:07:25.756 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.756 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.756 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:25.756 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.151038+0000", 2024-09-21T22:07:25.756 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.032415807999999997 2024-09-21T22:07:25.756 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.756 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.756 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:25.756 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.151053+0000", 2024-09-21T22:07:25.756 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5475e-05 2024-09-21T22:07:25.756 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.756 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.757 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:25.757 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.151077+0000", 2024-09-21T22:07:25.757 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3828e-05 2024-09-21T22:07:25.757 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:25.757 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:25.757 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:25.757 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.757 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.757 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4840.0:694 2.5 2:b1d55633:::rbd_data.126483d75f2d.000000000000000b:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:07:25.757 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:25.162829+0000", 2024-09-21T22:07:25.757 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.51697265699999995, 2024-09-21T22:07:25.757 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.045653405000000001, 2024-09-21T22:07:25.757 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:25.757 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:25.758 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:25.758 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:25.758 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4840", 2024-09-21T22:07:25.758 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:25.758 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 694 2024-09-21T22:07:25.758 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.758 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:25.758 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.758 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:25.758 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.162829+0000", 2024-09-21T22:07:25.758 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:25.758 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.758 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.758 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:25.759 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.162829+0000", 2024-09-21T22:07:25.759 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:25.759 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.759 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.759 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:25.759 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.162831+0000", 2024-09-21T22:07:25.759 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7799999999999999e-06 2024-09-21T22:07:25.759 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.759 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.759 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:25.759 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.163904+0000", 2024-09-21T22:07:25.759 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001073673 2024-09-21T22:07:25.759 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.759 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.760 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:25.760 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.163937+0000", 2024-09-21T22:07:25.760 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2100000000000001e-05 2024-09-21T22:07:25.760 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.760 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.760 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:25.760 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.163952+0000", 2024-09-21T22:07:25.760 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5517000000000001e-05 2024-09-21T22:07:25.760 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.760 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.760 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:25.760 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.164060+0000", 2024-09-21T22:07:25.760 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000107909 2024-09-21T22:07:25.760 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.761 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.761 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:25.761 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.164162+0000", 2024-09-21T22:07:25.761 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000101619 2024-09-21T22:07:25.761 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.761 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.761 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:25.761 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.164318+0000", 2024-09-21T22:07:25.761 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015683599999999999 2024-09-21T22:07:25.761 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.761 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.761 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:25.761 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.207369+0000", 2024-09-21T22:07:25.761 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.043050401000000002 2024-09-21T22:07:25.762 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.762 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.762 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:25.762 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.208256+0000", 2024-09-21T22:07:25.762 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00088747800000000003 2024-09-21T22:07:25.762 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.762 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.762 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:25.762 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.208350+0000", 2024-09-21T22:07:25.762 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4129000000000006e-05 2024-09-21T22:07:25.762 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.762 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.762 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:25.762 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.208482+0000", 2024-09-21T22:07:25.763 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000131963 2024-09-21T22:07:25.763 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:25.763 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:25.763 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:25.763 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.763 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.763 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4840.0:696 2.5 2:b1d55633:::rbd_data.126483d75f2d.000000000000000b:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:07:25.763 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:25.218641+0000", 2024-09-21T22:07:25.763 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.46116072200000002, 2024-09-21T22:07:25.763 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058363669999999999, 2024-09-21T22:07:25.763 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:25.763 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:25.763 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:25.764 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:25.764 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4840", 2024-09-21T22:07:25.764 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:25.764 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 696 2024-09-21T22:07:25.764 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.764 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:25.764 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.764 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:25.764 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.218641+0000", 2024-09-21T22:07:25.764 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:25.764 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.764 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.764 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:25.764 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.218641+0000", 2024-09-21T22:07:25.765 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:25.765 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.765 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.765 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:25.765 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.218642+0000", 2024-09-21T22:07:25.765 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.266e-06 2024-09-21T22:07:25.765 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.765 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.765 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:25.765 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.219432+0000", 2024-09-21T22:07:25.765 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00078986300000000002 2024-09-21T22:07:25.765 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.765 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.766 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:25.766 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.219450+0000", 2024-09-21T22:07:25.766 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7526000000000001e-05 2024-09-21T22:07:25.766 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.766 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.766 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:25.766 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.219459+0000", 2024-09-21T22:07:25.766 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3000000000000007e-06 2024-09-21T22:07:25.766 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.766 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.766 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:25.766 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.219655+0000", 2024-09-21T22:07:25.766 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019599099999999999 2024-09-21T22:07:25.766 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.767 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.767 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:25.767 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.219917+0000", 2024-09-21T22:07:25.767 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026198200000000001 2024-09-21T22:07:25.767 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.767 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.767 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:25.767 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.220198+0000", 2024-09-21T22:07:25.767 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00028154500000000002 2024-09-21T22:07:25.767 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.767 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.767 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:25.767 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.268856+0000", 2024-09-21T22:07:25.767 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.048657687999999998 2024-09-21T22:07:25.768 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.768 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.768 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:25.768 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.276903+0000", 2024-09-21T22:07:25.768 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0080472249999999999 2024-09-21T22:07:25.768 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.768 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.768 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:25.768 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.276958+0000", 2024-09-21T22:07:25.768 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4184999999999997e-05 2024-09-21T22:07:25.768 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.768 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.768 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:25.768 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.277005+0000", 2024-09-21T22:07:25.769 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7098999999999997e-05 2024-09-21T22:07:25.769 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:25.769 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:25.769 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:25.769 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.769 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.769 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4840.0:698 2.5 2:b1d55633:::rbd_data.126483d75f2d.000000000000000b:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:07:25.769 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:25.289380+0000", 2024-09-21T22:07:25.769 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.39042217499999998, 2024-09-21T22:07:25.769 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.055691072000000001, 2024-09-21T22:07:25.769 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:25.769 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:25.769 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:25.769 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:25.770 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4840", 2024-09-21T22:07:25.770 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:25.770 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 698 2024-09-21T22:07:25.770 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.770 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:25.770 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.770 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:25.770 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.289380+0000", 2024-09-21T22:07:25.770 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:25.770 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.770 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.770 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:25.770 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.289380+0000", 2024-09-21T22:07:25.771 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:25.771 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.771 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.771 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:25.771 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.289381+0000", 2024-09-21T22:07:25.771 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3540000000000001e-06 2024-09-21T22:07:25.771 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.771 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.771 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:25.771 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.290221+0000", 2024-09-21T22:07:25.771 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00083999900000000002 2024-09-21T22:07:25.771 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.771 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.771 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:25.772 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.290238+0000", 2024-09-21T22:07:25.772 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7130999999999999e-05 2024-09-21T22:07:25.772 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.772 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.772 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:25.772 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.290247+0000", 2024-09-21T22:07:25.772 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6500000000000002e-06 2024-09-21T22:07:25.772 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.772 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.772 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:25.772 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.290320+0000", 2024-09-21T22:07:25.772 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3530999999999998e-05 2024-09-21T22:07:25.772 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.772 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.773 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:25.773 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.290433+0000", 2024-09-21T22:07:25.773 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000112744 2024-09-21T22:07:25.773 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.773 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.773 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:25.773 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.290605+0000", 2024-09-21T22:07:25.773 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017253700000000001 2024-09-21T22:07:25.773 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.773 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.773 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:25.773 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.320545+0000", 2024-09-21T22:07:25.773 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029939582999999999 2024-09-21T22:07:25.773 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.773 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.774 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:25.774 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.344991+0000", 2024-09-21T22:07:25.774 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024445956000000001 2024-09-21T22:07:25.774 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.774 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.774 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:25.774 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.345034+0000", 2024-09-21T22:07:25.774 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2806000000000003e-05 2024-09-21T22:07:25.774 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.774 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.774 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:25.774 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.345071+0000", 2024-09-21T22:07:25.774 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6780999999999999e-05 2024-09-21T22:07:25.774 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:25.775 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:25.775 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:25.775 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.775 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.775 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4840.0:700 2.5 2:a70ae99f:::rbd_data.126483d75f2d.000000000000000c:head [write 131072~917504 in=917504b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:07:25.775 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:25.352433+0000", 2024-09-21T22:07:25.775 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.32736857699999999, 2024-09-21T22:07:25.775 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.030906125999999999, 2024-09-21T22:07:25.775 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:25.775 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:25.775 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:25.775 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:25.775 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4840", 2024-09-21T22:07:25.775 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:25.775 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 700 2024-09-21T22:07:25.776 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.776 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:25.776 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.776 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:25.776 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.352433+0000", 2024-09-21T22:07:25.776 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:25.776 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.776 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.776 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:25.776 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.352433+0000", 2024-09-21T22:07:25.776 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:25.776 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.776 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.776 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:25.777 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.352436+0000", 2024-09-21T22:07:25.777 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6429999999999999e-06 2024-09-21T22:07:25.777 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.777 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.777 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:25.777 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.353322+0000", 2024-09-21T22:07:25.777 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000886395 2024-09-21T22:07:25.777 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.777 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.777 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:25.777 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.353353+0000", 2024-09-21T22:07:25.777 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0685999999999997e-05 2024-09-21T22:07:25.777 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.777 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.778 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:25.778 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.353371+0000", 2024-09-21T22:07:25.778 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7873e-05 2024-09-21T22:07:25.778 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.778 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.778 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:25.778 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.353432+0000", 2024-09-21T22:07:25.778 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1365000000000006e-05 2024-09-21T22:07:25.778 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.778 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.778 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:25.778 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.353502+0000", 2024-09-21T22:07:25.778 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0018000000000006e-05 2024-09-21T22:07:25.778 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.779 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.779 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:25.779 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.353611+0000", 2024-09-21T22:07:25.779 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010871500000000001 2024-09-21T22:07:25.779 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.779 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.779 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:25.779 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.382395+0000", 2024-09-21T22:07:25.779 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.028784265 2024-09-21T22:07:25.779 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.779 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.779 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:25.779 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.383250+0000", 2024-09-21T22:07:25.779 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00085512900000000005 2024-09-21T22:07:25.780 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.780 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.780 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:25.780 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.383298+0000", 2024-09-21T22:07:25.780 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7435999999999998e-05 2024-09-21T22:07:25.780 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.780 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.780 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:25.780 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.383339+0000", 2024-09-21T22:07:25.780 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1600999999999997e-05 2024-09-21T22:07:25.780 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:25.780 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:25.780 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:25.780 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.780 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.781 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4840.0:702 2.5 2:a70ae99f:::rbd_data.126483d75f2d.000000000000000c:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:07:25.781 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:25.392746+0000", 2024-09-21T22:07:25.781 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.28705556799999998, 2024-09-21T22:07:25.781 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.037346635000000003, 2024-09-21T22:07:25.781 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:25.781 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:25.781 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:25.781 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:25.781 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4840", 2024-09-21T22:07:25.781 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:25.781 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 702 2024-09-21T22:07:25.781 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.781 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:25.781 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.781 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:25.782 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.392746+0000", 2024-09-21T22:07:25.782 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:25.782 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.782 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.782 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:25.782 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.392746+0000", 2024-09-21T22:07:25.782 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:25.782 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.782 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.782 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:25.782 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.392748+0000", 2024-09-21T22:07:25.782 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.395e-06 2024-09-21T22:07:25.782 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.782 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.783 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:25.783 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.393583+0000", 2024-09-21T22:07:25.783 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00083551000000000003 2024-09-21T22:07:25.783 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.783 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.783 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:25.783 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.393608+0000", 2024-09-21T22:07:25.783 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4729999999999999e-05 2024-09-21T22:07:25.783 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.783 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.783 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:25.783 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.393620+0000", 2024-09-21T22:07:25.783 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2548000000000001e-05 2024-09-21T22:07:25.783 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.784 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.784 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:25.784 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.393668+0000", 2024-09-21T22:07:25.784 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7587999999999999e-05 2024-09-21T22:07:25.784 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.784 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.784 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:25.784 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.393724+0000", 2024-09-21T22:07:25.784 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6023999999999998e-05 2024-09-21T22:07:25.784 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.784 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.784 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:25.784 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.393809+0000", 2024-09-21T22:07:25.784 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5016000000000001e-05 2024-09-21T22:07:25.785 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.785 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.785 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:25.785 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.415742+0000", 2024-09-21T22:07:25.785 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021933540000000001 2024-09-21T22:07:25.785 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.785 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.785 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:25.785 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.430019+0000", 2024-09-21T22:07:25.785 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014276577 2024-09-21T22:07:25.785 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.785 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.785 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:25.785 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.430059+0000", 2024-09-21T22:07:25.786 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9966999999999999e-05 2024-09-21T22:07:25.786 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.786 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.786 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:25.786 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.430093+0000", 2024-09-21T22:07:25.786 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3739999999999999e-05 2024-09-21T22:07:25.786 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:25.786 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:25.786 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:25.786 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.786 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.786 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4840.0:704 2.5 2:a70ae99f:::rbd_data.126483d75f2d.000000000000000c:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:07:25.786 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:25.438295+0000", 2024-09-21T22:07:25.786 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.24150618099999999, 2024-09-21T22:07:25.787 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.030415626000000001, 2024-09-21T22:07:25.787 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:25.787 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:25.787 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:25.787 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:25.787 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4840", 2024-09-21T22:07:25.787 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:25.787 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 704 2024-09-21T22:07:25.787 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.787 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:25.787 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.787 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:25.787 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.438295+0000", 2024-09-21T22:07:25.787 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:25.788 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.788 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.788 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:25.788 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.438295+0000", 2024-09-21T22:07:25.788 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:25.788 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.788 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.788 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:25.788 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.438297+0000", 2024-09-21T22:07:25.788 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.48e-06 2024-09-21T22:07:25.788 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.788 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.788 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:25.788 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.439138+0000", 2024-09-21T22:07:25.788 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00084115500000000001 2024-09-21T22:07:25.789 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.789 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.789 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:25.789 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.439160+0000", 2024-09-21T22:07:25.789 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1588000000000001e-05 2024-09-21T22:07:25.789 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.789 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.789 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:25.789 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.439170+0000", 2024-09-21T22:07:25.789 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0482000000000001e-05 2024-09-21T22:07:25.789 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.789 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.789 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:25.789 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.439245+0000", 2024-09-21T22:07:25.790 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5172000000000005e-05 2024-09-21T22:07:25.790 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.790 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.790 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:25.790 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.439308+0000", 2024-09-21T22:07:25.790 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2724000000000005e-05 2024-09-21T22:07:25.790 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.790 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.790 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:25.790 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.439399+0000", 2024-09-21T22:07:25.790 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0818999999999999e-05 2024-09-21T22:07:25.790 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.790 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.790 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:25.791 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.463906+0000", 2024-09-21T22:07:25.791 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024507542 2024-09-21T22:07:25.791 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.791 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.791 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:25.791 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.468633+0000", 2024-09-21T22:07:25.791 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0047264170000000001 2024-09-21T22:07:25.791 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.791 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.791 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:25.791 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.468676+0000", 2024-09-21T22:07:25.791 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3075999999999999e-05 2024-09-21T22:07:25.791 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.791 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.792 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:25.792 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.468711+0000", 2024-09-21T22:07:25.792 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5170999999999999e-05 2024-09-21T22:07:25.792 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:25.792 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:25.792 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:25.792 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.792 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.792 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4840.0:706 2.5 2:a70ae99f:::rbd_data.126483d75f2d.000000000000000c:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:07:25.792 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:25.476388+0000", 2024-09-21T22:07:25.792 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.20341327400000001, 2024-09-21T22:07:25.792 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.031232883999999999, 2024-09-21T22:07:25.792 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:25.792 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:25.793 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:25.793 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:25.793 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4840", 2024-09-21T22:07:25.793 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:25.793 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 706 2024-09-21T22:07:25.793 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.793 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:25.793 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.793 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:25.793 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.476388+0000", 2024-09-21T22:07:25.793 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:25.793 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.793 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.793 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:25.794 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.476388+0000", 2024-09-21T22:07:25.794 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:25.794 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.794 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.794 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:25.794 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.476390+0000", 2024-09-21T22:07:25.794 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.133e-06 2024-09-21T22:07:25.794 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.794 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.794 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:25.794 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.477229+0000", 2024-09-21T22:07:25.794 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00083948499999999995 2024-09-21T22:07:25.794 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.795 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.795 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:25.795 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.477242+0000", 2024-09-21T22:07:25.795 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3101999999999999e-05 2024-09-21T22:07:25.795 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.795 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.795 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:25.795 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.477248+0000", 2024-09-21T22:07:25.795 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3110000000000002e-06 2024-09-21T22:07:25.795 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.795 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.795 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:25.795 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.477313+0000", 2024-09-21T22:07:25.796 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4375999999999995e-05 2024-09-21T22:07:25.796 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.796 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.796 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:25.796 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.477374+0000", 2024-09-21T22:07:25.796 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1498e-05 2024-09-21T22:07:25.796 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.796 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.796 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:25.796 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.477463+0000", 2024-09-21T22:07:25.796 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9005000000000006e-05 2024-09-21T22:07:25.796 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.796 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.796 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:25.796 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.499921+0000", 2024-09-21T22:07:25.797 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022457969000000001 2024-09-21T22:07:25.797 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.797 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.797 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:25.797 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.507525+0000", 2024-09-21T22:07:25.797 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0076035 2024-09-21T22:07:25.797 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.797 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.797 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:25.797 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.507581+0000", 2024-09-21T22:07:25.797 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6127e-05 2024-09-21T22:07:25.797 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:25.797 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:25.797 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:25.797 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.507621+0000", 2024-09-21T22:07:25.798 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0377999999999999e-05 2024-09-21T22:07:25.798 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:25.798 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:25.798 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:25.798 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:25.798 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:25.798 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:07:25.800 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T22:07:25.800 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:07:25.772+0000 7f2078a4a7c0 -1 osd.5 41 log_to_monitors true 2024-09-21T22:07:25.800 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:25.796+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:25.900 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:25.896+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:25.987 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:07:25.988 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:07:25.988 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-09-21T22:07:25.988 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:07:26.001 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:25.998+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:26.002 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T22:07:26.101 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:26.098+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:26.173 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:07:26.173 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:07:26.173 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-21T22:07:26.173 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-21T22:07:26.173 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:07:26.188 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T22:07:26.191 INFO:tasks.workunit.client.0.smithi157.stderr:64+0 records in 2024-09-21T22:07:26.192 INFO:tasks.workunit.client.0.smithi157.stderr:64+0 records out 2024-09-21T22:07:26.192 INFO:tasks.workunit.client.0.smithi157.stderr:67108864 bytes (67 MB, 64 MiB) copied, 3.60343 s, 18.6 MB/s 2024-09-21T22:07:26.194 INFO:tasks.workunit.client.0.smithi157.stderr:++ dd if=/tmp/tmp.GqKuv7rsZi/data bs=1M 2024-09-21T22:07:26.194 INFO:tasks.workunit.client.0.smithi157.stderr:++ md5sum 2024-09-21T22:07:26.205 INFO:teuthology.orchestra.run.smithi039.stderr:noscrub is set 2024-09-21T22:07:26.234 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:26.232+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:26.246 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:07:26.244+0000 7f206c828640 -1 osd.5 41 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T22:07:26.293 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T22:07:26.303 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:26.300+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:26.318 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:07:26.319 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-21T22:07:26.319 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-21T22:07:26.319 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-21T22:07:26.319 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.319 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4887.0:1 3.3 3:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-21T22:07:26.319 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:17.151514+0000", 2024-09-21T22:07:26.319 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.1666265960000004, 2024-09-21T22:07:26.319 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.01157512, 2024-09-21T22:07:26.319 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:26.319 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:26.319 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:26.319 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:26.319 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4887", 2024-09-21T22:07:26.319 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/3431621434", 2024-09-21T22:07:26.320 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1 2024-09-21T22:07:26.320 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.320 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:26.320 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.320 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:26.320 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.151514+0000", 2024-09-21T22:07:26.320 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:26.320 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.320 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.320 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:26.320 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.151514+0000", 2024-09-21T22:07:26.320 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:26.320 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.320 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.321 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:26.321 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.151515+0000", 2024-09-21T22:07:26.321 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.469e-06 2024-09-21T22:07:26.321 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.321 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.321 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:26.321 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.151522+0000", 2024-09-21T22:07:26.321 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1330000000000001e-06 2024-09-21T22:07:26.321 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.321 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.321 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:26.321 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.151536+0000", 2024-09-21T22:07:26.321 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3563e-05 2024-09-21T22:07:26.321 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.322 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.322 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:26.322 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.151574+0000", 2024-09-21T22:07:26.322 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8467999999999998e-05 2024-09-21T22:07:26.322 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.322 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.322 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:26.322 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.151665+0000", 2024-09-21T22:07:26.322 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0668000000000007e-05 2024-09-21T22:07:26.322 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.322 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.322 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-21T22:07:26.322 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.151676+0000", 2024-09-21T22:07:26.323 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1443999999999999e-05 2024-09-21T22:07:26.323 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.323 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.323 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:26.323 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.161694+0000", 2024-09-21T22:07:26.323 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010017402999999999 2024-09-21T22:07:26.323 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.323 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.323 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:26.323 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.161805+0000", 2024-09-21T22:07:26.323 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011122500000000001 2024-09-21T22:07:26.323 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.323 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.323 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:26.324 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.161937+0000", 2024-09-21T22:07:26.324 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013177299999999999 2024-09-21T22:07:26.324 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.324 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.324 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:26.324 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.162555+0000", 2024-09-21T22:07:26.324 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00061786199999999995 2024-09-21T22:07:26.324 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.324 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.324 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:26.324 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.163021+0000", 2024-09-21T22:07:26.324 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000465749 2024-09-21T22:07:26.324 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.324 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.324 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:26.325 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.163059+0000", 2024-09-21T22:07:26.325 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8238000000000002e-05 2024-09-21T22:07:26.325 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.325 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.325 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:26.325 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.163089+0000", 2024-09-21T22:07:26.325 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0125e-05 2024-09-21T22:07:26.325 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:26.325 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:26.325 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:26.325 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.325 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.325 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4887.0:1 3.3 e59/58 ondisk, result = 0)", 2024-09-21T22:07:26.326 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:17.162939+0000", 2024-09-21T22:07:26.326 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.1552012050000009, 2024-09-21T22:07:26.326 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015730900000000001, 2024-09-21T22:07:26.326 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:26.326 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:26.326 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:07:26.326 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:26.326 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.326 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:26.327 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.162939+0000", 2024-09-21T22:07:26.327 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:26.327 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.327 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.327 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:26.327 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.162939+0000", 2024-09-21T22:07:26.327 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:26.327 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.327 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.327 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:26.327 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.162940+0000", 2024-09-21T22:07:26.327 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4499999999999996e-07 2024-09-21T22:07:26.327 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.327 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.328 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:26.328 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.162945+0000", 2024-09-21T22:07:26.328 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.758e-06 2024-09-21T22:07:26.328 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.328 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.328 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:26.328 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.162957+0000", 2024-09-21T22:07:26.328 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2259000000000001e-05 2024-09-21T22:07:26.328 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.328 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.328 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:26.328 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.162966+0000", 2024-09-21T22:07:26.328 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3270000000000007e-06 2024-09-21T22:07:26.328 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.328 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.329 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:26.329 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.163010+0000", 2024-09-21T22:07:26.329 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.422e-05 2024-09-21T22:07:26.329 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.329 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.329 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:26.329 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.163016+0000", 2024-09-21T22:07:26.329 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8579999999999996e-06 2024-09-21T22:07:26.329 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.329 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.329 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:26.329 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.163096+0000", 2024-09-21T22:07:26.329 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0142000000000006e-05 2024-09-21T22:07:26.329 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:26.330 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:26.330 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:26.330 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.330 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.330 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4840.0:675 2.7 e63/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 63'56, mlcod=63'54)", 2024-09-21T22:07:26.330 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.618817+0000", 2024-09-21T22:07:26.330 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 1.6993238079999999, 2024-09-21T22:07:26.330 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0060329809999999998, 2024-09-21T22:07:26.330 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:26.330 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:26.330 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:26.330 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:26.330 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.330 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:26.330 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.618817+0000", 2024-09-21T22:07:26.331 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:26.331 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.331 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.331 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:26.331 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.618817+0000", 2024-09-21T22:07:26.331 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:26.331 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.331 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.331 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:26.331 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.618819+0000", 2024-09-21T22:07:26.331 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3070000000000001e-06 2024-09-21T22:07:26.331 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.331 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.331 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:26.332 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.618934+0000", 2024-09-21T22:07:26.332 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000114669 2024-09-21T22:07:26.332 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.332 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.332 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:26.332 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.618969+0000", 2024-09-21T22:07:26.332 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5985999999999997e-05 2024-09-21T22:07:26.332 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.332 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.332 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:26.332 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.618985+0000", 2024-09-21T22:07:26.332 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5840000000000001e-05 2024-09-21T22:07:26.332 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.332 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.332 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:26.333 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.619101+0000", 2024-09-21T22:07:26.333 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000115591 2024-09-21T22:07:26.333 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.333 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.333 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:26.333 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.619136+0000", 2024-09-21T22:07:26.333 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5287000000000003e-05 2024-09-21T22:07:26.333 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.333 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.333 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:26.333 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.624766+0000", 2024-09-21T22:07:26.333 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0056301500000000004 2024-09-21T22:07:26.333 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.333 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.334 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:26.334 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.624850+0000", 2024-09-21T22:07:26.334 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3151000000000004e-05 2024-09-21T22:07:26.334 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:26.334 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:26.334 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:26.334 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.334 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.334 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4840.0:676 2.7 e63/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 63'57, mlcod=63'55)", 2024-09-21T22:07:26.334 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.627276+0000", 2024-09-21T22:07:26.334 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 1.6908640479999999, 2024-09-21T22:07:26.334 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.037500787000000001, 2024-09-21T22:07:26.334 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:26.335 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:26.335 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:26.335 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:26.335 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.335 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:26.335 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.627276+0000", 2024-09-21T22:07:26.335 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:26.335 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.335 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.335 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:26.335 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.627276+0000", 2024-09-21T22:07:26.335 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:26.335 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.335 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.335 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:26.336 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.627277+0000", 2024-09-21T22:07:26.336 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.068e-06 2024-09-21T22:07:26.336 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.336 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.336 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:26.336 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.628148+0000", 2024-09-21T22:07:26.336 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00087069999999999997 2024-09-21T22:07:26.336 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.336 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.336 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:26.336 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.628170+0000", 2024-09-21T22:07:26.336 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2147999999999999e-05 2024-09-21T22:07:26.336 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.336 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.336 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:26.337 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.628179+0000", 2024-09-21T22:07:26.337 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2450000000000004e-06 2024-09-21T22:07:26.337 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.337 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.337 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:26.337 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.628279+0000", 2024-09-21T22:07:26.337 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9298999999999999e-05 2024-09-21T22:07:26.337 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.337 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.337 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:26.337 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.628305+0000", 2024-09-21T22:07:26.337 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6316999999999999e-05 2024-09-21T22:07:26.337 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.337 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.338 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:26.338 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.664714+0000", 2024-09-21T22:07:26.338 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.036408923000000003 2024-09-21T22:07:26.338 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.338 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.338 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:26.338 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.664777+0000", 2024-09-21T22:07:26.338 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3087000000000001e-05 2024-09-21T22:07:26.338 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:26.338 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:26.338 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:26.338 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.338 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.338 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4840.0:677 2.7 e63/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 63'58, mlcod=63'56)", 2024-09-21T22:07:26.338 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.667210+0000", 2024-09-21T22:07:26.339 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 1.6509308090000001, 2024-09-21T22:07:26.339 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010046522, 2024-09-21T22:07:26.339 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:26.339 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:26.339 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:26.339 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:26.339 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.339 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:26.339 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.667210+0000", 2024-09-21T22:07:26.339 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:26.339 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.339 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.339 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:26.339 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.667210+0000", 2024-09-21T22:07:26.340 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:26.340 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.340 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.340 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:26.340 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.667211+0000", 2024-09-21T22:07:26.340 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3379999999999999e-06 2024-09-21T22:07:26.340 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.340 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.340 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:26.340 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.667272+0000", 2024-09-21T22:07:26.340 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1257e-05 2024-09-21T22:07:26.340 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.340 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.340 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:26.340 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.667295+0000", 2024-09-21T22:07:26.341 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2637999999999999e-05 2024-09-21T22:07:26.341 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.341 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.341 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:26.341 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.667305+0000", 2024-09-21T22:07:26.341 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0044000000000001e-05 2024-09-21T22:07:26.341 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.341 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.341 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:26.341 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.667425+0000", 2024-09-21T22:07:26.341 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000120605 2024-09-21T22:07:26.341 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.341 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.341 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:26.342 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.667476+0000", 2024-09-21T22:07:26.342 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0843000000000001e-05 2024-09-21T22:07:26.342 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.342 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.342 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:26.342 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.677180+0000", 2024-09-21T22:07:26.342 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0097035860000000002 2024-09-21T22:07:26.342 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.342 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.342 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:26.342 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.677256+0000", 2024-09-21T22:07:26.342 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6211000000000006e-05 2024-09-21T22:07:26.342 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:26.342 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:26.343 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:26.343 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.343 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.343 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4840.0:678 2.7 e63/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 63'59, mlcod=63'57)", 2024-09-21T22:07:26.343 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.679689+0000", 2024-09-21T22:07:26.343 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 1.6384515479999999, 2024-09-21T22:07:26.343 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053308406000000003, 2024-09-21T22:07:26.343 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:26.343 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:26.343 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:26.343 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:26.343 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.343 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:26.343 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.679689+0000", 2024-09-21T22:07:26.344 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:26.344 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.344 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.344 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:26.344 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.679689+0000", 2024-09-21T22:07:26.344 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:26.344 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.344 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.344 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:26.344 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.679691+0000", 2024-09-21T22:07:26.344 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9400000000000001e-06 2024-09-21T22:07:26.344 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.344 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.344 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:26.344 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.680245+0000", 2024-09-21T22:07:26.345 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00055382999999999995 2024-09-21T22:07:26.345 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.345 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.345 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:26.345 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.680276+0000", 2024-09-21T22:07:26.345 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1123e-05 2024-09-21T22:07:26.345 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.345 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.345 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:26.345 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.680290+0000", 2024-09-21T22:07:26.345 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4056999999999999e-05 2024-09-21T22:07:26.345 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.345 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.345 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:26.345 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.680418+0000", 2024-09-21T22:07:26.346 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000127792 2024-09-21T22:07:26.346 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.346 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.346 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:26.346 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.680465+0000", 2024-09-21T22:07:26.346 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7360000000000001e-05 2024-09-21T22:07:26.346 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.346 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.346 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:26.346 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.732923+0000", 2024-09-21T22:07:26.346 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.052458312999999999 2024-09-21T22:07:26.346 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.346 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.346 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:26.347 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.732997+0000", 2024-09-21T22:07:26.347 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3991000000000004e-05 2024-09-21T22:07:26.347 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:26.347 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:26.347 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:26.347 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.347 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.347 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4840.0:679 2.7 e63/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 63'60, mlcod=63'58)", 2024-09-21T22:07:26.347 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.735615+0000", 2024-09-21T22:07:26.347 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 1.582525395, 2024-09-21T22:07:26.347 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0077878399999999999, 2024-09-21T22:07:26.347 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:26.347 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:26.347 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:26.348 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:26.348 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.348 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:26.348 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.735615+0000", 2024-09-21T22:07:26.348 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:26.348 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.348 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.348 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:26.348 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.735615+0000", 2024-09-21T22:07:26.348 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:26.348 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.348 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.348 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:26.348 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.735617+0000", 2024-09-21T22:07:26.349 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7999999999999999e-06 2024-09-21T22:07:26.349 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.349 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.349 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:26.349 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.735706+0000", 2024-09-21T22:07:26.349 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8839999999999996e-05 2024-09-21T22:07:26.349 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.349 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.349 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:26.349 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.735736+0000", 2024-09-21T22:07:26.349 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9955999999999999e-05 2024-09-21T22:07:26.349 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.349 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.349 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:26.350 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.735755+0000", 2024-09-21T22:07:26.350 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9283000000000002e-05 2024-09-21T22:07:26.350 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.350 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.350 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:26.350 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.735857+0000", 2024-09-21T22:07:26.350 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000101681 2024-09-21T22:07:26.350 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.350 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.350 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:26.350 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.735894+0000", 2024-09-21T22:07:26.350 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7487999999999998e-05 2024-09-21T22:07:26.350 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.350 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.350 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:26.351 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.743321+0000", 2024-09-21T22:07:26.351 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0074270050000000004 2024-09-21T22:07:26.351 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.351 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.351 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:26.351 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.743403+0000", 2024-09-21T22:07:26.351 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1786999999999995e-05 2024-09-21T22:07:26.351 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:26.351 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:26.351 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:26.351 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.351 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.351 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4840.0:680 2.7 e63/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 63'61, mlcod=63'59)", 2024-09-21T22:07:26.351 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.745996+0000", 2024-09-21T22:07:26.352 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 1.572144035, 2024-09-21T22:07:26.352 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.061804132999999997, 2024-09-21T22:07:26.352 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:26.352 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:26.352 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:26.352 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:26.352 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.352 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:26.352 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.745996+0000", 2024-09-21T22:07:26.352 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:26.352 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.352 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.352 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:26.352 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.745996+0000", 2024-09-21T22:07:26.353 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:26.353 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.353 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.353 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:26.353 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.745998+0000", 2024-09-21T22:07:26.353 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.717e-06 2024-09-21T22:07:26.353 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.353 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.353 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:26.353 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.746523+0000", 2024-09-21T22:07:26.353 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00052525399999999996 2024-09-21T22:07:26.353 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.353 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.353 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:26.353 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.746554+0000", 2024-09-21T22:07:26.354 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0586000000000002e-05 2024-09-21T22:07:26.354 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.354 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.354 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:26.354 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.746577+0000", 2024-09-21T22:07:26.354 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3079e-05 2024-09-21T22:07:26.354 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.354 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.354 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:26.354 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.746689+0000", 2024-09-21T22:07:26.354 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011177 2024-09-21T22:07:26.354 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.355 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.355 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:26.355 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.746722+0000", 2024-09-21T22:07:26.355 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2870000000000002e-05 2024-09-21T22:07:26.355 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.355 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.355 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:26.355 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.807740+0000", 2024-09-21T22:07:26.355 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.061018612 2024-09-21T22:07:26.355 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.355 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.355 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:26.355 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.807800+0000", 2024-09-21T22:07:26.355 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0244999999999997e-05 2024-09-21T22:07:26.355 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:26.356 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:26.356 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:26.356 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.356 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.356 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4840.0:681 2.7 e63/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 63'62, mlcod=63'60)", 2024-09-21T22:07:26.356 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.810646+0000", 2024-09-21T22:07:26.356 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 1.507494611, 2024-09-21T22:07:26.356 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010715685000000001, 2024-09-21T22:07:26.356 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:26.356 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:26.356 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:26.356 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:26.356 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.356 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:26.357 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.810646+0000", 2024-09-21T22:07:26.357 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:26.357 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.357 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.357 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:26.357 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.810646+0000", 2024-09-21T22:07:26.357 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:26.357 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.357 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.357 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:26.357 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.810649+0000", 2024-09-21T22:07:26.357 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8569999999999999e-06 2024-09-21T22:07:26.357 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.357 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.358 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:26.358 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.810780+0000", 2024-09-21T22:07:26.358 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013114600000000001 2024-09-21T22:07:26.358 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.358 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.358 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:26.358 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.810819+0000", 2024-09-21T22:07:26.358 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9363000000000002e-05 2024-09-21T22:07:26.358 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.358 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.358 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:26.358 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.810839+0000", 2024-09-21T22:07:26.358 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9865999999999999e-05 2024-09-21T22:07:26.358 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.359 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.359 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:26.359 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.810992+0000", 2024-09-21T22:07:26.359 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015333299999999999 2024-09-21T22:07:26.359 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.359 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.359 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:26.359 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.811037+0000", 2024-09-21T22:07:26.359 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.464e-05 2024-09-21T22:07:26.359 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.359 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.359 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:26.359 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.821280+0000", 2024-09-21T22:07:26.359 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010243254 2024-09-21T22:07:26.360 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.360 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.360 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:26.360 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.821361+0000", 2024-09-21T22:07:26.360 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1225999999999998e-05 2024-09-21T22:07:26.360 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:26.360 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:26.360 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:26.360 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.360 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.360 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4840.0:682 2.7 e63/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 63'63, mlcod=63'61)", 2024-09-21T22:07:26.360 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.824126+0000", 2024-09-21T22:07:26.360 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 1.4940144659999999, 2024-09-21T22:07:26.360 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.038159327, 2024-09-21T22:07:26.361 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:26.361 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:26.361 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:26.361 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:26.361 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.361 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:26.361 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.824126+0000", 2024-09-21T22:07:26.361 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:26.361 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.361 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.361 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:26.361 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.824126+0000", 2024-09-21T22:07:26.361 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:26.361 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.362 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.362 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:26.362 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.824128+0000", 2024-09-21T22:07:26.362 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9659999999999999e-06 2024-09-21T22:07:26.362 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.362 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.362 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:26.362 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.824842+0000", 2024-09-21T22:07:26.362 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00071423999999999999 2024-09-21T22:07:26.362 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.362 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.362 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:26.362 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.824883+0000", 2024-09-21T22:07:26.362 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1131999999999999e-05 2024-09-21T22:07:26.363 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.363 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.363 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:26.363 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.824931+0000", 2024-09-21T22:07:26.363 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7327000000000003e-05 2024-09-21T22:07:26.363 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.363 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.363 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:26.363 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.825058+0000", 2024-09-21T22:07:26.363 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012790499999999999 2024-09-21T22:07:26.363 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.363 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.363 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:26.363 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.825087+0000", 2024-09-21T22:07:26.364 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8192e-05 2024-09-21T22:07:26.364 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.364 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.364 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:26.364 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.862225+0000", 2024-09-21T22:07:26.364 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.037138309000000001 2024-09-21T22:07:26.364 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:26.364 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:26.364 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:26.364 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.862285+0000", 2024-09-21T22:07:26.364 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0256e-05 2024-09-21T22:07:26.364 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:26.364 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:26.364 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:26.365 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:26.365 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:26.365 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:07:26.366 INFO:tasks.workunit.client.0.smithi157.stderr:64+0 records in 2024-09-21T22:07:26.366 INFO:tasks.workunit.client.0.smithi157.stderr:64+0 records out 2024-09-21T22:07:26.366 INFO:tasks.workunit.client.0.smithi157.stderr:67108864 bytes (67 MB, 64 MiB) copied, 0.135255 s, 496 MB/s 2024-09-21T22:07:26.366 INFO:tasks.workunit.client.0.smithi157.stderr:++ md5sum 2024-09-21T22:07:26.366 INFO:tasks.workunit.client.0.smithi157.stderr:++ rbd -p rbd --no-progress export testrbdnbd21553 - 2024-09-21T22:07:26.367 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T22:07:26.412 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:26.410+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:26.417 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-21T22:07:26.410+0000 7ff8a72a0640 0 -- 172.21.15.157:0/2160753649 >> [v2:172.21.15.39:6816/2876495139,v1:172.21.15.39:6817/2876495139] conn(0x7ff884008c60 msgr2=0x7ff88400b0f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-21T22:07:26.456 INFO:teuthology.orchestra.run.smithi039.stdout:osd.0: {} 2024-09-21T22:07:26.457 INFO:teuthology.orchestra.run.smithi039.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T22:07:26.475 INFO:teuthology.orchestra.run.smithi039.stdout:osd.1: {} 2024-09-21T22:07:26.475 INFO:teuthology.orchestra.run.smithi039.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T22:07:26.492 INFO:teuthology.orchestra.run.smithi039.stdout:osd.2: {} 2024-09-21T22:07:26.492 INFO:teuthology.orchestra.run.smithi039.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T22:07:26.504 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:26.502+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:26.506 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:26.506 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:26.513 INFO:teuthology.orchestra.run.smithi039.stdout:osd.3: {} 2024-09-21T22:07:26.513 INFO:teuthology.orchestra.run.smithi039.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T22:07:26.521 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T22:07:26.534 INFO:teuthology.orchestra.run.smithi039.stdout:osd.4: {} 2024-09-21T22:07:26.535 INFO:teuthology.orchestra.run.smithi039.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T22:07:26.535 INFO:teuthology.orchestra.run.smithi039.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-09-21T22:07:26.538 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T22:07:26.539 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T22:07:26.554 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-21T22:07:26.550+0000 7ff8a72a0640 0 -- 172.21.15.157:0/2160753649 >> [v2:172.21.15.39:6816/2876495139,v1:172.21.15.39:6817/2876495139] conn(0x7ff88c03b710 msgr2=0x7ff88c03db20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-21T22:07:26.606 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:26.604+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:26.610 INFO:tasks.workunit.client.0.smithi157.stderr:+ '[' 'efadd545db6c9d89a00e0fd884457af5 -' = 'efadd545db6c9d89a00e0fd884457af5 -' ']' 2024-09-21T22:07:26.610 INFO:tasks.workunit.client.0.smithi157.stderr:+ unmap_device /dev/nbd0 21953 2024-09-21T22:07:26.610 INFO:tasks.workunit.client.0.smithi157.stderr:+ local args=/dev/nbd0 2024-09-21T22:07:26.610 INFO:tasks.workunit.client.0.smithi157.stderr:+ local pid=21953 2024-09-21T22:07:26.610 INFO:tasks.workunit.client.0.smithi157.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd0 2024-09-21T22:07:26.610 INFO:tasks.workunit.client.0.smithi157.stderr:+ local cmd 2024-09-21T22:07:26.610 INFO:tasks.workunit.client.0.smithi157.stderr:++ id -u 2024-09-21T22:07:26.612 INFO:tasks.workunit.client.0.smithi157.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T22:07:26.612 INFO:tasks.workunit.client.0.smithi157.stderr:++ which rbd 2024-09-21T22:07:26.614 INFO:tasks.workunit.client.0.smithi157.stderr:+ cmd=/usr/bin/rbd 2024-09-21T22:07:26.614 INFO:tasks.workunit.client.0.smithi157.stderr:+ shift 2024-09-21T22:07:26.614 INFO:tasks.workunit.client.0.smithi157.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd0 2024-09-21T22:07:26.650 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:26.650 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:26.665 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T22:07:26.703 INFO:tasks.workunit.client.0.smithi157.stderr:+ rbd device --device-type nbd list 2024-09-21T22:07:26.704 INFO:tasks.workunit.client.0.smithi157.stderr:+ expect_false grep '^21953\b' 2024-09-21T22:07:26.704 INFO:tasks.workunit.client.0.smithi157.stderr:+ grep '^21953\b' 2024-09-21T22:07:26.707 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:26.704+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:26.752 INFO:tasks.workunit.client.0.smithi157.stderr:+ return 0 2024-09-21T22:07:26.753 INFO:tasks.workunit.client.0.smithi157.stderr:+ ps -C rbd-nbd 2024-09-21T22:07:26.753 INFO:tasks.workunit.client.0.smithi157.stderr:+ expect_false grep '^ *21953\b' 2024-09-21T22:07:26.753 INFO:tasks.workunit.client.0.smithi157.stderr:+ grep '^ *21953\b' 2024-09-21T22:07:26.759 INFO:tasks.workunit.client.0.smithi157.stderr:+ return 0 2024-09-21T22:07:26.759 INFO:tasks.workunit.client.0.smithi157.stderr:+ sleep 0.5 2024-09-21T22:07:26.792 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:26.793 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:26.807 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T22:07:26.808 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:26.804+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:26.910 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:26.906+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:26.926 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:26.927 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:26.942 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T22:07:27.028 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:27.026+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:27.103 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:27.103 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:27.111 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:27.108+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:27.117 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T22:07:27.213 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:27.210+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:27.261 INFO:tasks.workunit.client.0.smithi157.stderr:++ _sudo rbd device --device-type nbd --options notrim map rbd/testrbdnbd21553 2024-09-21T22:07:27.261 INFO:tasks.workunit.client.0.smithi157.stderr:++ local cmd 2024-09-21T22:07:27.261 INFO:tasks.workunit.client.0.smithi157.stderr:+++ id -u 2024-09-21T22:07:27.263 INFO:tasks.workunit.client.0.smithi157.stderr:++ '[' 1000 -eq 0 ']' 2024-09-21T22:07:27.263 INFO:tasks.workunit.client.0.smithi157.stderr:+++ which rbd 2024-09-21T22:07:27.265 INFO:tasks.workunit.client.0.smithi157.stderr:++ cmd=/usr/bin/rbd 2024-09-21T22:07:27.266 INFO:tasks.workunit.client.0.smithi157.stderr:++ shift 2024-09-21T22:07:27.266 INFO:tasks.workunit.client.0.smithi157.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd --options notrim map rbd/testrbdnbd21553 2024-09-21T22:07:27.293 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:27.293 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:27.307 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T22:07:27.352 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:27.350+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:27.415 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:27.412+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:27.478 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:27.478 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:27.492 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T22:07:27.516 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:27.516+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:27.617 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:27.614+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:27.617 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:27.618 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:27.632 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T22:07:27.718 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:27.712+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:27.808 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:27.809 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:27.819 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:27.814+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:27.823 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T22:07:27.920 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:27.918+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:27.958 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:27.958 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:27.972 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T22:07:28.021 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:28.016+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:28.100 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:28.100 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:28.114 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T22:07:28.122 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:28.118+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:28.223 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:28.220+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:28.229 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-21T22:07:28.304 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:28.304 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:28.319 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T22:07:28.323 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:28.318+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:28.357 INFO:tasks.workunit.client.0.smithi157.stderr:+ DEV=/dev/nbd0 2024-09-21T22:07:28.357 INFO:tasks.workunit.client.0.smithi157.stderr:+ get_pid rbd 2024-09-21T22:07:28.357 INFO:tasks.workunit.client.0.smithi157.stderr:+ local pool=rbd 2024-09-21T22:07:28.357 INFO:tasks.workunit.client.0.smithi157.stderr:+ local ns= 2024-09-21T22:07:28.358 INFO:tasks.workunit.client.0.smithi157.stderr:++ rbd device --device-type nbd --format xml list 2024-09-21T22:07:28.358 INFO:tasks.workunit.client.0.smithi157.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd21553'\''][device='\''/dev/nbd0'\'']/id' 2024-09-21T22:07:28.424 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:28.422+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:28.428 INFO:tasks.workunit.client.0.smithi157.stderr:+ PID=22066 2024-09-21T22:07:28.428 INFO:tasks.workunit.client.0.smithi157.stderr:+ test -n 22066 2024-09-21T22:07:28.428 INFO:tasks.workunit.client.0.smithi157.stderr:+ ps -p 22066 -C rbd-nbd 2024-09-21T22:07:28.434 INFO:tasks.workunit.client.0.smithi157.stdout: PID TTY TIME CMD 2024-09-21T22:07:28.434 INFO:tasks.workunit.client.0.smithi157.stdout: 22066 ? 00:00:00 rbd-nbd 2024-09-21T22:07:28.435 INFO:tasks.workunit.client.0.smithi157.stderr:++ rbd -p rbd --format xml du testrbdnbd21553 2024-09-21T22:07:28.435 INFO:tasks.workunit.client.0.smithi157.stderr:++ xmlstarlet sel -t -m //stats/images/image/provisioned_size -v . 2024-09-21T22:07:28.495 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:28.495 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:28.510 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T22:07:28.540 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T22:07:28.554 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:28.552+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:28.626 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:28.622+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:28.637 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:28.637 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:28.652 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T22:07:28.689 INFO:teuthology.orchestra.run.smithi039.stdout:osd.0: {} 2024-09-21T22:07:28.689 INFO:teuthology.orchestra.run.smithi039.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T22:07:28.705 INFO:teuthology.orchestra.run.smithi039.stdout:osd.1: {} 2024-09-21T22:07:28.706 INFO:teuthology.orchestra.run.smithi039.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T22:07:28.724 INFO:teuthology.orchestra.run.smithi039.stdout:osd.2: {} 2024-09-21T22:07:28.724 INFO:teuthology.orchestra.run.smithi039.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T22:07:28.742 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:28.740+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:28.743 INFO:teuthology.orchestra.run.smithi039.stdout:osd.3: {} 2024-09-21T22:07:28.744 INFO:teuthology.orchestra.run.smithi039.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T22:07:28.763 INFO:teuthology.orchestra.run.smithi039.stdout:osd.4: {} 2024-09-21T22:07:28.764 INFO:teuthology.orchestra.run.smithi039.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T22:07:28.782 INFO:teuthology.orchestra.run.smithi039.stdout:osd.5: {} 2024-09-21T22:07:28.782 INFO:teuthology.orchestra.run.smithi039.stderr:osd.5: osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_enable_op_tracker = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' 2024-09-21T22:07:28.822 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:07:28.822 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-21T22:07:28.822 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-21T22:07:28.823 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-21T22:07:28.823 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.823 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4887.0:1 3.3 3:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-21T22:07:28.823 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:17.151514+0000", 2024-09-21T22:07:28.823 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 11.670138197, 2024-09-21T22:07:28.823 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.01157512, 2024-09-21T22:07:28.823 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:28.823 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:28.823 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:28.823 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:28.823 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4887", 2024-09-21T22:07:28.824 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/3431621434", 2024-09-21T22:07:28.824 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1 2024-09-21T22:07:28.824 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.824 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:28.824 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.824 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:28.824 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.151514+0000", 2024-09-21T22:07:28.824 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:28.824 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.824 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.824 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:28.824 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.151514+0000", 2024-09-21T22:07:28.824 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:28.825 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.825 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.825 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:28.825 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.151515+0000", 2024-09-21T22:07:28.825 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.469e-06 2024-09-21T22:07:28.825 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.825 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.825 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:28.825 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.151522+0000", 2024-09-21T22:07:28.825 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1330000000000001e-06 2024-09-21T22:07:28.825 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.825 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.825 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:28.825 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.151536+0000", 2024-09-21T22:07:28.826 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3563e-05 2024-09-21T22:07:28.826 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.826 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.826 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:28.826 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.151574+0000", 2024-09-21T22:07:28.826 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8467999999999998e-05 2024-09-21T22:07:28.826 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.826 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.826 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:28.826 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.151665+0000", 2024-09-21T22:07:28.826 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0668000000000007e-05 2024-09-21T22:07:28.826 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.826 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.826 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-21T22:07:28.827 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.151676+0000", 2024-09-21T22:07:28.827 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1443999999999999e-05 2024-09-21T22:07:28.827 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.827 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.827 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:28.827 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.161694+0000", 2024-09-21T22:07:28.827 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010017402999999999 2024-09-21T22:07:28.827 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.827 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.827 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:28.827 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.161805+0000", 2024-09-21T22:07:28.827 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011122500000000001 2024-09-21T22:07:28.827 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.827 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.827 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:28.828 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.161937+0000", 2024-09-21T22:07:28.828 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013177299999999999 2024-09-21T22:07:28.828 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.828 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.828 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:28.828 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.162555+0000", 2024-09-21T22:07:28.828 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00061786199999999995 2024-09-21T22:07:28.828 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.828 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.828 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:28.828 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.163021+0000", 2024-09-21T22:07:28.828 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000465749 2024-09-21T22:07:28.828 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.828 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.829 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:28.829 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.163059+0000", 2024-09-21T22:07:28.829 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8238000000000002e-05 2024-09-21T22:07:28.829 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.829 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.829 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:28.829 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.163089+0000", 2024-09-21T22:07:28.829 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0125e-05 2024-09-21T22:07:28.829 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:28.829 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:28.829 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:28.829 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.829 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.829 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4887.0:1 3.3 e59/58 ondisk, result = 0)", 2024-09-21T22:07:28.830 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:17.162939+0000", 2024-09-21T22:07:28.830 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 11.658712806, 2024-09-21T22:07:28.830 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015730900000000001, 2024-09-21T22:07:28.830 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:28.830 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:28.830 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:07:28.830 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:28.830 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.830 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:28.830 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.162939+0000", 2024-09-21T22:07:28.830 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:28.830 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.830 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.830 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:28.831 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.162939+0000", 2024-09-21T22:07:28.831 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:28.831 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.831 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.831 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:28.831 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.162940+0000", 2024-09-21T22:07:28.831 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4499999999999996e-07 2024-09-21T22:07:28.831 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.831 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.831 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:28.831 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.162945+0000", 2024-09-21T22:07:28.831 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.758e-06 2024-09-21T22:07:28.831 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.832 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.832 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:28.832 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.162957+0000", 2024-09-21T22:07:28.832 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2259000000000001e-05 2024-09-21T22:07:28.832 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.832 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.832 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:28.832 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.162966+0000", 2024-09-21T22:07:28.832 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3270000000000007e-06 2024-09-21T22:07:28.832 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.832 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.832 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:28.832 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.163010+0000", 2024-09-21T22:07:28.832 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.422e-05 2024-09-21T22:07:28.833 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.833 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.833 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:28.833 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.163016+0000", 2024-09-21T22:07:28.833 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8579999999999996e-06 2024-09-21T22:07:28.833 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.833 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.833 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:28.833 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.163096+0000", 2024-09-21T22:07:28.833 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0142000000000006e-05 2024-09-21T22:07:28.833 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:28.833 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:28.833 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:28.833 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.834 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.834 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4840.0:675 2.7 e63/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 63'56, mlcod=63'54)", 2024-09-21T22:07:28.834 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.618817+0000", 2024-09-21T22:07:28.834 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.2028354090000004, 2024-09-21T22:07:28.834 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0060329809999999998, 2024-09-21T22:07:28.834 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:28.834 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:28.834 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:28.834 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:28.834 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.834 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:28.834 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.618817+0000", 2024-09-21T22:07:28.834 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:28.834 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.835 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.835 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:28.835 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.618817+0000", 2024-09-21T22:07:28.835 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:28.835 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.835 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.835 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:28.835 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.618819+0000", 2024-09-21T22:07:28.835 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3070000000000001e-06 2024-09-21T22:07:28.835 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.835 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.835 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:28.835 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.618934+0000", 2024-09-21T22:07:28.835 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000114669 2024-09-21T22:07:28.835 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.836 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.836 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:28.836 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.618969+0000", 2024-09-21T22:07:28.836 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5985999999999997e-05 2024-09-21T22:07:28.836 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.836 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.836 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:28.836 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.618985+0000", 2024-09-21T22:07:28.836 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5840000000000001e-05 2024-09-21T22:07:28.836 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.836 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.836 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:28.836 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.619101+0000", 2024-09-21T22:07:28.836 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000115591 2024-09-21T22:07:28.836 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.837 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.837 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:28.837 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.619136+0000", 2024-09-21T22:07:28.837 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5287000000000003e-05 2024-09-21T22:07:28.837 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.837 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.837 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:28.837 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.624766+0000", 2024-09-21T22:07:28.837 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0056301500000000004 2024-09-21T22:07:28.837 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.837 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.837 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:28.838 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.624850+0000", 2024-09-21T22:07:28.838 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3151000000000004e-05 2024-09-21T22:07:28.838 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:28.838 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:28.838 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:28.838 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.838 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.838 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4840.0:676 2.7 e63/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 63'57, mlcod=63'55)", 2024-09-21T22:07:28.838 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.627276+0000", 2024-09-21T22:07:28.838 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.1943756490000004, 2024-09-21T22:07:28.838 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.037500787000000001, 2024-09-21T22:07:28.838 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:28.839 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:28.839 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:28.839 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:28.839 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.839 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:28.839 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.627276+0000", 2024-09-21T22:07:28.839 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:28.839 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.839 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.839 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:28.839 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.627276+0000", 2024-09-21T22:07:28.839 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:28.839 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.839 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.840 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:28.840 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.627277+0000", 2024-09-21T22:07:28.840 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.068e-06 2024-09-21T22:07:28.840 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.840 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.840 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:28.840 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.628148+0000", 2024-09-21T22:07:28.840 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00087069999999999997 2024-09-21T22:07:28.840 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.840 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.840 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:28.840 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.628170+0000", 2024-09-21T22:07:28.840 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2147999999999999e-05 2024-09-21T22:07:28.840 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.841 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.841 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:28.841 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.628179+0000", 2024-09-21T22:07:28.841 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2450000000000004e-06 2024-09-21T22:07:28.841 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.841 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.841 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:28.841 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.628279+0000", 2024-09-21T22:07:28.841 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9298999999999999e-05 2024-09-21T22:07:28.841 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.841 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.841 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:28.841 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.628305+0000", 2024-09-21T22:07:28.841 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6316999999999999e-05 2024-09-21T22:07:28.841 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.842 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.842 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:28.842 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.664714+0000", 2024-09-21T22:07:28.842 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.036408923000000003 2024-09-21T22:07:28.842 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.842 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.842 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:28.842 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.664777+0000", 2024-09-21T22:07:28.842 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3087000000000001e-05 2024-09-21T22:07:28.842 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:28.842 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:28.842 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:28.842 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.843 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.843 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4840.0:677 2.7 e63/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 63'58, mlcod=63'56)", 2024-09-21T22:07:28.843 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.667210+0000", 2024-09-21T22:07:28.843 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.1544424099999997, 2024-09-21T22:07:28.843 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010046522, 2024-09-21T22:07:28.843 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:28.843 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:28.843 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:28.843 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:28.843 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.843 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:28.843 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.667210+0000", 2024-09-21T22:07:28.843 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:28.843 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.844 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.844 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:28.844 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.667210+0000", 2024-09-21T22:07:28.844 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:28.844 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.844 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.844 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:28.844 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.667211+0000", 2024-09-21T22:07:28.844 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3379999999999999e-06 2024-09-21T22:07:28.844 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.844 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.844 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:28.844 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.667272+0000", 2024-09-21T22:07:28.844 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1257e-05 2024-09-21T22:07:28.844 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.845 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.845 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:28.845 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.667295+0000", 2024-09-21T22:07:28.845 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2637999999999999e-05 2024-09-21T22:07:28.845 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.845 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.845 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:28.845 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.667305+0000", 2024-09-21T22:07:28.845 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0044000000000001e-05 2024-09-21T22:07:28.845 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.845 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.845 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:28.845 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.667425+0000", 2024-09-21T22:07:28.845 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000120605 2024-09-21T22:07:28.846 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.846 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.846 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:28.846 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.667476+0000", 2024-09-21T22:07:28.846 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0843000000000001e-05 2024-09-21T22:07:28.846 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.846 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.846 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:28.846 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.677180+0000", 2024-09-21T22:07:28.846 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0097035860000000002 2024-09-21T22:07:28.846 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.846 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.846 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:28.846 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.677256+0000", 2024-09-21T22:07:28.847 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6211000000000006e-05 2024-09-21T22:07:28.847 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:28.847 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:28.847 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:28.847 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.847 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.847 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4840.0:678 2.7 e63/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 63'59, mlcod=63'57)", 2024-09-21T22:07:28.847 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.679689+0000", 2024-09-21T22:07:28.847 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.1419631490000004, 2024-09-21T22:07:28.847 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053308406000000003, 2024-09-21T22:07:28.847 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:28.847 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:28.847 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:28.848 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:28.848 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.848 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:28.848 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.679689+0000", 2024-09-21T22:07:28.848 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:28.848 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.848 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.848 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:28.848 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.679689+0000", 2024-09-21T22:07:28.848 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:28.848 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.848 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.848 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:28.848 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.679691+0000", 2024-09-21T22:07:28.849 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9400000000000001e-06 2024-09-21T22:07:28.849 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.849 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.849 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:28.849 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.680245+0000", 2024-09-21T22:07:28.849 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00055382999999999995 2024-09-21T22:07:28.849 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.849 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.849 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:28.849 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.680276+0000", 2024-09-21T22:07:28.849 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1123e-05 2024-09-21T22:07:28.849 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.849 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.849 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:28.849 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.680290+0000", 2024-09-21T22:07:28.850 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4056999999999999e-05 2024-09-21T22:07:28.850 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.850 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.850 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:28.850 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.680418+0000", 2024-09-21T22:07:28.850 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000127792 2024-09-21T22:07:28.850 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.850 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.850 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:28.850 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.680465+0000", 2024-09-21T22:07:28.850 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7360000000000001e-05 2024-09-21T22:07:28.850 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.850 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.850 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:28.851 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.732923+0000", 2024-09-21T22:07:28.851 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.052458312999999999 2024-09-21T22:07:28.851 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.851 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.851 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:28.851 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.732997+0000", 2024-09-21T22:07:28.851 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3991000000000004e-05 2024-09-21T22:07:28.851 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:28.851 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:28.851 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:28.851 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.851 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.851 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4840.0:679 2.7 e63/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 63'60, mlcod=63'58)", 2024-09-21T22:07:28.852 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.735615+0000", 2024-09-21T22:07:28.852 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.0860369959999998, 2024-09-21T22:07:28.852 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0077878399999999999, 2024-09-21T22:07:28.852 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:28.852 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:28.852 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:28.852 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:28.852 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.852 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:28.852 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.735615+0000", 2024-09-21T22:07:28.852 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:28.852 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.852 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.852 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:28.853 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.735615+0000", 2024-09-21T22:07:28.853 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:28.853 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.853 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.853 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:28.853 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.735617+0000", 2024-09-21T22:07:28.853 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7999999999999999e-06 2024-09-21T22:07:28.853 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.853 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.853 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:28.853 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.735706+0000", 2024-09-21T22:07:28.853 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8839999999999996e-05 2024-09-21T22:07:28.853 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.853 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.854 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:28.854 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.735736+0000", 2024-09-21T22:07:28.854 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9955999999999999e-05 2024-09-21T22:07:28.854 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.854 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.854 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:28.854 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.735755+0000", 2024-09-21T22:07:28.854 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9283000000000002e-05 2024-09-21T22:07:28.854 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.854 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.854 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:28.854 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.735857+0000", 2024-09-21T22:07:28.854 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000101681 2024-09-21T22:07:28.855 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.855 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.855 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:28.855 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.735894+0000", 2024-09-21T22:07:28.855 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7487999999999998e-05 2024-09-21T22:07:28.855 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.855 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.855 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:28.855 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.743321+0000", 2024-09-21T22:07:28.855 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0074270050000000004 2024-09-21T22:07:28.855 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.855 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.855 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:28.856 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.743403+0000", 2024-09-21T22:07:28.856 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1786999999999995e-05 2024-09-21T22:07:28.856 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:28.856 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:28.856 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:28.856 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.856 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.856 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4840.0:680 2.7 e63/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 63'61, mlcod=63'59)", 2024-09-21T22:07:28.856 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.745996+0000", 2024-09-21T22:07:28.856 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.0756556359999996, 2024-09-21T22:07:28.856 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.061804132999999997, 2024-09-21T22:07:28.856 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:28.857 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:28.857 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:28.857 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:28.857 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.857 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:28.857 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.745996+0000", 2024-09-21T22:07:28.857 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:28.857 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.857 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.857 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:28.857 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.745996+0000", 2024-09-21T22:07:28.857 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:28.857 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.858 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.858 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:28.858 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.745998+0000", 2024-09-21T22:07:28.858 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.717e-06 2024-09-21T22:07:28.858 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.858 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.858 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:28.858 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.746523+0000", 2024-09-21T22:07:28.858 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00052525399999999996 2024-09-21T22:07:28.858 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.859 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.859 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:28.859 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.746554+0000", 2024-09-21T22:07:28.859 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0586000000000002e-05 2024-09-21T22:07:28.859 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.859 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.859 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:28.859 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.746577+0000", 2024-09-21T22:07:28.859 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3079e-05 2024-09-21T22:07:28.859 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.859 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.859 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:28.859 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.746689+0000", 2024-09-21T22:07:28.860 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011177 2024-09-21T22:07:28.860 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.860 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.860 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:28.860 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.746722+0000", 2024-09-21T22:07:28.860 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2870000000000002e-05 2024-09-21T22:07:28.860 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.860 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.860 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:28.860 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.807740+0000", 2024-09-21T22:07:28.860 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.061018612 2024-09-21T22:07:28.860 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.860 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.861 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:28.861 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.807800+0000", 2024-09-21T22:07:28.861 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0244999999999997e-05 2024-09-21T22:07:28.861 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:28.861 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:28.861 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:28.861 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.861 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.861 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4840.0:681 2.7 e63/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 63'62, mlcod=63'60)", 2024-09-21T22:07:28.861 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.810646+0000", 2024-09-21T22:07:28.861 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.0110062119999998, 2024-09-21T22:07:28.861 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010715685000000001, 2024-09-21T22:07:28.861 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:28.862 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:28.862 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:28.862 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:28.862 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.862 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:28.862 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.810646+0000", 2024-09-21T22:07:28.862 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:28.862 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.862 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.862 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:28.862 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.810646+0000", 2024-09-21T22:07:28.862 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:28.862 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.862 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.863 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:28.863 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.810649+0000", 2024-09-21T22:07:28.863 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8569999999999999e-06 2024-09-21T22:07:28.863 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.863 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.863 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:28.863 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.810780+0000", 2024-09-21T22:07:28.863 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013114600000000001 2024-09-21T22:07:28.863 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.863 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.863 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:28.863 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.810819+0000", 2024-09-21T22:07:28.863 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9363000000000002e-05 2024-09-21T22:07:28.864 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.864 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.864 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:28.864 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.810839+0000", 2024-09-21T22:07:28.864 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9865999999999999e-05 2024-09-21T22:07:28.864 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.864 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.864 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:28.864 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.810992+0000", 2024-09-21T22:07:28.864 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015333299999999999 2024-09-21T22:07:28.864 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.864 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.864 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:28.864 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.811037+0000", 2024-09-21T22:07:28.864 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.464e-05 2024-09-21T22:07:28.865 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.865 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.865 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:28.865 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.821280+0000", 2024-09-21T22:07:28.865 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010243254 2024-09-21T22:07:28.865 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.865 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.865 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:28.865 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.821361+0000", 2024-09-21T22:07:28.865 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1225999999999998e-05 2024-09-21T22:07:28.865 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:28.865 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:28.865 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:28.865 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.865 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.865 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4840.0:682 2.7 e63/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 63'63, mlcod=63'61)", 2024-09-21T22:07:28.866 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.824126+0000", 2024-09-21T22:07:28.866 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.9975260669999999, 2024-09-21T22:07:28.866 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.038159327, 2024-09-21T22:07:28.866 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:28.866 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:28.866 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:28.866 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:28.866 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.866 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:28.866 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.824126+0000", 2024-09-21T22:07:28.866 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:28.866 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.866 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.867 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:28.867 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.824126+0000", 2024-09-21T22:07:28.867 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:28.867 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.867 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.867 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:28.867 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.824128+0000", 2024-09-21T22:07:28.867 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9659999999999999e-06 2024-09-21T22:07:28.867 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.867 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.867 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:28.867 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.824842+0000", 2024-09-21T22:07:28.867 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00071423999999999999 2024-09-21T22:07:28.868 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.868 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.868 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:28.868 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.824883+0000", 2024-09-21T22:07:28.868 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1131999999999999e-05 2024-09-21T22:07:28.868 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.868 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.868 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:28.868 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.824931+0000", 2024-09-21T22:07:28.868 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7327000000000003e-05 2024-09-21T22:07:28.868 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.868 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.868 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:28.868 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.825058+0000", 2024-09-21T22:07:28.868 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012790499999999999 2024-09-21T22:07:28.868 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.869 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.869 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:28.869 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.825087+0000", 2024-09-21T22:07:28.869 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8192e-05 2024-09-21T22:07:28.869 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.869 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.869 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:28.869 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.862225+0000", 2024-09-21T22:07:28.869 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.037138309000000001 2024-09-21T22:07:28.869 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:28.869 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:28.869 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:28.869 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.862285+0000", 2024-09-21T22:07:28.869 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0256e-05 2024-09-21T22:07:28.870 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:28.870 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:28.870 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:28.870 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:28.870 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:28.870 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:07:28.871 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T22:07:28.872 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:28.868+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:28.972 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:28.970+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:29.002 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:07:29.002 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:07:29.002 INFO:teuthology.orchestra.run.smithi039.stdout: "num_ops": 0 2024-09-21T22:07:29.002 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:07:29.016 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T22:07:29.104 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:29.102+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:29.173 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:29.170+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:29.179 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:07:29.179 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:07:29.179 INFO:teuthology.orchestra.run.smithi039.stdout: "complaint_time": 30, 2024-09-21T22:07:29.179 INFO:teuthology.orchestra.run.smithi039.stdout: "num_blocked_ops": 0 2024-09-21T22:07:29.179 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:07:29.195 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T22:07:29.257 INFO:teuthology.orchestra.run.smithi039.stderr:nodeep-scrub is set 2024-09-21T22:07:29.274 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:29.270+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:29.282 INFO:tasks.workunit.client.0.smithi157.stderr:+ provisioned=67108864 2024-09-21T22:07:29.283 INFO:tasks.workunit.client.0.smithi157.stderr:++ rbd -p rbd --format xml du testrbdnbd21553 2024-09-21T22:07:29.283 INFO:tasks.workunit.client.0.smithi157.stderr:++ xmlstarlet sel -t -m //stats/images/image/used_size -v . 2024-09-21T22:07:29.363 INFO:tasks.workunit.client.0.smithi157.stderr:+ used=67108864 2024-09-21T22:07:29.363 INFO:tasks.workunit.client.0.smithi157.stderr:+ '[' 67108864 -eq 67108864 ']' 2024-09-21T22:07:29.363 INFO:tasks.workunit.client.0.smithi157.stderr:+ expect_false _sudo blkdiscard /dev/nbd0 2024-09-21T22:07:29.363 INFO:tasks.workunit.client.0.smithi157.stderr:+ _sudo blkdiscard /dev/nbd0 2024-09-21T22:07:29.363 INFO:tasks.workunit.client.0.smithi157.stderr:+ local cmd 2024-09-21T22:07:29.363 INFO:tasks.workunit.client.0.smithi157.stderr:++ id -u 2024-09-21T22:07:29.364 INFO:tasks.workunit.client.0.smithi157.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T22:07:29.364 INFO:tasks.workunit.client.0.smithi157.stderr:++ which blkdiscard 2024-09-21T22:07:29.366 INFO:tasks.workunit.client.0.smithi157.stderr:+ cmd=/usr/sbin/blkdiscard 2024-09-21T22:07:29.366 INFO:tasks.workunit.client.0.smithi157.stderr:+ shift 2024-09-21T22:07:29.366 INFO:tasks.workunit.client.0.smithi157.stderr:+ sudo -nE /usr/sbin/blkdiscard /dev/nbd0 2024-09-21T22:07:29.370 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:07:29.370 INFO:teuthology.orchestra.run.smithi039.stdout: "size": 20, 2024-09-21T22:07:29.371 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 600, 2024-09-21T22:07:29.371 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [ 2024-09-21T22:07:29.371 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.371 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.1 54/53 [PushOp(2:837ddd83:::rbd_data.126483d75f2d.000000000000000d:head, version: 52'1, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:837ddd83:::rbd_data.126483d75f2d.000000000000000d:head@52'1, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T22:07:29.371 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:11.071135+0000", 2024-09-21T22:07:29.371 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 18.295550810999998, 2024-09-21T22:07:29.371 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.025278432, 2024-09-21T22:07:29.371 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:29.371 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:29.371 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:29.371 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:29.371 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.371 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:29.372 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.071135+0000", 2024-09-21T22:07:29.372 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:29.372 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.372 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.372 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:29.372 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.071135+0000", 2024-09-21T22:07:29.372 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:29.372 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.372 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.372 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:29.372 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.071137+0000", 2024-09-21T22:07:29.373 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.0430000000000002e-06 2024-09-21T22:07:29.373 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.373 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.373 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:29.373 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.071783+0000", 2024-09-21T22:07:29.373 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00064543399999999996 2024-09-21T22:07:29.373 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.373 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.373 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:29.373 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.071820+0000", 2024-09-21T22:07:29.373 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.7588e-05 2024-09-21T22:07:29.373 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.374 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.374 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:29.374 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.071846+0000", 2024-09-21T22:07:29.374 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5673999999999998e-05 2024-09-21T22:07:29.374 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.374 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.374 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:29.374 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.071967+0000", 2024-09-21T22:07:29.374 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00012071899999999999 2024-09-21T22:07:29.374 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.374 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.374 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:29.374 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.071983+0000", 2024-09-21T22:07:29.374 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6013999999999999e-05 2024-09-21T22:07:29.375 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.375 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.375 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:29.375 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.096414+0000", 2024-09-21T22:07:29.375 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.024430960000000002 2024-09-21T22:07:29.375 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:29.375 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:29.375 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:29.375 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.375 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.375 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.1 54/53 [PushOp(2:837ddd83:::rbd_data.126483d75f2d.000000000000000d:head, version: 52'1, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:837ddd83:::rbd_data.126483d75f2d.000000000000000d:head@52'1, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:07:29.375 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:11.099645+0000", 2024-09-21T22:07:29.375 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 18.267041249999998, 2024-09-21T22:07:29.375 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.026929337000000001, 2024-09-21T22:07:29.376 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:29.376 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:29.376 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:29.376 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:29.376 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.376 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:29.376 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.099645+0000", 2024-09-21T22:07:29.376 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:29.376 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.376 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.376 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:29.376 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.099645+0000", 2024-09-21T22:07:29.376 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:29.376 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.377 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.377 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:29.377 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.099646+0000", 2024-09-21T22:07:29.377 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.221e-06 2024-09-21T22:07:29.377 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.377 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.377 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:29.377 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.100121+0000", 2024-09-21T22:07:29.377 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00047480500000000002 2024-09-21T22:07:29.377 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.377 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.377 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:29.377 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.100144+0000", 2024-09-21T22:07:29.378 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.2812000000000001e-05 2024-09-21T22:07:29.378 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.378 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.378 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:29.378 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.100178+0000", 2024-09-21T22:07:29.378 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.4379999999999999e-05 2024-09-21T22:07:29.378 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.378 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.378 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:29.378 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.100313+0000", 2024-09-21T22:07:29.378 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013434499999999999 2024-09-21T22:07:29.378 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.378 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.378 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:29.379 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.100323+0000", 2024-09-21T22:07:29.379 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0930999999999999e-05 2024-09-21T22:07:29.379 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.379 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.379 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:29.379 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.126574+0000", 2024-09-21T22:07:29.379 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.026250842999999999 2024-09-21T22:07:29.379 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:29.379 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:29.379 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:29.379 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.379 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.379 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.1 54/53 [PushOp(2:837ddd83:::rbd_data.126483d75f2d.000000000000000d:head, version: 52'1, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:837ddd83:::rbd_data.126483d75f2d.000000000000000d:head@52'1, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:07:29.380 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:11.129410+0000", 2024-09-21T22:07:29.380 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 18.237276067, 2024-09-21T22:07:29.380 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.027274217, 2024-09-21T22:07:29.380 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:29.380 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:29.380 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:29.380 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:29.380 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.380 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:29.380 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.129410+0000", 2024-09-21T22:07:29.380 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:29.380 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.380 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.380 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:29.380 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.129410+0000", 2024-09-21T22:07:29.381 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:29.381 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.381 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.381 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:29.381 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.129412+0000", 2024-09-21T22:07:29.381 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6160000000000001e-06 2024-09-21T22:07:29.381 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.381 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.381 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:29.381 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.130407+0000", 2024-09-21T22:07:29.381 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00099554299999999999 2024-09-21T22:07:29.381 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.381 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.381 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:29.381 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.130454+0000", 2024-09-21T22:07:29.382 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.6545000000000002e-05 2024-09-21T22:07:29.382 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.382 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.382 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:29.382 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.130508+0000", 2024-09-21T22:07:29.382 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.3940999999999997e-05 2024-09-21T22:07:29.382 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.382 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.382 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:29.382 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.130658+0000", 2024-09-21T22:07:29.382 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000149732 2024-09-21T22:07:29.382 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.383 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.383 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:29.383 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.130673+0000", 2024-09-21T22:07:29.383 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5608e-05 2024-09-21T22:07:29.383 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.383 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.383 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:29.383 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.156684+0000", 2024-09-21T22:07:29.383 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.026011231999999999 2024-09-21T22:07:29.383 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:29.383 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:29.383 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:29.384 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.384 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.384 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.1 54/53 [PushOp(2:837ddd83:::rbd_data.126483d75f2d.000000000000000d:head, version: 52'1, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:837ddd83:::rbd_data.126483d75f2d.000000000000000d:head@52'1, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 4194304, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:07:29.384 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:11.159382+0000", 2024-09-21T22:07:29.384 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 18.207304679, 2024-09-21T22:07:29.384 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.026585683999999998, 2024-09-21T22:07:29.384 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:29.384 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:29.384 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:29.384 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:29.384 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.384 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:29.385 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.159382+0000", 2024-09-21T22:07:29.385 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:29.385 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.385 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.385 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:29.385 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.159382+0000", 2024-09-21T22:07:29.385 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:29.385 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.385 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.385 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:29.385 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.159383+0000", 2024-09-21T22:07:29.386 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.544e-06 2024-09-21T22:07:29.386 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.386 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.386 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:29.386 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.160369+0000", 2024-09-21T22:07:29.386 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00098620700000000001 2024-09-21T22:07:29.386 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.386 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.386 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:29.386 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.160415+0000", 2024-09-21T22:07:29.386 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.6074e-05 2024-09-21T22:07:29.387 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.387 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.387 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:29.387 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.160463+0000", 2024-09-21T22:07:29.387 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8087999999999998e-05 2024-09-21T22:07:29.387 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.387 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.387 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:29.387 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.160577+0000", 2024-09-21T22:07:29.387 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000113893 2024-09-21T22:07:29.387 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.387 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.387 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:29.388 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.160592+0000", 2024-09-21T22:07:29.388 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.465e-05 2024-09-21T22:07:29.388 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.388 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.388 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:29.388 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.185967+0000", 2024-09-21T22:07:29.388 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.025375228 2024-09-21T22:07:29.388 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:29.388 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:29.388 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:29.388 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.389 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.389 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.1 54/53 [PushOp(2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head, version: 52'2, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head@52'2, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T22:07:29.389 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:11.189894+0000", 2024-09-21T22:07:29.389 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 18.176792714000001, 2024-09-21T22:07:29.389 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.025252653, 2024-09-21T22:07:29.389 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:29.389 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:29.389 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:29.389 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:29.389 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.389 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:29.390 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.189894+0000", 2024-09-21T22:07:29.390 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:29.390 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.390 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.390 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:29.390 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.189894+0000", 2024-09-21T22:07:29.390 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:29.390 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.390 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.390 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:29.390 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.189895+0000", 2024-09-21T22:07:29.391 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.916e-06 2024-09-21T22:07:29.391 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.391 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.391 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:29.391 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.190890+0000", 2024-09-21T22:07:29.391 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00099456900000000005 2024-09-21T22:07:29.391 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.391 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.391 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:29.391 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.190939+0000", 2024-09-21T22:07:29.391 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.9132000000000003e-05 2024-09-21T22:07:29.392 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.392 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.392 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:29.392 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.190989+0000", 2024-09-21T22:07:29.392 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.9913999999999997e-05 2024-09-21T22:07:29.392 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.392 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.392 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:29.392 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.191122+0000", 2024-09-21T22:07:29.393 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013264 2024-09-21T22:07:29.393 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.393 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.393 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:29.393 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.191137+0000", 2024-09-21T22:07:29.393 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4871999999999999e-05 2024-09-21T22:07:29.393 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.393 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.393 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:29.393 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.215146+0000", 2024-09-21T22:07:29.393 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.024009610000000001 2024-09-21T22:07:29.394 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:29.394 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:29.394 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:29.394 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.394 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.394 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.1 54/53 [PushOp(2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head, version: 52'2, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head@52'2, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 4194304, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:07:29.394 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:11.273163+0000", 2024-09-21T22:07:29.394 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 18.093522801999999, 2024-09-21T22:07:29.394 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.025286874000000001, 2024-09-21T22:07:29.394 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:29.394 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:29.394 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:29.395 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:29.395 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.395 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:29.395 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.273163+0000", 2024-09-21T22:07:29.395 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:29.395 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.395 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.395 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:29.395 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.273163+0000", 2024-09-21T22:07:29.395 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:29.395 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.395 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.396 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:29.396 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.273166+0000", 2024-09-21T22:07:29.396 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5969999999999999e-06 2024-09-21T22:07:29.396 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.396 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.396 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:29.396 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.274383+0000", 2024-09-21T22:07:29.396 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001217274 2024-09-21T22:07:29.396 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.396 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.396 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:29.397 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.274430+0000", 2024-09-21T22:07:29.397 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.6848000000000003e-05 2024-09-21T22:07:29.397 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.397 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.397 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:29.397 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.274465+0000", 2024-09-21T22:07:29.397 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.5231999999999997e-05 2024-09-21T22:07:29.397 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.397 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.397 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:29.397 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.274561+0000", 2024-09-21T22:07:29.397 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.5962e-05 2024-09-21T22:07:29.398 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.398 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.398 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:29.398 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.274571+0000", 2024-09-21T22:07:29.398 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.5449999999999992e-06 2024-09-21T22:07:29.398 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.398 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.398 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:29.398 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:11.298450+0000", 2024-09-21T22:07:29.398 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.023879416000000001 2024-09-21T22:07:29.398 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:29.398 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:29.399 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:29.399 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.399 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.399 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:692 2.5 e63/37 2:b1d55633:::rbd_data.126483d75f2d.000000000000000b:head v 63'24, mlcod=57'22)", 2024-09-21T22:07:29.399 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.091267+0000", 2024-09-21T22:07:29.399 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 4.2754188700000002, 2024-09-21T22:07:29.399 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.025586098000000002, 2024-09-21T22:07:29.399 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:29.399 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:29.399 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:29.399 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:29.400 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.400 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:29.400 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.091267+0000", 2024-09-21T22:07:29.400 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:29.400 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.400 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.400 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:29.400 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.091267+0000", 2024-09-21T22:07:29.400 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:29.400 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.400 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.400 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:29.401 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.091269+0000", 2024-09-21T22:07:29.401 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9099999999999999e-06 2024-09-21T22:07:29.401 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.401 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.401 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:29.401 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.092518+0000", 2024-09-21T22:07:29.401 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001249165 2024-09-21T22:07:29.401 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.401 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.401 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:29.402 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.092549+0000", 2024-09-21T22:07:29.402 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.0369e-05 2024-09-21T22:07:29.402 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.402 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.402 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:29.402 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.092566+0000", 2024-09-21T22:07:29.402 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7122e-05 2024-09-21T22:07:29.402 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.402 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.402 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:29.402 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.092628+0000", 2024-09-21T22:07:29.402 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.2087000000000004e-05 2024-09-21T22:07:29.402 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.403 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.403 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:29.403 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.092648+0000", 2024-09-21T22:07:29.403 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.0057000000000001e-05 2024-09-21T22:07:29.403 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.403 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.403 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:29.403 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.116791+0000", 2024-09-21T22:07:29.403 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.024143248999999999 2024-09-21T22:07:29.403 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.403 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.403 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:29.404 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.116853+0000", 2024-09-21T22:07:29.404 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.2138999999999996e-05 2024-09-21T22:07:29.404 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:29.404 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:29.404 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:29.404 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.404 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.404 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:694 2.5 e63/37 2:b1d55633:::rbd_data.126483d75f2d.000000000000000b:head v 63'26, mlcod=63'24)", 2024-09-21T22:07:29.404 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.163199+0000", 2024-09-21T22:07:29.404 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 4.2034875239999998, 2024-09-21T22:07:29.404 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.034676672999999998, 2024-09-21T22:07:29.404 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:29.404 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:29.404 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:29.405 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:29.405 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.405 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:29.405 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.163199+0000", 2024-09-21T22:07:29.405 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:29.405 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.405 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.405 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:29.405 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.163199+0000", 2024-09-21T22:07:29.405 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:29.405 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.405 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.405 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:29.406 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.163200+0000", 2024-09-21T22:07:29.406 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.703e-06 2024-09-21T22:07:29.406 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.406 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.406 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:29.406 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.164013+0000", 2024-09-21T22:07:29.406 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000812447 2024-09-21T22:07:29.406 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.406 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.406 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:29.406 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.164042+0000", 2024-09-21T22:07:29.407 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9006e-05 2024-09-21T22:07:29.407 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.407 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.407 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:29.407 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.164064+0000", 2024-09-21T22:07:29.407 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1719999999999999e-05 2024-09-21T22:07:29.407 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.407 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.407 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:29.407 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.164143+0000", 2024-09-21T22:07:29.407 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.9283999999999995e-05 2024-09-21T22:07:29.407 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.407 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.408 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:29.408 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.164165+0000", 2024-09-21T22:07:29.408 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.2379e-05 2024-09-21T22:07:29.408 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.408 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.408 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:29.408 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.197827+0000", 2024-09-21T22:07:29.408 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.033661782000000001 2024-09-21T22:07:29.408 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.408 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.408 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:29.408 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.197875+0000", 2024-09-21T22:07:29.408 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8352000000000001e-05 2024-09-21T22:07:29.409 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:29.409 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:29.409 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:29.409 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.409 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.409 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:696 2.5 e63/37 2:b1d55633:::rbd_data.126483d75f2d.000000000000000b:head v 63'28, mlcod=63'26)", 2024-09-21T22:07:29.409 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.218913+0000", 2024-09-21T22:07:29.409 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 4.1477730780000002, 2024-09-21T22:07:29.409 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.056187599999999997, 2024-09-21T22:07:29.409 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:29.409 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:29.409 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:29.409 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:29.409 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.410 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:29.410 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.218913+0000", 2024-09-21T22:07:29.410 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:29.410 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.410 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.410 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:29.410 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.218913+0000", 2024-09-21T22:07:29.410 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:29.410 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.410 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.410 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:29.410 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.218914+0000", 2024-09-21T22:07:29.410 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.1930000000000001e-06 2024-09-21T22:07:29.411 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.411 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.411 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:29.411 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.219721+0000", 2024-09-21T22:07:29.411 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00080701500000000001 2024-09-21T22:07:29.411 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.411 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.411 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:29.411 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.219743+0000", 2024-09-21T22:07:29.411 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1364000000000001e-05 2024-09-21T22:07:29.411 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.411 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.412 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:29.412 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.219759+0000", 2024-09-21T22:07:29.412 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.611e-05 2024-09-21T22:07:29.412 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.412 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.412 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:29.412 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.219908+0000", 2024-09-21T22:07:29.412 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000149252 2024-09-21T22:07:29.412 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.412 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.412 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:29.412 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.219951+0000", 2024-09-21T22:07:29.412 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.2518999999999997e-05 2024-09-21T22:07:29.413 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.413 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.413 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:29.413 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.275031+0000", 2024-09-21T22:07:29.413 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.055080241000000002 2024-09-21T22:07:29.413 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.413 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.413 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:29.413 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.275101+0000", 2024-09-21T22:07:29.413 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.9906000000000005e-05 2024-09-21T22:07:29.413 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:29.414 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:29.414 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:29.414 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.414 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.414 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:698 2.5 e63/37 2:b1d55633:::rbd_data.126483d75f2d.000000000000000b:head v 63'30, mlcod=63'28)", 2024-09-21T22:07:29.414 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.289319+0000", 2024-09-21T22:07:29.414 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 4.0773668690000004, 2024-09-21T22:07:29.414 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.053930944000000001, 2024-09-21T22:07:29.414 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:29.414 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:29.414 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:29.415 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:29.415 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.415 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:29.415 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.289319+0000", 2024-09-21T22:07:29.415 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:29.415 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.415 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.415 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:29.415 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.289319+0000", 2024-09-21T22:07:29.415 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:29.415 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.416 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.416 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:29.416 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.289321+0000", 2024-09-21T22:07:29.416 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.221e-06 2024-09-21T22:07:29.416 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.416 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.416 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:29.416 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.290091+0000", 2024-09-21T22:07:29.416 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00077069300000000003 2024-09-21T22:07:29.416 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.416 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.417 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:29.417 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.290109+0000", 2024-09-21T22:07:29.417 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7218999999999999e-05 2024-09-21T22:07:29.417 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.417 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.417 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:29.417 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.290122+0000", 2024-09-21T22:07:29.417 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3678e-05 2024-09-21T22:07:29.417 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.417 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.417 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:29.418 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.290274+0000", 2024-09-21T22:07:29.418 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00015205899999999999 2024-09-21T22:07:29.418 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.418 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.418 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:29.418 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.290313+0000", 2024-09-21T22:07:29.418 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.8636000000000001e-05 2024-09-21T22:07:29.418 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.418 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.418 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:29.418 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.343179+0000", 2024-09-21T22:07:29.419 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.052866199000000003 2024-09-21T22:07:29.419 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.419 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.419 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:29.419 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.343250+0000", 2024-09-21T22:07:29.419 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.1239e-05 2024-09-21T22:07:29.419 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:29.419 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:29.419 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:29.419 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.419 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.420 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:702 2.5 e63/37 2:a70ae99f:::rbd_data.126483d75f2d.000000000000000c:head v 63'34, mlcod=63'32)", 2024-09-21T22:07:29.420 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.392532+0000", 2024-09-21T22:07:29.420 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 3.9741542810000001, 2024-09-21T22:07:29.420 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.035703990999999997, 2024-09-21T22:07:29.420 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:29.420 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:29.420 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:29.420 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:29.420 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.420 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:29.420 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.392532+0000", 2024-09-21T22:07:29.421 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:29.421 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.421 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.421 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:29.421 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.392532+0000", 2024-09-21T22:07:29.421 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:29.421 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.421 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.421 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:29.421 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.392534+0000", 2024-09-21T22:07:29.421 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.64e-06 2024-09-21T22:07:29.421 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.422 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.422 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:29.422 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.393321+0000", 2024-09-21T22:07:29.422 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00078705699999999997 2024-09-21T22:07:29.422 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.422 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.422 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:29.422 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.393347+0000", 2024-09-21T22:07:29.422 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.6735999999999999e-05 2024-09-21T22:07:29.422 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.422 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.423 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:29.423 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.393361+0000", 2024-09-21T22:07:29.423 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3346000000000001e-05 2024-09-21T22:07:29.423 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.423 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.423 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:29.423 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.393469+0000", 2024-09-21T22:07:29.423 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000108039 2024-09-21T22:07:29.423 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.423 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.423 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:29.424 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.393512+0000", 2024-09-21T22:07:29.424 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.2862999999999999e-05 2024-09-21T22:07:29.424 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.424 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.424 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:29.424 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.428170+0000", 2024-09-21T22:07:29.424 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.034658858000000001 2024-09-21T22:07:29.424 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.424 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.424 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:29.425 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.428236+0000", 2024-09-21T22:07:29.425 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.5451999999999996e-05 2024-09-21T22:07:29.425 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:29.425 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:29.425 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:29.425 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.425 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.425 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:704 2.5 e63/37 2:a70ae99f:::rbd_data.126483d75f2d.000000000000000c:head v 63'36, mlcod=63'34)", 2024-09-21T22:07:29.425 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.438174+0000", 2024-09-21T22:07:29.425 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 3.9285121470000002, 2024-09-21T22:07:29.425 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.028699802, 2024-09-21T22:07:29.426 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:29.426 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:29.426 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:29.426 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:29.426 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.426 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:29.426 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.438174+0000", 2024-09-21T22:07:29.426 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:29.426 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.426 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.426 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:29.427 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.438174+0000", 2024-09-21T22:07:29.427 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:29.427 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.427 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.427 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:29.427 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.438176+0000", 2024-09-21T22:07:29.427 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.784e-06 2024-09-21T22:07:29.427 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.427 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.427 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:29.427 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.438985+0000", 2024-09-21T22:07:29.427 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00080882000000000003 2024-09-21T22:07:29.428 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.428 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.428 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:29.428 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.439013+0000", 2024-09-21T22:07:29.428 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8316000000000001e-05 2024-09-21T22:07:29.428 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.428 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.428 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:29.428 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.439029+0000", 2024-09-21T22:07:29.428 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6079e-05 2024-09-21T22:07:29.428 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.428 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.428 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:29.428 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.439116+0000", 2024-09-21T22:07:29.429 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.6648000000000001e-05 2024-09-21T22:07:29.429 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.429 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.429 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:29.429 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.439147+0000", 2024-09-21T22:07:29.429 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.1365000000000002e-05 2024-09-21T22:07:29.429 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.429 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.429 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:29.429 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.466816+0000", 2024-09-21T22:07:29.429 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.027668567000000002 2024-09-21T22:07:29.429 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.429 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.429 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:29.430 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.466874+0000", 2024-09-21T22:07:29.430 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.8223000000000001e-05 2024-09-21T22:07:29.430 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:29.430 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:29.430 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:29.430 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.430 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.430 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:706 2.5 e63/37 2:a70ae99f:::rbd_data.126483d75f2d.000000000000000c:head v 63'38, mlcod=63'36)", 2024-09-21T22:07:29.430 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.476192+0000", 2024-09-21T22:07:29.430 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 3.8904938740000001, 2024-09-21T22:07:29.430 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.029459526, 2024-09-21T22:07:29.430 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:29.430 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:29.430 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:29.431 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:29.431 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.431 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:29.431 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.476192+0000", 2024-09-21T22:07:29.431 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:29.431 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.431 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.431 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:29.431 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.476192+0000", 2024-09-21T22:07:29.431 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:29.431 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.431 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.431 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:29.431 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.476194+0000", 2024-09-21T22:07:29.432 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.697e-06 2024-09-21T22:07:29.432 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.432 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.432 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:29.432 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.476965+0000", 2024-09-21T22:07:29.432 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00077113099999999999 2024-09-21T22:07:29.432 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.432 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.432 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:29.432 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.476993+0000", 2024-09-21T22:07:29.432 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.7769000000000001e-05 2024-09-21T22:07:29.432 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.432 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.432 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:29.433 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.477008+0000", 2024-09-21T22:07:29.433 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5e-05 2024-09-21T22:07:29.433 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.433 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.433 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:29.433 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.477073+0000", 2024-09-21T22:07:29.433 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.5161999999999997e-05 2024-09-21T22:07:29.433 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.433 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.433 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:29.433 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.477099+0000", 2024-09-21T22:07:29.433 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5995000000000001e-05 2024-09-21T22:07:29.433 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.433 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.434 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:29.434 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.505578+0000", 2024-09-21T22:07:29.434 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.028478659999999999 2024-09-21T22:07:29.434 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.434 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.434 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:29.434 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.505652+0000", 2024-09-21T22:07:29.434 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.4111999999999998e-05 2024-09-21T22:07:29.434 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:29.434 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:29.434 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:29.434 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.434 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.434 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:708 2.1 e63/53 2:837ddd83:::rbd_data.126483d75f2d.000000000000000d:head v 63'6, mlcod=57'4)", 2024-09-21T22:07:29.435 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.517007+0000", 2024-09-21T22:07:29.435 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 3.8496793550000001, 2024-09-21T22:07:29.435 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.032495584000000001, 2024-09-21T22:07:29.435 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:29.435 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:29.435 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:29.435 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:29.435 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.435 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:29.435 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.517007+0000", 2024-09-21T22:07:29.435 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:29.435 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.435 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.435 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:29.436 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.517007+0000", 2024-09-21T22:07:29.436 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:29.436 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.436 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.436 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:29.436 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.517008+0000", 2024-09-21T22:07:29.436 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.604e-06 2024-09-21T22:07:29.436 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.436 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.436 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:29.436 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.517603+0000", 2024-09-21T22:07:29.436 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00059461900000000001 2024-09-21T22:07:29.436 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.436 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.436 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:29.437 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.517632+0000", 2024-09-21T22:07:29.437 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8663e-05 2024-09-21T22:07:29.437 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.437 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.437 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:29.437 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.517647+0000", 2024-09-21T22:07:29.437 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4833e-05 2024-09-21T22:07:29.437 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.437 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.437 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:29.437 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.517714+0000", 2024-09-21T22:07:29.437 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.7669000000000006e-05 2024-09-21T22:07:29.437 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.437 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.437 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:29.438 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.517737+0000", 2024-09-21T22:07:29.438 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.2469000000000001e-05 2024-09-21T22:07:29.438 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.438 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.438 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:29.438 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.549370+0000", 2024-09-21T22:07:29.438 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.031633221000000003 2024-09-21T22:07:29.438 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.438 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.438 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:29.438 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.549502+0000", 2024-09-21T22:07:29.438 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013250600000000001 2024-09-21T22:07:29.438 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:29.439 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:29.439 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:29.439 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.439 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.439 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:710 2.1 e63/53 2:837ddd83:::rbd_data.126483d75f2d.000000000000000d:head v 63'8, mlcod=63'6)", 2024-09-21T22:07:29.439 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.566410+0000", 2024-09-21T22:07:29.439 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 3.8002762300000001, 2024-09-21T22:07:29.439 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.029498893000000002, 2024-09-21T22:07:29.439 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:29.439 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:29.439 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:29.439 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:29.439 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.439 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:29.440 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.566410+0000", 2024-09-21T22:07:29.440 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:29.440 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.440 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.440 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:29.440 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.566410+0000", 2024-09-21T22:07:29.440 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:29.440 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.440 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.440 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:29.440 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.566412+0000", 2024-09-21T22:07:29.440 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1940000000000001e-06 2024-09-21T22:07:29.440 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.440 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.441 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:29.441 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.567148+0000", 2024-09-21T22:07:29.441 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00073535500000000004 2024-09-21T22:07:29.441 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.441 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.441 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:29.441 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.567175+0000", 2024-09-21T22:07:29.441 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.7900000000000001e-05 2024-09-21T22:07:29.441 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.441 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.441 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:29.441 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.567189+0000", 2024-09-21T22:07:29.441 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3495e-05 2024-09-21T22:07:29.441 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.442 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.442 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:29.442 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.567265+0000", 2024-09-21T22:07:29.442 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.5568000000000006e-05 2024-09-21T22:07:29.442 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.442 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.442 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:29.442 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.567288+0000", 2024-09-21T22:07:29.442 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3949999999999999e-05 2024-09-21T22:07:29.442 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.442 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.442 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:29.442 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.595827+0000", 2024-09-21T22:07:29.442 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.028538500000000001 2024-09-21T22:07:29.443 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.443 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.443 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:29.443 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.595909+0000", 2024-09-21T22:07:29.443 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.1930999999999999e-05 2024-09-21T22:07:29.443 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:29.443 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:29.443 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:29.443 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.443 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.443 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:712 2.1 e63/53 2:837ddd83:::rbd_data.126483d75f2d.000000000000000d:head v 63'10, mlcod=63'8)", 2024-09-21T22:07:29.443 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.604029+0000", 2024-09-21T22:07:29.443 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 3.7626574160000001, 2024-09-21T22:07:29.443 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.037982371000000001, 2024-09-21T22:07:29.444 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:29.444 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:29.444 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:29.444 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:29.444 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.444 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:29.444 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.604029+0000", 2024-09-21T22:07:29.444 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:29.444 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.444 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.444 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:29.444 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.604029+0000", 2024-09-21T22:07:29.444 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:29.444 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.445 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.445 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:29.445 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.604031+0000", 2024-09-21T22:07:29.445 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8870000000000001e-06 2024-09-21T22:07:29.445 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.445 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.445 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:29.445 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.604395+0000", 2024-09-21T22:07:29.445 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00036422000000000001 2024-09-21T22:07:29.445 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.445 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.445 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:29.445 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.604424+0000", 2024-09-21T22:07:29.445 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8609e-05 2024-09-21T22:07:29.445 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.446 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.446 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:29.446 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.604437+0000", 2024-09-21T22:07:29.446 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3086e-05 2024-09-21T22:07:29.446 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.446 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.446 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:29.446 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.604509+0000", 2024-09-21T22:07:29.446 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.2448999999999997e-05 2024-09-21T22:07:29.446 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.446 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.446 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:29.446 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.604531+0000", 2024-09-21T22:07:29.446 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1602000000000001e-05 2024-09-21T22:07:29.447 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.447 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.447 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:29.447 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.641885+0000", 2024-09-21T22:07:29.447 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.037354545000000003 2024-09-21T22:07:29.447 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.447 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.447 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:29.447 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.642011+0000", 2024-09-21T22:07:29.447 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00012597300000000001 2024-09-21T22:07:29.447 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:29.447 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:29.447 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:29.447 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.448 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.448 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:716 2.1 e63/53 2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head v 63'14, mlcod=63'12)", 2024-09-21T22:07:29.448 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.707637+0000", 2024-09-21T22:07:29.448 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 3.6590494800000002, 2024-09-21T22:07:29.448 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.029259646, 2024-09-21T22:07:29.448 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:29.448 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:29.448 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:29.448 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:29.448 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.448 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:29.448 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.707637+0000", 2024-09-21T22:07:29.448 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:29.448 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.449 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.449 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:29.449 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.707637+0000", 2024-09-21T22:07:29.449 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:29.449 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.449 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.449 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:29.449 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.707638+0000", 2024-09-21T22:07:29.449 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.5899999999999995e-07 2024-09-21T22:07:29.449 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.449 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.449 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:29.449 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.708257+0000", 2024-09-21T22:07:29.449 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00061904699999999996 2024-09-21T22:07:29.449 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.450 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.450 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:29.450 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.708286+0000", 2024-09-21T22:07:29.450 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8972999999999999e-05 2024-09-21T22:07:29.450 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.450 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.450 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:29.450 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.708301+0000", 2024-09-21T22:07:29.450 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5712000000000001e-05 2024-09-21T22:07:29.450 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.450 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.450 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:29.450 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.708412+0000", 2024-09-21T22:07:29.451 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0001109 2024-09-21T22:07:29.451 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.451 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.451 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:29.451 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.708429+0000", 2024-09-21T22:07:29.451 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6985000000000001e-05 2024-09-21T22:07:29.451 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.451 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.451 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:29.451 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.736806+0000", 2024-09-21T22:07:29.451 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.028377263 2024-09-21T22:07:29.451 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.451 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.451 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:29.452 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.736896+0000", 2024-09-21T22:07:29.452 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.9907000000000005e-05 2024-09-21T22:07:29.452 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:29.452 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:29.452 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:29.452 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.452 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.452 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:718 2.1 e63/53 2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head v 63'16, mlcod=63'14)", 2024-09-21T22:07:29.452 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.759837+0000", 2024-09-21T22:07:29.452 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 3.6068497310000001, 2024-09-21T22:07:29.452 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.031201353000000001, 2024-09-21T22:07:29.452 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:29.452 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:29.452 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:29.453 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:29.453 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.453 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:29.453 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.759837+0000", 2024-09-21T22:07:29.453 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:29.453 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.453 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.453 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:29.453 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.759837+0000", 2024-09-21T22:07:29.453 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:29.453 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.453 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.453 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:29.453 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.759837+0000", 2024-09-21T22:07:29.454 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.2999999999999999e-07 2024-09-21T22:07:29.454 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.454 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.454 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:29.454 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.760443+0000", 2024-09-21T22:07:29.454 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00060515600000000003 2024-09-21T22:07:29.454 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.454 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.454 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:29.454 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.760472+0000", 2024-09-21T22:07:29.454 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9416000000000001e-05 2024-09-21T22:07:29.454 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.454 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.454 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:29.455 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.760485+0000", 2024-09-21T22:07:29.455 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3359e-05 2024-09-21T22:07:29.455 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.455 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.455 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:29.455 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.760616+0000", 2024-09-21T22:07:29.455 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013029400000000001 2024-09-21T22:07:29.455 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.455 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.455 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:29.455 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.760656+0000", 2024-09-21T22:07:29.455 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.0788000000000003e-05 2024-09-21T22:07:29.455 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.455 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.456 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:29.456 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.790947+0000", 2024-09-21T22:07:29.456 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.030290480000000002 2024-09-21T22:07:29.456 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.456 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.456 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:29.456 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.791038+0000", 2024-09-21T22:07:29.456 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.0929999999999998e-05 2024-09-21T22:07:29.456 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:29.456 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:29.456 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:29.456 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.456 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.456 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:720 2.1 e63/53 2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head v 63'18, mlcod=63'16)", 2024-09-21T22:07:29.456 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.805111+0000", 2024-09-21T22:07:29.457 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 3.5615753560000001, 2024-09-21T22:07:29.457 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.046082864000000001, 2024-09-21T22:07:29.457 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:29.457 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:29.457 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:29.457 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:29.457 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.457 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:29.457 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.805111+0000", 2024-09-21T22:07:29.457 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:29.457 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.457 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.457 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:29.457 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.805111+0000", 2024-09-21T22:07:29.458 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:29.458 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.458 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.458 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:29.458 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.805112+0000", 2024-09-21T22:07:29.458 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.02e-06 2024-09-21T22:07:29.458 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.458 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.458 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:29.458 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.805429+0000", 2024-09-21T22:07:29.458 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00031740799999999998 2024-09-21T22:07:29.458 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.458 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.458 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:29.459 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.805459+0000", 2024-09-21T22:07:29.459 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.955e-05 2024-09-21T22:07:29.459 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.459 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.459 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:29.459 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.805472+0000", 2024-09-21T22:07:29.459 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3533000000000001e-05 2024-09-21T22:07:29.459 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.459 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.459 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:29.459 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.805581+0000", 2024-09-21T22:07:29.459 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000108194 2024-09-21T22:07:29.459 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.459 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.460 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:29.460 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.805615+0000", 2024-09-21T22:07:29.460 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.4310000000000002e-05 2024-09-21T22:07:29.460 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.460 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.460 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:29.460 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.851146+0000", 2024-09-21T22:07:29.460 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.045531092000000002 2024-09-21T22:07:29.460 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.460 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.460 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:29.460 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.851194+0000", 2024-09-21T22:07:29.460 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.7756999999999997e-05 2024-09-21T22:07:29.460 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:29.461 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:29.461 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:29.461 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.461 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.461 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:723 2.1 e63/53 2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head v 63'20, mlcod=63'18)", 2024-09-21T22:07:29.461 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.862341+0000", 2024-09-21T22:07:29.461 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 3.5043454220000001, 2024-09-21T22:07:29.461 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.036454671000000001, 2024-09-21T22:07:29.461 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:29.461 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:29.461 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:29.461 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:29.461 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.462 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:29.462 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.862341+0000", 2024-09-21T22:07:29.462 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:29.462 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.462 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.462 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:29.462 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.862341+0000", 2024-09-21T22:07:29.462 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:29.462 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.462 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.462 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:29.462 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.862342+0000", 2024-09-21T22:07:29.462 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.0299999999999997e-07 2024-09-21T22:07:29.462 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.463 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.463 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:29.463 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.862721+0000", 2024-09-21T22:07:29.463 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00037889800000000002 2024-09-21T22:07:29.463 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.463 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.463 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:29.463 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.862738+0000", 2024-09-21T22:07:29.463 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6957999999999999e-05 2024-09-21T22:07:29.463 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.463 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.463 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:29.463 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.862747+0000", 2024-09-21T22:07:29.464 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.0729999999999994e-06 2024-09-21T22:07:29.464 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.464 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.464 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:29.464 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.862830+0000", 2024-09-21T22:07:29.464 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.3113000000000002e-05 2024-09-21T22:07:29.464 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.464 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.464 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:29.464 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.862860+0000", 2024-09-21T22:07:29.464 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9980000000000001e-05 2024-09-21T22:07:29.464 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.464 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.464 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:29.465 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.898705+0000", 2024-09-21T22:07:29.465 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.035845063000000003 2024-09-21T22:07:29.465 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.465 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.465 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:29.465 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.898795+0000", 2024-09-21T22:07:29.465 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.0682999999999999e-05 2024-09-21T22:07:29.465 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:29.465 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:29.465 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:29.465 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:29.465 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:29.465 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:07:29.467 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T22:07:29.467 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:29.462+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:29.468 INFO:tasks.workunit.client.0.smithi157.stderr:blkdiscard: /dev/nbd0: BLKDISCARD ioctl failed: Operation not supported 2024-09-21T22:07:29.468 INFO:tasks.workunit.client.0.smithi157.stderr:+ return 0 2024-09-21T22:07:29.468 INFO:tasks.workunit.client.0.smithi157.stderr:+ sync 2024-09-21T22:07:29.568 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:29.564+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:29.643 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:07:29.643 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:07:29.643 INFO:teuthology.orchestra.run.smithi039.stdout: "num_ops": 0 2024-09-21T22:07:29.643 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:07:29.658 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T22:07:29.704 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:29.702+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:29.769 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:29.764+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:29.783 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:07:29.783 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:07:29.783 INFO:teuthology.orchestra.run.smithi039.stdout: "complaint_time": 30, 2024-09-21T22:07:29.783 INFO:teuthology.orchestra.run.smithi039.stdout: "num_blocked_ops": 0 2024-09-21T22:07:29.783 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:07:29.799 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T22:07:29.888 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:29.886+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:29.971 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:29.966+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:29.972 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:07:29.972 INFO:teuthology.orchestra.run.smithi039.stdout: "size": 20, 2024-09-21T22:07:29.972 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 600, 2024-09-21T22:07:29.972 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [ 2024-09-21T22:07:29.973 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.973 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.3 55/54 [PushOp(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head, version: 52'2, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head@52'2, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T22:07:29.973 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.084379+0000", 2024-09-21T22:07:29.973 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 17.883588507999999, 2024-09-21T22:07:29.973 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.051020674000000002, 2024-09-21T22:07:29.973 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:29.973 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:29.973 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:29.973 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:29.973 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.973 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:29.973 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.084379+0000", 2024-09-21T22:07:29.974 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:29.974 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.974 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.974 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:29.974 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.084379+0000", 2024-09-21T22:07:29.974 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:29.974 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.974 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.974 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:29.974 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.084381+0000", 2024-09-21T22:07:29.974 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.73e-06 2024-09-21T22:07:29.974 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.974 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.974 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:29.975 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.085494+0000", 2024-09-21T22:07:29.975 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011131559999999999 2024-09-21T22:07:29.975 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.975 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.975 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:29.975 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.085542+0000", 2024-09-21T22:07:29.975 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8334999999999997e-05 2024-09-21T22:07:29.975 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.975 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.975 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:29.975 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.085591+0000", 2024-09-21T22:07:29.976 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.9165000000000001e-05 2024-09-21T22:07:29.976 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.976 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.976 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:29.976 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.085777+0000", 2024-09-21T22:07:29.976 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000186152 2024-09-21T22:07:29.976 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.976 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.976 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:29.976 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.085796+0000", 2024-09-21T22:07:29.976 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8612999999999999e-05 2024-09-21T22:07:29.976 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.976 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.976 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:29.976 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.135399+0000", 2024-09-21T22:07:29.977 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.049603522999999997 2024-09-21T22:07:29.977 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:29.977 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:29.977 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:29.977 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.977 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.977 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.3 55/54 [PushOp(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head, version: 52'2, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head@52'2, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:07:29.977 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.139554+0000", 2024-09-21T22:07:29.977 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 17.828413404999999, 2024-09-21T22:07:29.977 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.056215389999999997, 2024-09-21T22:07:29.977 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:29.977 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:29.977 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:29.977 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:29.977 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.977 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:29.978 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.139554+0000", 2024-09-21T22:07:29.978 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:29.978 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.978 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.978 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:29.978 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.139554+0000", 2024-09-21T22:07:29.978 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:29.978 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.978 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.978 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:29.978 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.139556+0000", 2024-09-21T22:07:29.978 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7999999999999999e-06 2024-09-21T22:07:29.978 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.978 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.979 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:29.979 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.140671+0000", 2024-09-21T22:07:29.979 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001115612 2024-09-21T22:07:29.979 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.979 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.979 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:29.979 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.140720+0000", 2024-09-21T22:07:29.979 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8631999999999998e-05 2024-09-21T22:07:29.979 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.979 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.979 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:29.979 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.140752+0000", 2024-09-21T22:07:29.979 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.2199000000000001e-05 2024-09-21T22:07:29.979 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.980 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.980 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:29.980 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.140900+0000", 2024-09-21T22:07:29.980 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00014786000000000001 2024-09-21T22:07:29.980 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.980 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.980 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:29.980 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.140917+0000", 2024-09-21T22:07:29.980 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6753999999999999e-05 2024-09-21T22:07:29.980 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.980 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.980 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:29.980 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.195769+0000", 2024-09-21T22:07:29.981 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.054852533000000002 2024-09-21T22:07:29.981 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:29.981 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:29.981 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:29.981 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.981 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.981 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.3 55/54 [PushOp(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head, version: 52'2, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head@52'2, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:07:29.981 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.199187+0000", 2024-09-21T22:07:29.981 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 17.768779860999999, 2024-09-21T22:07:29.981 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.051543975999999998, 2024-09-21T22:07:29.981 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:29.981 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:29.981 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:29.981 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:29.981 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.982 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:29.982 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.199187+0000", 2024-09-21T22:07:29.982 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:29.982 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.982 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.982 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:29.982 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.199187+0000", 2024-09-21T22:07:29.982 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:29.982 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.982 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.983 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:29.983 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.199189+0000", 2024-09-21T22:07:29.983 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7969999999999999e-06 2024-09-21T22:07:29.983 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.983 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.983 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:29.983 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.200355+0000", 2024-09-21T22:07:29.983 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011660139999999999 2024-09-21T22:07:29.983 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.983 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.983 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:29.983 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.200405+0000", 2024-09-21T22:07:29.983 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.0124000000000003e-05 2024-09-21T22:07:29.983 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.984 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.984 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:29.984 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.200457+0000", 2024-09-21T22:07:29.984 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.1177999999999997e-05 2024-09-21T22:07:29.984 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.984 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.984 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:29.984 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.200635+0000", 2024-09-21T22:07:29.984 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00017841299999999999 2024-09-21T22:07:29.984 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.984 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.984 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:29.984 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.200653+0000", 2024-09-21T22:07:29.985 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7944999999999998e-05 2024-09-21T22:07:29.985 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.985 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.985 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:29.985 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.250731+0000", 2024-09-21T22:07:29.985 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.050078505000000002 2024-09-21T22:07:29.985 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:29.985 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:29.985 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:29.985 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.985 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.985 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.3 55/54 [PushOp(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head, version: 52'2, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head@52'2, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 4194304, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:07:29.985 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.253966+0000", 2024-09-21T22:07:29.985 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 17.714001531000001, 2024-09-21T22:07:29.986 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.041100606999999997, 2024-09-21T22:07:29.986 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:29.986 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:29.986 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:29.986 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:29.986 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.986 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:29.986 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.253966+0000", 2024-09-21T22:07:29.986 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:29.986 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.986 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.986 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:29.986 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.253966+0000", 2024-09-21T22:07:29.986 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:29.987 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.987 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.987 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:29.987 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.253967+0000", 2024-09-21T22:07:29.987 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.423e-06 2024-09-21T22:07:29.987 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.987 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.987 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:29.987 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.255117+0000", 2024-09-21T22:07:29.987 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011496709999999999 2024-09-21T22:07:29.987 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.987 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.987 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:29.987 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.255167+0000", 2024-09-21T22:07:29.987 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.0315000000000002e-05 2024-09-21T22:07:29.988 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.988 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.988 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:29.988 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.255218+0000", 2024-09-21T22:07:29.988 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.0390999999999999e-05 2024-09-21T22:07:29.988 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.988 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.988 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:29.988 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.255391+0000", 2024-09-21T22:07:29.988 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00017332799999999999 2024-09-21T22:07:29.988 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.988 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.988 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:29.988 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.255405+0000", 2024-09-21T22:07:29.988 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3723999999999999e-05 2024-09-21T22:07:29.989 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.989 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.989 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:29.989 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.295066+0000", 2024-09-21T22:07:29.989 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.039661755 2024-09-21T22:07:29.989 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:29.989 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:29.989 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:29.989 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.989 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.989 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.3 55/54 [PushOp(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head, version: 52'3, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head@52'3, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T22:07:29.989 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.299195+0000", 2024-09-21T22:07:29.989 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 17.668772766, 2024-09-21T22:07:29.990 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.040955077999999999, 2024-09-21T22:07:29.990 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:29.990 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:29.990 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:29.990 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:29.990 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.990 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:29.990 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.299195+0000", 2024-09-21T22:07:29.990 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:29.990 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.990 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.990 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:29.990 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.299195+0000", 2024-09-21T22:07:29.991 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:29.991 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.991 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.991 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:29.991 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.299196+0000", 2024-09-21T22:07:29.991 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.604e-06 2024-09-21T22:07:29.991 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.991 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.991 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:29.991 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.300356+0000", 2024-09-21T22:07:29.991 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011602699999999999 2024-09-21T22:07:29.991 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.991 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.991 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:29.992 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.300406+0000", 2024-09-21T22:07:29.992 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.9144999999999998e-05 2024-09-21T22:07:29.992 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.992 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.992 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:29.992 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.300435+0000", 2024-09-21T22:07:29.992 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9886000000000002e-05 2024-09-21T22:07:29.992 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.992 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.992 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:29.992 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.300590+0000", 2024-09-21T22:07:29.992 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00015431200000000001 2024-09-21T22:07:29.992 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.992 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.992 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:29.993 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.300604+0000", 2024-09-21T22:07:29.993 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4559000000000001e-05 2024-09-21T22:07:29.993 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.993 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.993 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:29.993 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.340150+0000", 2024-09-21T22:07:29.993 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.039545301999999997 2024-09-21T22:07:29.993 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:29.993 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:29.993 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:29.993 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.993 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.993 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.3 55/54 [PushOp(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head, version: 52'3, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head@52'3, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:07:29.993 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.343754+0000", 2024-09-21T22:07:29.994 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 17.624213149999999, 2024-09-21T22:07:29.994 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.063560607000000005, 2024-09-21T22:07:29.994 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:29.994 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:29.994 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:29.994 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:29.994 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.994 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:29.994 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.343754+0000", 2024-09-21T22:07:29.994 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:29.994 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.994 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.994 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:29.994 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.343754+0000", 2024-09-21T22:07:29.994 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:29.995 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.995 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.995 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:29.995 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.343756+0000", 2024-09-21T22:07:29.995 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7069999999999999e-06 2024-09-21T22:07:29.995 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.995 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.995 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:29.995 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.344864+0000", 2024-09-21T22:07:29.995 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011085159999999999 2024-09-21T22:07:29.995 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.995 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.995 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:29.995 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.344914+0000", 2024-09-21T22:07:29.996 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.9462000000000002e-05 2024-09-21T22:07:29.996 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.996 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.996 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:29.996 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.344961+0000", 2024-09-21T22:07:29.996 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.7633999999999998e-05 2024-09-21T22:07:29.996 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.996 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.996 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:29.996 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.345171+0000", 2024-09-21T22:07:29.996 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000209652 2024-09-21T22:07:29.996 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.996 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.997 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:29.997 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.345193+0000", 2024-09-21T22:07:29.997 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1985999999999999e-05 2024-09-21T22:07:29.997 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.997 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.997 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:29.997 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.407315+0000", 2024-09-21T22:07:29.997 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.06212165 2024-09-21T22:07:29.997 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:29.997 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:29.997 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:29.997 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.997 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.997 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.3 55/54 [PushOp(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head, version: 52'3, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head@52'3, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:07:29.997 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.410551+0000", 2024-09-21T22:07:29.998 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 17.557416019000001, 2024-09-21T22:07:29.998 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.054517522999999998, 2024-09-21T22:07:29.998 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:29.998 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:29.998 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:29.998 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:29.998 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.998 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:29.998 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.410551+0000", 2024-09-21T22:07:29.998 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:29.998 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.998 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.998 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:29.998 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.410551+0000", 2024-09-21T22:07:29.999 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:29.999 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.999 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.999 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:29.999 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.410553+0000", 2024-09-21T22:07:29.999 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5200000000000001e-06 2024-09-21T22:07:29.999 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.999 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.999 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:29.999 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.411747+0000", 2024-09-21T22:07:29.999 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011946089999999999 2024-09-21T22:07:29.999 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:29.999 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:29.999 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:29.999 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.411796+0000", 2024-09-21T22:07:30.000 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8717999999999997e-05 2024-09-21T22:07:30.000 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.000 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.000 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:30.000 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.411844+0000", 2024-09-21T22:07:30.000 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8291999999999998e-05 2024-09-21T22:07:30.000 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.000 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.000 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:30.000 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.412066+0000", 2024-09-21T22:07:30.000 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00022115899999999999 2024-09-21T22:07:30.000 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.000 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.000 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:30.001 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.412081+0000", 2024-09-21T22:07:30.001 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5654e-05 2024-09-21T22:07:30.001 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.001 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.001 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:30.001 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.465069+0000", 2024-09-21T22:07:30.001 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.052987570999999997 2024-09-21T22:07:30.001 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:30.001 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:30.001 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:30.001 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.001 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.001 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.3 55/54 [PushOp(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head, version: 52'3, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head@52'3, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 4194304, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:07:30.001 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.468339+0000", 2024-09-21T22:07:30.001 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 17.499628354999999, 2024-09-21T22:07:30.002 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.064010024999999998, 2024-09-21T22:07:30.002 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:30.002 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:30.002 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:30.002 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:30.002 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.002 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:30.002 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.468339+0000", 2024-09-21T22:07:30.002 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:30.002 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.002 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.002 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:30.002 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.468339+0000", 2024-09-21T22:07:30.002 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:30.003 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.003 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.003 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:30.003 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.468340+0000", 2024-09-21T22:07:30.003 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4330000000000001e-06 2024-09-21T22:07:30.003 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.003 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.003 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:30.003 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.469484+0000", 2024-09-21T22:07:30.003 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001143535 2024-09-21T22:07:30.003 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.003 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.003 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:30.003 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.469533+0000", 2024-09-21T22:07:30.003 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8980999999999997e-05 2024-09-21T22:07:30.004 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.004 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.004 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:30.004 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.469581+0000", 2024-09-21T22:07:30.004 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8551999999999999e-05 2024-09-21T22:07:30.004 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.004 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.004 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:30.004 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.469804+0000", 2024-09-21T22:07:30.004 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00022287699999999999 2024-09-21T22:07:30.004 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.004 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.004 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:30.004 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.469829+0000", 2024-09-21T22:07:30.005 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4766e-05 2024-09-21T22:07:30.005 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.005 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.005 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:30.005 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.532349+0000", 2024-09-21T22:07:30.005 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.062519880999999999 2024-09-21T22:07:30.005 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:30.005 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:30.005 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:30.005 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.005 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.005 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4840.0:668 2.3 2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:07:30.005 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.409114+0000", 2024-09-21T22:07:30.005 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 5.5588531489999999, 2024-09-21T22:07:30.005 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.048703758999999999, 2024-09-21T22:07:30.006 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:30.006 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:30.006 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:30.006 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-21T22:07:30.006 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4840", 2024-09-21T22:07:30.006 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:30.006 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 668 2024-09-21T22:07:30.006 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.006 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:30.006 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.006 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:30.006 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.409114+0000", 2024-09-21T22:07:30.006 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:30.006 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.007 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.007 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:30.007 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.409114+0000", 2024-09-21T22:07:30.007 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:30.007 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.007 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.007 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:30.007 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.409115+0000", 2024-09-21T22:07:30.007 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.114e-06 2024-09-21T22:07:30.007 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.007 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.007 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:30.007 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.410308+0000", 2024-09-21T22:07:30.007 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011930949999999999 2024-09-21T22:07:30.008 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.008 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.008 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:30.008 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.410325+0000", 2024-09-21T22:07:30.008 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7088e-05 2024-09-21T22:07:30.008 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.008 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.008 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:30.008 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.410334+0000", 2024-09-21T22:07:30.008 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.2549999999999997e-06 2024-09-21T22:07:30.008 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.008 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.008 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:30.008 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.410393+0000", 2024-09-21T22:07:30.009 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.9327999999999998e-05 2024-09-21T22:07:30.009 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.009 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.009 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:30.009 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.410482+0000", 2024-09-21T22:07:30.009 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.9019999999999998e-05 2024-09-21T22:07:30.009 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.009 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.009 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-21T22:07:30.009 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.410617+0000", 2024-09-21T22:07:30.009 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000135232 2024-09-21T22:07:30.009 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.009 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.009 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:30.009 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.457051+0000", 2024-09-21T22:07:30.010 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.04643415 2024-09-21T22:07:30.010 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.010 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.010 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-21T22:07:30.010 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.457675+0000", 2024-09-21T22:07:30.010 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00062332700000000002 2024-09-21T22:07:30.010 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.010 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.010 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:30.010 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.457735+0000", 2024-09-21T22:07:30.010 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.029e-05 2024-09-21T22:07:30.010 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.010 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.010 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:30.010 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.457818+0000", 2024-09-21T22:07:30.011 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.2860000000000003e-05 2024-09-21T22:07:30.011 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:30.011 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:30.011 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:30.011 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.011 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.011 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4840.0:670 2.3 2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:07:30.011 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.466980+0000", 2024-09-21T22:07:30.011 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 5.5009870809999999, 2024-09-21T22:07:30.011 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.050913930000000003, 2024-09-21T22:07:30.011 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:30.011 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:30.011 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:30.011 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-21T22:07:30.012 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4840", 2024-09-21T22:07:30.012 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:30.012 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 670 2024-09-21T22:07:30.012 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.012 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:30.012 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.012 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:30.012 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.466980+0000", 2024-09-21T22:07:30.012 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:30.012 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.012 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.012 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:30.012 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.466980+0000", 2024-09-21T22:07:30.012 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:30.013 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.013 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.013 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:30.013 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.466981+0000", 2024-09-21T22:07:30.013 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0839999999999999e-06 2024-09-21T22:07:30.013 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.013 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.013 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:30.013 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.467815+0000", 2024-09-21T22:07:30.013 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00083330399999999999 2024-09-21T22:07:30.013 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.013 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.013 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:30.013 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.467832+0000", 2024-09-21T22:07:30.013 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7601e-05 2024-09-21T22:07:30.014 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.014 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.014 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:30.014 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.467840+0000", 2024-09-21T22:07:30.014 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.2220000000000002e-06 2024-09-21T22:07:30.014 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.014 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.014 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:30.014 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.467974+0000", 2024-09-21T22:07:30.014 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000133411 2024-09-21T22:07:30.014 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.014 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.014 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:30.014 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.468223+0000", 2024-09-21T22:07:30.015 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00024906400000000001 2024-09-21T22:07:30.015 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.015 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.015 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-21T22:07:30.015 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.468616+0000", 2024-09-21T22:07:30.015 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000393433 2024-09-21T22:07:30.015 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.015 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.015 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:30.015 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.517016+0000", 2024-09-21T22:07:30.015 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.048399708 2024-09-21T22:07:30.015 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.016 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.016 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-21T22:07:30.016 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.517693+0000", 2024-09-21T22:07:30.016 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00067681000000000002 2024-09-21T22:07:30.016 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.016 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.016 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:30.017 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.517789+0000", 2024-09-21T22:07:30.018 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.5778000000000004e-05 2024-09-21T22:07:30.018 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.018 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.018 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:30.018 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.517894+0000", 2024-09-21T22:07:30.018 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000105515 2024-09-21T22:07:30.018 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:30.018 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:30.018 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:30.018 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.018 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.018 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4840.0:672 2.3 2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:07:30.018 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.527127+0000", 2024-09-21T22:07:30.018 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 5.4408406190000003, 2024-09-21T22:07:30.019 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.047345602000000001, 2024-09-21T22:07:30.019 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:30.019 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:30.019 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:30.019 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-21T22:07:30.019 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4840", 2024-09-21T22:07:30.019 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:30.019 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 672 2024-09-21T22:07:30.019 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.019 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:30.019 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.019 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:30.019 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.527127+0000", 2024-09-21T22:07:30.019 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:30.020 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.020 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.020 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:30.020 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.527127+0000", 2024-09-21T22:07:30.020 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:30.020 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.020 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.020 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:30.020 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.527128+0000", 2024-09-21T22:07:30.020 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2810000000000001e-06 2024-09-21T22:07:30.020 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.020 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.020 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:30.020 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.527962+0000", 2024-09-21T22:07:30.021 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00083445199999999996 2024-09-21T22:07:30.021 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.021 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.021 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:30.021 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.527981+0000", 2024-09-21T22:07:30.021 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9009e-05 2024-09-21T22:07:30.021 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.021 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.021 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:30.021 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.527990+0000", 2024-09-21T22:07:30.021 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.8619999999999992e-06 2024-09-21T22:07:30.021 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.021 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.021 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:30.021 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.528061+0000", 2024-09-21T22:07:30.022 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.0226000000000002e-05 2024-09-21T22:07:30.022 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.022 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.022 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:30.022 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.528156+0000", 2024-09-21T22:07:30.022 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.5724999999999996e-05 2024-09-21T22:07:30.022 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.022 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.022 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-21T22:07:30.022 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.528303+0000", 2024-09-21T22:07:30.022 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000146374 2024-09-21T22:07:30.022 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.022 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.022 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:30.023 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.573772+0000", 2024-09-21T22:07:30.023 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.045469745999999998 2024-09-21T22:07:30.023 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.023 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.023 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-21T22:07:30.023 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.574299+0000", 2024-09-21T22:07:30.023 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00052649200000000004 2024-09-21T22:07:30.023 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.023 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.023 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:30.023 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.574366+0000", 2024-09-21T22:07:30.023 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.6872000000000006e-05 2024-09-21T22:07:30.023 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.023 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.023 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:30.024 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.574472+0000", 2024-09-21T22:07:30.024 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000106563 2024-09-21T22:07:30.024 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:30.024 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:30.024 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:30.024 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.024 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.024 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4840.0:686 2.3 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:07:30.024 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.943561+0000", 2024-09-21T22:07:30.024 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 5.0244063429999999, 2024-09-21T22:07:30.024 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.049021493999999999, 2024-09-21T22:07:30.024 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:30.024 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:30.024 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:30.025 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-21T22:07:30.025 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4840", 2024-09-21T22:07:30.025 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:30.025 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 686 2024-09-21T22:07:30.025 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.025 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:30.025 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.025 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:30.025 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.943561+0000", 2024-09-21T22:07:30.025 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:30.025 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.025 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.025 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:30.026 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.943561+0000", 2024-09-21T22:07:30.026 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:30.026 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.026 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.026 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:30.026 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.943563+0000", 2024-09-21T22:07:30.026 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9420000000000002e-06 2024-09-21T22:07:30.026 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.026 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.026 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:30.026 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.944546+0000", 2024-09-21T22:07:30.026 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00098357099999999992 2024-09-21T22:07:30.026 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.026 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.027 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:30.027 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.944565+0000", 2024-09-21T22:07:30.027 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8621e-05 2024-09-21T22:07:30.027 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.027 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.027 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:30.027 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.944575+0000", 2024-09-21T22:07:30.027 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.4639999999999995e-06 2024-09-21T22:07:30.027 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.027 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.027 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:30.027 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.944647+0000", 2024-09-21T22:07:30.027 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.2786000000000004e-05 2024-09-21T22:07:30.027 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.028 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.028 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:30.028 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.944746+0000", 2024-09-21T22:07:30.028 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.8196999999999995e-05 2024-09-21T22:07:30.028 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.028 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.028 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-21T22:07:30.028 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.944890+0000", 2024-09-21T22:07:30.028 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000144804 2024-09-21T22:07:30.028 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.028 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.028 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:30.028 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.992404+0000", 2024-09-21T22:07:30.028 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.047513211999999999 2024-09-21T22:07:30.029 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.029 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.029 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-21T22:07:30.029 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.992480+0000", 2024-09-21T22:07:30.029 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.6491999999999999e-05 2024-09-21T22:07:30.029 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.029 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.029 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:30.029 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.992521+0000", 2024-09-21T22:07:30.029 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.0608000000000002e-05 2024-09-21T22:07:30.029 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.029 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.029 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:30.029 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.992582+0000", 2024-09-21T22:07:30.030 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.1797000000000005e-05 2024-09-21T22:07:30.030 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:30.030 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:30.030 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:30.030 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.030 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.030 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4840.0:708 2.1 2:837ddd83:::rbd_data.126483d75f2d.000000000000000d:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:07:30.030 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.515313+0000", 2024-09-21T22:07:30.030 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 4.4526548180000001, 2024-09-21T22:07:30.030 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.041155283000000001, 2024-09-21T22:07:30.030 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:30.030 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:30.030 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:30.030 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-21T22:07:30.031 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4840", 2024-09-21T22:07:30.031 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:30.031 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 708 2024-09-21T22:07:30.031 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.031 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:30.031 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.031 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:30.031 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.515313+0000", 2024-09-21T22:07:30.031 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:30.031 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.031 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.031 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:30.031 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.515313+0000", 2024-09-21T22:07:30.031 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:30.031 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.032 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.032 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:30.032 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.515314+0000", 2024-09-21T22:07:30.032 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7969999999999999e-06 2024-09-21T22:07:30.032 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.032 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.032 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:30.032 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.516396+0000", 2024-09-21T22:07:30.032 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0010819919999999999 2024-09-21T22:07:30.032 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.032 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.032 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:30.032 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.516425+0000", 2024-09-21T22:07:30.032 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9179999999999998e-05 2024-09-21T22:07:30.032 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.033 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.033 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:30.033 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.516447+0000", 2024-09-21T22:07:30.033 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1024999999999999e-05 2024-09-21T22:07:30.033 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.033 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.033 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:30.033 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.516611+0000", 2024-09-21T22:07:30.033 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00016448 2024-09-21T22:07:30.033 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.033 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.033 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:30.033 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.516686+0000", 2024-09-21T22:07:30.033 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.4783999999999994e-05 2024-09-21T22:07:30.033 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.034 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.034 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:30.034 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.516798+0000", 2024-09-21T22:07:30.034 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00011200399999999999 2024-09-21T22:07:30.034 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.034 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.034 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:30.034 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.555373+0000", 2024-09-21T22:07:30.034 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.038574978000000003 2024-09-21T22:07:30.034 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.034 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.034 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-21T22:07:30.034 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.556297+0000", 2024-09-21T22:07:30.034 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00092460299999999997 2024-09-21T22:07:30.035 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.035 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.035 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:30.035 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.556370+0000", 2024-09-21T22:07:30.035 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.2476000000000002e-05 2024-09-21T22:07:30.035 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.035 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.035 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:30.035 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.556468+0000", 2024-09-21T22:07:30.035 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.7963999999999999e-05 2024-09-21T22:07:30.035 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:30.035 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:30.035 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:30.035 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.035 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.036 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4840.0:714 2.1 2:837ddd83:::rbd_data.126483d75f2d.000000000000000d:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:07:30.036 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.652912+0000", 2024-09-21T22:07:30.036 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 4.315055375, 2024-09-21T22:07:30.036 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.041243425, 2024-09-21T22:07:30.036 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:30.036 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:30.036 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:30.036 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-21T22:07:30.036 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4840", 2024-09-21T22:07:30.036 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:30.036 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 714 2024-09-21T22:07:30.036 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.036 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:30.037 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.037 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:30.037 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.652912+0000", 2024-09-21T22:07:30.037 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:30.037 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.037 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.037 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:30.037 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.652912+0000", 2024-09-21T22:07:30.037 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:30.037 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.037 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.037 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:30.037 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.652914+0000", 2024-09-21T22:07:30.037 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.685e-06 2024-09-21T22:07:30.038 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.038 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.038 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:30.038 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.653763+0000", 2024-09-21T22:07:30.038 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00084910499999999996 2024-09-21T22:07:30.038 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.038 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.038 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:30.038 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.653793+0000", 2024-09-21T22:07:30.038 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.0405000000000001e-05 2024-09-21T22:07:30.038 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.038 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.038 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:30.038 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.653808+0000", 2024-09-21T22:07:30.038 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5044e-05 2024-09-21T22:07:30.039 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.039 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.039 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:30.039 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.653923+0000", 2024-09-21T22:07:30.039 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00011453 2024-09-21T22:07:30.039 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.039 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.039 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:30.039 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.654063+0000", 2024-09-21T22:07:30.039 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000139896 2024-09-21T22:07:30.039 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.039 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.039 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:30.039 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.654220+0000", 2024-09-21T22:07:30.040 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00015757799999999999 2024-09-21T22:07:30.040 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.040 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.040 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:30.040 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.692653+0000", 2024-09-21T22:07:30.040 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.038432980999999998 2024-09-21T22:07:30.040 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.040 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.040 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-21T22:07:30.040 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.693934+0000", 2024-09-21T22:07:30.040 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0012810810000000001 2024-09-21T22:07:30.040 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.040 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.040 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:30.041 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.694044+0000", 2024-09-21T22:07:30.041 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000109279 2024-09-21T22:07:30.041 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.041 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.041 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:30.041 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.694155+0000", 2024-09-21T22:07:30.041 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000111841 2024-09-21T22:07:30.041 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:30.041 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:30.041 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:30.041 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.041 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.041 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4840.0:720 2.1 2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:07:30.041 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.803827+0000", 2024-09-21T22:07:30.041 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 4.1641400669999999, 2024-09-21T22:07:30.042 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.049890510999999998, 2024-09-21T22:07:30.042 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:30.042 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:30.042 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:30.042 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-21T22:07:30.042 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4840", 2024-09-21T22:07:30.042 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:30.042 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 720 2024-09-21T22:07:30.042 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.042 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:30.042 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.042 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:30.042 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.803827+0000", 2024-09-21T22:07:30.042 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:30.043 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.043 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.043 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:30.043 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.803827+0000", 2024-09-21T22:07:30.043 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:30.043 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.043 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.043 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:30.043 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.803828+0000", 2024-09-21T22:07:30.043 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.569999999999999e-07 2024-09-21T22:07:30.043 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.043 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.043 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:30.043 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.804689+0000", 2024-09-21T22:07:30.043 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00086121799999999997 2024-09-21T22:07:30.044 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.044 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.044 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:30.044 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.804706+0000", 2024-09-21T22:07:30.044 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6266999999999999e-05 2024-09-21T22:07:30.044 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.044 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.044 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:30.044 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.804713+0000", 2024-09-21T22:07:30.044 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.1010000000000003e-06 2024-09-21T22:07:30.044 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.044 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.044 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:30.044 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.804781+0000", 2024-09-21T22:07:30.044 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.7871000000000001e-05 2024-09-21T22:07:30.045 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.045 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.045 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:30.045 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.804859+0000", 2024-09-21T22:07:30.045 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.7891000000000004e-05 2024-09-21T22:07:30.045 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.045 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.045 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:30.045 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.804971+0000", 2024-09-21T22:07:30.045 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00011275100000000001 2024-09-21T22:07:30.045 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.045 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.045 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:30.045 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.852704+0000", 2024-09-21T22:07:30.045 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.047732361000000001 2024-09-21T22:07:30.046 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.046 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.046 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-21T22:07:30.046 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.853446+0000", 2024-09-21T22:07:30.046 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00074258799999999997 2024-09-21T22:07:30.046 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.046 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.046 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:30.046 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.853608+0000", 2024-09-21T22:07:30.046 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00016190900000000001 2024-09-21T22:07:30.046 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.046 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.046 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:30.046 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.853718+0000", 2024-09-21T22:07:30.047 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000109597 2024-09-21T22:07:30.047 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:30.047 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:30.047 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:30.047 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.047 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.047 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4840.0:723 2.1 2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:07:30.047 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.860543+0000", 2024-09-21T22:07:30.047 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 4.1074246329999999, 2024-09-21T22:07:30.047 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.062354957000000003, 2024-09-21T22:07:30.047 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:30.047 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:30.047 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:30.047 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-21T22:07:30.047 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4840", 2024-09-21T22:07:30.048 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:30.048 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 723 2024-09-21T22:07:30.048 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.048 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:30.048 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.048 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:30.048 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.860543+0000", 2024-09-21T22:07:30.048 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:30.048 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.048 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.048 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:30.048 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.860543+0000", 2024-09-21T22:07:30.048 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:30.048 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.048 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.049 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:30.049 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.860544+0000", 2024-09-21T22:07:30.049 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.1119999999999999e-06 2024-09-21T22:07:30.049 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.049 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.049 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:30.049 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.861400+0000", 2024-09-21T22:07:30.049 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000856393 2024-09-21T22:07:30.049 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.049 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.049 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:30.049 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.861419+0000", 2024-09-21T22:07:30.049 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8992e-05 2024-09-21T22:07:30.049 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.049 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.050 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:30.050 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.861428+0000", 2024-09-21T22:07:30.050 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.551e-06 2024-09-21T22:07:30.050 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.050 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.050 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:30.050 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.861555+0000", 2024-09-21T22:07:30.050 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000127286 2024-09-21T22:07:30.050 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.050 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.050 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:30.050 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.861816+0000", 2024-09-21T22:07:30.050 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00026100899999999998 2024-09-21T22:07:30.050 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.051 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.051 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:30.051 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.862189+0000", 2024-09-21T22:07:30.051 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00037261899999999998 2024-09-21T22:07:30.051 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.051 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.051 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:30.051 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.921514+0000", 2024-09-21T22:07:30.051 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.059325210000000003 2024-09-21T22:07:30.051 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.051 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.051 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-21T22:07:30.051 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.922721+0000", 2024-09-21T22:07:30.051 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0012066819999999999 2024-09-21T22:07:30.051 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.052 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.052 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:30.052 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.922797+0000", 2024-09-21T22:07:30.052 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.6162999999999996e-05 2024-09-21T22:07:30.052 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.052 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.052 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:30.052 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.922898+0000", 2024-09-21T22:07:30.052 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00010094000000000001 2024-09-21T22:07:30.052 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:30.052 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:30.052 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:30.052 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.052 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.052 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:725 2.0 e63/16 2:08b4f96a:::rbd_data.126483d75f2d.000000000000000f:head v 63'33, mlcod=63'31)", 2024-09-21T22:07:30.053 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.936778+0000", 2024-09-21T22:07:30.053 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 4.0311888910000002, 2024-09-21T22:07:30.053 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.047671937999999997, 2024-09-21T22:07:30.053 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:30.053 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:30.053 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:30.053 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:30.053 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.053 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:30.053 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.936778+0000", 2024-09-21T22:07:30.053 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:30.053 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.053 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.053 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:30.053 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.936778+0000", 2024-09-21T22:07:30.054 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:30.054 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.054 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.054 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:30.054 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.936780+0000", 2024-09-21T22:07:30.054 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7799999999999999e-06 2024-09-21T22:07:30.054 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.054 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.054 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:30.054 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.938475+0000", 2024-09-21T22:07:30.054 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0016951749999999999 2024-09-21T22:07:30.054 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.054 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.054 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:30.055 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.938506+0000", 2024-09-21T22:07:30.055 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.0865999999999999e-05 2024-09-21T22:07:30.055 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.055 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.055 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:30.055 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.938520+0000", 2024-09-21T22:07:30.055 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3825999999999999e-05 2024-09-21T22:07:30.055 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.055 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.055 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:30.055 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.938624+0000", 2024-09-21T22:07:30.055 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00010391900000000001 2024-09-21T22:07:30.055 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.055 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.055 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:30.056 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.938651+0000", 2024-09-21T22:07:30.056 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.7481999999999999e-05 2024-09-21T22:07:30.056 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.056 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.056 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:30.056 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.984369+0000", 2024-09-21T22:07:30.056 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.045717595 2024-09-21T22:07:30.056 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.056 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.056 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:30.056 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.984450+0000", 2024-09-21T22:07:30.056 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.1295e-05 2024-09-21T22:07:30.056 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:30.056 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:30.056 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:30.057 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.057 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.057 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:727 2.0 e63/16 2:08b4f96a:::rbd_data.126483d75f2d.000000000000000f:head v 63'35, mlcod=63'33)", 2024-09-21T22:07:30.057 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.998090+0000", 2024-09-21T22:07:30.057 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 3.969877715, 2024-09-21T22:07:30.057 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.052205704999999998, 2024-09-21T22:07:30.057 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:30.057 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:30.057 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:30.057 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:30.057 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.057 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:30.057 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.998090+0000", 2024-09-21T22:07:30.057 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:30.057 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.058 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.058 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:30.058 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.998090+0000", 2024-09-21T22:07:30.058 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:30.058 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.058 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.058 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:30.058 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.998091+0000", 2024-09-21T22:07:30.058 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6929999999999999e-06 2024-09-21T22:07:30.058 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.058 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.058 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:30.058 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.998940+0000", 2024-09-21T22:07:30.058 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00084875200000000004 2024-09-21T22:07:30.059 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.059 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.059 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:30.059 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.998970+0000", 2024-09-21T22:07:30.059 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9709999999999998e-05 2024-09-21T22:07:30.059 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.059 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.059 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:30.059 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.998984+0000", 2024-09-21T22:07:30.059 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3739e-05 2024-09-21T22:07:30.059 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.059 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.059 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:30.059 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.999097+0000", 2024-09-21T22:07:30.059 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000113484 2024-09-21T22:07:30.060 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.060 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.060 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:30.060 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.999135+0000", 2024-09-21T22:07:30.060 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.8405999999999998e-05 2024-09-21T22:07:30.060 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.060 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.060 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:30.060 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.050245+0000", 2024-09-21T22:07:30.060 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.051109935000000002 2024-09-21T22:07:30.060 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.060 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.060 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:30.060 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.050295+0000", 2024-09-21T22:07:30.060 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.9985999999999999e-05 2024-09-21T22:07:30.061 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:30.061 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:30.061 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:30.061 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.061 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.061 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:729 2.0 e63/16 2:08b4f96a:::rbd_data.126483d75f2d.000000000000000f:head v 63'37, mlcod=63'35)", 2024-09-21T22:07:30.061 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:26.062424+0000", 2024-09-21T22:07:30.061 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 3.9055430929999999, 2024-09-21T22:07:30.061 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.047312186999999999, 2024-09-21T22:07:30.061 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:30.061 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:30.061 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:30.061 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:30.061 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.061 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:30.062 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.062424+0000", 2024-09-21T22:07:30.062 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:30.062 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.062 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.062 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:30.062 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.062424+0000", 2024-09-21T22:07:30.062 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:30.062 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.062 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.062 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:30.062 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.062426+0000", 2024-09-21T22:07:30.062 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7129999999999999e-06 2024-09-21T22:07:30.062 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.063 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.063 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:30.063 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.063211+0000", 2024-09-21T22:07:30.063 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00078488500000000003 2024-09-21T22:07:30.063 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.063 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.063 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:30.063 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.063241+0000", 2024-09-21T22:07:30.063 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.0056000000000002e-05 2024-09-21T22:07:30.063 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.063 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.063 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:30.063 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.063255+0000", 2024-09-21T22:07:30.063 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4022999999999999e-05 2024-09-21T22:07:30.063 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.064 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.064 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:30.064 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.063384+0000", 2024-09-21T22:07:30.064 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00012920600000000001 2024-09-21T22:07:30.064 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.064 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.064 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:30.064 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.063421+0000", 2024-09-21T22:07:30.064 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.7242e-05 2024-09-21T22:07:30.064 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.064 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.064 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:30.064 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.109666+0000", 2024-09-21T22:07:30.064 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.046244446000000002 2024-09-21T22:07:30.064 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.065 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.065 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:30.065 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.109736+0000", 2024-09-21T22:07:30.065 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.0616000000000003e-05 2024-09-21T22:07:30.065 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:30.065 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:30.065 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:30.065 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.065 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.065 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:731 2.0 e63/16 2:08b4f96a:::rbd_data.126483d75f2d.000000000000000f:head v 63'39, mlcod=63'37)", 2024-09-21T22:07:30.065 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:26.135946+0000", 2024-09-21T22:07:30.065 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 3.8320215859999998, 2024-09-21T22:07:30.065 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.054290359000000003, 2024-09-21T22:07:30.065 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:30.065 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:30.066 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:30.066 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:30.066 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.066 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:30.066 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.135946+0000", 2024-09-21T22:07:30.066 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:30.066 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.066 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.066 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:30.066 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.135946+0000", 2024-09-21T22:07:30.066 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:30.066 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.066 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.066 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:30.066 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.135948+0000", 2024-09-21T22:07:30.067 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7999999999999999e-06 2024-09-21T22:07:30.067 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.067 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.067 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:30.067 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.136730+0000", 2024-09-21T22:07:30.067 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00078271899999999997 2024-09-21T22:07:30.067 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.067 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.067 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:30.067 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.136760+0000", 2024-09-21T22:07:30.067 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9876e-05 2024-09-21T22:07:30.067 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.067 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.067 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:30.067 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.136774+0000", 2024-09-21T22:07:30.068 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3478999999999999e-05 2024-09-21T22:07:30.068 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.068 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.068 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:30.068 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.136944+0000", 2024-09-21T22:07:30.068 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000170626 2024-09-21T22:07:30.068 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.068 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.068 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:30.068 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.136985+0000", 2024-09-21T22:07:30.068 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.0377999999999999e-05 2024-09-21T22:07:30.068 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.068 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.068 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:30.068 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.190137+0000", 2024-09-21T22:07:30.069 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.053152438000000003 2024-09-21T22:07:30.069 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.069 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.069 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:30.069 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.190236+0000", 2024-09-21T22:07:30.069 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.9042999999999993e-05 2024-09-21T22:07:30.069 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:30.069 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:30.069 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:30.069 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:30.069 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:30.069 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:07:30.070 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T22:07:30.072 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:30.068+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:30.173 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:30.170+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:30.246 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:07:30.247 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:07:30.247 INFO:teuthology.orchestra.run.smithi039.stdout: "num_ops": 0 2024-09-21T22:07:30.247 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:07:30.263 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T22:07:30.273 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:30.268+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:30.374 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:30.372+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:30.433 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:07:30.433 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:07:30.433 INFO:teuthology.orchestra.run.smithi039.stdout: "complaint_time": 30, 2024-09-21T22:07:30.433 INFO:teuthology.orchestra.run.smithi039.stdout: "num_blocked_ops": 0 2024-09-21T22:07:30.433 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:07:30.447 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T22:07:30.475 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:30.476+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:30.576 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:30.574+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:30.577 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:07:30.577 INFO:teuthology.orchestra.run.smithi039.stdout: "size": 20, 2024-09-21T22:07:30.577 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 600, 2024-09-21T22:07:30.577 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [ 2024-09-21T22:07:30.577 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.577 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPull(2.3 e55/54 cost 1048576)", 2024-09-21T22:07:30.577 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.081116+0000", 2024-09-21T22:07:30.577 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 18.492756217, 2024-09-21T22:07:30.578 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0030214159999999999, 2024-09-21T22:07:30.578 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:30.578 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:30.578 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-21T22:07:30.578 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:30.578 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.578 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:30.578 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.081116+0000", 2024-09-21T22:07:30.578 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:30.578 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.578 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.578 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:30.578 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.081116+0000", 2024-09-21T22:07:30.579 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:30.579 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.579 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.579 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:30.579 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.081118+0000", 2024-09-21T22:07:30.579 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.545e-06 2024-09-21T22:07:30.579 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.579 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.579 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:30.579 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.081133+0000", 2024-09-21T22:07:30.579 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5642e-05 2024-09-21T22:07:30.579 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.579 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.579 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:30.580 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.081155+0000", 2024-09-21T22:07:30.580 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1858e-05 2024-09-21T22:07:30.580 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.580 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.580 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:30.580 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.081171+0000", 2024-09-21T22:07:30.580 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5804e-05 2024-09-21T22:07:30.580 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.580 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.580 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:30.580 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.081291+0000", 2024-09-21T22:07:30.580 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000120553 2024-09-21T22:07:30.580 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.580 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.581 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:30.581 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.084137+0000", 2024-09-21T22:07:30.581 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.002846014 2024-09-21T22:07:30.581 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:30.581 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:30.581 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:30.581 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.581 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.581 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPull(2.3 e55/54 cost 1048576)", 2024-09-21T22:07:30.581 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.136357+0000", 2024-09-21T22:07:30.581 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 18.437515150999999, 2024-09-21T22:07:30.581 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.002843904, 2024-09-21T22:07:30.581 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:30.581 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:30.582 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-21T22:07:30.582 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:30.582 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.582 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:30.582 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.136357+0000", 2024-09-21T22:07:30.582 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:30.582 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.582 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.582 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:30.582 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.136357+0000", 2024-09-21T22:07:30.582 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:30.582 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.582 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.582 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:30.583 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.136359+0000", 2024-09-21T22:07:30.583 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.0669999999999999e-06 2024-09-21T22:07:30.583 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.583 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.583 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:30.583 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.136381+0000", 2024-09-21T22:07:30.583 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.2104999999999999e-05 2024-09-21T22:07:30.583 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.583 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.583 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:30.583 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.136412+0000", 2024-09-21T22:07:30.583 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.0602999999999998e-05 2024-09-21T22:07:30.583 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.583 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.583 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:30.584 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.136427+0000", 2024-09-21T22:07:30.584 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4832e-05 2024-09-21T22:07:30.584 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.584 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.584 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:30.584 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.136569+0000", 2024-09-21T22:07:30.584 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000142702 2024-09-21T22:07:30.584 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.584 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.584 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:30.584 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.139201+0000", 2024-09-21T22:07:30.584 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.002631595 2024-09-21T22:07:30.584 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:30.584 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:30.585 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:30.585 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.585 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.585 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:667 2.3 e63/54 2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head v 63'4, mlcod=0'0)", 2024-09-21T22:07:30.585 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.402454+0000", 2024-09-21T22:07:30.585 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 6.171418633, 2024-09-21T22:07:30.585 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0058375959999999996, 2024-09-21T22:07:30.585 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:30.585 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:30.585 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:30.585 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:30.585 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.585 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:30.585 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.402454+0000", 2024-09-21T22:07:30.586 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:30.586 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.586 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.586 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:30.586 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.402454+0000", 2024-09-21T22:07:30.586 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:30.586 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.586 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.586 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:30.586 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.402456+0000", 2024-09-21T22:07:30.586 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1179999999999999e-06 2024-09-21T22:07:30.586 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.586 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.586 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:30.587 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.402618+0000", 2024-09-21T22:07:30.587 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00016243 2024-09-21T22:07:30.587 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.587 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.587 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:30.587 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.402643+0000", 2024-09-21T22:07:30.587 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4638999999999999e-05 2024-09-21T22:07:30.587 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.587 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.587 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:30.587 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.402658+0000", 2024-09-21T22:07:30.587 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5211000000000001e-05 2024-09-21T22:07:30.587 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.587 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.588 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:30.588 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.402756+0000", 2024-09-21T22:07:30.588 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.7880000000000005e-05 2024-09-21T22:07:30.588 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.588 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.588 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:30.588 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.402779+0000", 2024-09-21T22:07:30.588 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3524999999999999e-05 2024-09-21T22:07:30.588 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.588 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.588 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:30.588 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.408233+0000", 2024-09-21T22:07:30.588 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0054534700000000002 2024-09-21T22:07:30.588 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.589 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.589 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:30.589 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.408291+0000", 2024-09-21T22:07:30.589 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.8322999999999997e-05 2024-09-21T22:07:30.589 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:30.589 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:30.589 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:30.589 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.589 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.589 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:668 2.3 e63/54 2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head v 63'5, mlcod=52'3)", 2024-09-21T22:07:30.589 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.410724+0000", 2024-09-21T22:07:30.589 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 6.1631482159999997, 2024-09-21T22:07:30.589 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.031657809000000002, 2024-09-21T22:07:30.589 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:30.589 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:30.590 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:30.590 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:30.590 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.590 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:30.590 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.410724+0000", 2024-09-21T22:07:30.590 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:30.590 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.590 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.590 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:30.590 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.410724+0000", 2024-09-21T22:07:30.590 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:30.590 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.590 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.590 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:30.591 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.410725+0000", 2024-09-21T22:07:30.591 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.4300000000000001e-07 2024-09-21T22:07:30.591 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.591 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.591 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:30.591 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.411436+0000", 2024-09-21T22:07:30.591 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00071118600000000004 2024-09-21T22:07:30.591 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.591 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.591 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:30.591 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.411455+0000", 2024-09-21T22:07:30.591 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8357999999999999e-05 2024-09-21T22:07:30.592 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.592 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.592 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:30.592 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.411470+0000", 2024-09-21T22:07:30.592 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5753999999999999e-05 2024-09-21T22:07:30.592 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.592 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.592 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:30.592 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.411537+0000", 2024-09-21T22:07:30.592 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.6766000000000004e-05 2024-09-21T22:07:30.592 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.592 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.592 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:30.592 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.411571+0000", 2024-09-21T22:07:30.593 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.3555e-05 2024-09-21T22:07:30.593 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.593 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.593 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:30.593 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.442317+0000", 2024-09-21T22:07:30.593 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.030746393 2024-09-21T22:07:30.593 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.593 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.593 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:30.593 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.442382+0000", 2024-09-21T22:07:30.593 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.4853999999999999e-05 2024-09-21T22:07:30.593 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:30.593 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:30.593 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:30.593 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.594 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.594 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:669 2.3 e63/54 2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head v 63'6, mlcod=63'4)", 2024-09-21T22:07:30.594 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.459924+0000", 2024-09-21T22:07:30.594 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 6.113948658, 2024-09-21T22:07:30.594 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0062657900000000002, 2024-09-21T22:07:30.594 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:30.594 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:30.594 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:30.594 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:30.594 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.594 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:30.594 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.459924+0000", 2024-09-21T22:07:30.594 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:30.595 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.595 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.595 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:30.595 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.459924+0000", 2024-09-21T22:07:30.595 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:30.595 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.595 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.595 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:30.595 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.459925+0000", 2024-09-21T22:07:30.595 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8169999999999999e-06 2024-09-21T22:07:30.595 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.595 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.595 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:30.595 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.459983+0000", 2024-09-21T22:07:30.596 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.7567e-05 2024-09-21T22:07:30.596 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.596 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.596 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:30.596 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.460011+0000", 2024-09-21T22:07:30.596 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8455000000000001e-05 2024-09-21T22:07:30.596 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.596 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.596 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:30.596 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.460021+0000", 2024-09-21T22:07:30.596 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.9410000000000002e-06 2024-09-21T22:07:30.596 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.596 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.596 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:30.597 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.460093+0000", 2024-09-21T22:07:30.597 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.1231999999999998e-05 2024-09-21T22:07:30.597 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.597 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.597 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:30.597 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.460124+0000", 2024-09-21T22:07:30.597 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.1335999999999999e-05 2024-09-21T22:07:30.597 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.597 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.597 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:30.597 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.466130+0000", 2024-09-21T22:07:30.597 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0060055519999999999 2024-09-21T22:07:30.597 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.597 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.597 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:30.598 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.466189+0000", 2024-09-21T22:07:30.598 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.9889999999999997e-05 2024-09-21T22:07:30.598 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:30.598 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:30.598 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:30.598 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.598 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.598 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:670 2.3 e63/54 2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head v 63'7, mlcod=63'5)", 2024-09-21T22:07:30.598 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.468764+0000", 2024-09-21T22:07:30.599 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 6.1051086769999996, 2024-09-21T22:07:30.599 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.037963433999999997, 2024-09-21T22:07:30.599 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:30.599 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:30.599 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:30.599 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:30.599 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.599 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:30.599 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.468764+0000", 2024-09-21T22:07:30.599 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:30.599 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.599 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.600 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:30.600 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.468764+0000", 2024-09-21T22:07:30.600 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:30.600 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.600 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.600 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:30.600 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.468765+0000", 2024-09-21T22:07:30.600 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.9699999999999994e-07 2024-09-21T22:07:30.600 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.600 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.600 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:30.600 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.469295+0000", 2024-09-21T22:07:30.600 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00053003600000000001 2024-09-21T22:07:30.600 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.601 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.601 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:30.601 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.469311+0000", 2024-09-21T22:07:30.601 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6841999999999999e-05 2024-09-21T22:07:30.601 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.601 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.601 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:30.601 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.469324+0000", 2024-09-21T22:07:30.601 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2373e-05 2024-09-21T22:07:30.601 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.601 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.601 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:30.601 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.469391+0000", 2024-09-21T22:07:30.601 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.6880999999999999e-05 2024-09-21T22:07:30.602 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.602 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.602 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:30.602 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.469419+0000", 2024-09-21T22:07:30.602 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8742999999999999e-05 2024-09-21T22:07:30.602 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.602 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.602 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:30.602 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.506662+0000", 2024-09-21T22:07:30.602 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.037242994000000001 2024-09-21T22:07:30.602 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.603 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.603 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:30.603 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.506727+0000", 2024-09-21T22:07:30.603 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.4567999999999996e-05 2024-09-21T22:07:30.603 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:30.603 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:30.603 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:30.603 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.603 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.603 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:671 2.3 e63/54 2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head v 63'8, mlcod=63'6)", 2024-09-21T22:07:30.603 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.519938+0000", 2024-09-21T22:07:30.603 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 6.0539341379999998, 2024-09-21T22:07:30.603 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0058323419999999999, 2024-09-21T22:07:30.603 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:30.603 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:30.604 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:30.604 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:30.604 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.604 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:30.604 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.519938+0000", 2024-09-21T22:07:30.604 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:30.604 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.604 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.604 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:30.604 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.519938+0000", 2024-09-21T22:07:30.604 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:30.604 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.604 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.605 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:30.605 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.519940+0000", 2024-09-21T22:07:30.605 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.666e-06 2024-09-21T22:07:30.605 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.605 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.605 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:30.605 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.520031+0000", 2024-09-21T22:07:30.605 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.1051999999999995e-05 2024-09-21T22:07:30.605 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.605 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.605 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:30.605 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.520056+0000", 2024-09-21T22:07:30.605 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5459000000000001e-05 2024-09-21T22:07:30.605 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.605 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.606 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:30.606 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.520072+0000", 2024-09-21T22:07:30.606 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5801e-05 2024-09-21T22:07:30.606 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.606 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.606 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:30.606 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.520178+0000", 2024-09-21T22:07:30.606 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000105388 2024-09-21T22:07:30.606 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.606 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.606 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:30.607 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.520204+0000", 2024-09-21T22:07:30.607 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.6208e-05 2024-09-21T22:07:30.607 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.607 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.607 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:30.607 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.525735+0000", 2024-09-21T22:07:30.607 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0055308520000000002 2024-09-21T22:07:30.607 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.607 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.607 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:30.607 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.525770+0000", 2024-09-21T22:07:30.607 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.5916e-05 2024-09-21T22:07:30.607 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:30.607 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:30.608 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:30.608 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.608 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.608 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:672 2.3 e63/54 2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head v 63'9, mlcod=63'7)", 2024-09-21T22:07:30.608 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.528432+0000", 2024-09-21T22:07:30.608 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 6.0454407620000001, 2024-09-21T22:07:30.608 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.029763412999999999, 2024-09-21T22:07:30.608 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:30.608 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:30.608 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:30.608 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:30.608 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.608 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:30.608 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.528432+0000", 2024-09-21T22:07:30.609 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:30.609 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.609 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.609 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:30.609 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.528432+0000", 2024-09-21T22:07:30.609 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:30.609 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.609 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.609 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:30.609 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.528433+0000", 2024-09-21T22:07:30.609 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7179999999999999e-06 2024-09-21T22:07:30.609 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.609 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.609 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:30.610 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.528926+0000", 2024-09-21T22:07:30.610 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00049238500000000002 2024-09-21T22:07:30.610 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.610 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.610 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:30.610 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.528955+0000", 2024-09-21T22:07:30.610 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9037e-05 2024-09-21T22:07:30.610 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.610 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.610 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:30.610 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.528971+0000", 2024-09-21T22:07:30.610 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6622000000000001e-05 2024-09-21T22:07:30.610 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.610 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.611 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:30.611 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.529047+0000", 2024-09-21T22:07:30.611 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.5246000000000004e-05 2024-09-21T22:07:30.611 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.611 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.611 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:30.611 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.529074+0000", 2024-09-21T22:07:30.611 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.7532e-05 2024-09-21T22:07:30.611 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.611 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.611 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:30.611 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.558149+0000", 2024-09-21T22:07:30.611 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.029075192999999999 2024-09-21T22:07:30.612 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.612 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.612 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:30.612 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.558195+0000", 2024-09-21T22:07:30.612 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.5680000000000003e-05 2024-09-21T22:07:30.612 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:30.612 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:30.612 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:30.612 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.612 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.612 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:673 2.3 e63/54 2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head v 63'10, mlcod=63'8)", 2024-09-21T22:07:30.612 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.576513+0000", 2024-09-21T22:07:30.612 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 5.9973593269999999, 2024-09-21T22:07:30.612 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.003674666, 2024-09-21T22:07:30.613 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:30.613 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:30.613 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:30.613 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:30.613 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.613 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:30.613 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.576513+0000", 2024-09-21T22:07:30.613 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:30.614 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.614 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.614 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:30.614 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.576513+0000", 2024-09-21T22:07:30.614 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:30.614 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.614 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.615 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:30.615 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.576514+0000", 2024-09-21T22:07:30.615 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.1299999999999998e-07 2024-09-21T22:07:30.615 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.615 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.615 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:30.615 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.576567+0000", 2024-09-21T22:07:30.615 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.2710000000000002e-05 2024-09-21T22:07:30.615 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.615 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.615 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:30.615 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.576579+0000", 2024-09-21T22:07:30.615 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2035000000000001e-05 2024-09-21T22:07:30.616 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.616 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.616 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:30.616 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.576585+0000", 2024-09-21T22:07:30.616 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.6050000000000003e-06 2024-09-21T22:07:30.616 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.616 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.616 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:30.616 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.576638+0000", 2024-09-21T22:07:30.616 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.2726999999999998e-05 2024-09-21T22:07:30.616 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.617 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.617 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:30.617 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.576656+0000", 2024-09-21T22:07:30.617 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8084000000000002e-05 2024-09-21T22:07:30.617 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.617 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.617 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:30.617 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.580156+0000", 2024-09-21T22:07:30.617 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0035002700000000002 2024-09-21T22:07:30.617 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.617 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.617 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:30.618 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.580188+0000", 2024-09-21T22:07:30.618 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.1322000000000003e-05 2024-09-21T22:07:30.618 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:30.618 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:30.618 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:30.618 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.618 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.618 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:674 2.3 e63/54 2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head v 63'11, mlcod=63'9)", 2024-09-21T22:07:30.618 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.584683+0000", 2024-09-21T22:07:30.618 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 5.989188886, 2024-09-21T22:07:30.618 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.028028352999999999, 2024-09-21T22:07:30.619 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:30.619 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:30.619 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:30.619 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:30.619 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.619 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:30.619 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.584683+0000", 2024-09-21T22:07:30.619 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:30.619 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.619 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.619 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:30.619 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.584683+0000", 2024-09-21T22:07:30.620 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:30.620 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.620 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.620 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:30.620 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.584684+0000", 2024-09-21T22:07:30.620 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.4600000000000004e-07 2024-09-21T22:07:30.620 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.620 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.620 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:30.620 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.585041+0000", 2024-09-21T22:07:30.620 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000356752 2024-09-21T22:07:30.620 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.621 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.621 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:30.621 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.585054+0000", 2024-09-21T22:07:30.621 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3387000000000001e-05 2024-09-21T22:07:30.621 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.621 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.621 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:30.621 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.585065+0000", 2024-09-21T22:07:30.621 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.1124999999999999e-05 2024-09-21T22:07:30.621 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.621 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.621 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:30.622 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.585142+0000", 2024-09-21T22:07:30.622 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.6743999999999996e-05 2024-09-21T22:07:30.622 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.622 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.622 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:30.622 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.585170+0000", 2024-09-21T22:07:30.622 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.7569999999999999e-05 2024-09-21T22:07:30.622 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.622 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.622 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:30.622 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.612637+0000", 2024-09-21T22:07:30.623 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.027467803999999998 2024-09-21T22:07:30.623 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.623 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.623 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:30.623 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.612712+0000", 2024-09-21T22:07:30.623 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.4325000000000004e-05 2024-09-21T22:07:30.623 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:30.623 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:30.623 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:30.623 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.623 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.623 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:683 2.3 e63/54 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head v 63'12, mlcod=63'10)", 2024-09-21T22:07:30.624 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.888242+0000", 2024-09-21T22:07:30.624 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 5.6856302679999997, 2024-09-21T22:07:30.624 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0085158290000000008, 2024-09-21T22:07:30.624 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:30.624 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:30.624 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:30.624 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:30.624 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.624 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:30.624 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.888242+0000", 2024-09-21T22:07:30.624 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:30.625 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.625 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.625 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:30.625 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.888242+0000", 2024-09-21T22:07:30.625 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:30.625 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.625 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.625 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:30.625 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.888244+0000", 2024-09-21T22:07:30.625 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.553e-06 2024-09-21T22:07:30.625 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.625 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.626 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:30.626 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.888295+0000", 2024-09-21T22:07:30.626 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.1916000000000002e-05 2024-09-21T22:07:30.626 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.626 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.626 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:30.626 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.888319+0000", 2024-09-21T22:07:30.626 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3139999999999999e-05 2024-09-21T22:07:30.626 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.626 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.626 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:30.626 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.888332+0000", 2024-09-21T22:07:30.627 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2993000000000001e-05 2024-09-21T22:07:30.627 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.627 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.627 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:30.627 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.888441+0000", 2024-09-21T22:07:30.627 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000109039 2024-09-21T22:07:30.627 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.627 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.627 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:30.627 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.888481+0000", 2024-09-21T22:07:30.627 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.9950000000000002e-05 2024-09-21T22:07:30.627 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.627 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.628 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:30.628 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.896688+0000", 2024-09-21T22:07:30.628 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0082078729999999992 2024-09-21T22:07:30.628 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.628 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.628 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:30.628 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.896758+0000", 2024-09-21T22:07:30.628 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.9364999999999997e-05 2024-09-21T22:07:30.628 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:30.628 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:30.628 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:30.628 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.629 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.629 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:684 2.3 e63/54 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head v 63'13, mlcod=63'11)", 2024-09-21T22:07:30.629 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.899243+0000", 2024-09-21T22:07:30.629 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 5.6746294910000001, 2024-09-21T22:07:30.629 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.033436192000000003, 2024-09-21T22:07:30.629 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:30.629 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:30.629 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:30.629 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:30.629 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.629 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:30.629 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.899243+0000", 2024-09-21T22:07:30.629 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:30.629 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.630 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.630 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:30.630 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.899243+0000", 2024-09-21T22:07:30.630 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:30.630 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.630 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.630 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:30.630 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.899244+0000", 2024-09-21T22:07:30.630 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.705e-06 2024-09-21T22:07:30.630 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.630 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.630 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:30.630 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.899604+0000", 2024-09-21T22:07:30.631 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00035930599999999999 2024-09-21T22:07:30.631 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.631 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.631 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:30.631 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.899622+0000", 2024-09-21T22:07:30.631 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8487000000000001e-05 2024-09-21T22:07:30.631 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.631 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.631 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:30.631 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.899634+0000", 2024-09-21T22:07:30.631 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2116999999999999e-05 2024-09-21T22:07:30.631 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.631 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.631 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:30.632 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.899721+0000", 2024-09-21T22:07:30.632 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.6718999999999994e-05 2024-09-21T22:07:30.632 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.632 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.632 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:30.632 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.899755+0000", 2024-09-21T22:07:30.632 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.4292000000000003e-05 2024-09-21T22:07:30.634 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.634 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.634 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:30.634 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.932545+0000", 2024-09-21T22:07:30.634 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.03278959 2024-09-21T22:07:30.634 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.634 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.634 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:30.634 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.932679+0000", 2024-09-21T22:07:30.634 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013397600000000001 2024-09-21T22:07:30.634 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:30.634 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:30.634 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:30.634 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.635 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.635 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:685 2.3 e63/54 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head v 63'14, mlcod=63'12)", 2024-09-21T22:07:30.635 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.935246+0000", 2024-09-21T22:07:30.635 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 5.6386266540000003, 2024-09-21T22:07:30.635 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0073590590000000003, 2024-09-21T22:07:30.635 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:30.635 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:30.635 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:30.635 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:30.635 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.635 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:30.635 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.935246+0000", 2024-09-21T22:07:30.635 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:30.636 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.636 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.636 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:30.636 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.935246+0000", 2024-09-21T22:07:30.636 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:30.636 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.636 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.636 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:30.636 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.935247+0000", 2024-09-21T22:07:30.636 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4470000000000001e-06 2024-09-21T22:07:30.636 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.636 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.636 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:30.636 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.935295+0000", 2024-09-21T22:07:30.637 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.7963000000000001e-05 2024-09-21T22:07:30.637 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.637 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.637 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:30.637 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.935313+0000", 2024-09-21T22:07:30.637 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8317000000000001e-05 2024-09-21T22:07:30.637 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.637 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.637 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:30.637 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.935352+0000", 2024-09-21T22:07:30.637 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.8552e-05 2024-09-21T22:07:30.637 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.637 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.637 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:30.638 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.935438+0000", 2024-09-21T22:07:30.638 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.6405999999999999e-05 2024-09-21T22:07:30.638 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.638 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.638 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:30.638 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.935471+0000", 2024-09-21T22:07:30.638 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.3142999999999998e-05 2024-09-21T22:07:30.638 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.638 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.638 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:30.638 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.942552+0000", 2024-09-21T22:07:30.638 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0070809510000000003 2024-09-21T22:07:30.638 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.639 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.639 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:30.639 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.942605+0000", 2024-09-21T22:07:30.639 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.2280000000000001e-05 2024-09-21T22:07:30.639 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:30.639 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:30.639 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:30.639 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.639 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.639 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:686 2.3 e63/54 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head v 63'15, mlcod=63'13)", 2024-09-21T22:07:30.639 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.945043+0000", 2024-09-21T22:07:30.639 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 5.6288293410000003, 2024-09-21T22:07:30.639 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.04716302, 2024-09-21T22:07:30.640 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:30.640 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:30.640 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:30.640 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:30.640 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.640 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:30.640 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.945043+0000", 2024-09-21T22:07:30.640 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:30.640 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.640 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.640 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:30.641 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.945043+0000", 2024-09-21T22:07:30.641 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:30.641 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.641 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.641 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:30.641 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.945045+0000", 2024-09-21T22:07:30.641 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8700000000000001e-06 2024-09-21T22:07:30.641 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.641 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.641 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:30.641 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.945899+0000", 2024-09-21T22:07:30.641 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00085401900000000002 2024-09-21T22:07:30.641 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.642 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.642 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:30.642 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.945928+0000", 2024-09-21T22:07:30.642 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9193999999999999e-05 2024-09-21T22:07:30.642 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.642 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.642 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:30.642 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.945948+0000", 2024-09-21T22:07:30.642 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9876000000000001e-05 2024-09-21T22:07:30.642 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.642 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.643 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:30.643 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.946060+0000", 2024-09-21T22:07:30.643 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00011175099999999999 2024-09-21T22:07:30.643 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.643 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.643 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:30.643 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.946103+0000", 2024-09-21T22:07:30.643 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.3090999999999998e-05 2024-09-21T22:07:30.643 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.643 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.643 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:30.643 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.992151+0000", 2024-09-21T22:07:30.643 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.046047948999999998 2024-09-21T22:07:30.644 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.644 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.644 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:30.644 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.992206+0000", 2024-09-21T22:07:30.644 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.5269999999999998e-05 2024-09-21T22:07:30.644 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:30.644 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:30.644 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:30.644 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.644 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.644 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:687 2.3 e63/54 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head v 63'16, mlcod=63'14)", 2024-09-21T22:07:30.644 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.994473+0000", 2024-09-21T22:07:30.644 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 5.5793997180000003, 2024-09-21T22:07:30.644 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0054660990000000003, 2024-09-21T22:07:30.645 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:30.645 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:30.645 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:30.645 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:30.645 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.645 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:30.645 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.994473+0000", 2024-09-21T22:07:30.645 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:30.645 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.645 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.645 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:30.645 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.994473+0000", 2024-09-21T22:07:30.645 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:30.645 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.646 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.646 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:30.646 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.994474+0000", 2024-09-21T22:07:30.646 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.463e-06 2024-09-21T22:07:30.646 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.646 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.646 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:30.646 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.994546+0000", 2024-09-21T22:07:30.646 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.2260000000000003e-05 2024-09-21T22:07:30.646 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.646 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.646 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:30.647 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.994564+0000", 2024-09-21T22:07:30.647 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7983999999999999e-05 2024-09-21T22:07:30.647 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.647 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.647 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:30.647 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.994573+0000", 2024-09-21T22:07:30.647 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.4710000000000007e-06 2024-09-21T22:07:30.647 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.647 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.647 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:30.647 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.994634+0000", 2024-09-21T22:07:30.647 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.0872000000000003e-05 2024-09-21T22:07:30.648 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.648 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.648 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:30.648 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.994658+0000", 2024-09-21T22:07:30.648 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4573e-05 2024-09-21T22:07:30.648 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.648 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.648 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:30.648 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.999894+0000", 2024-09-21T22:07:30.648 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0052362390000000002 2024-09-21T22:07:30.648 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.648 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.649 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:30.649 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.999939+0000", 2024-09-21T22:07:30.649 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.4236999999999997e-05 2024-09-21T22:07:30.649 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:30.649 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:30.649 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:30.649 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.649 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.649 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:688 2.3 e63/54 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head v 63'17, mlcod=63'15)", 2024-09-21T22:07:30.649 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.001812+0000", 2024-09-21T22:07:30.649 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 5.5720606369999999, 2024-09-21T22:07:30.650 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.028785011999999999, 2024-09-21T22:07:30.650 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:30.650 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:30.650 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:30.650 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:30.650 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.650 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:30.650 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.001812+0000", 2024-09-21T22:07:30.650 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:30.650 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.650 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.650 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:30.651 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.001812+0000", 2024-09-21T22:07:30.651 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:30.651 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.651 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.651 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:30.651 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.001813+0000", 2024-09-21T22:07:30.651 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2699999999999999e-06 2024-09-21T22:07:30.651 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.651 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.651 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:30.651 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.002252+0000", 2024-09-21T22:07:30.652 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00043878599999999998 2024-09-21T22:07:30.652 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.652 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.652 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:30.652 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.002271+0000", 2024-09-21T22:07:30.652 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9346000000000001e-05 2024-09-21T22:07:30.652 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.652 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.652 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:30.652 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.002282+0000", 2024-09-21T22:07:30.652 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0951000000000001e-05 2024-09-21T22:07:30.652 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.653 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.653 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:30.653 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.002398+0000", 2024-09-21T22:07:30.653 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000116412 2024-09-21T22:07:30.653 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.653 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.653 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:30.653 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.002425+0000", 2024-09-21T22:07:30.653 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.6291999999999998e-05 2024-09-21T22:07:30.653 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.653 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.654 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:30.654 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.030528+0000", 2024-09-21T22:07:30.654 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.028103390999999998 2024-09-21T22:07:30.654 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.654 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.654 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:30.654 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.030597+0000", 2024-09-21T22:07:30.654 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.8564000000000003e-05 2024-09-21T22:07:30.654 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:30.654 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:30.654 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:30.654 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.655 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.655 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:689 2.3 e63/54 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head v 63'18, mlcod=63'16)", 2024-09-21T22:07:30.655 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.032891+0000", 2024-09-21T22:07:30.655 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 5.5409810549999996, 2024-09-21T22:07:30.655 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.005061032, 2024-09-21T22:07:30.655 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:30.655 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:30.655 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:30.655 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:30.655 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.656 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:30.656 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.032891+0000", 2024-09-21T22:07:30.656 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:30.656 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.656 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.656 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:30.656 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.032891+0000", 2024-09-21T22:07:30.656 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:30.656 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.656 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.656 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:30.656 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.032893+0000", 2024-09-21T22:07:30.657 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3030000000000001e-06 2024-09-21T22:07:30.657 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.657 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.657 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:30.657 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.032952+0000", 2024-09-21T22:07:30.657 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.9070000000000002e-05 2024-09-21T22:07:30.657 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.657 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.657 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:30.657 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.032969+0000", 2024-09-21T22:07:30.657 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7839e-05 2024-09-21T22:07:30.657 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.658 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.658 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:30.658 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.032979+0000", 2024-09-21T22:07:30.658 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.5370000000000003e-06 2024-09-21T22:07:30.658 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.658 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.658 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:30.658 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.033041+0000", 2024-09-21T22:07:30.658 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.1562000000000005e-05 2024-09-21T22:07:30.658 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.658 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.659 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:30.659 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.033068+0000", 2024-09-21T22:07:30.659 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.7226999999999999e-05 2024-09-21T22:07:30.659 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.659 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.659 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:30.659 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.037908+0000", 2024-09-21T22:07:30.659 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0048401770000000002 2024-09-21T22:07:30.659 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.659 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.659 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:30.659 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.037952+0000", 2024-09-21T22:07:30.660 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.4317000000000003e-05 2024-09-21T22:07:30.660 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:30.660 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:30.660 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:30.660 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.660 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.660 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:690 2.3 e63/54 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head v 63'19, mlcod=63'17)", 2024-09-21T22:07:30.660 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.039813+0000", 2024-09-21T22:07:30.660 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 5.534058838, 2024-09-21T22:07:30.660 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.035351891000000003, 2024-09-21T22:07:30.660 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:30.660 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:30.660 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:30.660 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:30.660 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.661 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:30.661 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.039813+0000", 2024-09-21T22:07:30.661 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:30.661 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.661 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.661 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:30.661 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.039813+0000", 2024-09-21T22:07:30.661 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:30.661 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.661 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.661 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:30.661 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.039814+0000", 2024-09-21T22:07:30.661 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.5300000000000002e-07 2024-09-21T22:07:30.661 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.662 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.662 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:30.662 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.040166+0000", 2024-09-21T22:07:30.662 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00035177 2024-09-21T22:07:30.662 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.662 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.662 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:30.662 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.040185+0000", 2024-09-21T22:07:30.662 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9000000000000001e-05 2024-09-21T22:07:30.662 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.662 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.662 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:30.662 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.040195+0000", 2024-09-21T22:07:30.662 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0254999999999999e-05 2024-09-21T22:07:30.663 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.663 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.663 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:30.663 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.040262+0000", 2024-09-21T22:07:30.663 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.6198000000000006e-05 2024-09-21T22:07:30.663 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.663 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.663 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:30.663 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.040288+0000", 2024-09-21T22:07:30.663 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5931999999999998e-05 2024-09-21T22:07:30.663 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.663 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.663 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:30.663 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.075106+0000", 2024-09-21T22:07:30.663 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.034818546999999998 2024-09-21T22:07:30.664 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.664 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.664 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:30.664 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.075165+0000", 2024-09-21T22:07:30.664 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.9236e-05 2024-09-21T22:07:30.664 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:30.664 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:30.664 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:30.664 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.664 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.664 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPull(2.3 e67/66 cost 1048576)", 2024-09-21T22:07:30.664 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:29.414957+0000", 2024-09-21T22:07:30.664 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 1.1589154589999999, 2024-09-21T22:07:30.664 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0032647269999999998, 2024-09-21T22:07:30.664 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:30.665 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:30.665 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-21T22:07:30.665 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:30.665 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.665 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:30.665 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:29.414957+0000", 2024-09-21T22:07:30.665 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:30.665 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.665 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.665 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:30.665 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:29.414957+0000", 2024-09-21T22:07:30.665 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:30.665 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.665 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.666 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:30.666 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:29.414959+0000", 2024-09-21T22:07:30.666 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.396e-06 2024-09-21T22:07:30.666 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.666 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.666 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:30.666 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:29.414986+0000", 2024-09-21T22:07:30.666 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.72e-05 2024-09-21T22:07:30.666 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.666 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.666 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:30.666 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:29.415018+0000", 2024-09-21T22:07:30.666 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.2079000000000002e-05 2024-09-21T22:07:30.666 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.667 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.667 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:30.667 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:29.415035+0000", 2024-09-21T22:07:30.667 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6316000000000001e-05 2024-09-21T22:07:30.667 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.667 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.667 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:30.667 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:29.415201+0000", 2024-09-21T22:07:30.667 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00016583900000000001 2024-09-21T22:07:30.667 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.667 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.667 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:30.667 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:29.418222+0000", 2024-09-21T22:07:30.667 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0030208969999999998 2024-09-21T22:07:30.667 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:30.668 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:30.668 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:30.668 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.668 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.668 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPull(2.3 e67/66 cost 1048576)", 2024-09-21T22:07:30.668 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:29.486922+0000", 2024-09-21T22:07:30.668 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 1.086950522, 2024-09-21T22:07:30.668 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0029880940000000002, 2024-09-21T22:07:30.668 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:30.668 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:30.668 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-21T22:07:30.668 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:30.668 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.668 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:30.669 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:29.486922+0000", 2024-09-21T22:07:30.669 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:30.669 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.669 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.669 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:30.669 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:29.486922+0000", 2024-09-21T22:07:30.669 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:30.669 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.669 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.669 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:30.669 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:29.486924+0000", 2024-09-21T22:07:30.669 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.897e-06 2024-09-21T22:07:30.669 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.670 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.670 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:30.670 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:29.486975+0000", 2024-09-21T22:07:30.670 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.1275e-05 2024-09-21T22:07:30.670 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.670 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.670 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:30.670 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:29.487002+0000", 2024-09-21T22:07:30.670 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.7469000000000001e-05 2024-09-21T22:07:30.670 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.670 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.670 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:30.671 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:29.487016+0000", 2024-09-21T22:07:30.671 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3587000000000001e-05 2024-09-21T22:07:30.671 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.671 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.671 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:30.671 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:29.487158+0000", 2024-09-21T22:07:30.671 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00014215899999999999 2024-09-21T22:07:30.671 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:30.671 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:30.671 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:30.671 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:29.489910+0000", 2024-09-21T22:07:30.671 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0027517069999999999 2024-09-21T22:07:30.671 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:30.672 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:30.672 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:30.672 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:30.672 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:30.672 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:07:30.672 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T22:07:30.674 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T22:07:30.677 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:30.676+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:30.777 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:30.772+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:30.801 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T22:07:30.807 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:07:30.808 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:07:30.808 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-09-21T22:07:30.808 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:07:30.813 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:07:30.813 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:07:30.813 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-09-21T22:07:30.813 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:07:30.824 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T22:07:30.879 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:30.874+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:30.955 INFO:teuthology.orchestra.run.smithi039.stdout:osd.0: {} 2024-09-21T22:07:30.956 INFO:teuthology.orchestra.run.smithi039.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T22:07:30.972 INFO:teuthology.orchestra.run.smithi039.stdout:osd.1: {} 2024-09-21T22:07:30.972 INFO:teuthology.orchestra.run.smithi039.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T22:07:30.980 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:30.976+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:30.988 INFO:teuthology.orchestra.run.smithi039.stdout:osd.2: {} 2024-09-21T22:07:30.989 INFO:teuthology.orchestra.run.smithi039.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T22:07:30.996 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:07:30.996 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:07:30.996 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-21T22:07:30.996 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-21T22:07:30.996 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:07:31.009 INFO:teuthology.orchestra.run.smithi039.stdout:osd.3: {} 2024-09-21T22:07:31.009 INFO:teuthology.orchestra.run.smithi039.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T22:07:31.011 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T22:07:31.030 INFO:teuthology.orchestra.run.smithi039.stdout:osd.4: {} 2024-09-21T22:07:31.030 INFO:teuthology.orchestra.run.smithi039.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T22:07:31.048 INFO:teuthology.orchestra.run.smithi039.stdout:osd.5: {} 2024-09-21T22:07:31.048 INFO:teuthology.orchestra.run.smithi039.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T22:07:31.108 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:31.106+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:31.182 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:07:31.176+0000 7f2075a06640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31720) UID: 0 2024-09-21T22:07:31.186 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:31.186 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:31.202 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T22:07:31.280 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-21T22:07:31.290 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:31.288+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:31.365 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:31.366 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:31.380 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T22:07:31.415 INFO:tasks.workunit.client.0.smithi157.stderr:++ rbd -p rbd --format xml du testrbdnbd21553 2024-09-21T22:07:31.416 INFO:tasks.workunit.client.0.smithi157.stderr:++ xmlstarlet sel -t -m //stats/images/image/provisioned_size -v . 2024-09-21T22:07:31.426 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:07:31.424+0000 7f2075a06640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31720) UID: 0 2024-09-21T22:07:31.477 INFO:tasks.workunit.client.0.smithi157.stderr:+ provisioned=67108864 2024-09-21T22:07:31.478 INFO:tasks.workunit.client.0.smithi157.stderr:++ rbd -p rbd --format xml du testrbdnbd21553 2024-09-21T22:07:31.478 INFO:tasks.workunit.client.0.smithi157.stderr:++ xmlstarlet sel -t -m //stats/images/image/used_size -v . 2024-09-21T22:07:31.483 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:31.478+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:31.543 INFO:tasks.workunit.client.0.smithi157.stderr:+ used=67108864 2024-09-21T22:07:31.543 INFO:tasks.workunit.client.0.smithi157.stderr:+ '[' 67108864 -eq 67108864 ']' 2024-09-21T22:07:31.543 INFO:tasks.workunit.client.0.smithi157.stderr:+ unmap_device /dev/nbd0 22066 2024-09-21T22:07:31.544 INFO:tasks.workunit.client.0.smithi157.stderr:+ local args=/dev/nbd0 2024-09-21T22:07:31.544 INFO:tasks.workunit.client.0.smithi157.stderr:+ local pid=22066 2024-09-21T22:07:31.544 INFO:tasks.workunit.client.0.smithi157.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd0 2024-09-21T22:07:31.544 INFO:tasks.workunit.client.0.smithi157.stderr:+ local cmd 2024-09-21T22:07:31.544 INFO:tasks.workunit.client.0.smithi157.stderr:++ id -u 2024-09-21T22:07:31.544 INFO:tasks.workunit.client.0.smithi157.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T22:07:31.544 INFO:tasks.workunit.client.0.smithi157.stderr:++ which rbd 2024-09-21T22:07:31.545 INFO:tasks.workunit.client.0.smithi157.stderr:+ cmd=/usr/bin/rbd 2024-09-21T22:07:31.545 INFO:tasks.workunit.client.0.smithi157.stderr:+ shift 2024-09-21T22:07:31.546 INFO:tasks.workunit.client.0.smithi157.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd0 2024-09-21T22:07:31.546 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:31.546 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:31.560 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T22:07:31.606 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:31.604+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:31.620 INFO:tasks.workunit.client.0.smithi157.stderr:+ rbd device --device-type nbd list 2024-09-21T22:07:31.620 INFO:tasks.workunit.client.0.smithi157.stderr:+ expect_false grep '^22066\b' 2024-09-21T22:07:31.620 INFO:tasks.workunit.client.0.smithi157.stderr:+ grep '^22066\b' 2024-09-21T22:07:31.667 INFO:tasks.workunit.client.0.smithi157.stderr:+ return 0 2024-09-21T22:07:31.668 INFO:tasks.workunit.client.0.smithi157.stderr:+ ps -C rbd-nbd 2024-09-21T22:07:31.668 INFO:tasks.workunit.client.0.smithi157.stderr:+ expect_false grep '^ *22066\b' 2024-09-21T22:07:31.668 INFO:tasks.workunit.client.0.smithi157.stderr:+ grep '^ *22066\b' 2024-09-21T22:07:31.674 INFO:tasks.workunit.client.0.smithi157.stderr:+ return 0 2024-09-21T22:07:31.674 INFO:tasks.workunit.client.0.smithi157.stderr:+ sleep 0.5 2024-09-21T22:07:31.678 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:31.678 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:31.685 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:31.682+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:31.691 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T22:07:31.786 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:31.782+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:31.858 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:31.858 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:31.873 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T22:07:31.887 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:31.882+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:31.988 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:31.986+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:31.997 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:31.997 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:32.012 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T22:07:32.089 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:32.086+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:32.139 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:32.139 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:32.153 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T22:07:32.176 INFO:tasks.workunit.client.0.smithi157.stderr:++ _sudo rbd device --device-type nbd map rbd/testrbdnbd21553 2024-09-21T22:07:32.176 INFO:tasks.workunit.client.0.smithi157.stderr:++ local cmd 2024-09-21T22:07:32.177 INFO:tasks.workunit.client.0.smithi157.stderr:+++ id -u 2024-09-21T22:07:32.178 INFO:tasks.workunit.client.0.smithi157.stderr:++ '[' 1000 -eq 0 ']' 2024-09-21T22:07:32.178 INFO:tasks.workunit.client.0.smithi157.stderr:+++ which rbd 2024-09-21T22:07:32.181 INFO:tasks.workunit.client.0.smithi157.stderr:++ cmd=/usr/bin/rbd 2024-09-21T22:07:32.181 INFO:tasks.workunit.client.0.smithi157.stderr:++ shift 2024-09-21T22:07:32.181 INFO:tasks.workunit.client.0.smithi157.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map rbd/testrbdnbd21553 2024-09-21T22:07:32.200 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:32.198+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:32.269 INFO:teuthology.orchestra.run.smithi039.stderr:noscrub is unset 2024-09-21T22:07:32.290 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:32.286+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:32.294 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:32.294 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:32.309 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T22:07:32.323 INFO:tasks.workunit.client.0.smithi157.stderr:+ DEV=/dev/nbd0 2024-09-21T22:07:32.323 INFO:tasks.workunit.client.0.smithi157.stderr:+ get_pid rbd 2024-09-21T22:07:32.323 INFO:tasks.workunit.client.0.smithi157.stderr:+ local pool=rbd 2024-09-21T22:07:32.323 INFO:tasks.workunit.client.0.smithi157.stderr:+ local ns= 2024-09-21T22:07:32.324 INFO:tasks.workunit.client.0.smithi157.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd21553'\''][device='\''/dev/nbd0'\'']/id' 2024-09-21T22:07:32.324 INFO:tasks.workunit.client.0.smithi157.stderr:++ rbd device --device-type nbd --format xml list 2024-09-21T22:07:32.390 INFO:tasks.workunit.client.0.smithi157.stderr:+ PID=22222 2024-09-21T22:07:32.390 INFO:tasks.workunit.client.0.smithi157.stderr:+ test -n 22222 2024-09-21T22:07:32.390 INFO:tasks.workunit.client.0.smithi157.stderr:+ ps -p 22222 -C rbd-nbd 2024-09-21T22:07:32.396 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:32.394+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:32.396 INFO:tasks.workunit.client.0.smithi157.stdout: PID TTY TIME CMD 2024-09-21T22:07:32.396 INFO:tasks.workunit.client.0.smithi157.stdout: 22222 ? 00:00:00 rbd-nbd 2024-09-21T22:07:32.397 INFO:tasks.workunit.client.0.smithi157.stderr:++ rbd -p rbd --format xml du testrbdnbd21553 2024-09-21T22:07:32.397 INFO:tasks.workunit.client.0.smithi157.stderr:++ xmlstarlet sel -t -m //stats/images/image/provisioned_size -v . 2024-09-21T22:07:32.455 INFO:tasks.workunit.client.0.smithi157.stderr:+ provisioned=67108864 2024-09-21T22:07:32.455 INFO:tasks.workunit.client.0.smithi157.stderr:++ rbd -p rbd --format xml du testrbdnbd21553 2024-09-21T22:07:32.456 INFO:tasks.workunit.client.0.smithi157.stderr:++ xmlstarlet sel -t -m //stats/images/image/used_size -v . 2024-09-21T22:07:32.472 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:32.473 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:32.488 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T22:07:32.491 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:32.488+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:32.512 INFO:tasks.workunit.client.0.smithi157.stderr:+ used=67108864 2024-09-21T22:07:32.512 INFO:tasks.workunit.client.0.smithi157.stderr:+ '[' 67108864 -eq 67108864 ']' 2024-09-21T22:07:32.512 INFO:tasks.workunit.client.0.smithi157.stderr:+ _sudo blkdiscard /dev/nbd0 2024-09-21T22:07:32.512 INFO:tasks.workunit.client.0.smithi157.stderr:+ local cmd 2024-09-21T22:07:32.513 INFO:tasks.workunit.client.0.smithi157.stderr:++ id -u 2024-09-21T22:07:32.513 INFO:tasks.workunit.client.0.smithi157.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T22:07:32.513 INFO:tasks.workunit.client.0.smithi157.stderr:++ which blkdiscard 2024-09-21T22:07:32.514 INFO:tasks.workunit.client.0.smithi157.stderr:+ cmd=/usr/sbin/blkdiscard 2024-09-21T22:07:32.515 INFO:tasks.workunit.client.0.smithi157.stderr:+ shift 2024-09-21T22:07:32.515 INFO:tasks.workunit.client.0.smithi157.stderr:+ sudo -nE /usr/sbin/blkdiscard /dev/nbd0 2024-09-21T22:07:32.592 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:32.588+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:32.624 INFO:tasks.workunit.client.0.smithi157.stderr:+ sync 2024-09-21T22:07:32.662 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:32.662 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:32.676 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T22:07:32.720 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:32.718+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:32.783 INFO:tasks.workunit.client.0.smithi157.stderr:++ rbd -p rbd --format xml du testrbdnbd21553 2024-09-21T22:07:32.783 INFO:tasks.workunit.client.0.smithi157.stderr:++ xmlstarlet sel -t -m //stats/images/image/provisioned_size -v . 2024-09-21T22:07:32.794 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:32.790+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:32.806 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:32.806 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:32.821 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T22:07:32.851 INFO:tasks.workunit.client.0.smithi157.stderr:+ provisioned=67108864 2024-09-21T22:07:32.852 INFO:tasks.workunit.client.0.smithi157.stderr:++ rbd -p rbd --format xml du testrbdnbd21553 2024-09-21T22:07:32.852 INFO:tasks.workunit.client.0.smithi157.stderr:++ xmlstarlet sel -t -m //stats/images/image/used_size -v . 2024-09-21T22:07:32.895 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:07:32.892+0000 7f2075a06640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31720) UID: 0 2024-09-21T22:07:32.918 INFO:tasks.workunit.client.0.smithi157.stderr:+ used=0 2024-09-21T22:07:32.918 INFO:tasks.workunit.client.0.smithi157.stderr:+ '[' 0 -lt 67108864 ']' 2024-09-21T22:07:32.918 INFO:tasks.workunit.client.0.smithi157.stderr:+ unmap_device /dev/nbd0 22222 2024-09-21T22:07:32.918 INFO:tasks.workunit.client.0.smithi157.stderr:+ local args=/dev/nbd0 2024-09-21T22:07:32.918 INFO:tasks.workunit.client.0.smithi157.stderr:+ local pid=22222 2024-09-21T22:07:32.918 INFO:tasks.workunit.client.0.smithi157.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd0 2024-09-21T22:07:32.918 INFO:tasks.workunit.client.0.smithi157.stderr:+ local cmd 2024-09-21T22:07:32.918 INFO:tasks.workunit.client.0.smithi157.stderr:++ id -u 2024-09-21T22:07:32.919 INFO:tasks.workunit.client.0.smithi157.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T22:07:32.919 INFO:tasks.workunit.client.0.smithi157.stderr:++ which rbd 2024-09-21T22:07:32.920 INFO:tasks.workunit.client.0.smithi157.stderr:+ cmd=/usr/bin/rbd 2024-09-21T22:07:32.920 INFO:tasks.workunit.client.0.smithi157.stderr:+ shift 2024-09-21T22:07:32.920 INFO:tasks.workunit.client.0.smithi157.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd0 2024-09-21T22:07:32.993 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:32.993 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:32.996 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:32.994+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:33.007 INFO:tasks.workunit.client.0.smithi157.stderr:+ rbd device --device-type nbd list 2024-09-21T22:07:33.007 INFO:tasks.workunit.client.0.smithi157.stderr:+ expect_false grep '^22222\b' 2024-09-21T22:07:33.007 INFO:tasks.workunit.client.0.smithi157.stderr:+ grep '^22222\b' 2024-09-21T22:07:33.008 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T22:07:33.054 INFO:tasks.workunit.client.0.smithi157.stderr:+ return 0 2024-09-21T22:07:33.055 INFO:tasks.workunit.client.0.smithi157.stderr:+ ps -C rbd-nbd 2024-09-21T22:07:33.055 INFO:tasks.workunit.client.0.smithi157.stderr:+ expect_false grep '^ *22222\b' 2024-09-21T22:07:33.056 INFO:tasks.workunit.client.0.smithi157.stderr:+ grep '^ *22222\b' 2024-09-21T22:07:33.061 INFO:tasks.workunit.client.0.smithi157.stderr:+ return 0 2024-09-21T22:07:33.061 INFO:tasks.workunit.client.0.smithi157.stderr:+ sleep 0.5 2024-09-21T22:07:33.067 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T22:07:33.096 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:33.096+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:33.178 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:33.178 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:33.194 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T22:07:33.197 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:07:33.192+0000 7f2075a06640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31720) UID: 0 2024-09-21T22:07:33.235 INFO:teuthology.orchestra.run.smithi039.stdout:osd.0: {} 2024-09-21T22:07:33.236 INFO:teuthology.orchestra.run.smithi039.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T22:07:33.252 INFO:teuthology.orchestra.run.smithi039.stdout:osd.1: {} 2024-09-21T22:07:33.252 INFO:teuthology.orchestra.run.smithi039.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T22:07:33.270 INFO:teuthology.orchestra.run.smithi039.stdout:osd.2: {} 2024-09-21T22:07:33.271 INFO:teuthology.orchestra.run.smithi039.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T22:07:33.291 INFO:teuthology.orchestra.run.smithi039.stdout:osd.3: {} 2024-09-21T22:07:33.292 INFO:teuthology.orchestra.run.smithi039.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T22:07:33.298 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:33.294+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:33.313 INFO:teuthology.orchestra.run.smithi039.stdout:osd.4: {} 2024-09-21T22:07:33.314 INFO:teuthology.orchestra.run.smithi039.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T22:07:33.315 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:07:33.315 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:07:33.315 INFO:teuthology.orchestra.run.smithi039.stdout: "num_ops": 0 2024-09-21T22:07:33.316 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:07:33.331 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T22:07:33.335 INFO:teuthology.orchestra.run.smithi039.stdout:osd.5: {} 2024-09-21T22:07:33.335 INFO:teuthology.orchestra.run.smithi039.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T22:07:33.400 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:33.398+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:33.468 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:07:33.468 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:07:33.469 INFO:teuthology.orchestra.run.smithi039.stdout: "complaint_time": 30, 2024-09-21T22:07:33.469 INFO:teuthology.orchestra.run.smithi039.stdout: "num_blocked_ops": 0 2024-09-21T22:07:33.469 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:07:33.483 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T22:07:33.524 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:33.522+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:33.563 INFO:tasks.workunit.client.0.smithi157.stderr:++ _sudo rbd device -t nbd -o try-netlink map rbd/testrbdnbd21553 2024-09-21T22:07:33.563 INFO:tasks.workunit.client.0.smithi157.stderr:++ local cmd 2024-09-21T22:07:33.563 INFO:tasks.workunit.client.0.smithi157.stderr:+++ id -u 2024-09-21T22:07:33.564 INFO:tasks.workunit.client.0.smithi157.stderr:++ '[' 1000 -eq 0 ']' 2024-09-21T22:07:33.565 INFO:tasks.workunit.client.0.smithi157.stderr:+++ which rbd 2024-09-21T22:07:33.567 INFO:tasks.workunit.client.0.smithi157.stderr:++ cmd=/usr/bin/rbd 2024-09-21T22:07:33.567 INFO:tasks.workunit.client.0.smithi157.stderr:++ shift 2024-09-21T22:07:33.567 INFO:tasks.workunit.client.0.smithi157.stderr:++ sudo -nE /usr/bin/rbd device -t nbd -o try-netlink map rbd/testrbdnbd21553 2024-09-21T22:07:33.601 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:33.596+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:33.653 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:07:33.653 INFO:teuthology.orchestra.run.smithi039.stdout: "size": 20, 2024-09-21T22:07:33.653 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 600, 2024-09-21T22:07:33.653 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [ 2024-09-21T22:07:33.653 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.653 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPull(2.3 e55/54 cost 1048576)", 2024-09-21T22:07:33.653 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.081116+0000", 2024-09-21T22:07:33.653 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 21.569147010999998, 2024-09-21T22:07:33.653 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0030214159999999999, 2024-09-21T22:07:33.654 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:33.654 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:33.654 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-21T22:07:33.654 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:33.654 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.654 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:33.654 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.081116+0000", 2024-09-21T22:07:33.654 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:33.654 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.654 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.654 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:33.654 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.081116+0000", 2024-09-21T22:07:33.655 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:33.655 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.655 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.655 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:33.655 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.081118+0000", 2024-09-21T22:07:33.655 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.545e-06 2024-09-21T22:07:33.655 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.655 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.655 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:33.655 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.081133+0000", 2024-09-21T22:07:33.655 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5642e-05 2024-09-21T22:07:33.655 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.655 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.655 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:33.656 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.081155+0000", 2024-09-21T22:07:33.656 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1858e-05 2024-09-21T22:07:33.656 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.656 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.656 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:33.656 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.081171+0000", 2024-09-21T22:07:33.656 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5804e-05 2024-09-21T22:07:33.656 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.656 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.656 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:33.656 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.081291+0000", 2024-09-21T22:07:33.656 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000120553 2024-09-21T22:07:33.656 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.656 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.657 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:33.657 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.084137+0000", 2024-09-21T22:07:33.657 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.002846014 2024-09-21T22:07:33.657 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:33.657 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:33.657 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:33.657 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.657 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.657 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPull(2.3 e55/54 cost 1048576)", 2024-09-21T22:07:33.657 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.136357+0000", 2024-09-21T22:07:33.657 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 21.513905945000001, 2024-09-21T22:07:33.657 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.002843904, 2024-09-21T22:07:33.657 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:33.657 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:33.657 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-21T22:07:33.658 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:33.658 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.658 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:33.658 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.136357+0000", 2024-09-21T22:07:33.658 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:33.658 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.658 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.658 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:33.658 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.136357+0000", 2024-09-21T22:07:33.658 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:33.658 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.658 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.658 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:33.659 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.136359+0000", 2024-09-21T22:07:33.659 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.0669999999999999e-06 2024-09-21T22:07:33.659 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.659 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.659 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:33.659 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.136381+0000", 2024-09-21T22:07:33.659 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.2104999999999999e-05 2024-09-21T22:07:33.659 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.659 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.659 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:33.659 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.136412+0000", 2024-09-21T22:07:33.659 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.0602999999999998e-05 2024-09-21T22:07:33.659 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.659 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.659 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:33.660 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.136427+0000", 2024-09-21T22:07:33.660 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4832e-05 2024-09-21T22:07:33.660 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.660 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.660 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:33.660 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.136569+0000", 2024-09-21T22:07:33.660 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000142702 2024-09-21T22:07:33.660 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.660 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.660 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:33.660 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.139201+0000", 2024-09-21T22:07:33.660 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.002631595 2024-09-21T22:07:33.660 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:33.660 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:33.661 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:33.661 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.661 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.661 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:667 2.3 e63/54 2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head v 63'4, mlcod=0'0)", 2024-09-21T22:07:33.661 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.402454+0000", 2024-09-21T22:07:33.661 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 9.247809427, 2024-09-21T22:07:33.661 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0058375959999999996, 2024-09-21T22:07:33.661 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:33.661 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:33.661 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:33.661 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:33.661 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.661 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:33.661 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.402454+0000", 2024-09-21T22:07:33.662 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:33.662 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.662 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.662 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:33.662 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.402454+0000", 2024-09-21T22:07:33.662 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:33.662 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.662 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.662 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:33.662 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.402456+0000", 2024-09-21T22:07:33.662 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1179999999999999e-06 2024-09-21T22:07:33.662 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.663 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.663 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:33.663 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.402618+0000", 2024-09-21T22:07:33.663 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00016243 2024-09-21T22:07:33.663 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.663 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.663 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:33.663 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.402643+0000", 2024-09-21T22:07:33.663 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4638999999999999e-05 2024-09-21T22:07:33.663 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.663 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.663 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:33.663 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.402658+0000", 2024-09-21T22:07:33.663 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5211000000000001e-05 2024-09-21T22:07:33.664 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.664 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.664 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:33.664 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.402756+0000", 2024-09-21T22:07:33.664 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.7880000000000005e-05 2024-09-21T22:07:33.664 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.664 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.664 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:33.664 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.402779+0000", 2024-09-21T22:07:33.664 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3524999999999999e-05 2024-09-21T22:07:33.664 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.664 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.665 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:33.665 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.408233+0000", 2024-09-21T22:07:33.665 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0054534700000000002 2024-09-21T22:07:33.665 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.665 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.665 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:33.665 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.408291+0000", 2024-09-21T22:07:33.665 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.8322999999999997e-05 2024-09-21T22:07:33.665 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:33.665 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:33.665 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:33.665 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.665 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.665 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:668 2.3 e63/54 2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head v 63'5, mlcod=52'3)", 2024-09-21T22:07:33.665 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.410724+0000", 2024-09-21T22:07:33.666 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 9.2395390099999997, 2024-09-21T22:07:33.666 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.031657809000000002, 2024-09-21T22:07:33.666 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:33.666 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:33.666 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:33.666 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:33.666 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.666 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:33.666 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.410724+0000", 2024-09-21T22:07:33.666 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:33.666 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.666 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.666 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:33.666 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.410724+0000", 2024-09-21T22:07:33.667 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:33.667 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.667 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.667 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:33.667 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.410725+0000", 2024-09-21T22:07:33.667 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.4300000000000001e-07 2024-09-21T22:07:33.667 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.667 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.667 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:33.667 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.411436+0000", 2024-09-21T22:07:33.667 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00071118600000000004 2024-09-21T22:07:33.667 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.667 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.667 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:33.667 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.411455+0000", 2024-09-21T22:07:33.668 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8357999999999999e-05 2024-09-21T22:07:33.668 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.668 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.668 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:33.668 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.411470+0000", 2024-09-21T22:07:33.668 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5753999999999999e-05 2024-09-21T22:07:33.668 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.668 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.668 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:33.668 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.411537+0000", 2024-09-21T22:07:33.668 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.6766000000000004e-05 2024-09-21T22:07:33.668 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.668 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.668 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:33.669 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.411571+0000", 2024-09-21T22:07:33.669 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.3555e-05 2024-09-21T22:07:33.669 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.669 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.669 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:33.669 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.442317+0000", 2024-09-21T22:07:33.669 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.030746393 2024-09-21T22:07:33.669 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.669 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.669 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:33.669 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.442382+0000", 2024-09-21T22:07:33.669 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.4853999999999999e-05 2024-09-21T22:07:33.669 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:33.669 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:33.670 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:33.670 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.670 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.670 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:669 2.3 e63/54 2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head v 63'6, mlcod=63'4)", 2024-09-21T22:07:33.670 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.459924+0000", 2024-09-21T22:07:33.670 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 9.1903394519999999, 2024-09-21T22:07:33.670 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0062657900000000002, 2024-09-21T22:07:33.670 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:33.670 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:33.670 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:33.670 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:33.670 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.670 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:33.670 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.459924+0000", 2024-09-21T22:07:33.671 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:33.671 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.671 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.671 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:33.671 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.459924+0000", 2024-09-21T22:07:33.671 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:33.671 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.671 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.671 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:33.671 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.459925+0000", 2024-09-21T22:07:33.671 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8169999999999999e-06 2024-09-21T22:07:33.671 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.671 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.671 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:33.671 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.459983+0000", 2024-09-21T22:07:33.672 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.7567e-05 2024-09-21T22:07:33.672 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.672 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.672 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:33.672 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.460011+0000", 2024-09-21T22:07:33.672 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8455000000000001e-05 2024-09-21T22:07:33.672 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.672 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.672 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:33.672 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.460021+0000", 2024-09-21T22:07:33.672 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.9410000000000002e-06 2024-09-21T22:07:33.672 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.672 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.672 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:33.672 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.460093+0000", 2024-09-21T22:07:33.673 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.1231999999999998e-05 2024-09-21T22:07:33.673 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.673 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.673 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:33.673 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.460124+0000", 2024-09-21T22:07:33.673 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.1335999999999999e-05 2024-09-21T22:07:33.673 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.673 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.673 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:33.673 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.466130+0000", 2024-09-21T22:07:33.673 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0060055519999999999 2024-09-21T22:07:33.673 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.673 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.673 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:33.674 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.466189+0000", 2024-09-21T22:07:33.674 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.9889999999999997e-05 2024-09-21T22:07:33.674 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:33.674 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:33.674 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:33.674 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.674 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.674 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:670 2.3 e63/54 2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head v 63'7, mlcod=63'5)", 2024-09-21T22:07:33.674 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.468764+0000", 2024-09-21T22:07:33.674 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 9.1814994710000004, 2024-09-21T22:07:33.674 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.037963433999999997, 2024-09-21T22:07:33.674 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:33.674 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:33.674 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:33.674 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:33.675 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.675 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:33.675 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.468764+0000", 2024-09-21T22:07:33.675 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:33.675 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.675 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.675 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:33.675 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.468764+0000", 2024-09-21T22:07:33.675 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:33.675 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.675 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.675 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:33.675 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.468765+0000", 2024-09-21T22:07:33.675 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.9699999999999994e-07 2024-09-21T22:07:33.676 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.676 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.676 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:33.676 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.469295+0000", 2024-09-21T22:07:33.676 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00053003600000000001 2024-09-21T22:07:33.676 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.676 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.676 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:33.676 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.469311+0000", 2024-09-21T22:07:33.676 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6841999999999999e-05 2024-09-21T22:07:33.676 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.676 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.676 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:33.676 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.469324+0000", 2024-09-21T22:07:33.676 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2373e-05 2024-09-21T22:07:33.677 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.677 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.677 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:33.677 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.469391+0000", 2024-09-21T22:07:33.677 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.6880999999999999e-05 2024-09-21T22:07:33.677 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.677 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.677 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:33.677 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.469419+0000", 2024-09-21T22:07:33.677 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8742999999999999e-05 2024-09-21T22:07:33.677 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.677 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.677 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:33.677 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.506662+0000", 2024-09-21T22:07:33.677 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.037242994000000001 2024-09-21T22:07:33.678 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.678 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.678 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:33.678 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.506727+0000", 2024-09-21T22:07:33.678 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.4567999999999996e-05 2024-09-21T22:07:33.678 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:33.678 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:33.678 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:33.678 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.678 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.678 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:671 2.3 e63/54 2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head v 63'8, mlcod=63'6)", 2024-09-21T22:07:33.678 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.519938+0000", 2024-09-21T22:07:33.678 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 9.1303249320000006, 2024-09-21T22:07:33.678 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0058323419999999999, 2024-09-21T22:07:33.679 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:33.679 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:33.679 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:33.679 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:33.679 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.679 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:33.679 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.519938+0000", 2024-09-21T22:07:33.679 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:33.679 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.679 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.679 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:33.680 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.519938+0000", 2024-09-21T22:07:33.680 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:33.680 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.680 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.680 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:33.680 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.519940+0000", 2024-09-21T22:07:33.680 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.666e-06 2024-09-21T22:07:33.680 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.680 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.680 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:33.680 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.520031+0000", 2024-09-21T22:07:33.680 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.1051999999999995e-05 2024-09-21T22:07:33.680 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.680 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.681 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:33.681 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.520056+0000", 2024-09-21T22:07:33.681 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5459000000000001e-05 2024-09-21T22:07:33.681 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.681 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.681 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:33.681 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.520072+0000", 2024-09-21T22:07:33.681 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5801e-05 2024-09-21T22:07:33.681 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.681 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.681 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:33.681 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.520178+0000", 2024-09-21T22:07:33.681 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000105388 2024-09-21T22:07:33.681 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.682 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.682 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:33.682 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.520204+0000", 2024-09-21T22:07:33.682 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.6208e-05 2024-09-21T22:07:33.682 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.682 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.682 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:33.682 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.525735+0000", 2024-09-21T22:07:33.682 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0055308520000000002 2024-09-21T22:07:33.682 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.682 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.682 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:33.682 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.525770+0000", 2024-09-21T22:07:33.682 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.5916e-05 2024-09-21T22:07:33.682 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:33.683 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:33.683 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:33.683 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.683 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.683 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:672 2.3 e63/54 2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head v 63'9, mlcod=63'7)", 2024-09-21T22:07:33.683 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.528432+0000", 2024-09-21T22:07:33.683 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 9.1218315560000001, 2024-09-21T22:07:33.683 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.029763412999999999, 2024-09-21T22:07:33.683 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:33.683 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:33.683 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:33.683 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:33.683 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.683 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:33.684 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.528432+0000", 2024-09-21T22:07:33.684 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:33.684 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.684 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.684 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:33.684 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.528432+0000", 2024-09-21T22:07:33.684 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:33.684 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.684 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.684 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:33.684 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.528433+0000", 2024-09-21T22:07:33.684 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7179999999999999e-06 2024-09-21T22:07:33.684 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.684 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.684 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:33.685 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.528926+0000", 2024-09-21T22:07:33.685 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00049238500000000002 2024-09-21T22:07:33.685 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.685 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.685 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:33.685 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.528955+0000", 2024-09-21T22:07:33.685 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9037e-05 2024-09-21T22:07:33.685 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.685 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.685 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:33.685 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.528971+0000", 2024-09-21T22:07:33.685 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6622000000000001e-05 2024-09-21T22:07:33.685 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.685 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.686 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:33.686 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.529047+0000", 2024-09-21T22:07:33.686 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.5246000000000004e-05 2024-09-21T22:07:33.686 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.686 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.686 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:33.686 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.529074+0000", 2024-09-21T22:07:33.686 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.7532e-05 2024-09-21T22:07:33.686 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.686 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.686 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:33.686 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.558149+0000", 2024-09-21T22:07:33.686 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.029075192999999999 2024-09-21T22:07:33.686 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.686 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.687 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:33.687 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.558195+0000", 2024-09-21T22:07:33.687 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.5680000000000003e-05 2024-09-21T22:07:33.687 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:33.687 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:33.687 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:33.687 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.687 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.687 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:673 2.3 e63/54 2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head v 63'10, mlcod=63'8)", 2024-09-21T22:07:33.687 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.576513+0000", 2024-09-21T22:07:33.687 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 9.0737501209999998, 2024-09-21T22:07:33.687 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.003674666, 2024-09-21T22:07:33.687 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:33.687 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:33.688 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:33.688 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:33.688 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.688 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:33.688 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.576513+0000", 2024-09-21T22:07:33.688 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:33.688 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.688 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.688 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:33.688 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.576513+0000", 2024-09-21T22:07:33.688 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:33.688 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.688 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.688 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:33.689 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.576514+0000", 2024-09-21T22:07:33.689 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.1299999999999998e-07 2024-09-21T22:07:33.689 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.689 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.689 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:33.689 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.576567+0000", 2024-09-21T22:07:33.689 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.2710000000000002e-05 2024-09-21T22:07:33.689 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.689 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.689 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:33.689 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.576579+0000", 2024-09-21T22:07:33.689 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2035000000000001e-05 2024-09-21T22:07:33.689 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.689 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.689 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:33.690 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.576585+0000", 2024-09-21T22:07:33.690 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.6050000000000003e-06 2024-09-21T22:07:33.690 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.690 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.690 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:33.690 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.576638+0000", 2024-09-21T22:07:33.690 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.2726999999999998e-05 2024-09-21T22:07:33.690 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.690 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.690 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:33.690 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.576656+0000", 2024-09-21T22:07:33.690 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8084000000000002e-05 2024-09-21T22:07:33.690 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.690 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.691 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:33.691 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.580156+0000", 2024-09-21T22:07:33.691 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0035002700000000002 2024-09-21T22:07:33.691 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.691 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.691 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:33.691 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.580188+0000", 2024-09-21T22:07:33.691 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.1322000000000003e-05 2024-09-21T22:07:33.691 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:33.691 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:33.691 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:33.691 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.691 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.691 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:674 2.3 e63/54 2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head v 63'11, mlcod=63'9)", 2024-09-21T22:07:33.691 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.584683+0000", 2024-09-21T22:07:33.692 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 9.0655796800000008, 2024-09-21T22:07:33.692 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.028028352999999999, 2024-09-21T22:07:33.692 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:33.692 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:33.692 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:33.692 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:33.692 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.692 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:33.692 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.584683+0000", 2024-09-21T22:07:33.692 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:33.692 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.693 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.693 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:33.693 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.584683+0000", 2024-09-21T22:07:33.693 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:33.693 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.693 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.693 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:33.693 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.584684+0000", 2024-09-21T22:07:33.693 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.4600000000000004e-07 2024-09-21T22:07:33.693 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.693 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.693 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:33.693 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.585041+0000", 2024-09-21T22:07:33.693 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000356752 2024-09-21T22:07:33.693 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.694 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.694 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:33.694 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.585054+0000", 2024-09-21T22:07:33.694 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3387000000000001e-05 2024-09-21T22:07:33.694 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.694 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.694 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:33.694 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.585065+0000", 2024-09-21T22:07:33.694 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.1124999999999999e-05 2024-09-21T22:07:33.694 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.694 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.694 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:33.695 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.585142+0000", 2024-09-21T22:07:33.695 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.6743999999999996e-05 2024-09-21T22:07:33.695 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.695 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.695 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:33.695 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.585170+0000", 2024-09-21T22:07:33.695 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.7569999999999999e-05 2024-09-21T22:07:33.695 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.695 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.695 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:33.695 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.612637+0000", 2024-09-21T22:07:33.695 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.027467803999999998 2024-09-21T22:07:33.695 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.695 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.696 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:33.696 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.612712+0000", 2024-09-21T22:07:33.696 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.4325000000000004e-05 2024-09-21T22:07:33.696 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:33.696 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:33.696 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:33.696 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.696 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.696 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:683 2.3 e63/54 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head v 63'12, mlcod=63'10)", 2024-09-21T22:07:33.696 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.888242+0000", 2024-09-21T22:07:33.696 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 8.7620210620000005, 2024-09-21T22:07:33.696 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0085158290000000008, 2024-09-21T22:07:33.696 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:33.696 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:33.697 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:33.697 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:33.697 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.697 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:33.697 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.888242+0000", 2024-09-21T22:07:33.697 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:33.697 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.697 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.697 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:33.697 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.888242+0000", 2024-09-21T22:07:33.697 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:33.697 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.697 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.697 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:33.697 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.888244+0000", 2024-09-21T22:07:33.698 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.553e-06 2024-09-21T22:07:33.698 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.698 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.698 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:33.698 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.888295+0000", 2024-09-21T22:07:33.698 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.1916000000000002e-05 2024-09-21T22:07:33.698 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.698 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.698 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:33.698 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.888319+0000", 2024-09-21T22:07:33.698 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3139999999999999e-05 2024-09-21T22:07:33.698 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.698 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.698 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:33.699 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.888332+0000", 2024-09-21T22:07:33.699 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2993000000000001e-05 2024-09-21T22:07:33.699 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.699 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.699 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:33.699 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.888441+0000", 2024-09-21T22:07:33.699 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000109039 2024-09-21T22:07:33.699 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.699 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.699 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:33.699 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.888481+0000", 2024-09-21T22:07:33.699 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.9950000000000002e-05 2024-09-21T22:07:33.699 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.699 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.699 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:33.700 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.896688+0000", 2024-09-21T22:07:33.700 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0082078729999999992 2024-09-21T22:07:33.700 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.700 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.700 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:33.700 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.896758+0000", 2024-09-21T22:07:33.700 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.9364999999999997e-05 2024-09-21T22:07:33.700 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:33.700 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:33.700 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:33.700 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.700 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.700 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:684 2.3 e63/54 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head v 63'13, mlcod=63'11)", 2024-09-21T22:07:33.700 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.899243+0000", 2024-09-21T22:07:33.701 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 8.7510202849999992, 2024-09-21T22:07:33.701 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.033436192000000003, 2024-09-21T22:07:33.701 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:33.701 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:33.701 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:33.701 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:33.701 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.701 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:33.701 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.899243+0000", 2024-09-21T22:07:33.701 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:33.701 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.701 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.701 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:33.701 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.899243+0000", 2024-09-21T22:07:33.701 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:33.702 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.702 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.702 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:33.702 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.899244+0000", 2024-09-21T22:07:33.702 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.705e-06 2024-09-21T22:07:33.702 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.702 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.702 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:33.702 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.899604+0000", 2024-09-21T22:07:33.702 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00035930599999999999 2024-09-21T22:07:33.702 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.702 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.702 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:33.702 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.899622+0000", 2024-09-21T22:07:33.703 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8487000000000001e-05 2024-09-21T22:07:33.703 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.703 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.703 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:33.703 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.899634+0000", 2024-09-21T22:07:33.703 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2116999999999999e-05 2024-09-21T22:07:33.703 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.703 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.703 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:33.703 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.899721+0000", 2024-09-21T22:07:33.703 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.6718999999999994e-05 2024-09-21T22:07:33.703 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.703 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.703 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:33.704 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.899755+0000", 2024-09-21T22:07:33.704 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.4292000000000003e-05 2024-09-21T22:07:33.704 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.704 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.704 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:33.704 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.932545+0000", 2024-09-21T22:07:33.704 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.03278959 2024-09-21T22:07:33.704 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.704 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.704 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:33.704 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.932679+0000", 2024-09-21T22:07:33.704 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013397600000000001 2024-09-21T22:07:33.704 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:33.704 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:33.704 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:33.705 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.705 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.705 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:685 2.3 e63/54 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head v 63'14, mlcod=63'12)", 2024-09-21T22:07:33.705 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.935246+0000", 2024-09-21T22:07:33.705 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 8.7150174479999993, 2024-09-21T22:07:33.705 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0073590590000000003, 2024-09-21T22:07:33.705 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:33.705 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:33.705 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:33.705 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:33.705 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.705 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:33.705 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.935246+0000", 2024-09-21T22:07:33.705 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:33.706 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.706 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.706 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:33.706 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.935246+0000", 2024-09-21T22:07:33.706 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:33.706 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.706 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.706 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:33.706 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.935247+0000", 2024-09-21T22:07:33.706 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4470000000000001e-06 2024-09-21T22:07:33.706 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.706 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.706 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:33.706 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.935295+0000", 2024-09-21T22:07:33.706 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.7963000000000001e-05 2024-09-21T22:07:33.707 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.707 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.707 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:33.707 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.935313+0000", 2024-09-21T22:07:33.707 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8317000000000001e-05 2024-09-21T22:07:33.707 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.707 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.707 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:33.707 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.935352+0000", 2024-09-21T22:07:33.707 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.8552e-05 2024-09-21T22:07:33.707 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.707 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.707 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:33.707 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.935438+0000", 2024-09-21T22:07:33.707 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.6405999999999999e-05 2024-09-21T22:07:33.708 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.708 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.708 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:33.708 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.935471+0000", 2024-09-21T22:07:33.709 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.3142999999999998e-05 2024-09-21T22:07:33.709 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.709 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.710 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:33.710 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.942552+0000", 2024-09-21T22:07:33.710 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0070809510000000003 2024-09-21T22:07:33.710 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.710 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.710 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:33.710 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.942605+0000", 2024-09-21T22:07:33.710 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.2280000000000001e-05 2024-09-21T22:07:33.710 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:33.710 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:33.710 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:33.710 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.710 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.710 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:686 2.3 e63/54 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head v 63'15, mlcod=63'13)", 2024-09-21T22:07:33.711 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.945043+0000", 2024-09-21T22:07:33.711 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 8.7052201349999994, 2024-09-21T22:07:33.711 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.04716302, 2024-09-21T22:07:33.711 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:33.711 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:33.711 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:33.711 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:33.711 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.711 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:33.711 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.945043+0000", 2024-09-21T22:07:33.711 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:33.711 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.711 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.711 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:33.712 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.945043+0000", 2024-09-21T22:07:33.712 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:33.712 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.712 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.712 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:33.712 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.945045+0000", 2024-09-21T22:07:33.712 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8700000000000001e-06 2024-09-21T22:07:33.712 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.712 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.712 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:33.712 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.945899+0000", 2024-09-21T22:07:33.712 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00085401900000000002 2024-09-21T22:07:33.712 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.712 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.713 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:33.713 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.945928+0000", 2024-09-21T22:07:33.713 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9193999999999999e-05 2024-09-21T22:07:33.713 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.713 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.713 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:33.713 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.945948+0000", 2024-09-21T22:07:33.713 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9876000000000001e-05 2024-09-21T22:07:33.713 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.713 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.713 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:33.713 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.946060+0000", 2024-09-21T22:07:33.713 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00011175099999999999 2024-09-21T22:07:33.713 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.713 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.714 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:33.714 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.946103+0000", 2024-09-21T22:07:33.714 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.3090999999999998e-05 2024-09-21T22:07:33.714 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.714 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.714 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:33.714 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.992151+0000", 2024-09-21T22:07:33.714 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.046047948999999998 2024-09-21T22:07:33.714 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.714 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.714 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:33.714 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.992206+0000", 2024-09-21T22:07:33.714 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.5269999999999998e-05 2024-09-21T22:07:33.714 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:33.715 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:33.715 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:33.715 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.715 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.715 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:687 2.3 e63/54 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head v 63'16, mlcod=63'14)", 2024-09-21T22:07:33.715 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.994473+0000", 2024-09-21T22:07:33.715 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 8.6557905119999994, 2024-09-21T22:07:33.715 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0054660990000000003, 2024-09-21T22:07:33.715 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:33.715 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:33.715 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:33.715 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:33.715 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.715 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:33.716 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.994473+0000", 2024-09-21T22:07:33.716 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:33.716 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.716 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.716 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:33.716 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.994473+0000", 2024-09-21T22:07:33.716 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:33.716 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.716 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.716 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:33.716 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.994474+0000", 2024-09-21T22:07:33.717 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.463e-06 2024-09-21T22:07:33.717 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.717 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.717 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:33.717 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.994546+0000", 2024-09-21T22:07:33.717 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.2260000000000003e-05 2024-09-21T22:07:33.717 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.717 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.717 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:33.717 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.994564+0000", 2024-09-21T22:07:33.717 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7983999999999999e-05 2024-09-21T22:07:33.717 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.717 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.717 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:33.718 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.994573+0000", 2024-09-21T22:07:33.718 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.4710000000000007e-06 2024-09-21T22:07:33.718 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.718 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.718 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:33.718 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.994634+0000", 2024-09-21T22:07:33.718 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.0872000000000003e-05 2024-09-21T22:07:33.718 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.718 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.718 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:33.718 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.994658+0000", 2024-09-21T22:07:33.718 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4573e-05 2024-09-21T22:07:33.718 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.718 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.718 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:33.719 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.999894+0000", 2024-09-21T22:07:33.719 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0052362390000000002 2024-09-21T22:07:33.719 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.719 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.719 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:33.719 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.999939+0000", 2024-09-21T22:07:33.719 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.4236999999999997e-05 2024-09-21T22:07:33.719 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:33.719 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:33.719 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:33.719 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.719 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.719 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:688 2.3 e63/54 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head v 63'17, mlcod=63'15)", 2024-09-21T22:07:33.719 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.001812+0000", 2024-09-21T22:07:33.720 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 8.6484514309999998, 2024-09-21T22:07:33.720 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.028785011999999999, 2024-09-21T22:07:33.720 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:33.720 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:33.720 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:33.720 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:33.720 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.720 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:33.720 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.001812+0000", 2024-09-21T22:07:33.720 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:33.720 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.720 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.721 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:33.721 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.001812+0000", 2024-09-21T22:07:33.721 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:33.721 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.721 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.721 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:33.721 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.001813+0000", 2024-09-21T22:07:33.721 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2699999999999999e-06 2024-09-21T22:07:33.721 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.721 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.721 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:33.721 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.002252+0000", 2024-09-21T22:07:33.721 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00043878599999999998 2024-09-21T22:07:33.722 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.722 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.722 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:33.722 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.002271+0000", 2024-09-21T22:07:33.722 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9346000000000001e-05 2024-09-21T22:07:33.722 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.722 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.722 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:33.722 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.002282+0000", 2024-09-21T22:07:33.722 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0951000000000001e-05 2024-09-21T22:07:33.722 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.722 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.722 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:33.722 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.002398+0000", 2024-09-21T22:07:33.723 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000116412 2024-09-21T22:07:33.723 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.723 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.723 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:33.723 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.002425+0000", 2024-09-21T22:07:33.723 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.6291999999999998e-05 2024-09-21T22:07:33.723 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.723 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.723 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:33.723 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.030528+0000", 2024-09-21T22:07:33.723 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.028103390999999998 2024-09-21T22:07:33.724 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.724 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.724 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:33.724 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.030597+0000", 2024-09-21T22:07:33.724 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.8564000000000003e-05 2024-09-21T22:07:33.724 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:33.724 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:33.724 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:33.724 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.724 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.724 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:689 2.3 e63/54 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head v 63'18, mlcod=63'16)", 2024-09-21T22:07:33.724 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.032891+0000", 2024-09-21T22:07:33.725 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 8.6173718489999995, 2024-09-21T22:07:33.725 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.005061032, 2024-09-21T22:07:33.725 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:33.725 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:33.725 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:33.725 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:33.725 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.725 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:33.725 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.032891+0000", 2024-09-21T22:07:33.725 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:33.725 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.725 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.725 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:33.725 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.032891+0000", 2024-09-21T22:07:33.726 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:33.726 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.726 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.726 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:33.726 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.032893+0000", 2024-09-21T22:07:33.726 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3030000000000001e-06 2024-09-21T22:07:33.726 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.726 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.726 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:33.726 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.032952+0000", 2024-09-21T22:07:33.726 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.9070000000000002e-05 2024-09-21T22:07:33.726 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.726 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.726 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:33.727 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.032969+0000", 2024-09-21T22:07:33.727 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7839e-05 2024-09-21T22:07:33.727 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.727 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.727 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:33.727 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.032979+0000", 2024-09-21T22:07:33.727 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.5370000000000003e-06 2024-09-21T22:07:33.727 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.727 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.727 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:33.727 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.033041+0000", 2024-09-21T22:07:33.727 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.1562000000000005e-05 2024-09-21T22:07:33.727 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.727 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.728 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:33.728 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.033068+0000", 2024-09-21T22:07:33.728 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.7226999999999999e-05 2024-09-21T22:07:33.728 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.728 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.728 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:33.728 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.037908+0000", 2024-09-21T22:07:33.728 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0048401770000000002 2024-09-21T22:07:33.728 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.728 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.728 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:33.728 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.037952+0000", 2024-09-21T22:07:33.728 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.4317000000000003e-05 2024-09-21T22:07:33.728 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:33.728 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:33.729 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:33.729 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.729 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.729 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:690 2.3 e63/54 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head v 63'19, mlcod=63'17)", 2024-09-21T22:07:33.729 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.039813+0000", 2024-09-21T22:07:33.729 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 8.6104496319999999, 2024-09-21T22:07:33.729 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.035351891000000003, 2024-09-21T22:07:33.729 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:33.729 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:33.729 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:33.729 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:33.729 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.729 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:33.729 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.039813+0000", 2024-09-21T22:07:33.730 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:33.730 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.730 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.730 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:33.730 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.039813+0000", 2024-09-21T22:07:33.730 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:33.730 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.730 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.730 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:33.730 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.039814+0000", 2024-09-21T22:07:33.730 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.5300000000000002e-07 2024-09-21T22:07:33.730 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.730 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.730 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:33.731 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.040166+0000", 2024-09-21T22:07:33.731 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00035177 2024-09-21T22:07:33.731 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.731 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.731 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:33.731 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.040185+0000", 2024-09-21T22:07:33.731 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9000000000000001e-05 2024-09-21T22:07:33.731 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.731 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.731 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:33.731 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.040195+0000", 2024-09-21T22:07:33.731 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0254999999999999e-05 2024-09-21T22:07:33.731 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.731 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.731 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:33.732 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.040262+0000", 2024-09-21T22:07:33.732 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.6198000000000006e-05 2024-09-21T22:07:33.732 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.732 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.732 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:33.732 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.040288+0000", 2024-09-21T22:07:33.732 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5931999999999998e-05 2024-09-21T22:07:33.732 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.732 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.732 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:33.732 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.075106+0000", 2024-09-21T22:07:33.732 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.034818546999999998 2024-09-21T22:07:33.732 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.732 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.732 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:33.733 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.075165+0000", 2024-09-21T22:07:33.733 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.9236e-05 2024-09-21T22:07:33.733 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:33.733 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:33.733 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:33.733 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.733 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.733 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPull(2.3 e67/66 cost 1048576)", 2024-09-21T22:07:33.733 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:29.414957+0000", 2024-09-21T22:07:33.733 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 4.2353062530000001, 2024-09-21T22:07:33.733 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0032647269999999998, 2024-09-21T22:07:33.733 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:33.733 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:33.733 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-21T22:07:33.734 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:33.734 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.734 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:33.734 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:29.414957+0000", 2024-09-21T22:07:33.734 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:33.734 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.734 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.734 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:33.734 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:29.414957+0000", 2024-09-21T22:07:33.734 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:33.734 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.734 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.734 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:33.734 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:29.414959+0000", 2024-09-21T22:07:33.734 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.396e-06 2024-09-21T22:07:33.735 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.735 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.735 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:33.735 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:29.414986+0000", 2024-09-21T22:07:33.735 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.72e-05 2024-09-21T22:07:33.735 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.735 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.735 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:33.735 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:29.415018+0000", 2024-09-21T22:07:33.735 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.2079000000000002e-05 2024-09-21T22:07:33.735 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.735 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.735 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:33.735 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:29.415035+0000", 2024-09-21T22:07:33.735 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6316000000000001e-05 2024-09-21T22:07:33.736 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.736 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.736 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:33.736 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:29.415201+0000", 2024-09-21T22:07:33.736 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00016583900000000001 2024-09-21T22:07:33.736 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.736 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.736 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:33.736 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:29.418222+0000", 2024-09-21T22:07:33.736 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0030208969999999998 2024-09-21T22:07:33.736 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:33.736 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:33.736 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:33.736 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.737 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.737 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPull(2.3 e67/66 cost 1048576)", 2024-09-21T22:07:33.737 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:29.486922+0000", 2024-09-21T22:07:33.737 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 4.1633413160000003, 2024-09-21T22:07:33.737 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0029880940000000002, 2024-09-21T22:07:33.737 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:33.737 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:33.737 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-21T22:07:33.737 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:33.737 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.737 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:33.737 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:29.486922+0000", 2024-09-21T22:07:33.737 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:33.737 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.737 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.737 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:33.738 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:29.486922+0000", 2024-09-21T22:07:33.738 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:33.738 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.738 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.738 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:33.738 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:29.486924+0000", 2024-09-21T22:07:33.738 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.897e-06 2024-09-21T22:07:33.738 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.738 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.738 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:33.738 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:29.486975+0000", 2024-09-21T22:07:33.738 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.1275e-05 2024-09-21T22:07:33.738 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.738 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.739 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:33.739 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:29.487002+0000", 2024-09-21T22:07:33.739 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.7469000000000001e-05 2024-09-21T22:07:33.739 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.739 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.739 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:33.739 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:29.487016+0000", 2024-09-21T22:07:33.739 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3587000000000001e-05 2024-09-21T22:07:33.739 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.739 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.739 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:33.739 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:29.487158+0000", 2024-09-21T22:07:33.739 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00014215899999999999 2024-09-21T22:07:33.739 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:33.739 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:33.740 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:33.740 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:29.489910+0000", 2024-09-21T22:07:33.740 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0027517069999999999 2024-09-21T22:07:33.740 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:33.740 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:33.740 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:33.740 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:33.740 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:33.740 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:07:33.742 INFO:tasks.workunit.client.0.smithi157.stderr:+ DEV=/dev/nbd0 2024-09-21T22:07:33.742 INFO:tasks.workunit.client.0.smithi157.stderr:+ get_pid rbd 2024-09-21T22:07:33.742 INFO:tasks.workunit.client.0.smithi157.stderr:+ local pool=rbd 2024-09-21T22:07:33.742 INFO:tasks.workunit.client.0.smithi157.stderr:+ local ns= 2024-09-21T22:07:33.742 INFO:tasks.workunit.client.0.smithi157.stderr:++ rbd device --device-type nbd --format xml list 2024-09-21T22:07:33.742 INFO:tasks.workunit.client.0.smithi157.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd21553'\''][device='\''/dev/nbd0'\'']/id' 2024-09-21T22:07:33.742 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T22:07:33.743 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:33.736+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:33.775 INFO:tasks.workunit.client.0.smithi157.stderr:+ PID=22378 2024-09-21T22:07:33.775 INFO:tasks.workunit.client.0.smithi157.stderr:+ test -n 22378 2024-09-21T22:07:33.775 INFO:tasks.workunit.client.0.smithi157.stderr:+ ps -p 22378 -C rbd-nbd 2024-09-21T22:07:33.781 INFO:tasks.workunit.client.0.smithi157.stdout: PID TTY TIME CMD 2024-09-21T22:07:33.781 INFO:tasks.workunit.client.0.smithi157.stdout: 22378 ? 00:00:00 rbd-nbd 2024-09-21T22:07:33.781 INFO:tasks.workunit.client.0.smithi157.stderr:++ basename /dev/nbd0 2024-09-21T22:07:33.782 INFO:tasks.workunit.client.0.smithi157.stderr:+ devname=nbd0 2024-09-21T22:07:33.782 INFO:tasks.workunit.client.0.smithi157.stderr:++ awk -v dev=nbd0 '$4 == dev {print $3}' /proc/partitions 2024-09-21T22:07:33.785 INFO:tasks.workunit.client.0.smithi157.stderr:+ blocks=65536 2024-09-21T22:07:33.786 INFO:tasks.workunit.client.0.smithi157.stderr:+ test -n 65536 2024-09-21T22:07:33.786 INFO:tasks.workunit.client.0.smithi157.stderr:+ rbd resize rbd/testrbdnbd21553 --size 128M 2024-09-21T22:07:33.842 INFO:tasks.workunit.client.0.smithi157.stderr: Resizing image: 100% complete...done. 2024-09-21T22:07:33.844 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:33.840+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:33.855 INFO:tasks.workunit.client.0.smithi157.stderr:+ rbd info rbd/testrbdnbd21553 2024-09-21T22:07:33.889 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:07:33.889 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:07:33.889 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-09-21T22:07:33.889 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:07:33.899 INFO:tasks.workunit.client.0.smithi157.stdout:rbd image 'testrbdnbd21553': 2024-09-21T22:07:33.899 INFO:tasks.workunit.client.0.smithi157.stdout: size 128 MiB in 32 objects 2024-09-21T22:07:33.899 INFO:tasks.workunit.client.0.smithi157.stdout: order 22 (4 MiB objects) 2024-09-21T22:07:33.899 INFO:tasks.workunit.client.0.smithi157.stdout: snapshot_count: 0 2024-09-21T22:07:33.899 INFO:tasks.workunit.client.0.smithi157.stdout: id: 126483d75f2d 2024-09-21T22:07:33.899 INFO:tasks.workunit.client.0.smithi157.stdout: block_name_prefix: rbd_data.126483d75f2d 2024-09-21T22:07:33.899 INFO:tasks.workunit.client.0.smithi157.stdout: format: 2 2024-09-21T22:07:33.899 INFO:tasks.workunit.client.0.smithi157.stdout: features: layering, exclusive-lock, object-map, fast-diff, deep-flatten 2024-09-21T22:07:33.900 INFO:tasks.workunit.client.0.smithi157.stdout: op_features: 2024-09-21T22:07:33.900 INFO:tasks.workunit.client.0.smithi157.stdout: flags: 2024-09-21T22:07:33.900 INFO:tasks.workunit.client.0.smithi157.stdout: create_timestamp: Sat Sep 21 22:07:09 2024 2024-09-21T22:07:33.900 INFO:tasks.workunit.client.0.smithi157.stdout: access_timestamp: Sat Sep 21 22:07:09 2024 2024-09-21T22:07:33.900 INFO:tasks.workunit.client.0.smithi157.stdout: modify_timestamp: Sat Sep 21 22:07:09 2024 2024-09-21T22:07:33.904 INFO:tasks.workunit.client.0.smithi157.stderr:++ awk -v dev=nbd0 '$4 == dev {print $3}' /proc/partitions 2024-09-21T22:07:33.905 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T22:07:33.905 INFO:tasks.workunit.client.0.smithi157.stderr:+ blocks2=131072 2024-09-21T22:07:33.905 INFO:tasks.workunit.client.0.smithi157.stderr:+ test -n 131072 2024-09-21T22:07:33.906 INFO:tasks.workunit.client.0.smithi157.stderr:+ test 131072 -eq 131072 2024-09-21T22:07:33.906 INFO:tasks.workunit.client.0.smithi157.stderr:+ rbd resize rbd/testrbdnbd21553 --allow-shrink --size 64M 2024-09-21T22:07:33.948 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:33.944+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:33.964 INFO:tasks.workunit.client.0.smithi157.stderr: Resizing image: 50% complete... Resizing image: 53% complete... Resizing image: 56% complete... Resizing image: 59% complete... Resizing image: 62% complete... Resizing image: 65% complete... Resizing image: 68% complete... Resizing image: 71% complete... Resizing image: 75% complete... Resizing image: 78% complete... Resizing image: 81% complete... Resizing image: 84% complete... Resizing image: 87% complete... Resizing image: 90% complete... Resizing image: 93% complete... Resizing image: 96% complete... Resizing image: 100% complete...done. 2024-09-21T22:07:33.977 INFO:tasks.workunit.client.0.smithi157.stderr:++ awk -v dev=nbd0 '$4 == dev {print $3}' /proc/partitions 2024-09-21T22:07:33.979 INFO:tasks.workunit.client.0.smithi157.stderr:+ blocks2=65536 2024-09-21T22:07:33.979 INFO:tasks.workunit.client.0.smithi157.stderr:+ test -n 65536 2024-09-21T22:07:33.979 INFO:tasks.workunit.client.0.smithi157.stderr:+ test 65536 -eq 65536 2024-09-21T22:07:33.979 INFO:tasks.workunit.client.0.smithi157.stderr:+ unmap_device /dev/nbd0 22378 2024-09-21T22:07:33.979 INFO:tasks.workunit.client.0.smithi157.stderr:+ local args=/dev/nbd0 2024-09-21T22:07:33.979 INFO:tasks.workunit.client.0.smithi157.stderr:+ local pid=22378 2024-09-21T22:07:33.979 INFO:tasks.workunit.client.0.smithi157.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd0 2024-09-21T22:07:33.979 INFO:tasks.workunit.client.0.smithi157.stderr:+ local cmd 2024-09-21T22:07:33.979 INFO:tasks.workunit.client.0.smithi157.stderr:++ id -u 2024-09-21T22:07:33.980 INFO:tasks.workunit.client.0.smithi157.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T22:07:33.980 INFO:tasks.workunit.client.0.smithi157.stderr:++ which rbd 2024-09-21T22:07:33.981 INFO:tasks.workunit.client.0.smithi157.stderr:+ cmd=/usr/bin/rbd 2024-09-21T22:07:33.981 INFO:tasks.workunit.client.0.smithi157.stderr:+ shift 2024-09-21T22:07:33.981 INFO:tasks.workunit.client.0.smithi157.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd0 2024-09-21T22:07:34.046 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:34.042+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:34.056 INFO:tasks.workunit.client.0.smithi157.stderr:+ rbd device --device-type nbd list 2024-09-21T22:07:34.056 INFO:tasks.workunit.client.0.smithi157.stderr:+ expect_false grep '^22378\b' 2024-09-21T22:07:34.056 INFO:tasks.workunit.client.0.smithi157.stderr:+ grep '^22378\b' 2024-09-21T22:07:34.075 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:07:34.075 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:07:34.075 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-21T22:07:34.075 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-21T22:07:34.075 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:07:34.089 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T22:07:34.103 INFO:tasks.workunit.client.0.smithi157.stderr:+ return 0 2024-09-21T22:07:34.104 INFO:tasks.workunit.client.0.smithi157.stderr:+ ps -C rbd-nbd 2024-09-21T22:07:34.104 INFO:tasks.workunit.client.0.smithi157.stderr:+ expect_false grep '^ *22378\b' 2024-09-21T22:07:34.104 INFO:tasks.workunit.client.0.smithi157.stderr:+ grep '^ *22378\b' 2024-09-21T22:07:34.109 INFO:tasks.workunit.client.0.smithi157.stderr:+ return 0 2024-09-21T22:07:34.109 INFO:tasks.workunit.client.0.smithi157.stderr:+ sleep 0.5 2024-09-21T22:07:34.146 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:34.142+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:34.216 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:07:34.216 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-21T22:07:34.216 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-21T22:07:34.216 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-21T22:07:34.216 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.216 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:07:34.216 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.036355+0000", 2024-09-21T22:07:34.217 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 92.176543735999999, 2024-09-21T22:07:34.217 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.062441934999999997, 2024-09-21T22:07:34.217 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:34.217 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:34.217 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:34.217 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:34.217 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:07:34.217 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:07:34.217 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1 2024-09-21T22:07:34.217 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.217 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:34.217 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.217 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:34.217 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036355+0000", 2024-09-21T22:07:34.218 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:34.218 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.218 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.218 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:34.218 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036355+0000", 2024-09-21T22:07:34.218 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:34.218 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.218 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.218 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:34.218 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036357+0000", 2024-09-21T22:07:34.218 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.446e-06 2024-09-21T22:07:34.218 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.218 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.218 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:34.219 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036373+0000", 2024-09-21T22:07:34.219 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5447e-05 2024-09-21T22:07:34.219 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.219 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.219 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:34.219 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036399+0000", 2024-09-21T22:07:34.219 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6423000000000001e-05 2024-09-21T22:07:34.219 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.219 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.219 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:34.219 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036420+0000", 2024-09-21T22:07:34.219 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1129999999999999e-05 2024-09-21T22:07:34.219 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.220 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.220 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:34.220 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036557+0000", 2024-09-21T22:07:34.220 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013686899999999999 2024-09-21T22:07:34.220 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.220 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.220 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-21T22:07:34.220 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036585+0000", 2024-09-21T22:07:34.220 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8362999999999999e-05 2024-09-21T22:07:34.220 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.220 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.220 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:34.220 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.095068+0000", 2024-09-21T22:07:34.220 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058482972000000001 2024-09-21T22:07:34.221 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.221 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.221 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:34.221 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.095516+0000", 2024-09-21T22:07:34.221 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00044802999999999998 2024-09-21T22:07:34.221 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.221 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.221 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:34.221 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.095869+0000", 2024-09-21T22:07:34.221 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000352104 2024-09-21T22:07:34.221 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.221 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.221 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:34.221 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.097055+0000", 2024-09-21T22:07:34.222 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011865599999999999 2024-09-21T22:07:34.222 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.222 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.222 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:34.222 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.098637+0000", 2024-09-21T22:07:34.222 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001581458 2024-09-21T22:07:34.222 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.222 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.222 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:34.222 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.098728+0000", 2024-09-21T22:07:34.222 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1043000000000002e-05 2024-09-21T22:07:34.222 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.222 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.222 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:34.223 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.098797+0000", 2024-09-21T22:07:34.223 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9090000000000004e-05 2024-09-21T22:07:34.223 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:34.223 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:34.223 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:34.223 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.223 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.223 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4840.0:662 2.0 2:13c3ab2f:::rbd_data.126483d75f2d.0000000000000007:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:07:34.223 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.283980+0000", 2024-09-21T22:07:34.223 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.9289181739999997, 2024-09-21T22:07:34.223 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.036679130999999997, 2024-09-21T22:07:34.223 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:34.223 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:34.224 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:34.224 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:34.224 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4840", 2024-09-21T22:07:34.224 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:34.224 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 662 2024-09-21T22:07:34.224 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.224 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:34.224 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.224 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:34.224 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.283980+0000", 2024-09-21T22:07:34.224 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:34.224 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.224 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.224 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:34.224 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.283980+0000", 2024-09-21T22:07:34.225 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:34.225 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.225 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.225 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:34.225 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.283981+0000", 2024-09-21T22:07:34.225 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0920000000000001e-06 2024-09-21T22:07:34.225 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.225 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.225 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:34.225 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.284737+0000", 2024-09-21T22:07:34.225 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00075525500000000003 2024-09-21T22:07:34.225 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.225 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.225 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:34.226 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.284755+0000", 2024-09-21T22:07:34.226 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8428999999999999e-05 2024-09-21T22:07:34.226 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.226 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.226 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:34.226 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.284764+0000", 2024-09-21T22:07:34.226 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5839999999999995e-06 2024-09-21T22:07:34.226 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.226 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.226 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:34.226 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.284828+0000", 2024-09-21T22:07:34.226 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4709000000000007e-05 2024-09-21T22:07:34.226 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.226 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.226 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:34.227 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.284910+0000", 2024-09-21T22:07:34.227 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1478999999999997e-05 2024-09-21T22:07:34.227 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.227 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.227 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-09-21T22:07:34.227 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.285045+0000", 2024-09-21T22:07:34.227 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013465300000000001 2024-09-21T22:07:34.227 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.227 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.227 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:34.227 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.319249+0000", 2024-09-21T22:07:34.227 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.034204171999999998 2024-09-21T22:07:34.227 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.227 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.228 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:34.228 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.320488+0000", 2024-09-21T22:07:34.228 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001239789 2024-09-21T22:07:34.228 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.228 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.228 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:34.228 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.320595+0000", 2024-09-21T22:07:34.228 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010697299999999999 2024-09-21T22:07:34.228 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.228 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.228 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:34.228 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.320659+0000", 2024-09-21T22:07:34.228 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3996000000000002e-05 2024-09-21T22:07:34.228 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:34.228 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:34.229 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:34.229 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.229 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.229 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4840.0:664 2.0 2:13c3ab2f:::rbd_data.126483d75f2d.0000000000000007:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:07:34.229 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.328703+0000", 2024-09-21T22:07:34.229 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.8841952909999993, 2024-09-21T22:07:34.229 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.032680873999999999, 2024-09-21T22:07:34.229 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:34.229 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:34.229 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:34.229 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:34.229 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4840", 2024-09-21T22:07:34.229 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:34.229 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 664 2024-09-21T22:07:34.229 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.230 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:34.230 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.230 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:34.230 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.328703+0000", 2024-09-21T22:07:34.230 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:34.230 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.230 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.230 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:34.230 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.328703+0000", 2024-09-21T22:07:34.230 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:34.230 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.230 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.230 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:34.230 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.328705+0000", 2024-09-21T22:07:34.231 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3459999999999999e-06 2024-09-21T22:07:34.231 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.231 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.231 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:34.231 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.329688+0000", 2024-09-21T22:07:34.231 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00098393799999999996 2024-09-21T22:07:34.231 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.231 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.231 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:34.231 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.329720+0000", 2024-09-21T22:07:34.231 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1779999999999997e-05 2024-09-21T22:07:34.231 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.231 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.231 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:34.232 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.329736+0000", 2024-09-21T22:07:34.232 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5396e-05 2024-09-21T22:07:34.232 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.232 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.232 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:34.232 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.329851+0000", 2024-09-21T22:07:34.232 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011579299999999999 2024-09-21T22:07:34.232 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.232 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.232 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:34.232 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.329982+0000", 2024-09-21T22:07:34.232 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000130176 2024-09-21T22:07:34.232 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.232 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.232 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-09-21T22:07:34.233 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.330150+0000", 2024-09-21T22:07:34.233 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016869699999999999 2024-09-21T22:07:34.233 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.233 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.233 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:34.233 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.360455+0000", 2024-09-21T22:07:34.233 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.030304714 2024-09-21T22:07:34.233 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.233 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.233 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:34.233 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.361223+0000", 2024-09-21T22:07:34.233 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00076771200000000004 2024-09-21T22:07:34.233 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.233 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.234 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:34.234 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.361318+0000", 2024-09-21T22:07:34.234 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5413000000000003e-05 2024-09-21T22:07:34.234 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.234 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.234 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:34.234 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.361384+0000", 2024-09-21T22:07:34.234 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5908999999999995e-05 2024-09-21T22:07:34.234 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:34.234 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:34.234 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:34.234 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.234 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.234 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4840.0:666 2.0 2:13c3ab2f:::rbd_data.126483d75f2d.0000000000000007:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:07:34.234 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.370777+0000", 2024-09-21T22:07:34.235 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.8421213139999999, 2024-09-21T22:07:34.235 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.031078489000000001, 2024-09-21T22:07:34.235 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:34.235 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:34.235 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:34.235 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:34.235 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4840", 2024-09-21T22:07:34.235 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:34.235 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 666 2024-09-21T22:07:34.235 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.235 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:34.235 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.235 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:34.235 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.370777+0000", 2024-09-21T22:07:34.235 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:34.236 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.236 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.236 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:34.236 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.370777+0000", 2024-09-21T22:07:34.236 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:34.236 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.236 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.236 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:34.236 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.370778+0000", 2024-09-21T22:07:34.236 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5799999999999998e-07 2024-09-21T22:07:34.236 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.236 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.236 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:34.236 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.371759+0000", 2024-09-21T22:07:34.237 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00098045500000000009 2024-09-21T22:07:34.237 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.237 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.237 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:34.237 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.371774+0000", 2024-09-21T22:07:34.237 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5227e-05 2024-09-21T22:07:34.237 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.237 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.237 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:34.237 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.371781+0000", 2024-09-21T22:07:34.237 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8809999999999998e-06 2024-09-21T22:07:34.237 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.237 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.237 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:34.237 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.371848+0000", 2024-09-21T22:07:34.238 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7772000000000001e-05 2024-09-21T22:07:34.238 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.238 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.238 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:34.238 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.371918+0000", 2024-09-21T22:07:34.238 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9709999999999995e-05 2024-09-21T22:07:34.238 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.238 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.238 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-09-21T22:07:34.238 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.372006+0000", 2024-09-21T22:07:34.238 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8246999999999997e-05 2024-09-21T22:07:34.238 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.238 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.238 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:34.238 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.401009+0000", 2024-09-21T22:07:34.239 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.02900266 2024-09-21T22:07:34.239 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.239 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.239 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:34.239 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.401750+0000", 2024-09-21T22:07:34.239 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000740903 2024-09-21T22:07:34.239 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.239 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.239 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:34.239 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.401812+0000", 2024-09-21T22:07:34.239 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2144000000000007e-05 2024-09-21T22:07:34.239 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.239 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.239 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:34.240 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.401856+0000", 2024-09-21T22:07:34.240 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3532000000000003e-05 2024-09-21T22:07:34.240 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:34.240 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:34.240 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:34.240 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.240 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.240 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4840.0:676 2.7 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:07:34.240 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.625676+0000", 2024-09-21T22:07:34.240 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.5872227799999994, 2024-09-21T22:07:34.240 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.039450085000000003, 2024-09-21T22:07:34.240 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:34.240 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:34.240 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:34.240 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:34.241 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4840", 2024-09-21T22:07:34.241 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:34.241 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 676 2024-09-21T22:07:34.241 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.241 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:34.241 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.241 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:34.241 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.625676+0000", 2024-09-21T22:07:34.241 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:34.241 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.241 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.241 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:34.241 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.625676+0000", 2024-09-21T22:07:34.241 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:34.241 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.241 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.242 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:34.242 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.625677+0000", 2024-09-21T22:07:34.242 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5969999999999999e-06 2024-09-21T22:07:34.242 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.242 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.242 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:34.242 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.626810+0000", 2024-09-21T22:07:34.242 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001132267 2024-09-21T22:07:34.242 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.242 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.242 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:34.242 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.626842+0000", 2024-09-21T22:07:34.242 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2938999999999998e-05 2024-09-21T22:07:34.242 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.243 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.243 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:34.243 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.626857+0000", 2024-09-21T22:07:34.243 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4987000000000001e-05 2024-09-21T22:07:34.243 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.243 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.243 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:34.243 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.626930+0000", 2024-09-21T22:07:34.243 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2908000000000001e-05 2024-09-21T22:07:34.243 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.243 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.243 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:34.243 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.626990+0000", 2024-09-21T22:07:34.243 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9397000000000001e-05 2024-09-21T22:07:34.244 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.244 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.244 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-09-21T22:07:34.244 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.627082+0000", 2024-09-21T22:07:34.244 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1914000000000001e-05 2024-09-21T22:07:34.244 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.244 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.244 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:34.244 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.652887+0000", 2024-09-21T22:07:34.244 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025804911999999999 2024-09-21T22:07:34.244 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.244 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.244 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:34.244 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.665017+0000", 2024-09-21T22:07:34.244 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.01213008 2024-09-21T22:07:34.245 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.245 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.245 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:34.245 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.665075+0000", 2024-09-21T22:07:34.245 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8566000000000001e-05 2024-09-21T22:07:34.245 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.245 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.245 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:34.245 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.665126+0000", 2024-09-21T22:07:34.245 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0518e-05 2024-09-21T22:07:34.245 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:34.245 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:34.245 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:34.245 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.245 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.246 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4840.0:678 2.7 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:07:34.246 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.678289+0000", 2024-09-21T22:07:34.246 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.5346096809999992, 2024-09-21T22:07:34.246 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.055089659999999999, 2024-09-21T22:07:34.246 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:34.246 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:34.246 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:34.246 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:34.246 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4840", 2024-09-21T22:07:34.246 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:34.246 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 678 2024-09-21T22:07:34.246 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.246 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:34.246 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.247 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:34.247 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.678289+0000", 2024-09-21T22:07:34.247 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:34.247 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.247 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.247 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:34.247 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.678289+0000", 2024-09-21T22:07:34.247 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:34.247 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.247 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.247 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:34.247 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.678291+0000", 2024-09-21T22:07:34.247 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7400000000000001e-06 2024-09-21T22:07:34.247 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.247 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.248 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:34.248 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.679129+0000", 2024-09-21T22:07:34.248 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00083801700000000004 2024-09-21T22:07:34.248 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.248 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.248 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:34.248 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.679161+0000", 2024-09-21T22:07:34.248 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2398999999999999e-05 2024-09-21T22:07:34.248 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.248 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.248 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:34.248 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.679176+0000", 2024-09-21T22:07:34.248 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5396999999999998e-05 2024-09-21T22:07:34.248 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.248 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.249 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:34.249 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.679256+0000", 2024-09-21T22:07:34.249 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9271999999999996e-05 2024-09-21T22:07:34.249 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.249 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.249 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:34.249 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.679333+0000", 2024-09-21T22:07:34.249 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7682999999999995e-05 2024-09-21T22:07:34.249 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.250 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.250 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-09-21T22:07:34.250 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.679457+0000", 2024-09-21T22:07:34.250 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012327900000000001 2024-09-21T22:07:34.250 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.250 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.250 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:34.250 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.713209+0000", 2024-09-21T22:07:34.250 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.033752058000000001 2024-09-21T22:07:34.250 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.250 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.250 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:34.250 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.733243+0000", 2024-09-21T22:07:34.250 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020034337999999999 2024-09-21T22:07:34.251 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.251 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.251 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:34.251 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.733320+0000", 2024-09-21T22:07:34.251 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7354999999999994e-05 2024-09-21T22:07:34.251 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.251 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.251 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:34.251 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.733378+0000", 2024-09-21T22:07:34.251 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8121999999999997e-05 2024-09-21T22:07:34.251 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:34.251 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:34.251 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:34.251 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.251 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.252 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4840.0:680 2.7 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:07:34.252 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.744506+0000", 2024-09-21T22:07:34.252 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.4683928819999998, 2024-09-21T22:07:34.252 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.063865055000000004, 2024-09-21T22:07:34.252 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:34.252 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:34.252 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:34.252 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:34.252 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4840", 2024-09-21T22:07:34.252 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:34.252 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 680 2024-09-21T22:07:34.252 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.252 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:34.252 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.252 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:34.253 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.744506+0000", 2024-09-21T22:07:34.253 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:34.253 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.253 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.253 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:34.253 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.744506+0000", 2024-09-21T22:07:34.253 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:34.253 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.253 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.253 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:34.253 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.744508+0000", 2024-09-21T22:07:34.253 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.103e-06 2024-09-21T22:07:34.253 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.253 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.254 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:34.254 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.745371+0000", 2024-09-21T22:07:34.254 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00086322199999999997 2024-09-21T22:07:34.254 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.254 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.254 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:34.254 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.745401+0000", 2024-09-21T22:07:34.254 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9955999999999999e-05 2024-09-21T22:07:34.254 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.254 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.254 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:34.254 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.745416+0000", 2024-09-21T22:07:34.254 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4873e-05 2024-09-21T22:07:34.254 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.255 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.255 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:34.255 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.745509+0000", 2024-09-21T22:07:34.255 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3171000000000006e-05 2024-09-21T22:07:34.255 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.255 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.255 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:34.255 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.745632+0000", 2024-09-21T22:07:34.255 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012272699999999999 2024-09-21T22:07:34.255 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.255 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.255 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-09-21T22:07:34.255 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.745789+0000", 2024-09-21T22:07:34.255 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015781399999999999 2024-09-21T22:07:34.255 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.255 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.256 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:34.256 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.785494+0000", 2024-09-21T22:07:34.256 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.039704401 2024-09-21T22:07:34.256 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.256 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.256 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:34.256 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.808090+0000", 2024-09-21T22:07:34.256 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022596207 2024-09-21T22:07:34.256 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.256 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.256 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:34.256 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.808246+0000", 2024-09-21T22:07:34.256 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015567200000000001 2024-09-21T22:07:34.256 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.257 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.257 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:34.257 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.808371+0000", 2024-09-21T22:07:34.257 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012490900000000001 2024-09-21T22:07:34.257 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:34.257 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:34.257 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:34.257 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.257 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.257 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4840.0:682 2.7 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:07:34.257 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.822415+0000", 2024-09-21T22:07:34.257 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.3904833189999994, 2024-09-21T22:07:34.257 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.065044633000000004, 2024-09-21T22:07:34.257 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:34.258 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:34.258 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:34.258 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:34.258 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4840", 2024-09-21T22:07:34.258 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:34.258 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 682 2024-09-21T22:07:34.258 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.258 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:34.258 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.258 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:34.258 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.822415+0000", 2024-09-21T22:07:34.258 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:34.258 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.258 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.259 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:34.259 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.822415+0000", 2024-09-21T22:07:34.259 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:34.259 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.259 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.259 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:34.259 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.822417+0000", 2024-09-21T22:07:34.259 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8029999999999999e-06 2024-09-21T22:07:34.259 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.259 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.259 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:34.259 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.823308+0000", 2024-09-21T22:07:34.259 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00089146500000000003 2024-09-21T22:07:34.259 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.259 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.260 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:34.260 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.823339+0000", 2024-09-21T22:07:34.260 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0493000000000001e-05 2024-09-21T22:07:34.260 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.260 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.260 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:34.260 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.823354+0000", 2024-09-21T22:07:34.260 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5472999999999999e-05 2024-09-21T22:07:34.260 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.260 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.260 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:34.260 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.823476+0000", 2024-09-21T22:07:34.260 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000121209 2024-09-21T22:07:34.260 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.261 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.261 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:34.261 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.823650+0000", 2024-09-21T22:07:34.261 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017394599999999999 2024-09-21T22:07:34.261 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.261 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.261 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-09-21T22:07:34.261 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.823873+0000", 2024-09-21T22:07:34.261 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022314899999999999 2024-09-21T22:07:34.261 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.261 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.261 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:34.261 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.884396+0000", 2024-09-21T22:07:34.261 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.060523674999999999 2024-09-21T22:07:34.261 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.262 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.262 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:34.262 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.887238+0000", 2024-09-21T22:07:34.262 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028420450000000001 2024-09-21T22:07:34.262 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.262 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.262 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:34.262 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.887369+0000", 2024-09-21T22:07:34.262 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013010699999999999 2024-09-21T22:07:34.262 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.262 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.262 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:34.262 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.887460+0000", 2024-09-21T22:07:34.262 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1267999999999994e-05 2024-09-21T22:07:34.263 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:34.263 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:34.263 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:34.263 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.263 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.263 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4840.0:692 2.5 2:b1d55633:::rbd_data.126483d75f2d.000000000000000b:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:07:34.263 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:25.091048+0000", 2024-09-21T22:07:34.263 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.1218507849999995, 2024-09-21T22:07:34.263 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.060917092999999999, 2024-09-21T22:07:34.263 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:34.263 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:34.263 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:34.263 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:34.263 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4840", 2024-09-21T22:07:34.264 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:34.264 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 692 2024-09-21T22:07:34.264 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.264 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:34.264 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.264 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:34.264 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.091048+0000", 2024-09-21T22:07:34.264 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:34.264 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.264 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.264 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:34.264 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.091048+0000", 2024-09-21T22:07:34.264 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:34.264 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.264 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.265 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:34.265 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.091050+0000", 2024-09-21T22:07:34.265 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.03e-06 2024-09-21T22:07:34.265 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.265 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.265 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:34.265 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.091989+0000", 2024-09-21T22:07:34.265 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00093932100000000001 2024-09-21T22:07:34.265 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.265 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.265 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:34.265 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.092023+0000", 2024-09-21T22:07:34.265 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4310000000000002e-05 2024-09-21T22:07:34.265 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.265 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.266 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:34.266 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.092040+0000", 2024-09-21T22:07:34.266 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6426000000000002e-05 2024-09-21T22:07:34.266 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.266 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.266 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:34.266 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.092175+0000", 2024-09-21T22:07:34.266 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013500299999999999 2024-09-21T22:07:34.266 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.266 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.266 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:34.266 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.092307+0000", 2024-09-21T22:07:34.266 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000132562 2024-09-21T22:07:34.266 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.267 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.267 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:34.267 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.092499+0000", 2024-09-21T22:07:34.267 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000191972 2024-09-21T22:07:34.267 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.267 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.267 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:34.267 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.151065+0000", 2024-09-21T22:07:34.267 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058565514999999999 2024-09-21T22:07:34.267 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.267 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.267 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:34.267 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.151844+0000", 2024-09-21T22:07:34.267 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00077930800000000004 2024-09-21T22:07:34.268 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.268 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.268 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:34.268 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.151916+0000", 2024-09-21T22:07:34.268 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1921999999999993e-05 2024-09-21T22:07:34.268 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.268 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.268 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:34.268 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.151965+0000", 2024-09-21T22:07:34.268 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8724000000000003e-05 2024-09-21T22:07:34.268 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:34.268 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:34.268 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:34.268 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.269 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.269 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4840.0:692 2.5 e63/37 ondisk, result = 0)", 2024-09-21T22:07:34.269 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:25.118569+0000", 2024-09-21T22:07:34.269 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.0943294229999996, 2024-09-21T22:07:34.269 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.032507991, 2024-09-21T22:07:34.269 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:34.270 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:34.270 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:07:34.271 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:34.271 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.271 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:34.271 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.118569+0000", 2024-09-21T22:07:34.271 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:34.271 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.271 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.271 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:34.271 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.118569+0000", 2024-09-21T22:07:34.271 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:34.271 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.271 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.271 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:34.271 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.118571+0000", 2024-09-21T22:07:34.271 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5430000000000001e-06 2024-09-21T22:07:34.272 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.272 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.272 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:34.272 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.118579+0000", 2024-09-21T22:07:34.272 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5799999999999992e-06 2024-09-21T22:07:34.272 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.272 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.272 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:34.272 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.118607+0000", 2024-09-21T22:07:34.272 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7733999999999999e-05 2024-09-21T22:07:34.272 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.272 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.272 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:34.272 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.118622+0000", 2024-09-21T22:07:34.273 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5023e-05 2024-09-21T22:07:34.273 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.273 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.273 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:34.273 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.151038+0000", 2024-09-21T22:07:34.273 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.032415807999999997 2024-09-21T22:07:34.273 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.273 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.273 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:34.273 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.151053+0000", 2024-09-21T22:07:34.273 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5475e-05 2024-09-21T22:07:34.273 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.273 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.273 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:34.273 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.151077+0000", 2024-09-21T22:07:34.274 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3828e-05 2024-09-21T22:07:34.274 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:34.274 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:34.274 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:34.274 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.274 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.274 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4840.0:694 2.5 2:b1d55633:::rbd_data.126483d75f2d.000000000000000b:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:07:34.274 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:25.162829+0000", 2024-09-21T22:07:34.274 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.0500694520000007, 2024-09-21T22:07:34.274 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.045653405000000001, 2024-09-21T22:07:34.274 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:34.274 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:34.274 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:34.274 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:34.274 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4840", 2024-09-21T22:07:34.275 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:34.275 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 694 2024-09-21T22:07:34.275 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.275 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:34.275 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.275 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:34.275 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.162829+0000", 2024-09-21T22:07:34.275 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:34.275 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.275 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.275 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:34.275 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.162829+0000", 2024-09-21T22:07:34.275 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:34.275 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.276 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.276 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:34.276 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.162831+0000", 2024-09-21T22:07:34.276 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7799999999999999e-06 2024-09-21T22:07:34.276 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.276 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.276 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:34.276 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.163904+0000", 2024-09-21T22:07:34.276 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001073673 2024-09-21T22:07:34.276 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.276 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.276 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:34.276 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.163937+0000", 2024-09-21T22:07:34.276 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2100000000000001e-05 2024-09-21T22:07:34.276 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.277 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.277 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:34.277 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.163952+0000", 2024-09-21T22:07:34.277 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5517000000000001e-05 2024-09-21T22:07:34.277 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.277 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.277 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:34.277 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.164060+0000", 2024-09-21T22:07:34.277 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000107909 2024-09-21T22:07:34.277 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.277 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.277 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:34.277 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.164162+0000", 2024-09-21T22:07:34.277 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000101619 2024-09-21T22:07:34.278 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.278 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.278 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:34.278 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.164318+0000", 2024-09-21T22:07:34.278 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015683599999999999 2024-09-21T22:07:34.278 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.278 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.278 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:34.278 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.207369+0000", 2024-09-21T22:07:34.278 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.043050401000000002 2024-09-21T22:07:34.278 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.278 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.278 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:34.278 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.208256+0000", 2024-09-21T22:07:34.279 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00088747800000000003 2024-09-21T22:07:34.279 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.279 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.279 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:34.279 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.208350+0000", 2024-09-21T22:07:34.279 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4129000000000006e-05 2024-09-21T22:07:34.279 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.279 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.279 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:34.279 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.208482+0000", 2024-09-21T22:07:34.279 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000131963 2024-09-21T22:07:34.279 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:34.279 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:34.279 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:34.279 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.280 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.280 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4840.0:696 2.5 2:b1d55633:::rbd_data.126483d75f2d.000000000000000b:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:07:34.280 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:25.218641+0000", 2024-09-21T22:07:34.280 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.9942575169999994, 2024-09-21T22:07:34.280 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058363669999999999, 2024-09-21T22:07:34.280 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:34.280 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:34.280 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:34.280 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:34.280 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4840", 2024-09-21T22:07:34.280 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:34.280 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 696 2024-09-21T22:07:34.280 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.280 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:34.280 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.281 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:34.281 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.218641+0000", 2024-09-21T22:07:34.281 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:34.281 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.281 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.281 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:34.281 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.218641+0000", 2024-09-21T22:07:34.281 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:34.281 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.281 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.281 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:34.281 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.218642+0000", 2024-09-21T22:07:34.281 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.266e-06 2024-09-21T22:07:34.281 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.281 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.282 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:34.282 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.219432+0000", 2024-09-21T22:07:34.282 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00078986300000000002 2024-09-21T22:07:34.282 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.282 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.282 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:34.282 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.219450+0000", 2024-09-21T22:07:34.282 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7526000000000001e-05 2024-09-21T22:07:34.282 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.282 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.282 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:34.282 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.219459+0000", 2024-09-21T22:07:34.282 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3000000000000007e-06 2024-09-21T22:07:34.282 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.283 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.283 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:34.283 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.219655+0000", 2024-09-21T22:07:34.283 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019599099999999999 2024-09-21T22:07:34.283 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.283 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.283 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:34.283 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.219917+0000", 2024-09-21T22:07:34.283 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026198200000000001 2024-09-21T22:07:34.283 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.283 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.283 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:34.283 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.220198+0000", 2024-09-21T22:07:34.283 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00028154500000000002 2024-09-21T22:07:34.283 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.284 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.284 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:34.284 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.268856+0000", 2024-09-21T22:07:34.284 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.048657687999999998 2024-09-21T22:07:34.284 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.284 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.284 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:34.284 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.276903+0000", 2024-09-21T22:07:34.284 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0080472249999999999 2024-09-21T22:07:34.284 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.284 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.284 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:34.284 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.276958+0000", 2024-09-21T22:07:34.284 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4184999999999997e-05 2024-09-21T22:07:34.284 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.285 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.285 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:34.285 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.277005+0000", 2024-09-21T22:07:34.285 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7098999999999997e-05 2024-09-21T22:07:34.285 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:34.285 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:34.285 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:34.285 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.285 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.285 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4840.0:698 2.5 2:b1d55633:::rbd_data.126483d75f2d.000000000000000b:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:07:34.285 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:25.289380+0000", 2024-09-21T22:07:34.285 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.9235189699999999, 2024-09-21T22:07:34.285 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.055691072000000001, 2024-09-21T22:07:34.285 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:34.285 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:34.286 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:34.286 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:34.286 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4840", 2024-09-21T22:07:34.286 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:34.286 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 698 2024-09-21T22:07:34.286 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.286 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:34.286 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.286 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:34.286 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.289380+0000", 2024-09-21T22:07:34.286 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:34.286 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.286 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.287 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:34.287 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.289380+0000", 2024-09-21T22:07:34.287 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:34.287 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.287 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.287 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:34.287 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.289381+0000", 2024-09-21T22:07:34.287 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3540000000000001e-06 2024-09-21T22:07:34.287 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.287 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.287 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:34.287 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.290221+0000", 2024-09-21T22:07:34.287 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00083999900000000002 2024-09-21T22:07:34.287 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.287 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.288 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:34.288 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.290238+0000", 2024-09-21T22:07:34.288 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7130999999999999e-05 2024-09-21T22:07:34.288 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.288 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.288 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:34.288 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.290247+0000", 2024-09-21T22:07:34.288 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6500000000000002e-06 2024-09-21T22:07:34.288 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.288 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.288 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:34.288 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.290320+0000", 2024-09-21T22:07:34.288 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3530999999999998e-05 2024-09-21T22:07:34.288 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.288 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.289 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:34.289 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.290433+0000", 2024-09-21T22:07:34.289 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000112744 2024-09-21T22:07:34.289 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.289 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.289 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:34.289 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.290605+0000", 2024-09-21T22:07:34.289 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017253700000000001 2024-09-21T22:07:34.289 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.289 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.289 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:34.289 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.320545+0000", 2024-09-21T22:07:34.289 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029939582999999999 2024-09-21T22:07:34.289 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.289 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.290 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:34.290 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.344991+0000", 2024-09-21T22:07:34.290 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024445956000000001 2024-09-21T22:07:34.290 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.290 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.290 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:34.290 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.345034+0000", 2024-09-21T22:07:34.290 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2806000000000003e-05 2024-09-21T22:07:34.290 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.290 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.290 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:34.290 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.345071+0000", 2024-09-21T22:07:34.290 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6780999999999999e-05 2024-09-21T22:07:34.290 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:34.290 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:34.291 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:34.291 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.291 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.291 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4840.0:700 2.5 2:a70ae99f:::rbd_data.126483d75f2d.000000000000000c:head [write 131072~917504 in=917504b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:07:34.291 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:25.352433+0000", 2024-09-21T22:07:34.291 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.8604653720000002, 2024-09-21T22:07:34.291 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.030906125999999999, 2024-09-21T22:07:34.291 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:34.291 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:34.291 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:34.291 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:34.291 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4840", 2024-09-21T22:07:34.291 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:34.291 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 700 2024-09-21T22:07:34.291 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.292 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:34.292 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.292 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:34.292 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.352433+0000", 2024-09-21T22:07:34.292 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:34.292 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.292 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.292 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:34.292 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.352433+0000", 2024-09-21T22:07:34.292 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:34.292 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.292 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.292 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:34.292 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.352436+0000", 2024-09-21T22:07:34.292 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6429999999999999e-06 2024-09-21T22:07:34.293 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.293 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.293 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:34.293 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.353322+0000", 2024-09-21T22:07:34.293 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000886395 2024-09-21T22:07:34.293 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.293 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.293 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:34.293 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.353353+0000", 2024-09-21T22:07:34.293 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0685999999999997e-05 2024-09-21T22:07:34.293 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.293 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.293 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:34.293 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.353371+0000", 2024-09-21T22:07:34.293 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7873e-05 2024-09-21T22:07:34.294 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.294 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.294 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:34.294 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.353432+0000", 2024-09-21T22:07:34.294 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1365000000000006e-05 2024-09-21T22:07:34.294 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.294 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.294 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:34.294 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.353502+0000", 2024-09-21T22:07:34.294 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0018000000000006e-05 2024-09-21T22:07:34.294 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.294 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.294 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:34.294 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.353611+0000", 2024-09-21T22:07:34.294 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010871500000000001 2024-09-21T22:07:34.295 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.295 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.295 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:34.295 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.382395+0000", 2024-09-21T22:07:34.295 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.028784265 2024-09-21T22:07:34.295 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.295 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.295 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:34.295 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.383250+0000", 2024-09-21T22:07:34.295 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00085512900000000005 2024-09-21T22:07:34.295 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.295 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.295 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:34.296 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.383298+0000", 2024-09-21T22:07:34.296 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7435999999999998e-05 2024-09-21T22:07:34.296 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.296 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.296 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:34.296 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.383339+0000", 2024-09-21T22:07:34.296 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1600999999999997e-05 2024-09-21T22:07:34.296 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:34.296 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:34.296 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:34.296 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.296 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.296 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4840.0:702 2.5 2:a70ae99f:::rbd_data.126483d75f2d.000000000000000c:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:07:34.296 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:25.392746+0000", 2024-09-21T22:07:34.297 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.8201523630000001, 2024-09-21T22:07:34.297 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.037346635000000003, 2024-09-21T22:07:34.297 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:34.297 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:34.297 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:34.297 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:34.297 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4840", 2024-09-21T22:07:34.297 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:34.297 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 702 2024-09-21T22:07:34.297 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.297 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:34.297 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.297 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:34.297 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.392746+0000", 2024-09-21T22:07:34.297 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:34.298 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.298 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.298 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:34.298 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.392746+0000", 2024-09-21T22:07:34.298 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:34.298 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.298 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.298 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:34.298 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.392748+0000", 2024-09-21T22:07:34.298 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.395e-06 2024-09-21T22:07:34.298 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.298 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.298 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:34.298 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.393583+0000", 2024-09-21T22:07:34.299 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00083551000000000003 2024-09-21T22:07:34.299 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.299 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.299 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:34.299 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.393608+0000", 2024-09-21T22:07:34.299 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4729999999999999e-05 2024-09-21T22:07:34.299 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.299 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.299 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:34.299 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.393620+0000", 2024-09-21T22:07:34.299 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2548000000000001e-05 2024-09-21T22:07:34.299 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.299 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.299 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:34.299 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.393668+0000", 2024-09-21T22:07:34.300 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7587999999999999e-05 2024-09-21T22:07:34.300 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.300 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.300 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:34.300 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.393724+0000", 2024-09-21T22:07:34.300 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6023999999999998e-05 2024-09-21T22:07:34.300 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.300 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.300 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:34.300 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.393809+0000", 2024-09-21T22:07:34.300 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5016000000000001e-05 2024-09-21T22:07:34.300 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.300 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.300 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:34.300 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.415742+0000", 2024-09-21T22:07:34.301 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021933540000000001 2024-09-21T22:07:34.301 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.301 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.301 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:34.301 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.430019+0000", 2024-09-21T22:07:34.301 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.014276577 2024-09-21T22:07:34.301 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.301 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.301 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:34.301 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.430059+0000", 2024-09-21T22:07:34.301 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9966999999999999e-05 2024-09-21T22:07:34.301 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.301 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.301 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:34.301 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.430093+0000", 2024-09-21T22:07:34.302 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3739999999999999e-05 2024-09-21T22:07:34.302 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:34.302 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:34.302 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:34.302 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.302 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.302 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4840.0:706 2.5 2:a70ae99f:::rbd_data.126483d75f2d.000000000000000c:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:07:34.302 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:25.476388+0000", 2024-09-21T22:07:34.302 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.7365100689999995, 2024-09-21T22:07:34.302 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.031232883999999999, 2024-09-21T22:07:34.302 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:34.302 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:34.302 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:34.302 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:34.303 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4840", 2024-09-21T22:07:34.303 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:34.303 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 706 2024-09-21T22:07:34.303 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.303 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:34.303 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.303 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:34.303 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.476388+0000", 2024-09-21T22:07:34.303 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:34.303 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.303 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.303 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:34.303 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.476388+0000", 2024-09-21T22:07:34.303 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:34.304 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.304 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.304 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:34.304 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.476390+0000", 2024-09-21T22:07:34.304 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.133e-06 2024-09-21T22:07:34.304 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.304 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.304 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:34.304 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.477229+0000", 2024-09-21T22:07:34.304 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00083948499999999995 2024-09-21T22:07:34.304 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.304 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.304 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:34.304 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.477242+0000", 2024-09-21T22:07:34.304 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3101999999999999e-05 2024-09-21T22:07:34.305 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.305 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.305 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:34.305 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.477248+0000", 2024-09-21T22:07:34.305 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3110000000000002e-06 2024-09-21T22:07:34.305 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.305 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.305 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:34.305 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.477313+0000", 2024-09-21T22:07:34.305 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4375999999999995e-05 2024-09-21T22:07:34.305 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.305 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.305 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:34.305 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.477374+0000", 2024-09-21T22:07:34.306 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1498e-05 2024-09-21T22:07:34.306 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.306 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.306 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:34.306 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.477463+0000", 2024-09-21T22:07:34.306 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9005000000000006e-05 2024-09-21T22:07:34.306 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.306 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.306 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:34.306 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.499921+0000", 2024-09-21T22:07:34.306 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022457969000000001 2024-09-21T22:07:34.306 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.306 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.306 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:34.306 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.507525+0000", 2024-09-21T22:07:34.307 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0076035 2024-09-21T22:07:34.307 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.307 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.307 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:34.307 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.507581+0000", 2024-09-21T22:07:34.307 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6127e-05 2024-09-21T22:07:34.307 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.307 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.307 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:34.307 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.507621+0000", 2024-09-21T22:07:34.307 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.0377999999999999e-05 2024-09-21T22:07:34.307 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:34.307 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:34.307 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:34.308 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.308 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.308 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4840.0:725 2.0 2:08b4f96a:::rbd_data.126483d75f2d.000000000000000f:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:07:34.308 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:25.936440+0000", 2024-09-21T22:07:34.308 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.2764584719999998, 2024-09-21T22:07:34.308 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.049962899999999998, 2024-09-21T22:07:34.308 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:34.308 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:34.308 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:34.308 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:34.308 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4840", 2024-09-21T22:07:34.308 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:34.308 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 725 2024-09-21T22:07:34.308 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.309 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:34.309 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.309 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:34.309 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.936440+0000", 2024-09-21T22:07:34.309 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:34.309 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.309 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.309 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:34.309 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.936440+0000", 2024-09-21T22:07:34.309 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:34.309 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.309 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.309 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:34.309 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.936442+0000", 2024-09-21T22:07:34.309 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0360000000000001e-06 2024-09-21T22:07:34.309 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.310 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.310 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:34.310 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.937560+0000", 2024-09-21T22:07:34.310 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011176300000000001 2024-09-21T22:07:34.310 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.310 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.310 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:34.310 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.937592+0000", 2024-09-21T22:07:34.310 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2743000000000001e-05 2024-09-21T22:07:34.310 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.310 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.310 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:34.310 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.937608+0000", 2024-09-21T22:07:34.311 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5477000000000001e-05 2024-09-21T22:07:34.311 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.311 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.311 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:34.311 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.937695+0000", 2024-09-21T22:07:34.311 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7220999999999997e-05 2024-09-21T22:07:34.311 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.311 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.311 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:34.311 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.937792+0000", 2024-09-21T22:07:34.311 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7261000000000003e-05 2024-09-21T22:07:34.311 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.311 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.311 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-09-21T22:07:34.311 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.937934+0000", 2024-09-21T22:07:34.312 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000141376 2024-09-21T22:07:34.312 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.312 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.312 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:34.312 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.975222+0000", 2024-09-21T22:07:34.312 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.037287995999999997 2024-09-21T22:07:34.312 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.312 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.312 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:34.312 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.986285+0000", 2024-09-21T22:07:34.312 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011063329 2024-09-21T22:07:34.312 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.312 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.312 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:34.312 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.986348+0000", 2024-09-21T22:07:34.313 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3330000000000005e-05 2024-09-21T22:07:34.313 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.313 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.313 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:34.313 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.986403+0000", 2024-09-21T22:07:34.313 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4500999999999999e-05 2024-09-21T22:07:34.313 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:34.313 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:34.313 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:34.313 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.313 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.313 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4840.0:727 2.0 2:08b4f96a:::rbd_data.126483d75f2d.000000000000000f:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:07:34.313 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:25.998176+0000", 2024-09-21T22:07:34.313 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.2147222749999997, 2024-09-21T22:07:34.313 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.054236470000000002, 2024-09-21T22:07:34.314 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:34.314 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:34.314 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:34.314 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:34.314 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4840", 2024-09-21T22:07:34.314 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:34.314 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 727 2024-09-21T22:07:34.314 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.314 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:34.314 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.314 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:34.314 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.998176+0000", 2024-09-21T22:07:34.314 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:34.314 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.315 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.315 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:34.315 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.998176+0000", 2024-09-21T22:07:34.315 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:34.315 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.315 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.315 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:34.315 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.998179+0000", 2024-09-21T22:07:34.315 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3870000000000002e-06 2024-09-21T22:07:34.315 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.315 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.315 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:34.315 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.999060+0000", 2024-09-21T22:07:34.315 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00088148800000000004 2024-09-21T22:07:34.316 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.316 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.316 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:34.316 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.999094+0000", 2024-09-21T22:07:34.316 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3779000000000003e-05 2024-09-21T22:07:34.316 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.316 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.316 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:34.316 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.999111+0000", 2024-09-21T22:07:34.316 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6990000000000002e-05 2024-09-21T22:07:34.316 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.316 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.316 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:34.316 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.999190+0000", 2024-09-21T22:07:34.316 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9354e-05 2024-09-21T22:07:34.317 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.317 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.317 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:34.317 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.999251+0000", 2024-09-21T22:07:34.317 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0733000000000003e-05 2024-09-21T22:07:34.317 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.317 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.317 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-09-21T22:07:34.317 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.999342+0000", 2024-09-21T22:07:34.317 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1376000000000001e-05 2024-09-21T22:07:34.317 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.317 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.317 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:34.318 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.032392+0000", 2024-09-21T22:07:34.318 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.033049626999999998 2024-09-21T22:07:34.318 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.318 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.318 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:34.318 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.052249+0000", 2024-09-21T22:07:34.318 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019856944000000001 2024-09-21T22:07:34.318 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.318 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.318 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:34.318 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.052342+0000", 2024-09-21T22:07:34.318 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2650000000000002e-05 2024-09-21T22:07:34.318 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.318 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.318 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:34.319 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.052413+0000", 2024-09-21T22:07:34.319 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1142000000000004e-05 2024-09-21T22:07:34.319 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:34.319 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:34.319 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:34.319 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.319 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.319 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4840.0:729 2.0 2:08b4f96a:::rbd_data.126483d75f2d.000000000000000f:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:07:34.319 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:26.062012+0000", 2024-09-21T22:07:34.319 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.1508862840000003, 2024-09-21T22:07:34.319 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.060656823999999998, 2024-09-21T22:07:34.319 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:34.319 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:34.319 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:34.319 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:34.320 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4840", 2024-09-21T22:07:34.320 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:34.320 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 729 2024-09-21T22:07:34.320 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.320 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:34.320 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.320 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:34.320 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.062012+0000", 2024-09-21T22:07:34.320 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:34.320 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.320 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.320 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:34.320 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.062012+0000", 2024-09-21T22:07:34.320 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:34.320 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.321 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.321 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:34.321 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.062013+0000", 2024-09-21T22:07:34.321 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.243e-06 2024-09-21T22:07:34.321 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.321 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.321 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:34.321 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.062827+0000", 2024-09-21T22:07:34.321 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00081374799999999999 2024-09-21T22:07:34.321 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.321 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.321 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:34.321 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.062845+0000", 2024-09-21T22:07:34.321 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.808e-05 2024-09-21T22:07:34.322 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.322 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.322 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:34.322 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.062854+0000", 2024-09-21T22:07:34.322 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1039999999999996e-06 2024-09-21T22:07:34.322 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.322 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.322 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:34.322 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.062970+0000", 2024-09-21T22:07:34.322 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011601 2024-09-21T22:07:34.322 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.322 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.322 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:34.322 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.063230+0000", 2024-09-21T22:07:34.323 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025918799999999999 2024-09-21T22:07:34.323 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.323 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.323 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-09-21T22:07:34.323 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.063641+0000", 2024-09-21T22:07:34.323 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00041152799999999998 2024-09-21T22:07:34.323 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.323 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.323 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:34.323 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.122015+0000", 2024-09-21T22:07:34.323 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058374246999999997 2024-09-21T22:07:34.323 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.323 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.323 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:34.323 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.122451+0000", 2024-09-21T22:07:34.324 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000435771 2024-09-21T22:07:34.324 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.324 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.324 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:34.324 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.122576+0000", 2024-09-21T22:07:34.324 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012469099999999999 2024-09-21T22:07:34.324 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.324 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.324 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:34.324 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.122669+0000", 2024-09-21T22:07:34.324 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3214000000000006e-05 2024-09-21T22:07:34.324 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:34.324 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:34.324 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:34.324 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.325 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.325 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4840.0:731 2.0 2:08b4f96a:::rbd_data.126483d75f2d.000000000000000f:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:07:34.325 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:26.135603+0000", 2024-09-21T22:07:34.325 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.0772957840000004, 2024-09-21T22:07:34.325 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.056838474999999999, 2024-09-21T22:07:34.325 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:34.325 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:34.325 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:34.325 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:34.325 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4840", 2024-09-21T22:07:34.325 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:34.325 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 731 2024-09-21T22:07:34.325 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.325 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:34.326 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.326 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:34.326 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.135603+0000", 2024-09-21T22:07:34.326 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:34.326 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.326 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.326 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:34.326 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.135603+0000", 2024-09-21T22:07:34.326 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:34.326 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.326 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.326 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:34.326 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.135605+0000", 2024-09-21T22:07:34.326 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9700000000000002e-06 2024-09-21T22:07:34.327 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.327 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.327 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:34.327 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.136554+0000", 2024-09-21T22:07:34.327 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00094886700000000005 2024-09-21T22:07:34.327 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.327 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.327 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:34.327 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.136582+0000", 2024-09-21T22:07:34.327 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8393000000000001e-05 2024-09-21T22:07:34.327 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.327 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.327 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:34.327 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.136596+0000", 2024-09-21T22:07:34.328 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.43e-05 2024-09-21T22:07:34.328 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.328 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.328 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:34.328 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.136691+0000", 2024-09-21T22:07:34.328 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4843e-05 2024-09-21T22:07:34.328 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.328 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.328 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:34.328 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.136873+0000", 2024-09-21T22:07:34.328 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018237099999999999 2024-09-21T22:07:34.328 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.329 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.329 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-09-21T22:07:34.329 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.137140+0000", 2024-09-21T22:07:34.329 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026642700000000002 2024-09-21T22:07:34.329 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.329 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.329 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:34.329 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.180580+0000", 2024-09-21T22:07:34.329 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.043440078 2024-09-21T22:07:34.329 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.329 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.329 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:34.329 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.192175+0000", 2024-09-21T22:07:34.329 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011594959 2024-09-21T22:07:34.330 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.330 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.330 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:34.330 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.192353+0000", 2024-09-21T22:07:34.330 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017850200000000001 2024-09-21T22:07:34.330 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.330 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.330 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:34.330 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.192441+0000", 2024-09-21T22:07:34.330 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7764999999999997e-05 2024-09-21T22:07:34.330 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:34.330 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:34.330 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:34.330 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:34.331 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:34.331 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:07:34.331 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T22:07:34.332 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-21T22:07:34.332 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:34.266+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:34.376 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:34.374+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:34.464 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:07:34.464 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:07:34.464 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-09-21T22:07:34.464 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:07:34.472 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:07:34.468+0000 7f2075a06640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31720) UID: 0 2024-09-21T22:07:34.479 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T22:07:34.573 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:34.570+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:34.611 INFO:tasks.workunit.client.0.smithi157.stderr:++ _sudo rbd --device-type nbd map --read-only rbd/testrbdnbd21553 2024-09-21T22:07:34.611 INFO:tasks.workunit.client.0.smithi157.stderr:++ local cmd 2024-09-21T22:07:34.611 INFO:tasks.workunit.client.0.smithi157.stderr:+++ id -u 2024-09-21T22:07:34.613 INFO:tasks.workunit.client.0.smithi157.stderr:++ '[' 1000 -eq 0 ']' 2024-09-21T22:07:34.614 INFO:tasks.workunit.client.0.smithi157.stderr:+++ which rbd 2024-09-21T22:07:34.616 INFO:tasks.workunit.client.0.smithi157.stderr:++ cmd=/usr/bin/rbd 2024-09-21T22:07:34.616 INFO:tasks.workunit.client.0.smithi157.stderr:++ shift 2024-09-21T22:07:34.616 INFO:tasks.workunit.client.0.smithi157.stderr:++ sudo -nE /usr/bin/rbd --device-type nbd map --read-only rbd/testrbdnbd21553 2024-09-21T22:07:34.648 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:07:34.648 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:07:34.648 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-21T22:07:34.648 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-21T22:07:34.648 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:07:34.662 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T22:07:34.673 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:34.670+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:34.759 INFO:tasks.workunit.client.0.smithi157.stderr:+ DEV=/dev/nbd0 2024-09-21T22:07:34.760 INFO:tasks.workunit.client.0.smithi157.stderr:++ rbd device --device-type nbd list 2024-09-21T22:07:34.760 INFO:tasks.workunit.client.0.smithi157.stderr:++ awk -v pool=rbd -v img=testrbdnbd21553 -v dev=/dev/nbd0 '$2 == pool && $3 == img && $5 == dev {print $1}' 2024-09-21T22:07:34.774 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:07:34.768+0000 7f2075a06640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31720) UID: 0 2024-09-21T22:07:34.797 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:07:34.797 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-21T22:07:34.797 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-21T22:07:34.797 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-21T22:07:34.797 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.798 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4887.0:1 3.3 3:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-21T22:07:34.798 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:17.151514+0000", 2024-09-21T22:07:34.798 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 17.643940521000001, 2024-09-21T22:07:34.798 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.01157512, 2024-09-21T22:07:34.798 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:34.798 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:34.798 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:34.798 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:34.798 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4887", 2024-09-21T22:07:34.798 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/3431621434", 2024-09-21T22:07:34.799 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1 2024-09-21T22:07:34.799 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.799 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:34.799 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.799 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:34.799 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.151514+0000", 2024-09-21T22:07:34.799 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:34.799 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.799 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.799 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:34.799 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.151514+0000", 2024-09-21T22:07:34.799 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:34.800 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.800 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.800 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:34.800 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.151515+0000", 2024-09-21T22:07:34.800 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.469e-06 2024-09-21T22:07:34.800 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.800 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.800 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:34.800 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.151522+0000", 2024-09-21T22:07:34.800 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1330000000000001e-06 2024-09-21T22:07:34.800 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.801 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.801 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:34.801 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.151536+0000", 2024-09-21T22:07:34.801 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3563e-05 2024-09-21T22:07:34.801 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.801 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.801 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:34.801 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.151574+0000", 2024-09-21T22:07:34.801 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8467999999999998e-05 2024-09-21T22:07:34.801 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.801 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.801 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:34.802 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.151665+0000", 2024-09-21T22:07:34.802 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0668000000000007e-05 2024-09-21T22:07:34.802 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.802 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.802 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-21T22:07:34.802 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.151676+0000", 2024-09-21T22:07:34.802 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1443999999999999e-05 2024-09-21T22:07:34.802 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.802 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.802 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:34.802 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.161694+0000", 2024-09-21T22:07:34.803 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010017402999999999 2024-09-21T22:07:34.803 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.803 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.803 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:34.803 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.161805+0000", 2024-09-21T22:07:34.803 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011122500000000001 2024-09-21T22:07:34.803 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.803 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.803 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:34.803 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.161937+0000", 2024-09-21T22:07:34.803 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013177299999999999 2024-09-21T22:07:34.803 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.804 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.804 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:34.804 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.162555+0000", 2024-09-21T22:07:34.804 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00061786199999999995 2024-09-21T22:07:34.804 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.804 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.804 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:34.804 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.163021+0000", 2024-09-21T22:07:34.804 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000465749 2024-09-21T22:07:34.804 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.805 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.805 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:34.805 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.163059+0000", 2024-09-21T22:07:34.805 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8238000000000002e-05 2024-09-21T22:07:34.805 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.805 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.805 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:34.805 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.163089+0000", 2024-09-21T22:07:34.805 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0125e-05 2024-09-21T22:07:34.805 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:34.805 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:34.806 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:34.806 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.806 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.806 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4887.0:1 3.3 e59/58 ondisk, result = 0)", 2024-09-21T22:07:34.806 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:17.162939+0000", 2024-09-21T22:07:34.806 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 17.632515130000002, 2024-09-21T22:07:34.806 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015730900000000001, 2024-09-21T22:07:34.806 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:34.806 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:34.806 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:07:34.806 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:34.806 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.807 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:34.807 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.162939+0000", 2024-09-21T22:07:34.807 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:34.807 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.807 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.807 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:34.807 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.162939+0000", 2024-09-21T22:07:34.807 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:34.807 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.807 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.807 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:34.808 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.162940+0000", 2024-09-21T22:07:34.808 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4499999999999996e-07 2024-09-21T22:07:34.808 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.808 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.808 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:34.808 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.162945+0000", 2024-09-21T22:07:34.808 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.758e-06 2024-09-21T22:07:34.808 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.808 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.808 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:34.808 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.162957+0000", 2024-09-21T22:07:34.808 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2259000000000001e-05 2024-09-21T22:07:34.809 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.809 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.809 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:34.809 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.162966+0000", 2024-09-21T22:07:34.809 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3270000000000007e-06 2024-09-21T22:07:34.809 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.809 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.809 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:34.809 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.163010+0000", 2024-09-21T22:07:34.809 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.422e-05 2024-09-21T22:07:34.809 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.809 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.810 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:34.810 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.163016+0000", 2024-09-21T22:07:34.810 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8579999999999996e-06 2024-09-21T22:07:34.810 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.810 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.810 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:34.810 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.163096+0000", 2024-09-21T22:07:34.810 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0142000000000006e-05 2024-09-21T22:07:34.810 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:34.810 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:34.810 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:34.811 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.811 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.811 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4840.0:675 2.7 e63/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 63'56, mlcod=63'54)", 2024-09-21T22:07:34.811 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.618817+0000", 2024-09-21T22:07:34.811 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.176637733, 2024-09-21T22:07:34.811 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0060329809999999998, 2024-09-21T22:07:34.811 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:34.811 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:34.811 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:34.811 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:34.811 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.811 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:34.812 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.618817+0000", 2024-09-21T22:07:34.812 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:34.812 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.812 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.812 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:34.812 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.618817+0000", 2024-09-21T22:07:34.812 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:34.812 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.812 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.812 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:34.812 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.618819+0000", 2024-09-21T22:07:34.812 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3070000000000001e-06 2024-09-21T22:07:34.813 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.813 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.813 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:34.813 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.618934+0000", 2024-09-21T22:07:34.813 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000114669 2024-09-21T22:07:34.813 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.813 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.813 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:34.813 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.618969+0000", 2024-09-21T22:07:34.813 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5985999999999997e-05 2024-09-21T22:07:34.813 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.814 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.814 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:34.814 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.618985+0000", 2024-09-21T22:07:34.814 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5840000000000001e-05 2024-09-21T22:07:34.814 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.814 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.814 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:34.814 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.619101+0000", 2024-09-21T22:07:34.814 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000115591 2024-09-21T22:07:34.814 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.814 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.815 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:34.815 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.619136+0000", 2024-09-21T22:07:34.815 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5287000000000003e-05 2024-09-21T22:07:34.815 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.815 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.815 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:34.815 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.624766+0000", 2024-09-21T22:07:34.815 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0056301500000000004 2024-09-21T22:07:34.815 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.815 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.815 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:34.815 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.624850+0000", 2024-09-21T22:07:34.816 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3151000000000004e-05 2024-09-21T22:07:34.816 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:34.816 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:34.816 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:34.816 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.816 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.816 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4840.0:676 2.7 e63/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 63'57, mlcod=63'55)", 2024-09-21T22:07:34.816 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.627276+0000", 2024-09-21T22:07:34.816 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.168177973000001, 2024-09-21T22:07:34.816 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.037500787000000001, 2024-09-21T22:07:34.816 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:34.816 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:34.817 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:34.817 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:34.817 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.817 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:34.817 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.627276+0000", 2024-09-21T22:07:34.817 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:34.817 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.817 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.817 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:34.817 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.627276+0000", 2024-09-21T22:07:34.817 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:34.817 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.818 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.818 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:34.818 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.627277+0000", 2024-09-21T22:07:34.818 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.068e-06 2024-09-21T22:07:34.818 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.818 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.818 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:34.818 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.628148+0000", 2024-09-21T22:07:34.818 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00087069999999999997 2024-09-21T22:07:34.818 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.818 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.818 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:34.819 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.628170+0000", 2024-09-21T22:07:34.819 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2147999999999999e-05 2024-09-21T22:07:34.819 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.819 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.819 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:34.819 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.628179+0000", 2024-09-21T22:07:34.819 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2450000000000004e-06 2024-09-21T22:07:34.819 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.819 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.819 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:34.819 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.628279+0000", 2024-09-21T22:07:34.819 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9298999999999999e-05 2024-09-21T22:07:34.820 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.820 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.820 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:34.820 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.628305+0000", 2024-09-21T22:07:34.820 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6316999999999999e-05 2024-09-21T22:07:34.820 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.820 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.820 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:34.820 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.664714+0000", 2024-09-21T22:07:34.820 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.036408923000000003 2024-09-21T22:07:34.820 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.820 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.820 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:34.821 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.664777+0000", 2024-09-21T22:07:34.821 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3087000000000001e-05 2024-09-21T22:07:34.821 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:34.821 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:34.821 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:34.821 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.821 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.821 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4840.0:677 2.7 e63/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 63'58, mlcod=63'56)", 2024-09-21T22:07:34.821 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.667210+0000", 2024-09-21T22:07:34.821 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.128244734000001, 2024-09-21T22:07:34.821 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010046522, 2024-09-21T22:07:34.821 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:34.822 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:34.822 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:34.822 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:34.822 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.822 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:34.822 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.667210+0000", 2024-09-21T22:07:34.822 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:34.822 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.822 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.822 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:34.822 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.667210+0000", 2024-09-21T22:07:34.822 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:34.823 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.823 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.823 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:34.823 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.667211+0000", 2024-09-21T22:07:34.823 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3379999999999999e-06 2024-09-21T22:07:34.823 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.823 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.823 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:34.823 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.667272+0000", 2024-09-21T22:07:34.823 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1257e-05 2024-09-21T22:07:34.823 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.824 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.824 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:34.824 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.667295+0000", 2024-09-21T22:07:34.824 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2637999999999999e-05 2024-09-21T22:07:34.824 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.824 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.824 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:34.824 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.667305+0000", 2024-09-21T22:07:34.824 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0044000000000001e-05 2024-09-21T22:07:34.824 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.824 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.824 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:34.825 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.667425+0000", 2024-09-21T22:07:34.825 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000120605 2024-09-21T22:07:34.825 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.825 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.825 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:34.825 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.667476+0000", 2024-09-21T22:07:34.825 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0843000000000001e-05 2024-09-21T22:07:34.825 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.825 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.826 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:34.826 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.677180+0000", 2024-09-21T22:07:34.826 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0097035860000000002 2024-09-21T22:07:34.826 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.826 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.826 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:34.826 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.677256+0000", 2024-09-21T22:07:34.826 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6211000000000006e-05 2024-09-21T22:07:34.826 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:34.826 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:34.826 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:34.827 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.827 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.827 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4840.0:678 2.7 e63/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 63'59, mlcod=63'57)", 2024-09-21T22:07:34.827 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.679689+0000", 2024-09-21T22:07:34.827 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.115765473, 2024-09-21T22:07:34.827 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053308406000000003, 2024-09-21T22:07:34.827 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:34.827 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:34.827 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:34.827 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:34.827 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.828 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:34.828 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.679689+0000", 2024-09-21T22:07:34.828 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:34.828 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.828 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.828 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:34.828 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.679689+0000", 2024-09-21T22:07:34.828 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:34.828 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.828 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.828 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:34.829 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.679691+0000", 2024-09-21T22:07:34.829 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9400000000000001e-06 2024-09-21T22:07:34.829 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.829 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.829 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:34.829 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.680245+0000", 2024-09-21T22:07:34.829 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00055382999999999995 2024-09-21T22:07:34.829 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.829 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.829 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:34.829 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.680276+0000", 2024-09-21T22:07:34.829 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1123e-05 2024-09-21T22:07:34.830 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.830 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.830 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:34.830 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.680290+0000", 2024-09-21T22:07:34.830 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4056999999999999e-05 2024-09-21T22:07:34.830 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.830 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.830 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:34.830 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.680418+0000", 2024-09-21T22:07:34.830 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000127792 2024-09-21T22:07:34.830 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.831 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.831 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:34.831 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.680465+0000", 2024-09-21T22:07:34.831 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7360000000000001e-05 2024-09-21T22:07:34.831 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.831 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.831 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:34.831 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.732923+0000", 2024-09-21T22:07:34.831 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.052458312999999999 2024-09-21T22:07:34.831 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.831 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.831 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:34.832 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.732997+0000", 2024-09-21T22:07:34.832 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3991000000000004e-05 2024-09-21T22:07:34.832 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:34.832 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:34.832 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:34.832 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.832 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.832 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4840.0:679 2.7 e63/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 63'60, mlcod=63'58)", 2024-09-21T22:07:34.832 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.735615+0000", 2024-09-21T22:07:34.832 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.05983932, 2024-09-21T22:07:34.832 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0077878399999999999, 2024-09-21T22:07:34.832 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:34.833 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:34.833 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:34.833 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:34.833 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.833 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:34.833 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.735615+0000", 2024-09-21T22:07:34.833 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:34.833 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.833 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.833 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:34.833 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.735615+0000", 2024-09-21T22:07:34.833 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:34.833 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.834 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.834 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:34.834 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.735617+0000", 2024-09-21T22:07:34.834 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7999999999999999e-06 2024-09-21T22:07:34.834 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.834 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.834 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:34.834 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.735706+0000", 2024-09-21T22:07:34.834 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8839999999999996e-05 2024-09-21T22:07:34.834 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.834 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.834 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:34.835 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.735736+0000", 2024-09-21T22:07:34.835 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9955999999999999e-05 2024-09-21T22:07:34.835 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.835 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.835 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:34.835 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.735755+0000", 2024-09-21T22:07:34.835 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9283000000000002e-05 2024-09-21T22:07:34.835 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.835 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.835 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:34.835 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.735857+0000", 2024-09-21T22:07:34.835 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000101681 2024-09-21T22:07:34.836 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.836 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.836 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:34.836 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.735894+0000", 2024-09-21T22:07:34.836 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7487999999999998e-05 2024-09-21T22:07:34.836 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.836 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.836 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:34.836 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.743321+0000", 2024-09-21T22:07:34.836 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0074270050000000004 2024-09-21T22:07:34.836 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.836 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.837 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:34.837 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.743403+0000", 2024-09-21T22:07:34.837 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1786999999999995e-05 2024-09-21T22:07:34.837 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:34.837 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:34.837 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:34.837 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.837 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.837 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4840.0:680 2.7 e63/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 63'61, mlcod=63'59)", 2024-09-21T22:07:34.837 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.745996+0000", 2024-09-21T22:07:34.837 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.04945796, 2024-09-21T22:07:34.837 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.061804132999999997, 2024-09-21T22:07:34.838 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:34.838 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:34.838 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:34.838 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:34.838 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.838 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:34.838 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.745996+0000", 2024-09-21T22:07:34.838 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:34.838 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.838 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.838 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:34.838 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.745996+0000", 2024-09-21T22:07:34.839 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:34.839 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.839 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.839 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:34.839 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.745998+0000", 2024-09-21T22:07:34.839 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.717e-06 2024-09-21T22:07:34.839 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.839 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.839 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:34.839 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.746523+0000", 2024-09-21T22:07:34.839 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00052525399999999996 2024-09-21T22:07:34.839 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.840 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.840 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:34.840 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.746554+0000", 2024-09-21T22:07:34.840 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0586000000000002e-05 2024-09-21T22:07:34.840 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.840 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.840 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:34.840 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.746577+0000", 2024-09-21T22:07:34.840 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3079e-05 2024-09-21T22:07:34.840 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.840 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.840 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:34.841 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.746689+0000", 2024-09-21T22:07:34.841 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011177 2024-09-21T22:07:34.841 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.841 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.841 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:34.841 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.746722+0000", 2024-09-21T22:07:34.841 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2870000000000002e-05 2024-09-21T22:07:34.841 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.841 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.841 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:34.841 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.807740+0000", 2024-09-21T22:07:34.841 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.061018612 2024-09-21T22:07:34.842 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.842 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.842 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:34.842 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.807800+0000", 2024-09-21T22:07:34.842 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0244999999999997e-05 2024-09-21T22:07:34.842 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:34.842 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:34.842 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:34.842 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.842 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.842 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4840.0:681 2.7 e63/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 63'62, mlcod=63'60)", 2024-09-21T22:07:34.843 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.810646+0000", 2024-09-21T22:07:34.843 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.9848085359999992, 2024-09-21T22:07:34.843 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010715685000000001, 2024-09-21T22:07:34.843 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:34.843 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:34.843 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:34.843 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:34.843 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.843 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:34.843 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.810646+0000", 2024-09-21T22:07:34.844 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:34.844 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.844 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.844 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:34.844 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.810646+0000", 2024-09-21T22:07:34.844 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:34.844 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.844 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.845 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:34.845 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.810649+0000", 2024-09-21T22:07:34.845 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8569999999999999e-06 2024-09-21T22:07:34.845 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.845 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.845 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:34.845 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.810780+0000", 2024-09-21T22:07:34.845 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013114600000000001 2024-09-21T22:07:34.845 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.845 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.845 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:34.846 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.810819+0000", 2024-09-21T22:07:34.846 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9363000000000002e-05 2024-09-21T22:07:34.846 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.846 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.846 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:34.846 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.810839+0000", 2024-09-21T22:07:34.846 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9865999999999999e-05 2024-09-21T22:07:34.846 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.846 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.846 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:34.846 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.810992+0000", 2024-09-21T22:07:34.846 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015333299999999999 2024-09-21T22:07:34.847 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.847 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.847 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:34.847 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.811037+0000", 2024-09-21T22:07:34.847 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.464e-05 2024-09-21T22:07:34.847 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.847 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.847 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:34.847 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.821280+0000", 2024-09-21T22:07:34.847 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010243254 2024-09-21T22:07:34.847 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.847 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.848 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:34.848 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.821361+0000", 2024-09-21T22:07:34.848 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1225999999999998e-05 2024-09-21T22:07:34.848 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:34.848 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:34.848 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:34.848 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.848 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.848 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4840.0:682 2.7 e63/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 63'63, mlcod=63'61)", 2024-09-21T22:07:34.848 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.824126+0000", 2024-09-21T22:07:34.848 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.9713283910000001, 2024-09-21T22:07:34.849 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.038159327, 2024-09-21T22:07:34.849 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:34.849 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:34.849 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:34.849 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:34.849 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.849 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:34.849 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.824126+0000", 2024-09-21T22:07:34.849 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:34.849 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.849 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.849 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:34.850 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.824126+0000", 2024-09-21T22:07:34.850 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:34.850 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.850 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.850 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:34.850 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.824128+0000", 2024-09-21T22:07:34.850 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9659999999999999e-06 2024-09-21T22:07:34.850 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.850 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.850 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:34.850 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.824842+0000", 2024-09-21T22:07:34.851 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00071423999999999999 2024-09-21T22:07:34.851 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.851 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.851 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:34.851 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.824883+0000", 2024-09-21T22:07:34.851 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1131999999999999e-05 2024-09-21T22:07:34.851 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.851 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.851 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:34.851 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.824931+0000", 2024-09-21T22:07:34.851 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7327000000000003e-05 2024-09-21T22:07:34.851 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.852 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.852 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:34.852 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.825058+0000", 2024-09-21T22:07:34.852 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012790499999999999 2024-09-21T22:07:34.852 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.852 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.852 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:34.852 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.825087+0000", 2024-09-21T22:07:34.852 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8192e-05 2024-09-21T22:07:34.852 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.852 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.852 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:34.853 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.862225+0000", 2024-09-21T22:07:34.853 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.037138309000000001 2024-09-21T22:07:34.853 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.853 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.853 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:34.853 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.862285+0000", 2024-09-21T22:07:34.853 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0256e-05 2024-09-21T22:07:34.853 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:34.853 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:34.853 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:34.853 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.853 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.854 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4963.0:15 2.7 e68/16 2:f45e3993:::rbd_object_map.126483d75f2d:head v 68'102, mlcod=68'100)", 2024-09-21T22:07:34.854 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:33.835091+0000", 2024-09-21T22:07:34.854 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.96036326699999996, 2024-09-21T22:07:34.854 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000910661, 2024-09-21T22:07:34.854 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:34.854 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:34.854 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:34.854 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:34.854 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.854 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:34.854 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.835091+0000", 2024-09-21T22:07:34.854 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:34.855 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.855 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.855 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:34.855 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.835091+0000", 2024-09-21T22:07:34.855 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:34.855 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.855 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.855 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:34.855 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.835093+0000", 2024-09-21T22:07:34.855 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7889999999999999e-06 2024-09-21T22:07:34.855 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.856 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.856 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:34.856 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.835103+0000", 2024-09-21T22:07:34.856 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0461e-05 2024-09-21T22:07:34.856 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.856 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.856 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:34.856 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.835125+0000", 2024-09-21T22:07:34.856 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1593000000000001e-05 2024-09-21T22:07:34.856 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.856 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.856 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:34.856 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.835145+0000", 2024-09-21T22:07:34.857 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0397000000000001e-05 2024-09-21T22:07:34.857 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.857 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.857 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:34.857 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.835221+0000", 2024-09-21T22:07:34.857 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5473000000000001e-05 2024-09-21T22:07:34.857 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.857 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.857 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:34.857 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.835245+0000", 2024-09-21T22:07:34.857 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4256e-05 2024-09-21T22:07:34.857 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.858 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.858 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:34.858 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.835948+0000", 2024-09-21T22:07:34.858 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00070343499999999998 2024-09-21T22:07:34.858 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.858 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.858 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:34.858 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.836002+0000", 2024-09-21T22:07:34.858 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3257000000000002e-05 2024-09-21T22:07:34.858 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:34.858 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:34.858 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:34.859 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.859 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.859 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4963.0:18 2.7 e68/16 2:f45e3993:::rbd_object_map.126483d75f2d:head v 68'103, mlcod=68'101)", 2024-09-21T22:07:34.859 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:33.837758+0000", 2024-09-21T22:07:34.859 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.95769608900000003, 2024-09-21T22:07:34.859 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0014775649999999999, 2024-09-21T22:07:34.859 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:34.859 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:34.859 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:34.859 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:34.859 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.859 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:34.860 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.837758+0000", 2024-09-21T22:07:34.860 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:34.860 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.860 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.860 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:34.860 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.837758+0000", 2024-09-21T22:07:34.860 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:34.860 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.860 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.860 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:34.860 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.837759+0000", 2024-09-21T22:07:34.860 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7199999999999997e-07 2024-09-21T22:07:34.861 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.861 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.861 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:34.861 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.837766+0000", 2024-09-21T22:07:34.861 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5969999999999997e-06 2024-09-21T22:07:34.861 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.861 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.861 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:34.861 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.837781+0000", 2024-09-21T22:07:34.861 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5602000000000001e-05 2024-09-21T22:07:34.861 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.861 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.862 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:34.862 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.837793+0000", 2024-09-21T22:07:34.862 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1816e-05 2024-09-21T22:07:34.862 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.862 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.862 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:34.862 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.837922+0000", 2024-09-21T22:07:34.862 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012941600000000001 2024-09-21T22:07:34.862 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.862 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.862 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:34.862 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.837953+0000", 2024-09-21T22:07:34.863 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0729999999999999e-05 2024-09-21T22:07:34.863 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.863 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.863 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:34.863 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.839185+0000", 2024-09-21T22:07:34.863 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0012320059999999999 2024-09-21T22:07:34.863 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.863 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.863 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:34.863 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.839236+0000", 2024-09-21T22:07:34.863 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0525999999999997e-05 2024-09-21T22:07:34.863 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:34.864 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:34.864 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:34.864 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.864 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.864 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4963.0:23 2.7 e68/16 2:f45e3993:::rbd_object_map.126483d75f2d:head v 68'104, mlcod=68'102)", 2024-09-21T22:07:34.864 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:33.844296+0000", 2024-09-21T22:07:34.864 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.95115781899999996, 2024-09-21T22:07:34.864 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0010385679999999999, 2024-09-21T22:07:34.864 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:34.864 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:34.864 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:34.864 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:34.865 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.865 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:34.865 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.844296+0000", 2024-09-21T22:07:34.865 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:34.865 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.865 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.865 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:34.865 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.844296+0000", 2024-09-21T22:07:34.865 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:34.865 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.865 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.866 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:34.866 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.844298+0000", 2024-09-21T22:07:34.866 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3230000000000001e-06 2024-09-21T22:07:34.866 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.866 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.866 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:34.866 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.844307+0000", 2024-09-21T22:07:34.866 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4350000000000003e-06 2024-09-21T22:07:34.866 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.866 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.866 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:34.866 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.844327+0000", 2024-09-21T22:07:34.867 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0136000000000002e-05 2024-09-21T22:07:34.867 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.867 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.867 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:34.867 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.844338+0000", 2024-09-21T22:07:34.867 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0474e-05 2024-09-21T22:07:34.867 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.867 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.867 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:34.867 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.844460+0000", 2024-09-21T22:07:34.867 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012226399999999999 2024-09-21T22:07:34.867 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.868 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.868 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:34.868 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.844488+0000", 2024-09-21T22:07:34.868 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7965000000000001e-05 2024-09-21T22:07:34.868 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.868 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.868 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:34.868 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.845290+0000", 2024-09-21T22:07:34.868 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00080154599999999999 2024-09-21T22:07:34.868 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.868 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.868 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:34.868 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.845335+0000", 2024-09-21T22:07:34.869 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5425e-05 2024-09-21T22:07:34.869 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:34.869 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:34.869 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:34.869 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.869 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.869 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4981.0:15 2.7 e68/16 2:f45e3993:::rbd_object_map.126483d75f2d:head v 68'105, mlcod=68'103)", 2024-09-21T22:07:34.869 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:33.956875+0000", 2024-09-21T22:07:34.869 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.838579509, 2024-09-21T22:07:34.869 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00093627499999999998, 2024-09-21T22:07:34.869 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:34.870 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:34.870 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:34.870 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:34.870 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.870 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:34.870 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.956875+0000", 2024-09-21T22:07:34.870 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:34.870 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.870 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.870 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:34.870 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.956875+0000", 2024-09-21T22:07:34.870 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:34.871 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.871 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.871 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:34.871 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.956876+0000", 2024-09-21T22:07:34.871 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9600000000000008e-07 2024-09-21T22:07:34.871 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.871 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.871 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:34.871 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.956884+0000", 2024-09-21T22:07:34.871 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7919999999999993e-06 2024-09-21T22:07:34.871 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.872 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.872 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:34.872 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.956899+0000", 2024-09-21T22:07:34.872 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5339999999999999e-05 2024-09-21T22:07:34.872 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.872 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.872 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:34.872 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.956906+0000", 2024-09-21T22:07:34.872 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1509999999999998e-06 2024-09-21T22:07:34.872 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.872 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.873 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:34.873 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.956985+0000", 2024-09-21T22:07:34.873 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8807999999999995e-05 2024-09-21T22:07:34.873 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.873 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.873 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:34.873 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.957004+0000", 2024-09-21T22:07:34.873 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9213000000000001e-05 2024-09-21T22:07:34.873 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.873 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.873 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:34.874 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.957781+0000", 2024-09-21T22:07:34.874 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00077705600000000004 2024-09-21T22:07:34.874 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.874 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.874 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:34.874 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.957811+0000", 2024-09-21T22:07:34.874 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9918999999999999e-05 2024-09-21T22:07:34.874 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:34.874 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:34.874 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:34.874 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.874 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.875 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4981.0:20 2.7 e68/16 2:f45e3993:::rbd_object_map.126483d75f2d:head v 68'106, mlcod=68'104)", 2024-09-21T22:07:34.875 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:33.962453+0000", 2024-09-21T22:07:34.875 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.83300080099999996, 2024-09-21T22:07:34.875 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001372926, 2024-09-21T22:07:34.875 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:34.875 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:34.875 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:34.875 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:34.875 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.875 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:34.875 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.962453+0000", 2024-09-21T22:07:34.875 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:34.876 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.876 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.876 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:34.876 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.962453+0000", 2024-09-21T22:07:34.876 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:34.876 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.876 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.876 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:34.876 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.962455+0000", 2024-09-21T22:07:34.876 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1400000000000001e-06 2024-09-21T22:07:34.876 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.876 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.876 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:34.877 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.962467+0000", 2024-09-21T22:07:34.877 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2502e-05 2024-09-21T22:07:34.877 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.877 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.877 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:34.877 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.962490+0000", 2024-09-21T22:07:34.877 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3166000000000001e-05 2024-09-21T22:07:34.877 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.877 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.878 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:34.878 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.962503+0000", 2024-09-21T22:07:34.878 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3118999999999999e-05 2024-09-21T22:07:34.878 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.878 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.878 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:34.878 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.962579+0000", 2024-09-21T22:07:34.878 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5412000000000002e-05 2024-09-21T22:07:34.878 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.878 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.878 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:34.878 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.962602+0000", 2024-09-21T22:07:34.879 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3496999999999999e-05 2024-09-21T22:07:34.879 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.879 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.879 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:34.879 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.963782+0000", 2024-09-21T22:07:34.879 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011801170000000001 2024-09-21T22:07:34.879 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.879 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.879 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:34.879 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.963826+0000", 2024-09-21T22:07:34.880 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3973000000000001e-05 2024-09-21T22:07:34.880 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:34.880 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:34.880 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:34.880 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.880 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.880 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4981.0:23 2.7 e68/16 2:f45e3993:::rbd_object_map.126483d75f2d:head v 68'107, mlcod=68'105)", 2024-09-21T22:07:34.880 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:33.966404+0000", 2024-09-21T22:07:34.880 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.82904986999999997, 2024-09-21T22:07:34.880 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00086599099999999998, 2024-09-21T22:07:34.880 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:34.880 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:34.881 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:34.881 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:34.881 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.881 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:34.881 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.966404+0000", 2024-09-21T22:07:34.881 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:34.881 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.881 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.881 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:34.881 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.966404+0000", 2024-09-21T22:07:34.881 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:34.881 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.881 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.882 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:34.882 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.966405+0000", 2024-09-21T22:07:34.882 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0529999999999999e-06 2024-09-21T22:07:34.882 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.882 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.882 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:34.882 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.966413+0000", 2024-09-21T22:07:34.882 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3849999999999996e-06 2024-09-21T22:07:34.882 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.882 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.882 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:34.882 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.966432+0000", 2024-09-21T22:07:34.883 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8947e-05 2024-09-21T22:07:34.883 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.883 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.883 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:34.883 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.966444+0000", 2024-09-21T22:07:34.883 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1929e-05 2024-09-21T22:07:34.883 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.883 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.883 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:34.883 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.966529+0000", 2024-09-21T22:07:34.883 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5030000000000004e-05 2024-09-21T22:07:34.884 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.884 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.884 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:34.884 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.966560+0000", 2024-09-21T22:07:34.884 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1590000000000001e-05 2024-09-21T22:07:34.884 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.884 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.884 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:34.884 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.967239+0000", 2024-09-21T22:07:34.884 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00067846499999999995 2024-09-21T22:07:34.884 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:34.884 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:34.885 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:34.885 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.967270+0000", 2024-09-21T22:07:34.885 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1591999999999998e-05 2024-09-21T22:07:34.885 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:34.885 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:34.885 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:34.885 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:34.885 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:34.885 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:07:34.887 INFO:tasks.workunit.client.0.smithi157.stderr:+ PID=22504 2024-09-21T22:07:34.887 INFO:tasks.workunit.client.0.smithi157.stderr:+ test -n 22504 2024-09-21T22:07:34.887 INFO:tasks.workunit.client.0.smithi157.stderr:+ ps -p 22504 -C rbd-nbd 2024-09-21T22:07:34.887 INFO:tasks.workunit.client.0.smithi157.stdout: PID TTY TIME CMD 2024-09-21T22:07:34.887 INFO:tasks.workunit.client.0.smithi157.stdout: 22504 ? 00:00:00 rbd-nbd 2024-09-21T22:07:34.888 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T22:07:34.889 INFO:tasks.workunit.client.0.smithi157.stderr:+ _sudo dd if=/dev/nbd0 of=/dev/null bs=1M 2024-09-21T22:07:34.889 INFO:tasks.workunit.client.0.smithi157.stderr:+ local cmd 2024-09-21T22:07:34.889 INFO:tasks.workunit.client.0.smithi157.stderr:++ id -u 2024-09-21T22:07:34.890 INFO:tasks.workunit.client.0.smithi157.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T22:07:34.890 INFO:tasks.workunit.client.0.smithi157.stderr:++ which dd 2024-09-21T22:07:34.890 INFO:tasks.workunit.client.0.smithi157.stderr:+ cmd=/usr/bin/dd 2024-09-21T22:07:34.890 INFO:tasks.workunit.client.0.smithi157.stderr:+ shift 2024-09-21T22:07:34.890 INFO:tasks.workunit.client.0.smithi157.stderr:+ sudo -nE /usr/bin/dd if=/dev/nbd0 of=/dev/null bs=1M 2024-09-21T22:07:34.890 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:34.882+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:34.987 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:07:34.984+0000 7f2075a06640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31720) UID: 0 2024-09-21T22:07:35.041 INFO:teuthology.orchestra.run.smithi039.stderr:nodeep-scrub is unset 2024-09-21T22:07:35.048 INFO:tasks.workunit.client.0.smithi157.stderr:64+0 records in 2024-09-21T22:07:35.048 INFO:tasks.workunit.client.0.smithi157.stderr:64+0 records out 2024-09-21T22:07:35.048 INFO:tasks.workunit.client.0.smithi157.stderr:67108864 bytes (67 MB, 64 MiB) copied, 0.210918 s, 318 MB/s 2024-09-21T22:07:35.050 INFO:tasks.workunit.client.0.smithi157.stderr:+ expect_false _sudo dd if=/tmp/tmp.GqKuv7rsZi/data of=/dev/nbd0 bs=1M oflag=direct 2024-09-21T22:07:35.050 INFO:tasks.workunit.client.0.smithi157.stderr:+ _sudo dd if=/tmp/tmp.GqKuv7rsZi/data of=/dev/nbd0 bs=1M oflag=direct 2024-09-21T22:07:35.050 INFO:tasks.workunit.client.0.smithi157.stderr:+ local cmd 2024-09-21T22:07:35.051 INFO:tasks.workunit.client.0.smithi157.stderr:++ id -u 2024-09-21T22:07:35.053 INFO:tasks.workunit.client.0.smithi157.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T22:07:35.053 INFO:tasks.workunit.client.0.smithi157.stderr:++ which dd 2024-09-21T22:07:35.055 INFO:tasks.workunit.client.0.smithi157.stderr:+ cmd=/usr/bin/dd 2024-09-21T22:07:35.056 INFO:tasks.workunit.client.0.smithi157.stderr:+ shift 2024-09-21T22:07:35.056 INFO:tasks.workunit.client.0.smithi157.stderr:+ sudo -nE /usr/bin/dd if=/tmp/tmp.GqKuv7rsZi/data of=/dev/nbd0 bs=1M oflag=direct 2024-09-21T22:07:35.065 INFO:tasks.workunit.client.0.smithi157.stderr:/usr/bin/dd: error writing '/dev/nbd0': Operation not permitted 2024-09-21T22:07:35.065 INFO:tasks.workunit.client.0.smithi157.stderr:1+0 records in 2024-09-21T22:07:35.065 INFO:tasks.workunit.client.0.smithi157.stderr:0+0 records out 2024-09-21T22:07:35.066 INFO:tasks.workunit.client.0.smithi157.stderr:0 bytes copied, 0.0011069 s, 0.0 kB/s 2024-09-21T22:07:35.067 INFO:tasks.workunit.client.0.smithi157.stderr:+ return 0 2024-09-21T22:07:35.067 INFO:tasks.workunit.client.0.smithi157.stderr:+ unmap_device /dev/nbd0 22504 2024-09-21T22:07:35.067 INFO:tasks.workunit.client.0.smithi157.stderr:+ local args=/dev/nbd0 2024-09-21T22:07:35.067 INFO:tasks.workunit.client.0.smithi157.stderr:+ local pid=22504 2024-09-21T22:07:35.067 INFO:tasks.workunit.client.0.smithi157.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd0 2024-09-21T22:07:35.067 INFO:tasks.workunit.client.0.smithi157.stderr:+ local cmd 2024-09-21T22:07:35.067 INFO:tasks.workunit.client.0.smithi157.stderr:++ id -u 2024-09-21T22:07:35.069 INFO:tasks.workunit.client.0.smithi157.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T22:07:35.069 INFO:tasks.workunit.client.0.smithi157.stderr:++ which rbd 2024-09-21T22:07:35.070 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:07:35.070 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:07:35.070 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-09-21T22:07:35.070 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:07:35.071 INFO:tasks.workunit.client.0.smithi157.stderr:+ cmd=/usr/bin/rbd 2024-09-21T22:07:35.071 INFO:tasks.workunit.client.0.smithi157.stderr:+ shift 2024-09-21T22:07:35.071 INFO:tasks.workunit.client.0.smithi157.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd0 2024-09-21T22:07:35.086 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T22:07:35.133 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:35.128+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:35.154 INFO:tasks.workunit.client.0.smithi157.stderr:+ rbd device --device-type nbd list 2024-09-21T22:07:35.155 INFO:tasks.workunit.client.0.smithi157.stderr:+ expect_false grep '^22504\b' 2024-09-21T22:07:35.155 INFO:tasks.workunit.client.0.smithi157.stderr:+ grep '^22504\b' 2024-09-21T22:07:35.188 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:35.186+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:35.207 INFO:tasks.workunit.client.0.smithi157.stderr:+ return 0 2024-09-21T22:07:35.207 INFO:tasks.workunit.client.0.smithi157.stderr:+ ps -C rbd-nbd 2024-09-21T22:07:35.208 INFO:tasks.workunit.client.0.smithi157.stderr:+ expect_false grep '^ *22504\b' 2024-09-21T22:07:35.208 INFO:tasks.workunit.client.0.smithi157.stderr:+ grep '^ *22504\b' 2024-09-21T22:07:35.213 INFO:tasks.workunit.client.0.smithi157.stderr:+ return 0 2024-09-21T22:07:35.214 INFO:tasks.workunit.client.0.smithi157.stderr:+ sleep 0.5 2024-09-21T22:07:35.252 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:07:35.253 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:07:35.253 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-21T22:07:35.253 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-21T22:07:35.253 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:07:35.267 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T22:07:35.289 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:35.286+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:35.354 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T22:07:35.390 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:07:35.390 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-21T22:07:35.390 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-21T22:07:35.390 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-21T22:07:35.390 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.390 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5031.0:3 2.2 2:53d5e6d0:::rbd_header.126483d75f2d:head [call rbd.get_size in=11b,call,call rbd.get_features in=15b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e65)", 2024-09-21T22:07:35.390 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:27.373436+0000", 2024-09-21T22:07:35.391 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.0136146099999994, 2024-09-21T22:07:35.391 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.86524013700000002, 2024-09-21T22:07:35.391 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:35.391 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:35.391 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:07:35.391 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:35.391 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5031", 2024-09-21T22:07:35.391 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/3232705659", 2024-09-21T22:07:35.391 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3 2024-09-21T22:07:35.391 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.391 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:35.392 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.392 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:35.392 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:27.373436+0000", 2024-09-21T22:07:35.392 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:35.392 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.392 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.392 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:35.392 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:27.373436+0000", 2024-09-21T22:07:35.392 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:35.392 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.392 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.392 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:35.393 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:27.373439+0000", 2024-09-21T22:07:35.393 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5969999999999999e-06 2024-09-21T22:07:35.393 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.393 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.393 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:35.393 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:27.373450+0000", 2024-09-21T22:07:35.393 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1326e-05 2024-09-21T22:07:35.393 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.393 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.393 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:35.393 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:27.373471+0000", 2024-09-21T22:07:35.393 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0169999999999998e-05 2024-09-21T22:07:35.394 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.394 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.394 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:35.394 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:27.373486+0000", 2024-09-21T22:07:35.394 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5483000000000001e-05 2024-09-21T22:07:35.394 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.394 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.394 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:35.394 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:27.373602+0000", 2024-09-21T22:07:35.394 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000115603 2024-09-21T22:07:35.394 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.395 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.395 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-21T22:07:35.395 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:27.373617+0000", 2024-09-21T22:07:35.395 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4956999999999999e-05 2024-09-21T22:07:35.395 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.395 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.395 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:35.395 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.234578+0000", 2024-09-21T22:07:35.395 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.86096097900000002 2024-09-21T22:07:35.395 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.395 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.395 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for missing object", 2024-09-21T22:07:35.396 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.234771+0000", 2024-09-21T22:07:35.396 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019394700000000001 2024-09-21T22:07:35.396 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.396 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.396 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:35.396 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.237470+0000", 2024-09-21T22:07:35.396 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026981319999999998 2024-09-21T22:07:35.396 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.396 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.396 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:35.396 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.237791+0000", 2024-09-21T22:07:35.396 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00032138499999999999 2024-09-21T22:07:35.396 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.397 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.397 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:35.397 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.238677+0000", 2024-09-21T22:07:35.397 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00088555800000000005 2024-09-21T22:07:35.397 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:35.397 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:35.397 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:35.397 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.397 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.397 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.2 66/65 [PushOp(2:4481fad7:ns::rbd_data.127f79bc213d.0000000000000006:head, version: 57'9, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:4481fad7:ns::rbd_data.127f79bc213d.0000000000000006:head@57'9, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T22:07:35.397 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:28.247056+0000", 2024-09-21T22:07:35.397 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 7.1399949469999999, 2024-09-21T22:07:35.397 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.045025012000000003, 2024-09-21T22:07:35.398 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:35.398 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:35.398 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:07:35.398 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:35.398 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.398 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:35.398 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.247056+0000", 2024-09-21T22:07:35.398 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:35.398 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.398 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.398 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:35.398 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.247056+0000", 2024-09-21T22:07:35.399 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:35.399 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.399 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.399 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:35.399 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.247057+0000", 2024-09-21T22:07:35.399 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2500000000000001e-06 2024-09-21T22:07:35.399 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.399 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.399 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:35.399 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.249384+0000", 2024-09-21T22:07:35.399 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0023265619999999999 2024-09-21T22:07:35.399 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.400 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.400 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:35.400 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.249429+0000", 2024-09-21T22:07:35.400 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5392999999999997e-05 2024-09-21T22:07:35.400 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.400 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.400 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:35.400 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.249476+0000", 2024-09-21T22:07:35.400 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6570000000000003e-05 2024-09-21T22:07:35.400 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.400 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.400 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:35.400 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.249611+0000", 2024-09-21T22:07:35.401 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000135143 2024-09-21T22:07:35.401 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.401 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.401 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:35.401 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.249626+0000", 2024-09-21T22:07:35.401 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.469e-05 2024-09-21T22:07:35.401 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.401 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.401 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:35.401 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.292081+0000", 2024-09-21T22:07:35.401 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.042455404000000002 2024-09-21T22:07:35.401 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:35.401 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:35.402 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:35.402 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.402 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.402 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5031.0:9 2.2 2:53d5e6d0:::rbd_header.126483d75f2d:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-21T22:07:35.402 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:28.250163+0000", 2024-09-21T22:07:35.402 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 7.1368875530000002, 2024-09-21T22:07:35.402 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.042986430999999999, 2024-09-21T22:07:35.402 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:35.402 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:35.402 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:07:35.402 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:35.402 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5031", 2024-09-21T22:07:35.403 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/3232705659", 2024-09-21T22:07:35.403 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 9 2024-09-21T22:07:35.403 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.403 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:35.403 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.403 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:35.403 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.250163+0000", 2024-09-21T22:07:35.403 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:35.403 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.403 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.403 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:35.403 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.250163+0000", 2024-09-21T22:07:35.403 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:35.403 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.404 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.404 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:35.404 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.250166+0000", 2024-09-21T22:07:35.404 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7039999999999999e-06 2024-09-21T22:07:35.404 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.404 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.404 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:35.404 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.250175+0000", 2024-09-21T22:07:35.404 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7560000000000005e-06 2024-09-21T22:07:35.404 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.404 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.404 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:35.404 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.250199+0000", 2024-09-21T22:07:35.404 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4049999999999998e-05 2024-09-21T22:07:35.404 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.405 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.405 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:35.405 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.250218+0000", 2024-09-21T22:07:35.405 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9236e-05 2024-09-21T22:07:35.405 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.405 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.405 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:35.405 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.292193+0000", 2024-09-21T22:07:35.405 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.041974694 2024-09-21T22:07:35.405 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.405 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.405 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:35.405 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.292492+0000", 2024-09-21T22:07:35.405 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00029900100000000002 2024-09-21T22:07:35.405 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.406 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.406 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:35.406 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.293150+0000", 2024-09-21T22:07:35.406 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00065799000000000001 2024-09-21T22:07:35.406 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:35.406 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:35.406 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:35.406 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.406 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.406 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5031.0:12 2.2 2:53d5e6d0:::rbd_header.126483d75f2d:head [watch watch cookie 140495695137040] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-09-21T22:07:35.406 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:28.295580+0000", 2024-09-21T22:07:35.406 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 7.0914706660000002, 2024-09-21T22:07:35.406 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.033090870000000001, 2024-09-21T22:07:35.407 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:35.407 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:35.407 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:35.407 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:35.407 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5031", 2024-09-21T22:07:35.407 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/3232705659", 2024-09-21T22:07:35.407 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 12 2024-09-21T22:07:35.407 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.407 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:35.407 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.407 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:35.407 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.295580+0000", 2024-09-21T22:07:35.407 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:35.407 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.407 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.408 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:35.408 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.295580+0000", 2024-09-21T22:07:35.408 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:35.408 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.408 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.408 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:35.408 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.295581+0000", 2024-09-21T22:07:35.408 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6799999999999999e-07 2024-09-21T22:07:35.408 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.408 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.408 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:35.408 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.295586+0000", 2024-09-21T22:07:35.408 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8659999999999997e-06 2024-09-21T22:07:35.408 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.408 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.409 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:35.409 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.295599+0000", 2024-09-21T22:07:35.409 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2897999999999999e-05 2024-09-21T22:07:35.409 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.409 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.409 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:35.409 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.295614+0000", 2024-09-21T22:07:35.409 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5406000000000001e-05 2024-09-21T22:07:35.409 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.409 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.409 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:35.409 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.295665+0000", 2024-09-21T22:07:35.409 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0501000000000003e-05 2024-09-21T22:07:35.409 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.409 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.410 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:35.410 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.295727+0000", 2024-09-21T22:07:35.410 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2073e-05 2024-09-21T22:07:35.410 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.410 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.410 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-09-21T22:07:35.410 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.295849+0000", 2024-09-21T22:07:35.410 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012190500000000001 2024-09-21T22:07:35.410 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.410 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.410 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:35.410 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.296429+0000", 2024-09-21T22:07:35.410 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00058019699999999996 2024-09-21T22:07:35.410 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.411 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.411 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:35.411 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.328308+0000", 2024-09-21T22:07:35.411 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.031878849000000001 2024-09-21T22:07:35.411 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.411 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.411 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:35.411 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.328421+0000", 2024-09-21T22:07:35.411 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000113322 2024-09-21T22:07:35.411 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.411 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.411 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:35.412 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.328671+0000", 2024-09-21T22:07:35.412 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024998499999999999 2024-09-21T22:07:35.412 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:35.412 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:35.412 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:35.412 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.412 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.412 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.2 66/65 [PushOp(2:4481fad7:ns::rbd_data.127f79bc213d.0000000000000006:head, version: 57'9, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:4481fad7:ns::rbd_data.127f79bc213d.0000000000000006:head@57'9, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:07:35.412 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:28.296301+0000", 2024-09-21T22:07:35.412 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 7.0907497450000001, 2024-09-21T22:07:35.412 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.031905291000000002, 2024-09-21T22:07:35.412 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:35.412 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:35.412 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:07:35.413 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:35.413 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.413 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:35.413 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.296301+0000", 2024-09-21T22:07:35.413 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:35.413 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.413 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.413 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:35.413 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.296301+0000", 2024-09-21T22:07:35.413 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:35.413 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.413 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.413 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:35.414 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.296315+0000", 2024-09-21T22:07:35.414 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3867e-05 2024-09-21T22:07:35.414 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.414 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.414 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:35.414 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.297432+0000", 2024-09-21T22:07:35.414 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011164759999999999 2024-09-21T22:07:35.414 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.414 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.414 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:35.414 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.297454+0000", 2024-09-21T22:07:35.414 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2813e-05 2024-09-21T22:07:35.415 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.415 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.415 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:35.415 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.297479+0000", 2024-09-21T22:07:35.415 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4817e-05 2024-09-21T22:07:35.415 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.415 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.415 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:35.415 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.297582+0000", 2024-09-21T22:07:35.415 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010261500000000001 2024-09-21T22:07:35.415 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.415 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.416 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:35.416 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.297600+0000", 2024-09-21T22:07:35.416 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8323000000000001e-05 2024-09-21T22:07:35.416 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.416 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.416 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:35.416 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.328207+0000", 2024-09-21T22:07:35.416 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.030606379999999999 2024-09-21T22:07:35.416 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:35.416 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:35.416 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:35.416 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.416 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.417 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.5031.0:12 2.2 e66/65 ondisk, result = 0)", 2024-09-21T22:07:35.417 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:28.297512+0000", 2024-09-21T22:07:35.417 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 7.0895391720000003, 2024-09-21T22:07:35.417 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.031170693999999999, 2024-09-21T22:07:35.417 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:35.417 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:35.417 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:07:35.417 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:35.417 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.417 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:35.417 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.297512+0000", 2024-09-21T22:07:35.418 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:35.418 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.418 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.418 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:35.418 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.297512+0000", 2024-09-21T22:07:35.418 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:35.418 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.418 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.418 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:35.418 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.297512+0000", 2024-09-21T22:07:35.418 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7700000000000005e-07 2024-09-21T22:07:35.419 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.419 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.419 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:35.419 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.297517+0000", 2024-09-21T22:07:35.419 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1039999999999998e-06 2024-09-21T22:07:35.419 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.419 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.419 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:35.419 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.297528+0000", 2024-09-21T22:07:35.419 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0633e-05 2024-09-21T22:07:35.419 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.420 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.420 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:35.420 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.297537+0000", 2024-09-21T22:07:35.420 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7199999999999995e-06 2024-09-21T22:07:35.420 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.420 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.420 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:35.420 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.328273+0000", 2024-09-21T22:07:35.420 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.030736574999999999 2024-09-21T22:07:35.420 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.420 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.420 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:35.420 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.328293+0000", 2024-09-21T22:07:35.420 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9304e-05 2024-09-21T22:07:35.421 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.421 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.421 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:35.421 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.328683+0000", 2024-09-21T22:07:35.421 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00038988100000000002 2024-09-21T22:07:35.421 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:35.421 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:35.421 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:35.421 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.421 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.421 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.2 66/65 [PushOp(2:4481fad7:ns::rbd_data.127f79bc213d.0000000000000006:head, version: 57'9, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:4481fad7:ns::rbd_data.127f79bc213d.0000000000000006:head@57'9, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:07:35.421 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:28.331166+0000", 2024-09-21T22:07:35.421 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 7.0558848029999996, 2024-09-21T22:07:35.421 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.039995378999999998, 2024-09-21T22:07:35.422 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:35.422 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:35.422 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:07:35.422 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:35.422 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.422 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:35.422 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.331166+0000", 2024-09-21T22:07:35.422 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:35.422 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.422 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.422 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:35.423 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.331166+0000", 2024-09-21T22:07:35.423 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:35.423 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.423 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.423 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:35.423 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.331167+0000", 2024-09-21T22:07:35.423 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9199999999999999e-07 2024-09-21T22:07:35.423 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.423 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.423 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:35.423 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.332127+0000", 2024-09-21T22:07:35.424 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00095969800000000004 2024-09-21T22:07:35.424 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.424 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.424 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:35.424 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.332149+0000", 2024-09-21T22:07:35.424 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1974999999999999e-05 2024-09-21T22:07:35.424 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.424 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.424 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:35.424 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.332161+0000", 2024-09-21T22:07:35.424 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2276000000000001e-05 2024-09-21T22:07:35.424 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.425 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.425 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:35.425 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.332290+0000", 2024-09-21T22:07:35.425 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000129214 2024-09-21T22:07:35.425 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.425 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.425 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:35.425 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.332312+0000", 2024-09-21T22:07:35.425 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1308999999999999e-05 2024-09-21T22:07:35.425 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.425 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.425 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:35.425 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.371162+0000", 2024-09-21T22:07:35.426 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.038850015000000002 2024-09-21T22:07:35.426 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:35.426 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:35.426 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:35.426 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.426 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.426 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.2 66/65 [PushOp(2:4481fad7:ns::rbd_data.127f79bc213d.0000000000000006:head, version: 57'9, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:4481fad7:ns::rbd_data.127f79bc213d.0000000000000006:head@57'9, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 4194304, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:07:35.426 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:28.373682+0000", 2024-09-21T22:07:35.426 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 7.0133689190000004, 2024-09-21T22:07:35.426 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026496644999999999, 2024-09-21T22:07:35.426 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:35.426 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:35.427 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:07:35.427 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:35.427 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.427 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:35.427 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.373682+0000", 2024-09-21T22:07:35.427 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:35.427 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.427 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.427 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:35.427 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.373682+0000", 2024-09-21T22:07:35.427 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:35.427 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.428 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.428 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:35.428 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.373684+0000", 2024-09-21T22:07:35.428 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.42e-06 2024-09-21T22:07:35.428 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.428 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.428 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:35.428 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.375520+0000", 2024-09-21T22:07:35.428 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001836414 2024-09-21T22:07:35.428 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.428 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.428 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:35.429 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.375596+0000", 2024-09-21T22:07:35.429 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6234000000000002e-05 2024-09-21T22:07:35.429 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.429 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.429 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:35.429 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.375638+0000", 2024-09-21T22:07:35.429 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1742000000000001e-05 2024-09-21T22:07:35.429 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.429 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.429 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:35.430 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.375763+0000", 2024-09-21T22:07:35.430 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000125408 2024-09-21T22:07:35.430 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.430 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.430 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:35.430 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.375780+0000", 2024-09-21T22:07:35.430 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6212999999999999e-05 2024-09-21T22:07:35.430 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.430 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.430 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:35.430 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.400179+0000", 2024-09-21T22:07:35.430 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024399213999999999 2024-09-21T22:07:35.431 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:35.431 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:35.431 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:35.431 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.431 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.431 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.2 66/65 [PushOp(2:53e0fad0:ns::rbd_data.127f79bc213d.000000000000000e:head, version: 57'10, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:53e0fad0:ns::rbd_data.127f79bc213d.000000000000000e:head@57'10, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T22:07:35.431 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:28.403213+0000", 2024-09-21T22:07:35.431 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 6.9838380359999999, 2024-09-21T22:07:35.431 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.030711980999999999, 2024-09-21T22:07:35.431 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:35.431 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:35.431 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:07:35.432 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:35.432 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.432 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:35.432 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.403213+0000", 2024-09-21T22:07:35.432 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:35.432 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.432 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.432 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:35.432 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.403213+0000", 2024-09-21T22:07:35.432 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:35.432 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.432 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.432 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:35.432 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.403214+0000", 2024-09-21T22:07:35.432 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3149999999999999e-06 2024-09-21T22:07:35.433 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.433 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.433 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:35.433 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.404316+0000", 2024-09-21T22:07:35.433 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011019999999999999 2024-09-21T22:07:35.433 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.433 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.433 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:35.433 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.404344+0000", 2024-09-21T22:07:35.433 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8056e-05 2024-09-21T22:07:35.433 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.433 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.433 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:35.433 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.404362+0000", 2024-09-21T22:07:35.434 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8091999999999999e-05 2024-09-21T22:07:35.434 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.434 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.434 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:35.434 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.404438+0000", 2024-09-21T22:07:35.434 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5164000000000001e-05 2024-09-21T22:07:35.434 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.434 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.434 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:35.434 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.404448+0000", 2024-09-21T22:07:35.434 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0855e-05 2024-09-21T22:07:35.434 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.434 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.434 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:35.434 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.433925+0000", 2024-09-21T22:07:35.435 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029476499 2024-09-21T22:07:35.435 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:35.435 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:35.435 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:35.435 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.435 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.435 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.2 66/65 [PushOp(2:53e0fad0:ns::rbd_data.127f79bc213d.000000000000000e:head, version: 57'10, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:53e0fad0:ns::rbd_data.127f79bc213d.000000000000000e:head@57'10, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:07:35.435 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:28.437075+0000", 2024-09-21T22:07:35.435 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 6.9499761470000001, 2024-09-21T22:07:35.435 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.027135738, 2024-09-21T22:07:35.435 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:35.435 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:35.435 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:07:35.435 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:35.436 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.436 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:35.436 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.437075+0000", 2024-09-21T22:07:35.436 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:35.436 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.436 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.436 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:35.436 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.437075+0000", 2024-09-21T22:07:35.436 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:35.436 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.436 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.436 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:35.436 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.437077+0000", 2024-09-21T22:07:35.436 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0839999999999999e-06 2024-09-21T22:07:35.436 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.437 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.437 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:35.437 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.438573+0000", 2024-09-21T22:07:35.437 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0014962720000000001 2024-09-21T22:07:35.437 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.437 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.437 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:35.437 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.438621+0000", 2024-09-21T22:07:35.437 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8108000000000001e-05 2024-09-21T22:07:35.437 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.437 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.437 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:35.437 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.438649+0000", 2024-09-21T22:07:35.437 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7186999999999999e-05 2024-09-21T22:07:35.438 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.438 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.438 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:35.438 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.438769+0000", 2024-09-21T22:07:35.438 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000120482 2024-09-21T22:07:35.438 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.438 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.438 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:35.438 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.438781+0000", 2024-09-21T22:07:35.438 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1907e-05 2024-09-21T22:07:35.438 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.438 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.438 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:35.439 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.464211+0000", 2024-09-21T22:07:35.439 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025429698000000001 2024-09-21T22:07:35.439 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:35.439 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:35.439 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:35.439 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.439 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.439 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.2 66/65 [PushOp(2:53e0fad0:ns::rbd_data.127f79bc213d.000000000000000e:head, version: 57'10, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:53e0fad0:ns::rbd_data.127f79bc213d.000000000000000e:head@57'10, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:07:35.439 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:28.467017+0000", 2024-09-21T22:07:35.439 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 6.9200335859999997, 2024-09-21T22:07:35.439 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026823830999999999, 2024-09-21T22:07:35.439 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:35.439 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:35.439 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:07:35.439 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:35.440 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.440 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:35.440 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.467017+0000", 2024-09-21T22:07:35.440 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:35.440 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.440 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.440 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:35.440 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.467017+0000", 2024-09-21T22:07:35.440 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:35.440 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.440 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.440 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:35.440 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.467019+0000", 2024-09-21T22:07:35.440 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.629e-06 2024-09-21T22:07:35.440 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.441 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.441 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:35.441 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.468119+0000", 2024-09-21T22:07:35.441 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001100345 2024-09-21T22:07:35.441 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.441 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.441 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:35.441 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.468150+0000", 2024-09-21T22:07:35.441 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0673000000000003e-05 2024-09-21T22:07:35.441 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.441 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.441 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:35.441 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.468172+0000", 2024-09-21T22:07:35.441 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2079999999999999e-05 2024-09-21T22:07:35.441 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.442 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.442 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:35.442 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.468269+0000", 2024-09-21T22:07:35.442 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6521000000000006e-05 2024-09-21T22:07:35.442 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.442 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.442 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:35.442 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.468281+0000", 2024-09-21T22:07:35.442 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2252000000000001e-05 2024-09-21T22:07:35.442 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.442 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.442 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:35.442 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.493841+0000", 2024-09-21T22:07:35.443 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025560330999999999 2024-09-21T22:07:35.443 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:35.443 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:35.443 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:35.443 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.443 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.443 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4811.0:3 2.3 2:c4c92e5a:::rbd_trash:head [call rbd.trash_list in=13b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-21T22:07:35.443 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:28.475306+0000", 2024-09-21T22:07:35.443 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 6.9117452589999999, 2024-09-21T22:07:35.443 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.77996670099999998, 2024-09-21T22:07:35.443 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:35.443 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:35.443 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:07:35.444 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:35.444 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4811", 2024-09-21T22:07:35.444 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/2028736390", 2024-09-21T22:07:35.444 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3 2024-09-21T22:07:35.444 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.444 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:35.444 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.444 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:35.444 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.475306+0000", 2024-09-21T22:07:35.444 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:35.444 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.444 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.444 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:35.445 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.475306+0000", 2024-09-21T22:07:35.445 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:35.445 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.445 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.445 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:35.445 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.475308+0000", 2024-09-21T22:07:35.445 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8959999999999999e-06 2024-09-21T22:07:35.445 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.445 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.446 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:35.446 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.475314+0000", 2024-09-21T22:07:35.446 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2280000000000003e-06 2024-09-21T22:07:35.446 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.446 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.446 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:35.446 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.475329+0000", 2024-09-21T22:07:35.446 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5054e-05 2024-09-21T22:07:35.446 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.446 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.446 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:35.447 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.475338+0000", 2024-09-21T22:07:35.447 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3770000000000002e-06 2024-09-21T22:07:35.447 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.447 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.447 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:35.447 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.475484+0000", 2024-09-21T22:07:35.447 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014612300000000001 2024-09-21T22:07:35.447 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.447 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.447 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-21T22:07:35.447 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.475506+0000", 2024-09-21T22:07:35.447 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1319999999999999e-05 2024-09-21T22:07:35.447 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.447 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.447 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:35.448 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.254433+0000", 2024-09-21T22:07:35.448 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.778927708 2024-09-21T22:07:35.448 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.448 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.448 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:35.448 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.254979+0000", 2024-09-21T22:07:35.448 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00054573100000000004 2024-09-21T22:07:35.448 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.448 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.448 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:35.448 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.255272+0000", 2024-09-21T22:07:35.448 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000293264 2024-09-21T22:07:35.448 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:35.448 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:35.449 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:35.449 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.449 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.449 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.3 67/66 [PushOp(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head, version: 63'11, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head@63'11, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T22:07:35.449 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:29.262595+0000", 2024-09-21T22:07:35.449 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 6.1244560760000004, 2024-09-21T22:07:35.449 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.036027272999999999, 2024-09-21T22:07:35.449 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:35.449 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:35.449 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:07:35.449 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:35.449 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.449 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:35.449 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.262595+0000", 2024-09-21T22:07:35.450 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:35.450 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.450 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.450 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:35.450 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.262595+0000", 2024-09-21T22:07:35.450 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:35.450 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.450 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.450 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:35.450 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.262596+0000", 2024-09-21T22:07:35.450 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4300000000000001e-06 2024-09-21T22:07:35.450 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.450 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.450 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:35.451 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.264505+0000", 2024-09-21T22:07:35.451 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019085650000000001 2024-09-21T22:07:35.451 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.451 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.451 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:35.451 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.264557+0000", 2024-09-21T22:07:35.451 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2040000000000003e-05 2024-09-21T22:07:35.451 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.451 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.451 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:35.451 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.264589+0000", 2024-09-21T22:07:35.451 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1636e-05 2024-09-21T22:07:35.451 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.451 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.451 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:35.452 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.264778+0000", 2024-09-21T22:07:35.452 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018935600000000001 2024-09-21T22:07:35.452 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.452 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.452 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:35.452 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.264803+0000", 2024-09-21T22:07:35.452 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4692999999999999e-05 2024-09-21T22:07:35.452 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.452 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.452 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:35.452 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.298622+0000", 2024-09-21T22:07:35.452 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.033819553000000002 2024-09-21T22:07:35.452 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:35.452 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:35.453 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:35.453 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.453 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.453 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.3 67/66 [PushOp(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head, version: 63'11, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head@63'11, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:07:35.453 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:29.301490+0000", 2024-09-21T22:07:35.453 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 6.0855612900000002, 2024-09-21T22:07:35.453 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.041178293999999997, 2024-09-21T22:07:35.453 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:35.453 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:35.453 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:07:35.453 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:35.453 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.453 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:35.453 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.301490+0000", 2024-09-21T22:07:35.453 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:35.454 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.454 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.454 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:35.454 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.301490+0000", 2024-09-21T22:07:35.454 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:35.454 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.454 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.454 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:35.454 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.301491+0000", 2024-09-21T22:07:35.454 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.513e-06 2024-09-21T22:07:35.454 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.454 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.454 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:35.454 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.302815+0000", 2024-09-21T22:07:35.455 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001323722 2024-09-21T22:07:35.455 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.455 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.455 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:35.455 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.302863+0000", 2024-09-21T22:07:35.455 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8449999999999999e-05 2024-09-21T22:07:35.455 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.455 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.455 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:35.455 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.302895+0000", 2024-09-21T22:07:35.455 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1072999999999999e-05 2024-09-21T22:07:35.455 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.455 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.455 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:35.455 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.303014+0000", 2024-09-21T22:07:35.456 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011907400000000001 2024-09-21T22:07:35.456 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.456 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.456 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:35.456 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.303029+0000", 2024-09-21T22:07:35.456 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5322e-05 2024-09-21T22:07:35.456 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.456 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.456 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:35.456 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.342668+0000", 2024-09-21T22:07:35.456 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.039639140000000003 2024-09-21T22:07:35.456 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:35.456 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:35.456 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:35.457 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.457 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.457 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.3 67/66 [PushOp(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head, version: 63'11, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head@63'11, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:07:35.457 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:29.345626+0000", 2024-09-21T22:07:35.457 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 6.0414245559999999, 2024-09-21T22:07:35.457 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.034820216000000001, 2024-09-21T22:07:35.457 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:35.457 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:35.457 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:07:35.457 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:35.457 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.457 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:35.457 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.345626+0000", 2024-09-21T22:07:35.457 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:35.457 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.458 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.458 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:35.458 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.345626+0000", 2024-09-21T22:07:35.458 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:35.458 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.458 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.458 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:35.458 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.345628+0000", 2024-09-21T22:07:35.458 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3799999999999999e-06 2024-09-21T22:07:35.458 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.458 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.458 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:35.458 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.347215+0000", 2024-09-21T22:07:35.458 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015868760000000001 2024-09-21T22:07:35.459 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.459 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.459 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:35.459 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.347283+0000", 2024-09-21T22:07:35.459 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8267000000000002e-05 2024-09-21T22:07:35.459 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.459 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.459 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:35.459 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.347336+0000", 2024-09-21T22:07:35.459 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3455999999999998e-05 2024-09-21T22:07:35.459 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.459 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.459 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:35.459 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.347463+0000", 2024-09-21T22:07:35.459 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012614399999999999 2024-09-21T22:07:35.460 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.460 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.460 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:35.460 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.347473+0000", 2024-09-21T22:07:35.460 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0462000000000001e-05 2024-09-21T22:07:35.460 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.460 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.460 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:35.460 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.380447+0000", 2024-09-21T22:07:35.460 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.032973631000000003 2024-09-21T22:07:35.460 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:35.460 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:35.460 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:35.460 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.460 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.461 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.3 67/66 [PushOp(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head, version: 63'11, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head@63'11, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 4194304, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:07:35.461 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:29.383677+0000", 2024-09-21T22:07:35.461 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 6.0033736790000001, 2024-09-21T22:07:35.461 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.031566611000000001, 2024-09-21T22:07:35.461 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:35.461 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:35.461 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:07:35.461 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:35.461 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.461 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:35.461 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.383677+0000", 2024-09-21T22:07:35.461 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:35.461 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.461 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.462 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:35.462 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.383677+0000", 2024-09-21T22:07:35.462 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:35.462 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.462 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.462 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:35.462 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.383679+0000", 2024-09-21T22:07:35.462 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4500000000000001e-06 2024-09-21T22:07:35.462 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.462 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.462 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:35.462 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.384982+0000", 2024-09-21T22:07:35.462 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001303069 2024-09-21T22:07:35.462 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.463 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.463 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:35.463 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.385028+0000", 2024-09-21T22:07:35.463 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5655999999999998e-05 2024-09-21T22:07:35.463 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.463 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.463 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:35.463 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.385056+0000", 2024-09-21T22:07:35.463 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8877000000000002e-05 2024-09-21T22:07:35.463 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.463 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.463 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:35.463 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.385161+0000", 2024-09-21T22:07:35.463 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000104325 2024-09-21T22:07:35.463 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.464 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.464 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:35.464 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.385173+0000", 2024-09-21T22:07:35.464 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2493e-05 2024-09-21T22:07:35.464 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.464 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.464 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:35.464 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.415244+0000", 2024-09-21T22:07:35.464 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.030070741000000002 2024-09-21T22:07:35.464 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:35.464 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:35.464 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:35.464 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.464 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.465 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.3 67/66 [PushOp(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head, version: 63'19, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head@63'19, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T22:07:35.465 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:29.420098+0000", 2024-09-21T22:07:35.465 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 5.9669527210000002, 2024-09-21T22:07:35.465 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.067283846999999994, 2024-09-21T22:07:35.465 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:35.465 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:35.465 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:07:35.465 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:35.465 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.465 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:35.465 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.420098+0000", 2024-09-21T22:07:35.465 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:35.465 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.465 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.466 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:35.466 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.420098+0000", 2024-09-21T22:07:35.466 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:35.466 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.466 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.466 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:35.466 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.420100+0000", 2024-09-21T22:07:35.466 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.716e-06 2024-09-21T22:07:35.466 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.466 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.466 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:35.466 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.421949+0000", 2024-09-21T22:07:35.466 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0018485190000000001 2024-09-21T22:07:35.467 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.467 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.467 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:35.467 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.421996+0000", 2024-09-21T22:07:35.467 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7089000000000003e-05 2024-09-21T22:07:35.467 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.467 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.467 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:35.467 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.422042+0000", 2024-09-21T22:07:35.467 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6010000000000002e-05 2024-09-21T22:07:35.467 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.467 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.467 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:35.468 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.422231+0000", 2024-09-21T22:07:35.468 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018960900000000001 2024-09-21T22:07:35.468 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.468 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.468 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:35.468 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.422254+0000", 2024-09-21T22:07:35.468 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2864e-05 2024-09-21T22:07:35.468 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.468 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.468 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:35.468 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.487382+0000", 2024-09-21T22:07:35.468 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.065128039999999998 2024-09-21T22:07:35.468 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:35.469 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:35.469 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:35.469 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.469 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.469 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.3 67/66 [PushOp(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head, version: 63'19, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head@63'19, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:07:35.469 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:29.491783+0000", 2024-09-21T22:07:35.469 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 5.8952683759999998, 2024-09-21T22:07:35.469 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.04476596, 2024-09-21T22:07:35.469 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:35.469 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:35.469 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:07:35.469 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:35.469 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.470 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:35.470 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.491783+0000", 2024-09-21T22:07:35.470 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:35.470 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.470 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.470 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:35.470 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.491783+0000", 2024-09-21T22:07:35.470 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:35.470 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.470 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.470 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:35.470 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.491784+0000", 2024-09-21T22:07:35.470 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.17e-06 2024-09-21T22:07:35.471 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.471 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.471 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:35.471 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.493126+0000", 2024-09-21T22:07:35.471 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0013424489999999999 2024-09-21T22:07:35.471 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.471 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.471 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:35.471 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.493170+0000", 2024-09-21T22:07:35.471 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4153000000000002e-05 2024-09-21T22:07:35.471 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.471 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.471 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:35.471 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.493218+0000", 2024-09-21T22:07:35.472 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7392999999999998e-05 2024-09-21T22:07:35.472 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.472 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.472 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:35.472 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.493392+0000", 2024-09-21T22:07:35.472 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017392699999999999 2024-09-21T22:07:35.472 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.472 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.472 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:35.472 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.493416+0000", 2024-09-21T22:07:35.472 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4117e-05 2024-09-21T22:07:35.473 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.473 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.473 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:35.473 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.536549+0000", 2024-09-21T22:07:35.473 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.043132750999999997 2024-09-21T22:07:35.473 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:35.473 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:35.473 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:35.473 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.473 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.473 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.3 67/66 [PushOp(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head, version: 63'19, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head@63'19, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:07:35.473 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:29.539831+0000", 2024-09-21T22:07:35.473 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 5.8472196160000003, 2024-09-21T22:07:35.473 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.036233222000000002, 2024-09-21T22:07:35.473 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:35.474 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:35.474 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:07:35.474 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:35.474 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.474 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:35.474 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.539831+0000", 2024-09-21T22:07:35.474 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:35.474 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.474 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.474 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:35.474 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.539831+0000", 2024-09-21T22:07:35.474 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:35.474 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.475 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.475 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:35.475 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.539833+0000", 2024-09-21T22:07:35.475 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5200000000000001e-06 2024-09-21T22:07:35.475 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.475 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.475 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:35.475 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.541130+0000", 2024-09-21T22:07:35.475 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0012972719999999999 2024-09-21T22:07:35.475 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.475 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.475 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:35.475 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.541179+0000", 2024-09-21T22:07:35.476 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8507000000000002e-05 2024-09-21T22:07:35.476 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.476 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.476 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:35.476 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.541226+0000", 2024-09-21T22:07:35.476 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7663000000000001e-05 2024-09-21T22:07:35.476 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.476 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.476 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:35.476 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.541361+0000", 2024-09-21T22:07:35.476 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000135065 2024-09-21T22:07:35.476 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.476 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.476 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:35.476 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.541374+0000", 2024-09-21T22:07:35.477 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2295e-05 2024-09-21T22:07:35.477 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.477 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.477 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:35.477 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.576065+0000", 2024-09-21T22:07:35.477 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.034690899999999997 2024-09-21T22:07:35.477 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:35.477 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:35.477 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:35.477 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.477 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.477 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.3 67/66 [PushOp(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head, version: 63'19, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head@63'19, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 4194304, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:07:35.477 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:29.579455+0000", 2024-09-21T22:07:35.477 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 5.8075963079999999, 2024-09-21T22:07:35.478 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.034295594999999998, 2024-09-21T22:07:35.478 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:35.478 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:35.478 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:07:35.478 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:35.478 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.478 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:35.478 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.579455+0000", 2024-09-21T22:07:35.478 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:35.478 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.478 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.478 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:35.478 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.579455+0000", 2024-09-21T22:07:35.478 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:35.478 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.479 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.479 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:35.479 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.579456+0000", 2024-09-21T22:07:35.479 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.404e-06 2024-09-21T22:07:35.479 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.479 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.479 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:35.479 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.580763+0000", 2024-09-21T22:07:35.479 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001307325 2024-09-21T22:07:35.479 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.479 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.479 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:35.479 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.580811+0000", 2024-09-21T22:07:35.479 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7843000000000003e-05 2024-09-21T22:07:35.480 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.480 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.480 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:35.480 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.580860+0000", 2024-09-21T22:07:35.480 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9036999999999998e-05 2024-09-21T22:07:35.480 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.480 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.480 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:35.480 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.580988+0000", 2024-09-21T22:07:35.480 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000127529 2024-09-21T22:07:35.480 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.480 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.480 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:35.480 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.580998+0000", 2024-09-21T22:07:35.480 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0617e-05 2024-09-21T22:07:35.480 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:35.481 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:35.481 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:35.481 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.613750+0000", 2024-09-21T22:07:35.481 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.032751839999999997 2024-09-21T22:07:35.481 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:35.481 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:35.481 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:35.481 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:35.481 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:35.481 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:07:35.482 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:35.398+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:35.483 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T22:07:35.512 INFO:teuthology.orchestra.run.smithi039.stdout:osd.0: {} 2024-09-21T22:07:35.512 INFO:teuthology.orchestra.run.smithi039.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T22:07:35.513 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:35.510+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:35.529 INFO:teuthology.orchestra.run.smithi039.stdout:osd.1: {} 2024-09-21T22:07:35.529 INFO:teuthology.orchestra.run.smithi039.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T22:07:35.547 INFO:teuthology.orchestra.run.smithi039.stdout:osd.2: {} 2024-09-21T22:07:35.548 INFO:teuthology.orchestra.run.smithi039.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T22:07:35.569 INFO:teuthology.orchestra.run.smithi039.stdout:osd.3: {} 2024-09-21T22:07:35.569 INFO:teuthology.orchestra.run.smithi039.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T22:07:35.590 INFO:teuthology.orchestra.run.smithi039.stdout:osd.4: {} 2024-09-21T22:07:35.590 INFO:teuthology.orchestra.run.smithi039.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T22:07:35.591 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:35.586+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:35.603 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:35.603 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:35.612 INFO:teuthology.orchestra.run.smithi039.stdout:osd.5: {} 2024-09-21T22:07:35.612 INFO:teuthology.orchestra.run.smithi039.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T22:07:35.618 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T22:07:35.692 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:35.688+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:35.715 INFO:tasks.workunit.client.0.smithi157.stderr:++ _sudo rbd --device-type nbd map --exclusive rbd/testrbdnbd21553 2024-09-21T22:07:35.715 INFO:tasks.workunit.client.0.smithi157.stderr:++ local cmd 2024-09-21T22:07:35.716 INFO:tasks.workunit.client.0.smithi157.stderr:+++ id -u 2024-09-21T22:07:35.717 INFO:tasks.workunit.client.0.smithi157.stderr:++ '[' 1000 -eq 0 ']' 2024-09-21T22:07:35.718 INFO:tasks.workunit.client.0.smithi157.stderr:+++ which rbd 2024-09-21T22:07:35.720 INFO:tasks.workunit.client.0.smithi157.stderr:++ cmd=/usr/bin/rbd 2024-09-21T22:07:35.721 INFO:tasks.workunit.client.0.smithi157.stderr:++ shift 2024-09-21T22:07:35.721 INFO:tasks.workunit.client.0.smithi157.stderr:++ sudo -nE /usr/bin/rbd --device-type nbd map --exclusive rbd/testrbdnbd21553 2024-09-21T22:07:35.793 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:35.790+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:35.801 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:35.801 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:35.816 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T22:07:35.830 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 5, 0] out_osds: [2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-09-21T22:07:35.831 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-09-21T22:07:35.879 INFO:tasks.workunit.client.0.smithi157.stderr:+ DEV=/dev/nbd0 2024-09-21T22:07:35.879 INFO:tasks.workunit.client.0.smithi157.stderr:+ get_pid rbd 2024-09-21T22:07:35.879 INFO:tasks.workunit.client.0.smithi157.stderr:+ local pool=rbd 2024-09-21T22:07:35.879 INFO:tasks.workunit.client.0.smithi157.stderr:+ local ns= 2024-09-21T22:07:35.879 INFO:tasks.workunit.client.0.smithi157.stderr:++ rbd device --device-type nbd --format xml list 2024-09-21T22:07:35.879 INFO:tasks.workunit.client.0.smithi157.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd21553'\''][device='\''/dev/nbd0'\'']/id' 2024-09-21T22:07:35.904 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:35.902+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:35.946 INFO:tasks.workunit.client.0.smithi157.stderr:+ PID=22576 2024-09-21T22:07:35.946 INFO:tasks.workunit.client.0.smithi157.stderr:+ test -n 22576 2024-09-21T22:07:35.946 INFO:tasks.workunit.client.0.smithi157.stderr:+ ps -p 22576 -C rbd-nbd 2024-09-21T22:07:35.952 INFO:tasks.workunit.client.0.smithi157.stdout: PID TTY TIME CMD 2024-09-21T22:07:35.952 INFO:tasks.workunit.client.0.smithi157.stdout: 22576 ? 00:00:00 rbd-nbd 2024-09-21T22:07:35.952 INFO:tasks.workunit.client.0.smithi157.stderr:+ _sudo dd if=/tmp/tmp.GqKuv7rsZi/data of=/dev/nbd0 bs=1M oflag=direct 2024-09-21T22:07:35.952 INFO:tasks.workunit.client.0.smithi157.stderr:+ local cmd 2024-09-21T22:07:35.952 INFO:tasks.workunit.client.0.smithi157.stderr:++ id -u 2024-09-21T22:07:35.953 INFO:tasks.workunit.client.0.smithi157.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T22:07:35.954 INFO:tasks.workunit.client.0.smithi157.stderr:++ which dd 2024-09-21T22:07:35.956 INFO:tasks.workunit.client.0.smithi157.stderr:+ cmd=/usr/bin/dd 2024-09-21T22:07:35.956 INFO:tasks.workunit.client.0.smithi157.stderr:+ shift 2024-09-21T22:07:35.956 INFO:tasks.workunit.client.0.smithi157.stderr:+ sudo -nE /usr/bin/dd if=/tmp/tmp.GqKuv7rsZi/data of=/dev/nbd0 bs=1M oflag=direct 2024-09-21T22:07:35.990 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:35.990 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:35.994 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:35.990+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:36.006 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T22:07:36.095 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:36.092+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:36.168 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:36.168 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:36.183 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T22:07:36.196 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:36.192+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:36.283 INFO:teuthology.orchestra.run.smithi039.stderr:reweighted osd.1 to 1 (10000) 2024-09-21T22:07:36.297 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:36.294+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:36.308 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-09-21T22:07:36.310 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:36.310 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:36.331 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T22:07:36.398 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:36.396+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:36.500 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:07:36.496+0000 7f2075a06640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31720) UID: 0 2024-09-21T22:07:36.502 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:36.502 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:36.519 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T22:07:36.601 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:36.598+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:36.640 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:36.640 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:36.657 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T22:07:36.702 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:36.696+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:36.803 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:36.800+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:36.825 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:36.826 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:36.839 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T22:07:36.904 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:36.900+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:36.982 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:36.982 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:36.996 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T22:07:37.004 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:37.002+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:37.064 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-21T22:07:37.108 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:37.106+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:37.135 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:37.135 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:37.149 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T22:07:37.206 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:07:37.200+0000 7f2075a06640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31720) UID: 0 2024-09-21T22:07:37.280 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:37.280 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:37.294 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T22:07:37.297 INFO:teuthology.orchestra.run.smithi039.stderr:reweighted osd.3 to 1 (10000) 2024-09-21T22:07:37.327 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-09-21T22:07:37.338 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:37.336+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:37.412 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:37.410+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:37.469 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:37.469 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:37.484 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T22:07:37.534 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:37.532+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:37.610 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:37.608+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:37.610 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:37.611 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:37.625 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T22:07:37.632 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T22:07:37.711 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:37.706+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:37.782 INFO:teuthology.orchestra.run.smithi039.stdout:osd.0: {} 2024-09-21T22:07:37.783 INFO:teuthology.orchestra.run.smithi039.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T22:07:37.795 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:37.795 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:37.799 INFO:teuthology.orchestra.run.smithi039.stdout:osd.1: {} 2024-09-21T22:07:37.800 INFO:teuthology.orchestra.run.smithi039.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T22:07:37.810 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T22:07:37.811 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:37.808+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:37.816 INFO:teuthology.orchestra.run.smithi039.stdout:osd.2: {} 2024-09-21T22:07:37.816 INFO:teuthology.orchestra.run.smithi039.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T22:07:37.835 INFO:teuthology.orchestra.run.smithi039.stdout:osd.3: {} 2024-09-21T22:07:37.835 INFO:teuthology.orchestra.run.smithi039.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T22:07:37.853 INFO:teuthology.orchestra.run.smithi039.stdout:osd.4: {} 2024-09-21T22:07:37.853 INFO:teuthology.orchestra.run.smithi039.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T22:07:37.871 INFO:teuthology.orchestra.run.smithi039.stdout:osd.5: {} 2024-09-21T22:07:37.871 INFO:teuthology.orchestra.run.smithi039.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T22:07:37.912 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:37.908+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:37.957 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:07:37.957 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-21T22:07:37.957 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-21T22:07:37.957 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-21T22:07:37.957 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.958 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4887.0:1 3.3 3:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-21T22:07:37.958 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:17.151514+0000", 2024-09-21T22:07:37.958 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 20.803243103, 2024-09-21T22:07:37.958 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.01157512, 2024-09-21T22:07:37.958 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:37.958 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:37.958 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:37.958 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:37.958 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4887", 2024-09-21T22:07:37.958 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/3431621434", 2024-09-21T22:07:37.958 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1 2024-09-21T22:07:37.958 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:37.958 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:37.958 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.958 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:37.959 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.151514+0000", 2024-09-21T22:07:37.959 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:37.959 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:37.959 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.959 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:37.959 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.151514+0000", 2024-09-21T22:07:37.959 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:37.959 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:37.959 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.959 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:37.959 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.151515+0000", 2024-09-21T22:07:37.959 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.469e-06 2024-09-21T22:07:37.959 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:37.959 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.960 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:37.960 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.151522+0000", 2024-09-21T22:07:37.960 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1330000000000001e-06 2024-09-21T22:07:37.960 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:37.960 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.960 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:37.960 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.151536+0000", 2024-09-21T22:07:37.960 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3563e-05 2024-09-21T22:07:37.960 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:37.960 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.960 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:37.960 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.151574+0000", 2024-09-21T22:07:37.960 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8467999999999998e-05 2024-09-21T22:07:37.960 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:37.960 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.961 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:37.961 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.151665+0000", 2024-09-21T22:07:37.961 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0668000000000007e-05 2024-09-21T22:07:37.961 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:37.961 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.961 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-21T22:07:37.961 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.151676+0000", 2024-09-21T22:07:37.961 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1443999999999999e-05 2024-09-21T22:07:37.961 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:37.961 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.961 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:37.961 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.161694+0000", 2024-09-21T22:07:37.961 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010017402999999999 2024-09-21T22:07:37.961 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:37.961 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.962 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:37.962 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.161805+0000", 2024-09-21T22:07:37.962 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011122500000000001 2024-09-21T22:07:37.962 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:37.962 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.962 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:37.962 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.161937+0000", 2024-09-21T22:07:37.962 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013177299999999999 2024-09-21T22:07:37.962 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:37.962 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.962 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:37.962 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.162555+0000", 2024-09-21T22:07:37.962 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00061786199999999995 2024-09-21T22:07:37.963 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:37.963 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.963 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:37.963 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.163021+0000", 2024-09-21T22:07:37.963 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000465749 2024-09-21T22:07:37.963 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:37.963 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.963 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:37.963 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.163059+0000", 2024-09-21T22:07:37.963 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8238000000000002e-05 2024-09-21T22:07:37.963 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:37.963 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.963 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:37.963 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.163089+0000", 2024-09-21T22:07:37.963 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0125e-05 2024-09-21T22:07:37.964 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:37.964 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:37.964 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:37.964 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:37.964 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.964 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.4887.0:1 3.3 e59/58 ondisk, result = 0)", 2024-09-21T22:07:37.964 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:17.162939+0000", 2024-09-21T22:07:37.964 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 20.791817712, 2024-09-21T22:07:37.964 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015730900000000001, 2024-09-21T22:07:37.964 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:37.964 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:37.964 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:07:37.964 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:37.964 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.965 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:37.965 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.162939+0000", 2024-09-21T22:07:37.965 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:37.965 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:37.965 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.965 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:37.965 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.162939+0000", 2024-09-21T22:07:37.965 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:37.965 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:37.965 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.965 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:37.965 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.162940+0000", 2024-09-21T22:07:37.965 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4499999999999996e-07 2024-09-21T22:07:37.965 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:37.965 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.966 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:37.966 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.162945+0000", 2024-09-21T22:07:37.966 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.758e-06 2024-09-21T22:07:37.966 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:37.966 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.966 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:37.966 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.162957+0000", 2024-09-21T22:07:37.966 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2259000000000001e-05 2024-09-21T22:07:37.966 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:37.966 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.966 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:37.966 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.162966+0000", 2024-09-21T22:07:37.966 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3270000000000007e-06 2024-09-21T22:07:37.966 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:37.966 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.967 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:37.967 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.163010+0000", 2024-09-21T22:07:37.967 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.422e-05 2024-09-21T22:07:37.967 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:37.967 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.967 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:37.967 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.163016+0000", 2024-09-21T22:07:37.967 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8579999999999996e-06 2024-09-21T22:07:37.967 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:37.967 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.967 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:37.967 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.163096+0000", 2024-09-21T22:07:37.967 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0142000000000006e-05 2024-09-21T22:07:37.967 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:37.967 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:37.968 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:37.968 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:37.968 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.968 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4840.0:675 2.7 e63/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 63'56, mlcod=63'54)", 2024-09-21T22:07:37.968 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.618817+0000", 2024-09-21T22:07:37.968 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 13.335940315, 2024-09-21T22:07:37.968 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0060329809999999998, 2024-09-21T22:07:37.968 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:37.968 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:37.968 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:37.968 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:37.968 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.968 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:37.968 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.618817+0000", 2024-09-21T22:07:37.968 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:37.969 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:37.969 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.969 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:37.969 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.618817+0000", 2024-09-21T22:07:37.969 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:37.969 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:37.969 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.969 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:37.969 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.618819+0000", 2024-09-21T22:07:37.969 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3070000000000001e-06 2024-09-21T22:07:37.969 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:37.969 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.969 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:37.969 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.618934+0000", 2024-09-21T22:07:37.969 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000114669 2024-09-21T22:07:37.970 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:37.970 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.970 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:37.970 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.618969+0000", 2024-09-21T22:07:37.970 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5985999999999997e-05 2024-09-21T22:07:37.970 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:37.970 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.970 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:37.970 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.618985+0000", 2024-09-21T22:07:37.970 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5840000000000001e-05 2024-09-21T22:07:37.970 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:37.970 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.970 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:37.970 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.619101+0000", 2024-09-21T22:07:37.970 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000115591 2024-09-21T22:07:37.971 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:37.971 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.971 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:37.971 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.619136+0000", 2024-09-21T22:07:37.971 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5287000000000003e-05 2024-09-21T22:07:37.971 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:37.971 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.971 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:37.971 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.624766+0000", 2024-09-21T22:07:37.971 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0056301500000000004 2024-09-21T22:07:37.971 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:37.971 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.971 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:37.971 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.624850+0000", 2024-09-21T22:07:37.972 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3151000000000004e-05 2024-09-21T22:07:37.972 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:37.972 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:37.972 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:37.972 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:37.972 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.972 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4840.0:676 2.7 e63/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 63'57, mlcod=63'55)", 2024-09-21T22:07:37.972 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.627276+0000", 2024-09-21T22:07:37.972 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 13.327480554999999, 2024-09-21T22:07:37.972 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.037500787000000001, 2024-09-21T22:07:37.972 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:37.972 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:37.972 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:37.972 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:37.972 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.973 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:37.973 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.627276+0000", 2024-09-21T22:07:37.973 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:37.973 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:37.973 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.973 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:37.973 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.627276+0000", 2024-09-21T22:07:37.973 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:37.973 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:37.973 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.973 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:37.973 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.627277+0000", 2024-09-21T22:07:37.973 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.068e-06 2024-09-21T22:07:37.973 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:37.973 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.974 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:37.974 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.628148+0000", 2024-09-21T22:07:37.974 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00087069999999999997 2024-09-21T22:07:37.974 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:37.974 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.974 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:37.974 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.628170+0000", 2024-09-21T22:07:37.974 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2147999999999999e-05 2024-09-21T22:07:37.974 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:37.974 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.974 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:37.974 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.628179+0000", 2024-09-21T22:07:37.974 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2450000000000004e-06 2024-09-21T22:07:37.974 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:37.974 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.975 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:37.975 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.628279+0000", 2024-09-21T22:07:37.975 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9298999999999999e-05 2024-09-21T22:07:37.975 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:37.975 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.975 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:37.975 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.628305+0000", 2024-09-21T22:07:37.975 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6316999999999999e-05 2024-09-21T22:07:37.975 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:37.975 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.975 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:37.975 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.664714+0000", 2024-09-21T22:07:37.975 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.036408923000000003 2024-09-21T22:07:37.975 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:37.975 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.976 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:37.976 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.664777+0000", 2024-09-21T22:07:37.976 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3087000000000001e-05 2024-09-21T22:07:37.976 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:37.976 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:37.976 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:37.976 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:37.976 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.976 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4840.0:677 2.7 e63/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 63'58, mlcod=63'56)", 2024-09-21T22:07:37.976 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.667210+0000", 2024-09-21T22:07:37.976 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 13.287547315999999, 2024-09-21T22:07:37.976 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010046522, 2024-09-21T22:07:37.976 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:37.976 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:37.976 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:37.977 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:37.977 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.977 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:37.977 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.667210+0000", 2024-09-21T22:07:37.977 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:37.977 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:37.977 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.977 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:37.977 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.667210+0000", 2024-09-21T22:07:37.977 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:37.977 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:37.977 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.977 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:37.977 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.667211+0000", 2024-09-21T22:07:37.977 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3379999999999999e-06 2024-09-21T22:07:37.978 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:37.978 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.978 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:37.978 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.667272+0000", 2024-09-21T22:07:37.978 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1257e-05 2024-09-21T22:07:37.978 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:37.978 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.978 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:37.978 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.667295+0000", 2024-09-21T22:07:37.978 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2637999999999999e-05 2024-09-21T22:07:37.978 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:37.978 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.978 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:37.978 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.667305+0000", 2024-09-21T22:07:37.979 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0044000000000001e-05 2024-09-21T22:07:37.979 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:37.979 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.979 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:37.979 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.667425+0000", 2024-09-21T22:07:37.979 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000120605 2024-09-21T22:07:37.979 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:37.979 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.979 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:37.979 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.667476+0000", 2024-09-21T22:07:37.979 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0843000000000001e-05 2024-09-21T22:07:37.979 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:37.979 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.979 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:37.979 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.677180+0000", 2024-09-21T22:07:37.980 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0097035860000000002 2024-09-21T22:07:37.980 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:37.980 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.980 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:37.980 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.677256+0000", 2024-09-21T22:07:37.980 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6211000000000006e-05 2024-09-21T22:07:37.980 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:37.980 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:37.980 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:37.980 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:37.980 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.980 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4840.0:678 2.7 e63/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 63'59, mlcod=63'57)", 2024-09-21T22:07:37.980 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.679689+0000", 2024-09-21T22:07:37.980 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 13.275068055, 2024-09-21T22:07:37.980 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053308406000000003, 2024-09-21T22:07:37.980 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:37.981 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:37.981 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:37.981 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:37.981 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.981 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:37.981 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.679689+0000", 2024-09-21T22:07:37.981 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:37.981 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:37.981 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.981 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:37.981 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.679689+0000", 2024-09-21T22:07:37.981 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:37.981 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:37.981 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.981 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:37.982 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.679691+0000", 2024-09-21T22:07:37.982 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9400000000000001e-06 2024-09-21T22:07:37.982 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:37.982 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.982 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:37.982 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.680245+0000", 2024-09-21T22:07:37.982 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00055382999999999995 2024-09-21T22:07:37.982 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:37.982 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.982 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:37.982 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.680276+0000", 2024-09-21T22:07:37.982 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1123e-05 2024-09-21T22:07:37.982 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:37.982 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.983 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:37.983 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.680290+0000", 2024-09-21T22:07:37.983 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4056999999999999e-05 2024-09-21T22:07:37.983 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:37.983 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.983 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:37.983 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.680418+0000", 2024-09-21T22:07:37.983 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000127792 2024-09-21T22:07:37.983 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:37.983 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.983 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:37.983 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.680465+0000", 2024-09-21T22:07:37.983 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7360000000000001e-05 2024-09-21T22:07:37.983 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:37.983 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.984 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:37.984 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.732923+0000", 2024-09-21T22:07:37.984 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.052458312999999999 2024-09-21T22:07:37.984 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:37.984 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.984 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:37.984 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.732997+0000", 2024-09-21T22:07:37.984 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3991000000000004e-05 2024-09-21T22:07:37.984 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:37.984 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:37.984 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:37.984 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:37.984 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.984 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4840.0:679 2.7 e63/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 63'60, mlcod=63'58)", 2024-09-21T22:07:37.984 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.735615+0000", 2024-09-21T22:07:37.985 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 13.219141902, 2024-09-21T22:07:37.985 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0077878399999999999, 2024-09-21T22:07:37.985 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:37.985 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:37.985 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:37.985 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:37.985 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.985 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:37.985 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.735615+0000", 2024-09-21T22:07:37.985 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:37.985 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:37.985 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.985 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:37.985 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.735615+0000", 2024-09-21T22:07:37.985 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:37.986 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:37.986 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.986 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:37.986 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.735617+0000", 2024-09-21T22:07:37.986 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7999999999999999e-06 2024-09-21T22:07:37.986 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:37.986 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.986 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:37.986 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.735706+0000", 2024-09-21T22:07:37.986 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8839999999999996e-05 2024-09-21T22:07:37.986 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:37.986 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.986 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:37.986 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.735736+0000", 2024-09-21T22:07:37.987 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9955999999999999e-05 2024-09-21T22:07:37.987 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:37.987 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.987 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:37.987 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.735755+0000", 2024-09-21T22:07:37.987 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9283000000000002e-05 2024-09-21T22:07:37.987 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:37.987 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.987 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:37.987 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.735857+0000", 2024-09-21T22:07:37.987 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000101681 2024-09-21T22:07:37.987 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:37.987 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.987 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:37.987 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.735894+0000", 2024-09-21T22:07:37.988 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7487999999999998e-05 2024-09-21T22:07:37.988 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:37.988 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.988 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:37.988 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.743321+0000", 2024-09-21T22:07:37.988 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0074270050000000004 2024-09-21T22:07:37.988 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:37.988 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.988 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:37.988 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.743403+0000", 2024-09-21T22:07:37.988 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1786999999999995e-05 2024-09-21T22:07:37.988 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:37.988 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:37.988 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:37.989 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:37.989 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.989 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4840.0:680 2.7 e63/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 63'61, mlcod=63'59)", 2024-09-21T22:07:37.989 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.745996+0000", 2024-09-21T22:07:37.989 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 13.208760542, 2024-09-21T22:07:37.989 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.061804132999999997, 2024-09-21T22:07:37.989 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:37.989 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:37.989 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:37.989 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:37.989 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.989 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:37.989 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.745996+0000", 2024-09-21T22:07:37.989 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:37.990 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:37.990 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.990 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:37.990 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.745996+0000", 2024-09-21T22:07:37.990 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:37.990 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:37.990 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.990 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:37.990 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.745998+0000", 2024-09-21T22:07:37.990 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.717e-06 2024-09-21T22:07:37.990 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:37.990 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.990 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:37.990 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.746523+0000", 2024-09-21T22:07:37.990 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00052525399999999996 2024-09-21T22:07:37.991 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:37.991 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.991 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:37.991 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.746554+0000", 2024-09-21T22:07:37.991 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0586000000000002e-05 2024-09-21T22:07:37.991 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:37.991 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.991 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:37.991 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.746577+0000", 2024-09-21T22:07:37.991 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3079e-05 2024-09-21T22:07:37.991 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:37.991 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.991 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:37.991 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.746689+0000", 2024-09-21T22:07:37.991 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011177 2024-09-21T22:07:37.992 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:37.992 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.992 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:37.992 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.746722+0000", 2024-09-21T22:07:37.992 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2870000000000002e-05 2024-09-21T22:07:37.992 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:37.992 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.992 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:37.992 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.807740+0000", 2024-09-21T22:07:37.992 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.061018612 2024-09-21T22:07:37.992 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:37.992 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.992 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:37.992 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.807800+0000", 2024-09-21T22:07:37.992 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0244999999999997e-05 2024-09-21T22:07:37.993 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:37.993 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:37.993 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:37.993 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:37.993 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.993 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4840.0:681 2.7 e63/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 63'62, mlcod=63'60)", 2024-09-21T22:07:37.993 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.810646+0000", 2024-09-21T22:07:37.993 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 13.144111118, 2024-09-21T22:07:37.993 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010715685000000001, 2024-09-21T22:07:37.993 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:37.993 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:37.993 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:37.993 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:37.993 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.993 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:37.994 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.810646+0000", 2024-09-21T22:07:37.994 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:37.994 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:37.994 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.994 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:37.994 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.810646+0000", 2024-09-21T22:07:37.994 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:37.994 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:37.994 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.994 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:37.994 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.810649+0000", 2024-09-21T22:07:37.994 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8569999999999999e-06 2024-09-21T22:07:37.994 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:37.994 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.995 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:37.995 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.810780+0000", 2024-09-21T22:07:37.995 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013114600000000001 2024-09-21T22:07:37.995 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:37.995 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.995 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:37.995 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.810819+0000", 2024-09-21T22:07:37.995 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9363000000000002e-05 2024-09-21T22:07:37.995 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:37.995 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.995 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:37.995 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.810839+0000", 2024-09-21T22:07:37.995 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9865999999999999e-05 2024-09-21T22:07:37.995 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:37.995 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.995 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:37.996 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.810992+0000", 2024-09-21T22:07:37.996 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015333299999999999 2024-09-21T22:07:37.996 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:37.996 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.996 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:37.996 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.811037+0000", 2024-09-21T22:07:37.996 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.464e-05 2024-09-21T22:07:37.996 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:37.996 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.996 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:37.996 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.821280+0000", 2024-09-21T22:07:37.996 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010243254 2024-09-21T22:07:37.996 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:37.996 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.996 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:37.997 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.821361+0000", 2024-09-21T22:07:37.997 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1225999999999998e-05 2024-09-21T22:07:37.997 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:37.997 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:37.997 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:37.997 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:37.997 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.997 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4840.0:682 2.7 e63/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 63'63, mlcod=63'61)", 2024-09-21T22:07:37.997 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.824126+0000", 2024-09-21T22:07:37.997 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 13.130630973000001, 2024-09-21T22:07:37.997 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.038159327, 2024-09-21T22:07:37.997 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:37.997 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:37.997 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:37.998 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:37.998 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.998 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:37.998 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.824126+0000", 2024-09-21T22:07:37.998 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:37.998 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:37.998 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.998 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:37.998 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.824126+0000", 2024-09-21T22:07:37.998 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:37.998 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:37.998 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.998 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:37.998 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.824128+0000", 2024-09-21T22:07:37.998 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9659999999999999e-06 2024-09-21T22:07:37.999 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:37.999 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.999 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:37.999 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.824842+0000", 2024-09-21T22:07:37.999 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00071423999999999999 2024-09-21T22:07:37.999 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:37.999 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.999 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:37.999 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.824883+0000", 2024-09-21T22:07:37.999 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1131999999999999e-05 2024-09-21T22:07:37.999 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:37.999 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:37.999 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:37.999 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.824931+0000", 2024-09-21T22:07:38.000 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7327000000000003e-05 2024-09-21T22:07:38.000 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.000 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.000 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:38.000 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.825058+0000", 2024-09-21T22:07:38.000 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012790499999999999 2024-09-21T22:07:38.000 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.000 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.000 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:38.000 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.825087+0000", 2024-09-21T22:07:38.000 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8192e-05 2024-09-21T22:07:38.000 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.000 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.000 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:38.000 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.862225+0000", 2024-09-21T22:07:38.001 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.037138309000000001 2024-09-21T22:07:38.001 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.001 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.001 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:38.001 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.862285+0000", 2024-09-21T22:07:38.001 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0256e-05 2024-09-21T22:07:38.001 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:38.001 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:38.001 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:38.001 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.001 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.001 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4963.0:15 2.7 e68/16 2:f45e3993:::rbd_object_map.126483d75f2d:head v 68'102, mlcod=68'100)", 2024-09-21T22:07:38.001 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:33.835091+0000", 2024-09-21T22:07:38.001 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.1196658490000004, 2024-09-21T22:07:38.001 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000910661, 2024-09-21T22:07:38.002 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:38.002 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:38.002 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:38.002 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:38.002 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.002 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:38.002 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.835091+0000", 2024-09-21T22:07:38.002 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:38.002 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.002 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.002 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:38.002 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.835091+0000", 2024-09-21T22:07:38.002 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:38.002 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.003 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.003 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:38.003 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.835093+0000", 2024-09-21T22:07:38.003 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7889999999999999e-06 2024-09-21T22:07:38.003 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.003 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.003 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:38.003 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.835103+0000", 2024-09-21T22:07:38.003 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0461e-05 2024-09-21T22:07:38.003 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.003 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.003 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:38.003 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.835125+0000", 2024-09-21T22:07:38.003 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1593000000000001e-05 2024-09-21T22:07:38.004 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.004 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.004 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:38.004 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.835145+0000", 2024-09-21T22:07:38.004 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0397000000000001e-05 2024-09-21T22:07:38.004 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.004 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.004 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:38.004 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.835221+0000", 2024-09-21T22:07:38.004 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5473000000000001e-05 2024-09-21T22:07:38.004 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.004 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.004 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:38.004 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.835245+0000", 2024-09-21T22:07:38.004 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4256e-05 2024-09-21T22:07:38.005 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.005 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.005 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:38.005 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.835948+0000", 2024-09-21T22:07:38.005 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00070343499999999998 2024-09-21T22:07:38.005 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.005 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.005 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:38.005 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.836002+0000", 2024-09-21T22:07:38.005 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3257000000000002e-05 2024-09-21T22:07:38.005 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:38.005 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:38.005 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:38.005 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.005 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.006 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4963.0:18 2.7 e68/16 2:f45e3993:::rbd_object_map.126483d75f2d:head v 68'103, mlcod=68'101)", 2024-09-21T22:07:38.006 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:33.837758+0000", 2024-09-21T22:07:38.006 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.1169986710000002, 2024-09-21T22:07:38.006 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0014775649999999999, 2024-09-21T22:07:38.006 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:38.006 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:38.006 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:38.006 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:38.006 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.006 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:38.006 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.837758+0000", 2024-09-21T22:07:38.006 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:38.006 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.006 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.007 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:38.007 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.837758+0000", 2024-09-21T22:07:38.007 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:38.007 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.007 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.007 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:38.007 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.837759+0000", 2024-09-21T22:07:38.007 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7199999999999997e-07 2024-09-21T22:07:38.007 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.007 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.007 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:38.007 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.837766+0000", 2024-09-21T22:07:38.007 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5969999999999997e-06 2024-09-21T22:07:38.007 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.008 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:38.008 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.837781+0000", 2024-09-21T22:07:38.008 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5602000000000001e-05 2024-09-21T22:07:38.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.008 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:38.008 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.837793+0000", 2024-09-21T22:07:38.008 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1816e-05 2024-09-21T22:07:38.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.008 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:38.008 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.837922+0000", 2024-09-21T22:07:38.008 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012941600000000001 2024-09-21T22:07:38.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.009 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.009 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:38.009 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.837953+0000", 2024-09-21T22:07:38.009 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0729999999999999e-05 2024-09-21T22:07:38.009 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.009 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.009 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:38.009 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.839185+0000", 2024-09-21T22:07:38.009 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0012320059999999999 2024-09-21T22:07:38.009 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.009 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.009 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:38.009 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.839236+0000", 2024-09-21T22:07:38.009 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0525999999999997e-05 2024-09-21T22:07:38.009 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:38.010 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:38.010 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:38.010 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.010 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.010 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4963.0:23 2.7 e68/16 2:f45e3993:::rbd_object_map.126483d75f2d:head v 68'104, mlcod=68'102)", 2024-09-21T22:07:38.010 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:33.844296+0000", 2024-09-21T22:07:38.010 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.1104604010000001, 2024-09-21T22:07:38.010 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0010385679999999999, 2024-09-21T22:07:38.010 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:38.010 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:38.010 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:38.010 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:38.010 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.010 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:38.011 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.844296+0000", 2024-09-21T22:07:38.011 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:38.011 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.011 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.011 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:38.011 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.844296+0000", 2024-09-21T22:07:38.011 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:38.011 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.011 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.011 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:38.011 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.844298+0000", 2024-09-21T22:07:38.011 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3230000000000001e-06 2024-09-21T22:07:38.011 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.011 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.011 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:38.012 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.844307+0000", 2024-09-21T22:07:38.012 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4350000000000003e-06 2024-09-21T22:07:38.012 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.012 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:38.012 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.844327+0000", 2024-09-21T22:07:38.012 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0136000000000002e-05 2024-09-21T22:07:38.012 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.012 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:38.012 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.844338+0000", 2024-09-21T22:07:38.012 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0474e-05 2024-09-21T22:07:38.012 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.013 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:38.013 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.844460+0000", 2024-09-21T22:07:38.013 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012226399999999999 2024-09-21T22:07:38.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.013 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.013 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:38.013 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.844488+0000", 2024-09-21T22:07:38.013 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7965000000000001e-05 2024-09-21T22:07:38.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.013 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.013 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:38.013 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.845290+0000", 2024-09-21T22:07:38.013 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00080154599999999999 2024-09-21T22:07:38.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.013 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.014 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:38.014 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.845335+0000", 2024-09-21T22:07:38.014 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5425e-05 2024-09-21T22:07:38.014 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:38.014 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:38.014 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:38.014 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.014 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.014 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4981.0:15 2.7 e68/16 2:f45e3993:::rbd_object_map.126483d75f2d:head v 68'105, mlcod=68'103)", 2024-09-21T22:07:38.014 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:33.956875+0000", 2024-09-21T22:07:38.014 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.9978820910000001, 2024-09-21T22:07:38.014 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00093627499999999998, 2024-09-21T22:07:38.014 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:38.014 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:38.014 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:38.015 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:38.015 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.015 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:38.015 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.956875+0000", 2024-09-21T22:07:38.015 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:38.015 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.015 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.015 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:38.015 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.956875+0000", 2024-09-21T22:07:38.015 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:38.015 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.015 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.015 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:38.015 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.956876+0000", 2024-09-21T22:07:38.015 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9600000000000008e-07 2024-09-21T22:07:38.016 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.016 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.016 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:38.016 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.956884+0000", 2024-09-21T22:07:38.016 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7919999999999993e-06 2024-09-21T22:07:38.016 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.016 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.016 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:38.016 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.956899+0000", 2024-09-21T22:07:38.016 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5339999999999999e-05 2024-09-21T22:07:38.016 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.016 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.016 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:38.016 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.956906+0000", 2024-09-21T22:07:38.017 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1509999999999998e-06 2024-09-21T22:07:38.017 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.017 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.017 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:38.017 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.956985+0000", 2024-09-21T22:07:38.017 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8807999999999995e-05 2024-09-21T22:07:38.017 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.017 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.017 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:38.017 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.957004+0000", 2024-09-21T22:07:38.017 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9213000000000001e-05 2024-09-21T22:07:38.017 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.017 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.017 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:38.017 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.957781+0000", 2024-09-21T22:07:38.018 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00077705600000000004 2024-09-21T22:07:38.018 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.018 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.018 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:38.018 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.957811+0000", 2024-09-21T22:07:38.018 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9918999999999999e-05 2024-09-21T22:07:38.018 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:38.018 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:38.018 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:38.018 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.018 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.018 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4981.0:20 2.7 e68/16 2:f45e3993:::rbd_object_map.126483d75f2d:head v 68'106, mlcod=68'104)", 2024-09-21T22:07:38.018 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:33.962453+0000", 2024-09-21T22:07:38.018 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.9923033829999999, 2024-09-21T22:07:38.019 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001372926, 2024-09-21T22:07:38.019 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:38.019 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:38.019 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:38.019 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:38.019 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.019 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:38.019 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.962453+0000", 2024-09-21T22:07:38.019 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:38.019 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.019 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.019 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:38.019 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.962453+0000", 2024-09-21T22:07:38.019 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:38.019 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.020 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.020 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:38.020 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.962455+0000", 2024-09-21T22:07:38.020 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1400000000000001e-06 2024-09-21T22:07:38.020 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.020 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.020 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:38.020 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.962467+0000", 2024-09-21T22:07:38.020 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2502e-05 2024-09-21T22:07:38.020 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.020 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.020 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:38.020 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.962490+0000", 2024-09-21T22:07:38.020 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3166000000000001e-05 2024-09-21T22:07:38.020 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.021 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.021 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:38.021 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.962503+0000", 2024-09-21T22:07:38.021 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3118999999999999e-05 2024-09-21T22:07:38.021 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.021 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.021 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:38.021 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.962579+0000", 2024-09-21T22:07:38.021 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5412000000000002e-05 2024-09-21T22:07:38.021 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.021 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.021 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:38.021 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.962602+0000", 2024-09-21T22:07:38.021 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3496999999999999e-05 2024-09-21T22:07:38.021 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.022 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.022 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:38.022 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.963782+0000", 2024-09-21T22:07:38.022 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011801170000000001 2024-09-21T22:07:38.022 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.022 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.022 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:38.022 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.963826+0000", 2024-09-21T22:07:38.022 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3973000000000001e-05 2024-09-21T22:07:38.022 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:38.022 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:38.022 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:38.022 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.022 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.022 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4981.0:23 2.7 e68/16 2:f45e3993:::rbd_object_map.126483d75f2d:head v 68'107, mlcod=68'105)", 2024-09-21T22:07:38.023 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:33.966404+0000", 2024-09-21T22:07:38.023 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.988352452, 2024-09-21T22:07:38.023 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00086599099999999998, 2024-09-21T22:07:38.023 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:38.023 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:38.023 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:38.023 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:38.023 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.023 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:38.023 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.966404+0000", 2024-09-21T22:07:38.023 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:38.023 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.023 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.023 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:38.023 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.966404+0000", 2024-09-21T22:07:38.024 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:38.024 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.024 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.024 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:38.024 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.966405+0000", 2024-09-21T22:07:38.024 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0529999999999999e-06 2024-09-21T22:07:38.024 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.024 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.024 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:38.024 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.966413+0000", 2024-09-21T22:07:38.024 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3849999999999996e-06 2024-09-21T22:07:38.024 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.024 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.024 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:38.024 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.966432+0000", 2024-09-21T22:07:38.025 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8947e-05 2024-09-21T22:07:38.025 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.025 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.025 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:38.025 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.966444+0000", 2024-09-21T22:07:38.025 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1929e-05 2024-09-21T22:07:38.025 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.025 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.025 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:38.025 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.966529+0000", 2024-09-21T22:07:38.025 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5030000000000004e-05 2024-09-21T22:07:38.025 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.025 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.025 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:38.025 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.966560+0000", 2024-09-21T22:07:38.026 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1590000000000001e-05 2024-09-21T22:07:38.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.026 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.026 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:38.026 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.967239+0000", 2024-09-21T22:07:38.026 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00067846499999999995 2024-09-21T22:07:38.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.026 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.026 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:38.026 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:33.967270+0000", 2024-09-21T22:07:38.026 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1591999999999998e-05 2024-09-21T22:07:38.026 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:38.026 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:38.027 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:38.027 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.027 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.027 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5008.0:114 2.7 e71/16 2:f45e3993:::rbd_object_map.126483d75f2d:head v 71'146, mlcod=71'144)", 2024-09-21T22:07:38.027 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:37.857883+0000", 2024-09-21T22:07:38.027 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.096873781000000006, 2024-09-21T22:07:38.027 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001130254, 2024-09-21T22:07:38.027 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:38.027 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:38.027 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:38.027 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:38.027 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.027 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:38.027 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.857883+0000", 2024-09-21T22:07:38.027 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:38.028 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.028 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.028 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:38.028 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.857883+0000", 2024-09-21T22:07:38.028 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:38.028 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.028 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.028 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:38.028 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.857884+0000", 2024-09-21T22:07:38.028 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.012e-06 2024-09-21T22:07:38.028 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.028 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.028 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:38.028 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.857891+0000", 2024-09-21T22:07:38.028 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8379999999999996e-06 2024-09-21T22:07:38.029 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.029 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.029 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:38.029 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.857904+0000", 2024-09-21T22:07:38.029 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3261999999999999e-05 2024-09-21T22:07:38.029 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.029 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.029 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:38.029 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.857919+0000", 2024-09-21T22:07:38.029 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4365999999999999e-05 2024-09-21T22:07:38.029 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.029 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.029 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:38.029 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.857992+0000", 2024-09-21T22:07:38.029 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3047999999999996e-05 2024-09-21T22:07:38.030 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.030 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.030 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:38.030 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.858020+0000", 2024-09-21T22:07:38.030 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8793999999999999e-05 2024-09-21T22:07:38.030 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.030 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.030 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:38.030 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.858971+0000", 2024-09-21T22:07:38.030 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00095029000000000001 2024-09-21T22:07:38.030 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.030 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.030 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:38.030 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.859013+0000", 2024-09-21T22:07:38.031 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2644e-05 2024-09-21T22:07:38.031 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:38.031 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:38.031 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:38.031 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.031 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.031 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5008.0:115 2.7 e71/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 71'147, mlcod=71'145)", 2024-09-21T22:07:38.031 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:37.860188+0000", 2024-09-21T22:07:38.031 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.094568939000000005, 2024-09-21T22:07:38.031 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0054120000000000001, 2024-09-21T22:07:38.031 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:38.031 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:38.031 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:38.031 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:38.031 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.032 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:38.032 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.860188+0000", 2024-09-21T22:07:38.032 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:38.032 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.032 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.032 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:38.032 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.860188+0000", 2024-09-21T22:07:38.032 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:38.032 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.032 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.032 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:38.032 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.860189+0000", 2024-09-21T22:07:38.032 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0800000000000004e-07 2024-09-21T22:07:38.032 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.033 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.033 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:38.033 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.860243+0000", 2024-09-21T22:07:38.033 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4666000000000001e-05 2024-09-21T22:07:38.033 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.033 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.033 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:38.033 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.860255+0000", 2024-09-21T22:07:38.033 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.171e-05 2024-09-21T22:07:38.033 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.033 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.033 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:38.033 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.860260+0000", 2024-09-21T22:07:38.033 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1159999999999998e-06 2024-09-21T22:07:38.033 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.034 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.034 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:38.034 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.860325+0000", 2024-09-21T22:07:38.034 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4796999999999996e-05 2024-09-21T22:07:38.034 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.034 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.034 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:38.034 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.860344+0000", 2024-09-21T22:07:38.034 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9403e-05 2024-09-21T22:07:38.034 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.034 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.034 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:38.034 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.865563+0000", 2024-09-21T22:07:38.034 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0052183280000000004 2024-09-21T22:07:38.034 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.035 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:38.035 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.865600+0000", 2024-09-21T22:07:38.035 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7172000000000003e-05 2024-09-21T22:07:38.035 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:38.035 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:38.035 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:38.035 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.035 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5008.0:116 2.7 e71/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 71'148, mlcod=71'146)", 2024-09-21T22:07:38.035 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:37.867612+0000", 2024-09-21T22:07:38.035 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.087144477999999997, 2024-09-21T22:07:38.035 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.045123155999999998, 2024-09-21T22:07:38.035 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:38.036 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:38.036 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:38.036 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:38.036 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.036 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:38.036 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.867612+0000", 2024-09-21T22:07:38.036 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:38.036 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.036 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.036 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:38.036 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.867612+0000", 2024-09-21T22:07:38.036 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:38.036 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.036 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.036 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:38.037 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.867613+0000", 2024-09-21T22:07:38.037 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1200000000000002e-07 2024-09-21T22:07:38.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.037 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.037 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:38.037 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.868135+0000", 2024-09-21T22:07:38.037 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00052132700000000003 2024-09-21T22:07:38.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.037 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.037 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:38.037 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.868150+0000", 2024-09-21T22:07:38.037 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5271e-05 2024-09-21T22:07:38.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.037 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.037 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:38.038 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.868156+0000", 2024-09-21T22:07:38.038 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4969999999999998e-06 2024-09-21T22:07:38.038 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.038 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.038 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:38.038 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.868213+0000", 2024-09-21T22:07:38.038 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7114999999999998e-05 2024-09-21T22:07:38.038 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.038 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.038 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:38.038 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.868238+0000", 2024-09-21T22:07:38.038 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4987999999999999e-05 2024-09-21T22:07:38.038 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.038 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.039 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:38.039 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.912677+0000", 2024-09-21T22:07:38.039 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.044439019000000003 2024-09-21T22:07:38.039 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.039 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.039 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:38.039 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.912736+0000", 2024-09-21T22:07:38.039 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8127000000000001e-05 2024-09-21T22:07:38.039 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:38.039 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:38.039 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:38.039 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.039 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.039 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5008.0:117 2.7 e71/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 71'149, mlcod=71'147)", 2024-09-21T22:07:38.039 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:37.914937+0000", 2024-09-21T22:07:38.040 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.039819465999999998, 2024-09-21T22:07:38.040 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0076223819999999996, 2024-09-21T22:07:38.040 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:38.040 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:38.040 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:38.040 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:38.040 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.040 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:38.040 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.914937+0000", 2024-09-21T22:07:38.040 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:38.040 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.040 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.040 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:38.040 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.914937+0000", 2024-09-21T22:07:38.040 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:38.041 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.041 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.041 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:38.041 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.914939+0000", 2024-09-21T22:07:38.041 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.826e-06 2024-09-21T22:07:38.041 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.041 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.041 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:38.041 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.915021+0000", 2024-09-21T22:07:38.041 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2244999999999996e-05 2024-09-21T22:07:38.041 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.041 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.041 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:38.041 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.915046+0000", 2024-09-21T22:07:38.041 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4794e-05 2024-09-21T22:07:38.041 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.042 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.042 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:38.042 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.915063+0000", 2024-09-21T22:07:38.042 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6773999999999999e-05 2024-09-21T22:07:38.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.042 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.042 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:38.042 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.915143+0000", 2024-09-21T22:07:38.042 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9516000000000002e-05 2024-09-21T22:07:38.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.042 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.042 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:38.042 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.915185+0000", 2024-09-21T22:07:38.042 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2324000000000003e-05 2024-09-21T22:07:38.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.043 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.043 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:38.043 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.922508+0000", 2024-09-21T22:07:38.043 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0073226419999999999 2024-09-21T22:07:38.043 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.043 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.043 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:38.043 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.922560+0000", 2024-09-21T22:07:38.043 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2261e-05 2024-09-21T22:07:38.043 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:38.043 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:38.043 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:38.043 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:38.043 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:38.043 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:07:38.045 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T22:07:38.045 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:38.040+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:38.145 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:38.142+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:38.214 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:07:38.214 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-21T22:07:38.214 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.214 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5008.0:129 2.3 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head [write 2228224~917504 in=917504b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-21T22:07:38.214 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:38.209297+0000", 2024-09-21T22:07:38.215 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.0054194940000000004, 2024-09-21T22:07:38.215 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0054540439999999999, 2024-09-21T22:07:38.215 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:38.215 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:38.215 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "waiting for sub ops", 2024-09-21T22:07:38.215 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:38.215 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5008", 2024-09-21T22:07:38.215 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/1400114666", 2024-09-21T22:07:38.215 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 129 2024-09-21T22:07:38.215 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.215 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:38.215 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.216 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:38.216 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.209297+0000", 2024-09-21T22:07:38.216 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:38.216 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.216 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.216 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:38.216 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.209297+0000", 2024-09-21T22:07:38.216 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:38.216 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.216 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.216 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:38.216 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.209299+0000", 2024-09-21T22:07:38.216 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.274e-06 2024-09-21T22:07:38.216 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.217 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.217 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:38.217 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.210077+0000", 2024-09-21T22:07:38.217 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00077849799999999997 2024-09-21T22:07:38.217 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.217 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.217 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:38.217 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.210094+0000", 2024-09-21T22:07:38.217 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6772000000000001e-05 2024-09-21T22:07:38.217 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.217 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.217 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:38.217 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.210103+0000", 2024-09-21T22:07:38.217 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6780000000000006e-06 2024-09-21T22:07:38.217 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.218 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.218 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:38.218 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.210254+0000", 2024-09-21T22:07:38.218 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015141699999999999 2024-09-21T22:07:38.218 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.218 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.218 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:38.218 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.210501+0000", 2024-09-21T22:07:38.218 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000246498 2024-09-21T22:07:38.218 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.218 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.218 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-09-21T22:07:38.218 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.210872+0000", 2024-09-21T22:07:38.218 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00037147500000000002 2024-09-21T22:07:38.219 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:38.219 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:38.219 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:38.219 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:38.219 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-21T22:07:38.219 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 1 2024-09-21T22:07:38.219 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:07:38.228 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T22:07:38.278 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:07:38.276+0000 7f2075a06640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31720) UID: 0 2024-09-21T22:07:38.326 INFO:teuthology.orchestra.run.smithi039.stderr:noscrub is set 2024-09-21T22:07:38.327 INFO:teuthology.orchestra.run.smithi039.stderr:reweighted osd.4 to 1 (10000) 2024-09-21T22:07:38.347 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:38.344+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:38.349 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:07:38.350 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:07:38.350 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-21T22:07:38.350 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-21T22:07:38.350 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:07:38.355 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-09-21T22:07:38.364 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T22:07:38.454 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:07:38.452+0000 7f2075a06640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31720) UID: 0 2024-09-21T22:07:38.532 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:07:38.532 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-21T22:07:38.532 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-21T22:07:38.532 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-21T22:07:38.532 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.532 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5031.0:3 2.2 2:53d5e6d0:::rbd_header.126483d75f2d:head [call rbd.get_size in=11b,call,call rbd.get_features in=15b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e65)", 2024-09-21T22:07:38.533 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:27.373436+0000", 2024-09-21T22:07:38.533 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 11.158049458000001, 2024-09-21T22:07:38.533 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.86524013700000002, 2024-09-21T22:07:38.533 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:38.533 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:38.533 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:07:38.533 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:38.533 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5031", 2024-09-21T22:07:38.533 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/3232705659", 2024-09-21T22:07:38.533 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3 2024-09-21T22:07:38.533 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.533 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:38.533 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.534 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:38.534 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:27.373436+0000", 2024-09-21T22:07:38.534 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:38.534 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.534 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.534 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:38.534 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:27.373436+0000", 2024-09-21T22:07:38.534 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:38.534 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.534 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.534 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:38.534 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:27.373439+0000", 2024-09-21T22:07:38.534 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5969999999999999e-06 2024-09-21T22:07:38.534 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.535 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.535 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:38.535 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:27.373450+0000", 2024-09-21T22:07:38.535 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1326e-05 2024-09-21T22:07:38.535 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.535 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.535 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:38.535 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:27.373471+0000", 2024-09-21T22:07:38.535 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0169999999999998e-05 2024-09-21T22:07:38.535 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.535 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.535 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:38.535 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:27.373486+0000", 2024-09-21T22:07:38.535 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5483000000000001e-05 2024-09-21T22:07:38.536 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.536 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.536 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:38.536 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:27.373602+0000", 2024-09-21T22:07:38.536 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000115603 2024-09-21T22:07:38.536 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.536 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.536 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-21T22:07:38.536 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:27.373617+0000", 2024-09-21T22:07:38.536 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4956999999999999e-05 2024-09-21T22:07:38.536 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.536 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.536 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:38.536 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.234578+0000", 2024-09-21T22:07:38.536 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.86096097900000002 2024-09-21T22:07:38.537 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.537 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.537 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for missing object", 2024-09-21T22:07:38.537 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.234771+0000", 2024-09-21T22:07:38.537 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019394700000000001 2024-09-21T22:07:38.537 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.537 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.537 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:38.537 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.237470+0000", 2024-09-21T22:07:38.537 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026981319999999998 2024-09-21T22:07:38.537 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.537 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.537 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:38.537 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.237791+0000", 2024-09-21T22:07:38.538 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00032138499999999999 2024-09-21T22:07:38.538 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.538 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.538 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:38.538 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.238677+0000", 2024-09-21T22:07:38.538 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00088555800000000005 2024-09-21T22:07:38.538 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:38.538 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:38.538 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:38.538 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.538 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.538 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.2 66/65 [PushOp(2:4481fad7:ns::rbd_data.127f79bc213d.0000000000000006:head, version: 57'9, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:4481fad7:ns::rbd_data.127f79bc213d.0000000000000006:head@57'9, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T22:07:38.538 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:28.247056+0000", 2024-09-21T22:07:38.538 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.284429794999999, 2024-09-21T22:07:38.539 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.045025012000000003, 2024-09-21T22:07:38.539 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:38.539 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:38.539 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:07:38.539 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:38.539 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.539 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:38.539 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.247056+0000", 2024-09-21T22:07:38.539 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:38.539 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.539 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.539 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:38.539 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.247056+0000", 2024-09-21T22:07:38.539 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:38.540 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.540 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.540 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:38.540 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.247057+0000", 2024-09-21T22:07:38.540 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2500000000000001e-06 2024-09-21T22:07:38.540 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.540 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.540 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:38.540 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.249384+0000", 2024-09-21T22:07:38.540 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0023265619999999999 2024-09-21T22:07:38.540 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.540 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.540 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:38.540 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.249429+0000", 2024-09-21T22:07:38.541 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5392999999999997e-05 2024-09-21T22:07:38.541 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.541 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.541 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:38.541 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.249476+0000", 2024-09-21T22:07:38.541 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6570000000000003e-05 2024-09-21T22:07:38.541 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.541 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.541 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:38.541 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.249611+0000", 2024-09-21T22:07:38.541 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000135143 2024-09-21T22:07:38.541 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.541 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.541 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:38.542 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.249626+0000", 2024-09-21T22:07:38.542 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.469e-05 2024-09-21T22:07:38.542 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.542 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.542 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:38.542 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.292081+0000", 2024-09-21T22:07:38.542 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.042455404000000002 2024-09-21T22:07:38.542 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:38.542 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:38.542 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:38.542 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.542 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.542 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5031.0:9 2.2 2:53d5e6d0:::rbd_header.126483d75f2d:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-21T22:07:38.542 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:28.250163+0000", 2024-09-21T22:07:38.543 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.281322401000001, 2024-09-21T22:07:38.543 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.042986430999999999, 2024-09-21T22:07:38.543 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:38.543 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:38.543 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:07:38.543 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:38.543 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5031", 2024-09-21T22:07:38.543 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/3232705659", 2024-09-21T22:07:38.543 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 9 2024-09-21T22:07:38.543 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.543 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:38.543 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.543 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:38.543 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.250163+0000", 2024-09-21T22:07:38.544 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:38.544 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.544 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.544 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:38.544 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.250163+0000", 2024-09-21T22:07:38.544 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:38.544 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.544 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.544 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:38.544 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.250166+0000", 2024-09-21T22:07:38.544 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7039999999999999e-06 2024-09-21T22:07:38.544 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.544 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.544 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:38.545 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.250175+0000", 2024-09-21T22:07:38.545 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7560000000000005e-06 2024-09-21T22:07:38.545 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.545 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.545 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:38.545 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.250199+0000", 2024-09-21T22:07:38.545 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4049999999999998e-05 2024-09-21T22:07:38.545 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.545 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.546 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:38.546 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.250218+0000", 2024-09-21T22:07:38.547 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9236e-05 2024-09-21T22:07:38.547 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.547 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.547 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:38.547 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.292193+0000", 2024-09-21T22:07:38.547 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.041974694 2024-09-21T22:07:38.547 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.547 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.547 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:38.547 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.292492+0000", 2024-09-21T22:07:38.547 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00029900100000000002 2024-09-21T22:07:38.547 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.547 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.547 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:38.547 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.293150+0000", 2024-09-21T22:07:38.548 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00065799000000000001 2024-09-21T22:07:38.548 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:38.548 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:38.548 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:38.548 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.548 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.548 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5031.0:12 2.2 2:53d5e6d0:::rbd_header.126483d75f2d:head [watch watch cookie 140495695137040] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-09-21T22:07:38.548 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:28.295580+0000", 2024-09-21T22:07:38.548 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.235905514000001, 2024-09-21T22:07:38.548 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.033090870000000001, 2024-09-21T22:07:38.548 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:38.548 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:38.548 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:38.548 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:38.548 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5031", 2024-09-21T22:07:38.549 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/3232705659", 2024-09-21T22:07:38.549 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 12 2024-09-21T22:07:38.549 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.549 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:38.549 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.549 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:38.549 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.295580+0000", 2024-09-21T22:07:38.549 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:38.549 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.549 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.549 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:38.549 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.295580+0000", 2024-09-21T22:07:38.549 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:38.549 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.550 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.550 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:38.550 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.295581+0000", 2024-09-21T22:07:38.550 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6799999999999999e-07 2024-09-21T22:07:38.550 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.550 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.550 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:38.550 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.295586+0000", 2024-09-21T22:07:38.550 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8659999999999997e-06 2024-09-21T22:07:38.550 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.550 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.550 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:38.550 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.295599+0000", 2024-09-21T22:07:38.550 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2897999999999999e-05 2024-09-21T22:07:38.550 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.551 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.551 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:38.551 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.295614+0000", 2024-09-21T22:07:38.551 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5406000000000001e-05 2024-09-21T22:07:38.551 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.551 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.551 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:38.551 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.295665+0000", 2024-09-21T22:07:38.551 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0501000000000003e-05 2024-09-21T22:07:38.553 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.554 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.554 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:38.554 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.295727+0000", 2024-09-21T22:07:38.554 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2073e-05 2024-09-21T22:07:38.554 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.554 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.554 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-09-21T22:07:38.554 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.295849+0000", 2024-09-21T22:07:38.554 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012190500000000001 2024-09-21T22:07:38.554 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.554 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.554 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:38.554 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.296429+0000", 2024-09-21T22:07:38.555 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00058019699999999996 2024-09-21T22:07:38.555 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.555 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.555 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:38.555 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.328308+0000", 2024-09-21T22:07:38.555 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.031878849000000001 2024-09-21T22:07:38.555 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.555 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.555 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:38.555 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.328421+0000", 2024-09-21T22:07:38.555 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000113322 2024-09-21T22:07:38.555 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.555 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.555 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:38.556 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.328671+0000", 2024-09-21T22:07:38.556 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024998499999999999 2024-09-21T22:07:38.556 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:38.556 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:38.556 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:38.556 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.556 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.556 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.2 66/65 [PushOp(2:4481fad7:ns::rbd_data.127f79bc213d.0000000000000006:head, version: 57'9, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:4481fad7:ns::rbd_data.127f79bc213d.0000000000000006:head@57'9, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:07:38.556 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:28.296301+0000", 2024-09-21T22:07:38.556 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.235184593, 2024-09-21T22:07:38.556 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.031905291000000002, 2024-09-21T22:07:38.556 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:38.556 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:38.556 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:07:38.556 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:38.557 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.557 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:38.557 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.296301+0000", 2024-09-21T22:07:38.557 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:38.557 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.557 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.557 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:38.557 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.296301+0000", 2024-09-21T22:07:38.557 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:38.557 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.557 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.557 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:38.557 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.296315+0000", 2024-09-21T22:07:38.557 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3867e-05 2024-09-21T22:07:38.557 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.557 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.558 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:38.558 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.297432+0000", 2024-09-21T22:07:38.558 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011164759999999999 2024-09-21T22:07:38.558 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.558 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.558 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:38.558 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.297454+0000", 2024-09-21T22:07:38.558 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2813e-05 2024-09-21T22:07:38.558 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.558 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.558 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:38.558 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.297479+0000", 2024-09-21T22:07:38.558 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4817e-05 2024-09-21T22:07:38.559 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.559 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.559 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:38.559 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.297582+0000", 2024-09-21T22:07:38.559 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010261500000000001 2024-09-21T22:07:38.559 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.559 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.559 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:38.559 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.297600+0000", 2024-09-21T22:07:38.559 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8323000000000001e-05 2024-09-21T22:07:38.559 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.559 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.559 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:38.559 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.328207+0000", 2024-09-21T22:07:38.560 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.030606379999999999 2024-09-21T22:07:38.560 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:38.560 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:38.560 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:38.560 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.560 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.560 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.5031.0:12 2.2 e66/65 ondisk, result = 0)", 2024-09-21T22:07:38.560 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:28.297512+0000", 2024-09-21T22:07:38.560 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.23397402, 2024-09-21T22:07:38.560 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.031170693999999999, 2024-09-21T22:07:38.560 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:38.560 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:38.560 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:07:38.561 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:38.561 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.561 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:38.561 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.297512+0000", 2024-09-21T22:07:38.561 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:38.561 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.561 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.561 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:38.561 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.297512+0000", 2024-09-21T22:07:38.561 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:38.561 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.561 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.561 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:38.561 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.297512+0000", 2024-09-21T22:07:38.562 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7700000000000005e-07 2024-09-21T22:07:38.562 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.562 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.562 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:38.562 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.297517+0000", 2024-09-21T22:07:38.562 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1039999999999998e-06 2024-09-21T22:07:38.562 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.562 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.562 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:38.562 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.297528+0000", 2024-09-21T22:07:38.562 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0633e-05 2024-09-21T22:07:38.562 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.562 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.562 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:38.562 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.297537+0000", 2024-09-21T22:07:38.563 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7199999999999995e-06 2024-09-21T22:07:38.563 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.563 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.563 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:38.563 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.328273+0000", 2024-09-21T22:07:38.563 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.030736574999999999 2024-09-21T22:07:38.563 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.563 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.563 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:38.563 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.328293+0000", 2024-09-21T22:07:38.563 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9304e-05 2024-09-21T22:07:38.563 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.564 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.564 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:38.564 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.328683+0000", 2024-09-21T22:07:38.564 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00038988100000000002 2024-09-21T22:07:38.564 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:38.564 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:38.564 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:38.564 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.564 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.565 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.2 66/65 [PushOp(2:4481fad7:ns::rbd_data.127f79bc213d.0000000000000006:head, version: 57'9, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:4481fad7:ns::rbd_data.127f79bc213d.0000000000000006:head@57'9, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:07:38.565 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:28.331166+0000", 2024-09-21T22:07:38.565 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.200319650999999, 2024-09-21T22:07:38.565 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.039995378999999998, 2024-09-21T22:07:38.565 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:38.565 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:38.565 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:07:38.565 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:38.565 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.565 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:38.565 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.331166+0000", 2024-09-21T22:07:38.565 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:38.565 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.565 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.565 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:38.566 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.331166+0000", 2024-09-21T22:07:38.566 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:38.566 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.566 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.566 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:38.566 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.331167+0000", 2024-09-21T22:07:38.566 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9199999999999999e-07 2024-09-21T22:07:38.566 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.566 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.566 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:38.566 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.332127+0000", 2024-09-21T22:07:38.566 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00095969800000000004 2024-09-21T22:07:38.566 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.566 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.567 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:38.567 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.332149+0000", 2024-09-21T22:07:38.567 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1974999999999999e-05 2024-09-21T22:07:38.567 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.567 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.567 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:38.567 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.332161+0000", 2024-09-21T22:07:38.567 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2276000000000001e-05 2024-09-21T22:07:38.567 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.567 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.567 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:38.567 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.332290+0000", 2024-09-21T22:07:38.567 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000129214 2024-09-21T22:07:38.567 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.568 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.568 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:38.568 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.332312+0000", 2024-09-21T22:07:38.568 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1308999999999999e-05 2024-09-21T22:07:38.568 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.568 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.568 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:38.568 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.371162+0000", 2024-09-21T22:07:38.568 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.038850015000000002 2024-09-21T22:07:38.568 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:38.568 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:38.568 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:38.568 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.569 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.569 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4811.0:3 2.3 2:c4c92e5a:::rbd_trash:head [call rbd.trash_list in=13b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-21T22:07:38.569 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:28.475306+0000", 2024-09-21T22:07:38.569 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.056180106999999, 2024-09-21T22:07:38.569 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.77996670099999998, 2024-09-21T22:07:38.569 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:38.569 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:38.569 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:07:38.569 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:38.569 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4811", 2024-09-21T22:07:38.569 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/2028736390", 2024-09-21T22:07:38.569 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3 2024-09-21T22:07:38.569 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.569 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:38.570 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.570 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:38.570 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.475306+0000", 2024-09-21T22:07:38.570 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:38.570 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.570 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.570 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:38.570 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.475306+0000", 2024-09-21T22:07:38.570 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:38.570 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.570 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.570 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:38.570 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.475308+0000", 2024-09-21T22:07:38.570 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8959999999999999e-06 2024-09-21T22:07:38.571 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.571 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.571 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:38.571 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.475314+0000", 2024-09-21T22:07:38.571 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2280000000000003e-06 2024-09-21T22:07:38.571 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.571 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.571 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:38.571 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.475329+0000", 2024-09-21T22:07:38.571 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5054e-05 2024-09-21T22:07:38.571 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.571 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.571 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:38.571 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.475338+0000", 2024-09-21T22:07:38.571 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3770000000000002e-06 2024-09-21T22:07:38.572 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.572 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.572 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:38.572 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.475484+0000", 2024-09-21T22:07:38.572 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014612300000000001 2024-09-21T22:07:38.572 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.572 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.572 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-21T22:07:38.572 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.475506+0000", 2024-09-21T22:07:38.572 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1319999999999999e-05 2024-09-21T22:07:38.572 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.572 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.572 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:38.572 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.254433+0000", 2024-09-21T22:07:38.573 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.778927708 2024-09-21T22:07:38.573 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.573 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.573 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:38.573 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.254979+0000", 2024-09-21T22:07:38.573 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00054573100000000004 2024-09-21T22:07:38.573 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.573 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.573 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:38.573 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.255272+0000", 2024-09-21T22:07:38.573 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000293264 2024-09-21T22:07:38.573 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:38.573 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:38.573 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:38.573 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.574 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.574 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.3 67/66 [PushOp(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head, version: 63'11, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head@63'11, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T22:07:38.574 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:29.262595+0000", 2024-09-21T22:07:38.574 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.2688909240000008, 2024-09-21T22:07:38.574 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.036027272999999999, 2024-09-21T22:07:38.574 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:38.574 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:38.574 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:07:38.574 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:38.574 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.574 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:38.574 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.262595+0000", 2024-09-21T22:07:38.574 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:38.575 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.575 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.575 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:38.575 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.262595+0000", 2024-09-21T22:07:38.575 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:38.575 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.575 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.575 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:38.575 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.262596+0000", 2024-09-21T22:07:38.575 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4300000000000001e-06 2024-09-21T22:07:38.575 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.575 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.575 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:38.575 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.264505+0000", 2024-09-21T22:07:38.575 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019085650000000001 2024-09-21T22:07:38.576 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.576 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.576 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:38.576 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.264557+0000", 2024-09-21T22:07:38.576 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2040000000000003e-05 2024-09-21T22:07:38.576 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.576 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.576 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:38.576 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.264589+0000", 2024-09-21T22:07:38.576 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1636e-05 2024-09-21T22:07:38.576 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.576 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.576 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:38.576 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.264778+0000", 2024-09-21T22:07:38.577 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018935600000000001 2024-09-21T22:07:38.577 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.577 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.577 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:38.577 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.264803+0000", 2024-09-21T22:07:38.577 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4692999999999999e-05 2024-09-21T22:07:38.577 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.577 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.577 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:38.577 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.298622+0000", 2024-09-21T22:07:38.577 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.033819553000000002 2024-09-21T22:07:38.577 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:38.577 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:38.577 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:38.578 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.578 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.578 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.3 67/66 [PushOp(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head, version: 63'11, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head@63'11, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:07:38.578 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:29.301490+0000", 2024-09-21T22:07:38.578 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.2299961380000006, 2024-09-21T22:07:38.578 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.041178293999999997, 2024-09-21T22:07:38.578 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:38.578 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:38.578 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:07:38.578 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:38.578 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.578 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:38.578 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.301490+0000", 2024-09-21T22:07:38.579 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:38.579 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.579 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.579 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:38.579 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.301490+0000", 2024-09-21T22:07:38.579 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:38.579 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.579 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.579 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:38.579 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.301491+0000", 2024-09-21T22:07:38.579 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.513e-06 2024-09-21T22:07:38.579 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.579 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.579 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:38.580 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.302815+0000", 2024-09-21T22:07:38.580 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001323722 2024-09-21T22:07:38.580 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.580 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.580 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:38.580 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.302863+0000", 2024-09-21T22:07:38.580 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8449999999999999e-05 2024-09-21T22:07:38.580 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.580 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.580 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:38.580 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.302895+0000", 2024-09-21T22:07:38.580 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1072999999999999e-05 2024-09-21T22:07:38.580 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.581 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.581 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:38.581 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.303014+0000", 2024-09-21T22:07:38.581 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011907400000000001 2024-09-21T22:07:38.581 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.581 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.581 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:38.581 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.303029+0000", 2024-09-21T22:07:38.581 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5322e-05 2024-09-21T22:07:38.581 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.581 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.581 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:38.581 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.342668+0000", 2024-09-21T22:07:38.581 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.039639140000000003 2024-09-21T22:07:38.582 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:38.582 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:38.582 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:38.582 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.582 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.582 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.3 67/66 [PushOp(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head, version: 63'11, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head@63'11, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:07:38.582 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:29.345626+0000", 2024-09-21T22:07:38.582 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.1858594040000003, 2024-09-21T22:07:38.582 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.034820216000000001, 2024-09-21T22:07:38.582 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:38.582 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:38.582 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:07:38.582 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:38.582 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.583 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:38.583 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.345626+0000", 2024-09-21T22:07:38.583 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:38.583 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.583 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.583 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:38.583 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.345626+0000", 2024-09-21T22:07:38.583 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:38.583 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.583 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.583 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:38.583 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.345628+0000", 2024-09-21T22:07:38.583 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3799999999999999e-06 2024-09-21T22:07:38.584 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.584 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.584 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:38.584 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.347215+0000", 2024-09-21T22:07:38.584 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015868760000000001 2024-09-21T22:07:38.584 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.584 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.584 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:38.584 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.347283+0000", 2024-09-21T22:07:38.584 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8267000000000002e-05 2024-09-21T22:07:38.584 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.584 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.584 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:38.585 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.347336+0000", 2024-09-21T22:07:38.585 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3455999999999998e-05 2024-09-21T22:07:38.585 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.585 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.585 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:38.585 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.347463+0000", 2024-09-21T22:07:38.585 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012614399999999999 2024-09-21T22:07:38.585 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.585 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.585 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:38.585 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.347473+0000", 2024-09-21T22:07:38.585 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0462000000000001e-05 2024-09-21T22:07:38.585 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.585 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.586 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:38.586 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.380447+0000", 2024-09-21T22:07:38.586 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.032973631000000003 2024-09-21T22:07:38.586 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:38.586 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:38.586 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:38.586 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.586 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.586 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.3 67/66 [PushOp(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head, version: 63'11, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head@63'11, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 4194304, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:07:38.586 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:29.383677+0000", 2024-09-21T22:07:38.586 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.1478085270000005, 2024-09-21T22:07:38.586 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.031566611000000001, 2024-09-21T22:07:38.587 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:38.587 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:38.587 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:07:38.587 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:38.587 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.587 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:38.587 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.383677+0000", 2024-09-21T22:07:38.587 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:38.587 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.587 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.587 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:38.587 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.383677+0000", 2024-09-21T22:07:38.587 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:38.587 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.588 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.588 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:38.588 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.383679+0000", 2024-09-21T22:07:38.588 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4500000000000001e-06 2024-09-21T22:07:38.588 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.588 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.588 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:38.588 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.384982+0000", 2024-09-21T22:07:38.588 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001303069 2024-09-21T22:07:38.588 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.588 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.588 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:38.588 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.385028+0000", 2024-09-21T22:07:38.588 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5655999999999998e-05 2024-09-21T22:07:38.589 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.589 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.589 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:38.589 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.385056+0000", 2024-09-21T22:07:38.589 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8877000000000002e-05 2024-09-21T22:07:38.589 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.589 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.589 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:38.589 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.385161+0000", 2024-09-21T22:07:38.589 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000104325 2024-09-21T22:07:38.589 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.589 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.589 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:38.589 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.385173+0000", 2024-09-21T22:07:38.590 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2493e-05 2024-09-21T22:07:38.590 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.590 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.590 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:38.590 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.415244+0000", 2024-09-21T22:07:38.590 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.030070741000000002 2024-09-21T22:07:38.590 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:38.590 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:38.590 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:38.590 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.590 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.590 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.3 67/66 [PushOp(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head, version: 63'19, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head@63'19, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T22:07:38.590 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:29.420098+0000", 2024-09-21T22:07:38.590 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.1113875689999997, 2024-09-21T22:07:38.591 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.067283846999999994, 2024-09-21T22:07:38.591 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:38.591 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:38.591 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:07:38.591 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:38.591 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.591 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:38.591 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.420098+0000", 2024-09-21T22:07:38.591 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:38.591 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.591 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.591 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:38.591 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.420098+0000", 2024-09-21T22:07:38.592 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:38.592 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.592 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.592 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:38.592 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.420100+0000", 2024-09-21T22:07:38.592 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.716e-06 2024-09-21T22:07:38.592 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.592 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.592 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:38.592 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.421949+0000", 2024-09-21T22:07:38.592 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0018485190000000001 2024-09-21T22:07:38.592 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.592 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.592 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:38.593 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.421996+0000", 2024-09-21T22:07:38.593 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7089000000000003e-05 2024-09-21T22:07:38.593 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.593 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.593 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:38.593 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.422042+0000", 2024-09-21T22:07:38.593 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6010000000000002e-05 2024-09-21T22:07:38.593 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.593 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.593 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:38.593 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.422231+0000", 2024-09-21T22:07:38.593 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018960900000000001 2024-09-21T22:07:38.593 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.593 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.593 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:38.594 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.422254+0000", 2024-09-21T22:07:38.594 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2864e-05 2024-09-21T22:07:38.594 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.594 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.594 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:38.594 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.487382+0000", 2024-09-21T22:07:38.594 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.065128039999999998 2024-09-21T22:07:38.594 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:38.594 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:38.594 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:38.594 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.594 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.594 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.3 67/66 [PushOp(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head, version: 63'19, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head@63'19, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:07:38.594 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:29.491783+0000", 2024-09-21T22:07:38.595 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.0397032240000001, 2024-09-21T22:07:38.595 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.04476596, 2024-09-21T22:07:38.595 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:38.595 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:38.595 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:07:38.595 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:38.595 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.595 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:38.595 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.491783+0000", 2024-09-21T22:07:38.595 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:38.595 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.595 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.595 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:38.595 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.491783+0000", 2024-09-21T22:07:38.596 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:38.596 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.596 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.596 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:38.596 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.491784+0000", 2024-09-21T22:07:38.596 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.17e-06 2024-09-21T22:07:38.596 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.596 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.596 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:38.596 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.493126+0000", 2024-09-21T22:07:38.596 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0013424489999999999 2024-09-21T22:07:38.596 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.596 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.596 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:38.596 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.493170+0000", 2024-09-21T22:07:38.597 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4153000000000002e-05 2024-09-21T22:07:38.597 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.597 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.597 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:38.597 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.493218+0000", 2024-09-21T22:07:38.597 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7392999999999998e-05 2024-09-21T22:07:38.597 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.597 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.597 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:38.597 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.493392+0000", 2024-09-21T22:07:38.597 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017392699999999999 2024-09-21T22:07:38.597 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.597 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.597 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:38.597 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.493416+0000", 2024-09-21T22:07:38.598 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4117e-05 2024-09-21T22:07:38.598 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.598 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.598 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:38.598 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.536549+0000", 2024-09-21T22:07:38.598 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.043132750999999997 2024-09-21T22:07:38.598 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:38.598 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:38.598 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:38.598 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.598 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.598 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.3 67/66 [PushOp(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head, version: 63'19, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head@63'19, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:07:38.598 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:29.539831+0000", 2024-09-21T22:07:38.598 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.9916544639999998, 2024-09-21T22:07:38.599 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.036233222000000002, 2024-09-21T22:07:38.599 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:38.599 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:38.599 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:07:38.599 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:38.599 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.599 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:38.599 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.539831+0000", 2024-09-21T22:07:38.599 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:38.599 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.599 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.599 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:38.599 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.539831+0000", 2024-09-21T22:07:38.599 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:38.599 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.600 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.600 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:38.600 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.539833+0000", 2024-09-21T22:07:38.600 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5200000000000001e-06 2024-09-21T22:07:38.600 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.600 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.600 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:38.600 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.541130+0000", 2024-09-21T22:07:38.600 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0012972719999999999 2024-09-21T22:07:38.600 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.600 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.600 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:38.600 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.541179+0000", 2024-09-21T22:07:38.600 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8507000000000002e-05 2024-09-21T22:07:38.600 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.601 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.601 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:38.601 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.541226+0000", 2024-09-21T22:07:38.601 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7663000000000001e-05 2024-09-21T22:07:38.601 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.601 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.601 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:38.601 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.541361+0000", 2024-09-21T22:07:38.601 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000135065 2024-09-21T22:07:38.601 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.601 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.601 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:38.601 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.541374+0000", 2024-09-21T22:07:38.601 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2295e-05 2024-09-21T22:07:38.602 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.602 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.602 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:38.602 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.576065+0000", 2024-09-21T22:07:38.602 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.034690899999999997 2024-09-21T22:07:38.602 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:38.602 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:38.602 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:38.602 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.602 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.602 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.3 67/66 [PushOp(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head, version: 63'19, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head@63'19, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 4194304, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:07:38.602 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:29.579455+0000", 2024-09-21T22:07:38.602 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.9520311560000003, 2024-09-21T22:07:38.602 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.034295594999999998, 2024-09-21T22:07:38.603 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:38.603 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:38.603 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:07:38.603 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:38.603 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.603 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:38.603 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.579455+0000", 2024-09-21T22:07:38.603 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:38.603 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.603 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.603 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:38.603 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.579455+0000", 2024-09-21T22:07:38.603 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:38.603 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.603 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.604 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:38.604 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.579456+0000", 2024-09-21T22:07:38.604 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.404e-06 2024-09-21T22:07:38.604 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.604 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.604 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:38.604 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.580763+0000", 2024-09-21T22:07:38.604 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001307325 2024-09-21T22:07:38.604 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.604 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.604 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:38.604 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.580811+0000", 2024-09-21T22:07:38.604 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7843000000000003e-05 2024-09-21T22:07:38.604 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.604 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.605 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:38.605 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.580860+0000", 2024-09-21T22:07:38.605 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9036999999999998e-05 2024-09-21T22:07:38.605 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.605 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.605 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:38.605 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.580988+0000", 2024-09-21T22:07:38.605 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000127529 2024-09-21T22:07:38.605 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.605 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.605 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:38.605 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.580998+0000", 2024-09-21T22:07:38.605 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0617e-05 2024-09-21T22:07:38.605 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.606 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.606 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:38.606 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.613750+0000", 2024-09-21T22:07:38.606 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.032751839999999997 2024-09-21T22:07:38.606 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:38.606 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:38.606 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:38.606 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.606 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.606 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5008.0:125 2.3 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-21T22:07:38.606 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:38.066298+0000", 2024-09-21T22:07:38.606 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.46518749199999998, 2024-09-21T22:07:38.606 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.062817136999999995, 2024-09-21T22:07:38.606 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:38.607 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:38.607 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:38.607 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:38.607 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5008", 2024-09-21T22:07:38.607 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/1400114666", 2024-09-21T22:07:38.607 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 125 2024-09-21T22:07:38.607 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.607 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:38.607 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.607 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:38.607 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.066298+0000", 2024-09-21T22:07:38.607 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:38.607 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.607 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.608 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:38.608 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.066298+0000", 2024-09-21T22:07:38.608 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:38.608 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.608 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.608 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:38.608 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.066300+0000", 2024-09-21T22:07:38.608 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.474e-06 2024-09-21T22:07:38.608 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.608 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.608 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:38.608 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.067443+0000", 2024-09-21T22:07:38.608 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011428009999999999 2024-09-21T22:07:38.608 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.609 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.609 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:38.609 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.067477+0000", 2024-09-21T22:07:38.609 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4059e-05 2024-09-21T22:07:38.609 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.609 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.609 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:38.609 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.067492+0000", 2024-09-21T22:07:38.609 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5054e-05 2024-09-21T22:07:38.609 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.609 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.609 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:38.609 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.067638+0000", 2024-09-21T22:07:38.610 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014583900000000001 2024-09-21T22:07:38.610 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.610 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.610 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:38.610 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.067882+0000", 2024-09-21T22:07:38.610 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024415500000000002 2024-09-21T22:07:38.610 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.610 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.610 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-09-21T22:07:38.610 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.068247+0000", 2024-09-21T22:07:38.610 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00036570800000000002 2024-09-21T22:07:38.610 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.610 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.611 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:38.611 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.119971+0000", 2024-09-21T22:07:38.611 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.051723866 2024-09-21T22:07:38.611 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.611 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.611 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:38.611 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.128849+0000", 2024-09-21T22:07:38.611 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0088779489999999996 2024-09-21T22:07:38.611 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.611 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.611 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:38.611 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.129000+0000", 2024-09-21T22:07:38.611 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015062499999999999 2024-09-21T22:07:38.611 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.612 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.612 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:38.612 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.129116+0000", 2024-09-21T22:07:38.612 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011560700000000001 2024-09-21T22:07:38.612 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:38.612 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:38.612 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:38.612 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.612 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.612 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5008.0:127 2.3 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-21T22:07:38.612 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:38.140308+0000", 2024-09-21T22:07:38.612 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.39117739000000001, 2024-09-21T22:07:38.612 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.055290512999999999, 2024-09-21T22:07:38.612 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:38.613 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:38.613 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:38.613 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:38.613 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5008", 2024-09-21T22:07:38.613 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/1400114666", 2024-09-21T22:07:38.613 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 127 2024-09-21T22:07:38.613 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.613 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:38.613 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.613 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:38.613 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.140308+0000", 2024-09-21T22:07:38.613 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:38.613 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.614 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.614 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:38.614 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.140308+0000", 2024-09-21T22:07:38.614 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:38.614 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.614 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.614 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:38.614 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.140310+0000", 2024-09-21T22:07:38.614 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6509999999999999e-06 2024-09-21T22:07:38.614 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.614 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.614 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:38.614 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.141504+0000", 2024-09-21T22:07:38.615 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011935750000000001 2024-09-21T22:07:38.615 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.615 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.615 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:38.615 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.141522+0000", 2024-09-21T22:07:38.615 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8349e-05 2024-09-21T22:07:38.615 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.615 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.615 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:38.615 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.141531+0000", 2024-09-21T22:07:38.615 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3239999999999992e-06 2024-09-21T22:07:38.615 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.615 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.615 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:38.616 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.141661+0000", 2024-09-21T22:07:38.616 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013001000000000001 2024-09-21T22:07:38.616 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.616 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.616 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:38.616 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.141907+0000", 2024-09-21T22:07:38.616 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024547499999999999 2024-09-21T22:07:38.616 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.616 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.616 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-09-21T22:07:38.616 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.142274+0000", 2024-09-21T22:07:38.616 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00036716400000000001 2024-09-21T22:07:38.616 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.617 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.617 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:38.617 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.189864+0000", 2024-09-21T22:07:38.617 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047589617000000001 2024-09-21T22:07:38.617 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.617 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.617 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:38.617 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.195406+0000", 2024-09-21T22:07:38.617 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0055425860000000004 2024-09-21T22:07:38.617 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.617 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.617 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:38.617 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.195507+0000", 2024-09-21T22:07:38.618 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000100862 2024-09-21T22:07:38.618 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.618 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.618 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:38.618 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.195599+0000", 2024-09-21T22:07:38.618 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1899999999999998e-05 2024-09-21T22:07:38.618 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:38.618 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:38.618 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:38.618 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.618 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.618 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5008.0:129 2.3 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head [write 2228224~917504 in=917504b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-21T22:07:38.618 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:38.209297+0000", 2024-09-21T22:07:38.619 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.32218850900000001, 2024-09-21T22:07:38.619 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.060441054000000001, 2024-09-21T22:07:38.619 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:38.619 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:38.619 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:38.619 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:38.619 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5008", 2024-09-21T22:07:38.619 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/1400114666", 2024-09-21T22:07:38.619 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 129 2024-09-21T22:07:38.619 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.619 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:38.619 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.619 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:38.619 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.209297+0000", 2024-09-21T22:07:38.620 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:38.620 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.620 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.620 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:38.620 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.209297+0000", 2024-09-21T22:07:38.620 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:38.620 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.620 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.620 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:38.620 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.209299+0000", 2024-09-21T22:07:38.620 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.274e-06 2024-09-21T22:07:38.620 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.620 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.620 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:38.620 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.210077+0000", 2024-09-21T22:07:38.621 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00077849799999999997 2024-09-21T22:07:38.621 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.621 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.621 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:38.621 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.210094+0000", 2024-09-21T22:07:38.621 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6772000000000001e-05 2024-09-21T22:07:38.621 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.621 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.621 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:38.621 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.210103+0000", 2024-09-21T22:07:38.621 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6780000000000006e-06 2024-09-21T22:07:38.621 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.621 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.621 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:38.621 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.210254+0000", 2024-09-21T22:07:38.622 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015141699999999999 2024-09-21T22:07:38.622 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.622 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.622 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:38.622 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.210501+0000", 2024-09-21T22:07:38.622 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000246498 2024-09-21T22:07:38.622 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.622 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.622 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-09-21T22:07:38.622 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.210872+0000", 2024-09-21T22:07:38.622 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00037147500000000002 2024-09-21T22:07:38.622 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.622 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.622 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:38.622 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.269019+0000", 2024-09-21T22:07:38.623 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058147114 2024-09-21T22:07:38.623 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.623 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.623 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:38.623 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.269638+0000", 2024-09-21T22:07:38.623 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00061896499999999997 2024-09-21T22:07:38.623 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.623 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.623 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:38.623 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.269693+0000", 2024-09-21T22:07:38.623 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4786e-05 2024-09-21T22:07:38.623 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.623 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.623 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:38.624 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.269738+0000", 2024-09-21T22:07:38.624 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5577000000000001e-05 2024-09-21T22:07:38.624 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:38.624 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:38.624 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:38.624 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.624 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.624 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5008.0:131 2.3 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-21T22:07:38.624 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:38.279879+0000", 2024-09-21T22:07:38.624 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.25160681899999998, 2024-09-21T22:07:38.624 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.054184296999999999, 2024-09-21T22:07:38.624 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:38.624 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:38.624 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:38.625 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:38.625 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5008", 2024-09-21T22:07:38.625 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/1400114666", 2024-09-21T22:07:38.625 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 131 2024-09-21T22:07:38.625 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.625 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:38.625 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.625 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:38.625 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.279879+0000", 2024-09-21T22:07:38.625 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:38.625 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.625 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.625 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:38.625 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.279879+0000", 2024-09-21T22:07:38.626 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:38.626 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.626 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.626 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:38.626 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.279880+0000", 2024-09-21T22:07:38.626 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.136e-06 2024-09-21T22:07:38.626 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.626 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.626 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:38.626 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.280636+0000", 2024-09-21T22:07:38.626 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00075625900000000001 2024-09-21T22:07:38.626 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.626 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.627 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:38.627 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.280648+0000", 2024-09-21T22:07:38.627 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2026e-05 2024-09-21T22:07:38.627 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.627 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.627 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:38.627 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.280654+0000", 2024-09-21T22:07:38.627 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8839999999999999e-06 2024-09-21T22:07:38.627 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.627 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.627 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:38.627 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.280739+0000", 2024-09-21T22:07:38.627 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4196000000000005e-05 2024-09-21T22:07:38.627 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.628 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.628 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:38.628 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.280808+0000", 2024-09-21T22:07:38.628 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9685999999999997e-05 2024-09-21T22:07:38.628 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.628 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.628 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-09-21T22:07:38.628 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.280900+0000", 2024-09-21T22:07:38.628 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1407000000000001e-05 2024-09-21T22:07:38.628 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.628 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.628 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:38.628 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.307730+0000", 2024-09-21T22:07:38.628 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026830568999999999 2024-09-21T22:07:38.629 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.629 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.629 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:38.629 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.333901+0000", 2024-09-21T22:07:38.629 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026170787000000001 2024-09-21T22:07:38.629 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.629 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.629 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:38.629 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.333996+0000", 2024-09-21T22:07:38.629 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4723999999999996e-05 2024-09-21T22:07:38.629 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:38.629 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:38.629 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:38.630 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.334063+0000", 2024-09-21T22:07:38.630 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7623e-05 2024-09-21T22:07:38.630 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:38.630 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:38.630 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:38.630 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:38.630 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:38.630 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:07:38.632 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T22:07:38.632 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:38.626+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:38.732 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:38.730+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:38.766 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:07:38.766 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [ 2024-09-21T22:07:38.766 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:38.766 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:152 2.1 e72/53 2:837ddd83:::rbd_data.126483d75f2d.000000000000000d:head v 72'24, mlcod=68'22)", 2024-09-21T22:07:38.766 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.755671+0000", 2024-09-21T22:07:38.766 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.009541852, 2024-09-21T22:07:38.766 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0095617819999999996, 2024-09-21T22:07:38.766 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:38.766 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:38.767 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:38.767 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:38.767 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:38.767 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:38.767 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.755671+0000", 2024-09-21T22:07:38.767 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:38.767 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:38.767 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:38.767 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:38.767 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.755671+0000", 2024-09-21T22:07:38.767 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:38.767 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:38.767 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:38.767 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:38.768 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.755672+0000", 2024-09-21T22:07:38.768 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2580000000000001e-06 2024-09-21T22:07:38.768 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:38.768 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:38.768 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:38.768 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.755994+0000", 2024-09-21T22:07:38.768 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00032109100000000002 2024-09-21T22:07:38.768 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:38.768 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:38.768 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:38.768 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.756011+0000", 2024-09-21T22:07:38.768 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7762999999999999e-05 2024-09-21T22:07:38.768 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:38.768 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:38.769 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:38.769 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.756020+0000", 2024-09-21T22:07:38.769 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.9109999999999999e-06 2024-09-21T22:07:38.769 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:38.769 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:38.769 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:38.769 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.756148+0000", 2024-09-21T22:07:38.769 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000127828 2024-09-21T22:07:38.769 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:38.769 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:38.769 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:38.769 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.756187+0000", 2024-09-21T22:07:38.769 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.8485999999999997e-05 2024-09-21T22:07:38.769 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:38.769 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:38.770 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:38.770 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:38.770 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T22:07:38.770 INFO:teuthology.orchestra.run.smithi039.stdout: "num_ops": 1 2024-09-21T22:07:38.770 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:07:38.780 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T22:07:38.833 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:38.832+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:38.933 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:07:38.932+0000 7f2075a06640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31720) UID: 0 2024-09-21T22:07:38.950 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:07:38.950 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:07:38.950 INFO:teuthology.orchestra.run.smithi039.stdout: "complaint_time": 30, 2024-09-21T22:07:38.950 INFO:teuthology.orchestra.run.smithi039.stdout: "num_blocked_ops": 0 2024-09-21T22:07:38.950 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:07:38.966 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T22:07:39.034 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:39.030+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:39.122 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:07:39.122 INFO:teuthology.orchestra.run.smithi039.stdout: "size": 20, 2024-09-21T22:07:39.122 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 600, 2024-09-21T22:07:39.122 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [ 2024-09-21T22:07:39.122 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.122 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:694 2.5 e63/37 2:b1d55633:::rbd_data.126483d75f2d.000000000000000b:head v 63'26, mlcod=63'24)", 2024-09-21T22:07:39.122 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.163199+0000", 2024-09-21T22:07:39.122 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 13.954630286, 2024-09-21T22:07:39.122 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.034676672999999998, 2024-09-21T22:07:39.122 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:39.123 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:39.123 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:39.123 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:39.123 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.123 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:39.123 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.163199+0000", 2024-09-21T22:07:39.123 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:39.123 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.123 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.123 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:39.123 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.163199+0000", 2024-09-21T22:07:39.123 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:39.123 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.123 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.124 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:39.124 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.163200+0000", 2024-09-21T22:07:39.124 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.703e-06 2024-09-21T22:07:39.124 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.124 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.124 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:39.124 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.164013+0000", 2024-09-21T22:07:39.124 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000812447 2024-09-21T22:07:39.124 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.124 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.124 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:39.124 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.164042+0000", 2024-09-21T22:07:39.124 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9006e-05 2024-09-21T22:07:39.124 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.125 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.125 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:39.125 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.164064+0000", 2024-09-21T22:07:39.125 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1719999999999999e-05 2024-09-21T22:07:39.125 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.125 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.125 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:39.125 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.164143+0000", 2024-09-21T22:07:39.125 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.9283999999999995e-05 2024-09-21T22:07:39.125 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.125 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.125 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:39.125 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.164165+0000", 2024-09-21T22:07:39.125 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.2379e-05 2024-09-21T22:07:39.125 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.126 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.126 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:39.126 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.197827+0000", 2024-09-21T22:07:39.126 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.033661782000000001 2024-09-21T22:07:39.126 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.126 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.126 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:39.126 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.197875+0000", 2024-09-21T22:07:39.126 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8352000000000001e-05 2024-09-21T22:07:39.126 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:39.126 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:39.126 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:39.126 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.127 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.127 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:696 2.5 e63/37 2:b1d55633:::rbd_data.126483d75f2d.000000000000000b:head v 63'28, mlcod=63'26)", 2024-09-21T22:07:39.127 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.218913+0000", 2024-09-21T22:07:39.127 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 13.898915840000001, 2024-09-21T22:07:39.127 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.056187599999999997, 2024-09-21T22:07:39.127 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:39.127 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:39.127 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:39.127 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:39.127 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.127 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:39.127 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.218913+0000", 2024-09-21T22:07:39.127 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:39.127 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.127 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.128 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:39.128 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.218913+0000", 2024-09-21T22:07:39.128 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:39.128 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.128 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.128 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:39.128 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.218914+0000", 2024-09-21T22:07:39.128 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.1930000000000001e-06 2024-09-21T22:07:39.128 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.128 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.128 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:39.128 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.219721+0000", 2024-09-21T22:07:39.128 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00080701500000000001 2024-09-21T22:07:39.128 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.129 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.129 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:39.129 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.219743+0000", 2024-09-21T22:07:39.129 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1364000000000001e-05 2024-09-21T22:07:39.129 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.129 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.129 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:39.129 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.219759+0000", 2024-09-21T22:07:39.129 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.611e-05 2024-09-21T22:07:39.129 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.129 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.129 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:39.129 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.219908+0000", 2024-09-21T22:07:39.129 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000149252 2024-09-21T22:07:39.129 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.130 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.130 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:39.130 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.219951+0000", 2024-09-21T22:07:39.130 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.2518999999999997e-05 2024-09-21T22:07:39.130 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.130 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.130 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:39.130 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.275031+0000", 2024-09-21T22:07:39.130 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.055080241000000002 2024-09-21T22:07:39.130 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.130 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.130 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:39.130 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.275101+0000", 2024-09-21T22:07:39.130 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.9906000000000005e-05 2024-09-21T22:07:39.130 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:39.131 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:39.131 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:39.131 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.131 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.131 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:698 2.5 e63/37 2:b1d55633:::rbd_data.126483d75f2d.000000000000000b:head v 63'30, mlcod=63'28)", 2024-09-21T22:07:39.131 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.289319+0000", 2024-09-21T22:07:39.131 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 13.828509630999999, 2024-09-21T22:07:39.131 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.053930944000000001, 2024-09-21T22:07:39.131 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:39.131 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:39.131 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:39.131 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:39.131 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.131 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:39.131 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.289319+0000", 2024-09-21T22:07:39.132 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:39.132 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.132 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.132 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:39.132 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.289319+0000", 2024-09-21T22:07:39.132 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:39.132 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.132 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.132 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:39.132 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.289321+0000", 2024-09-21T22:07:39.132 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.221e-06 2024-09-21T22:07:39.132 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.132 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.132 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:39.133 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.290091+0000", 2024-09-21T22:07:39.133 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00077069300000000003 2024-09-21T22:07:39.133 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.133 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.133 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:39.133 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.290109+0000", 2024-09-21T22:07:39.133 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7218999999999999e-05 2024-09-21T22:07:39.133 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.133 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.133 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:39.133 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.290122+0000", 2024-09-21T22:07:39.133 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3678e-05 2024-09-21T22:07:39.133 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.133 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.134 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:39.134 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.290274+0000", 2024-09-21T22:07:39.134 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00015205899999999999 2024-09-21T22:07:39.134 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.134 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.134 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:39.134 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.290313+0000", 2024-09-21T22:07:39.134 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.8636000000000001e-05 2024-09-21T22:07:39.134 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.134 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.134 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:39.134 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.343179+0000", 2024-09-21T22:07:39.134 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.052866199000000003 2024-09-21T22:07:39.134 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.135 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.135 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:39.135 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.343250+0000", 2024-09-21T22:07:39.135 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.1239e-05 2024-09-21T22:07:39.135 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:39.135 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:39.135 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:39.135 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.135 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.135 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:702 2.5 e63/37 2:a70ae99f:::rbd_data.126483d75f2d.000000000000000c:head v 63'34, mlcod=63'32)", 2024-09-21T22:07:39.135 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.392532+0000", 2024-09-21T22:07:39.135 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 13.725297042999999, 2024-09-21T22:07:39.135 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.035703990999999997, 2024-09-21T22:07:39.135 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:39.136 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:39.136 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:39.136 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:39.136 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.136 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:39.136 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.392532+0000", 2024-09-21T22:07:39.136 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:39.136 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.136 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.136 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:39.136 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.392532+0000", 2024-09-21T22:07:39.136 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:39.136 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.136 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.137 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:39.137 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.392534+0000", 2024-09-21T22:07:39.137 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.64e-06 2024-09-21T22:07:39.137 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.137 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.137 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:39.137 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.393321+0000", 2024-09-21T22:07:39.137 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00078705699999999997 2024-09-21T22:07:39.137 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.137 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.137 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:39.137 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.393347+0000", 2024-09-21T22:07:39.137 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.6735999999999999e-05 2024-09-21T22:07:39.137 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.137 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.138 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:39.138 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.393361+0000", 2024-09-21T22:07:39.138 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3346000000000001e-05 2024-09-21T22:07:39.138 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.138 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.138 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:39.138 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.393469+0000", 2024-09-21T22:07:39.138 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000108039 2024-09-21T22:07:39.138 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.138 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.138 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:39.138 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.393512+0000", 2024-09-21T22:07:39.138 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.2862999999999999e-05 2024-09-21T22:07:39.138 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.139 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.139 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:39.139 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.428170+0000", 2024-09-21T22:07:39.139 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.034658858000000001 2024-09-21T22:07:39.139 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.139 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.139 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:39.139 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.428236+0000", 2024-09-21T22:07:39.139 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.5451999999999996e-05 2024-09-21T22:07:39.139 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:39.139 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:39.139 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:39.139 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.139 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.139 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:708 2.1 e63/53 2:837ddd83:::rbd_data.126483d75f2d.000000000000000d:head v 63'6, mlcod=57'4)", 2024-09-21T22:07:39.139 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.517007+0000", 2024-09-21T22:07:39.140 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 13.600822117, 2024-09-21T22:07:39.140 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.032495584000000001, 2024-09-21T22:07:39.140 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:39.140 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:39.140 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:39.140 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:39.140 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.140 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:39.140 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.517007+0000", 2024-09-21T22:07:39.140 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:39.140 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.140 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.140 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:39.140 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.517007+0000", 2024-09-21T22:07:39.141 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:39.141 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.141 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.141 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:39.141 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.517008+0000", 2024-09-21T22:07:39.141 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.604e-06 2024-09-21T22:07:39.141 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.141 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.141 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:39.141 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.517603+0000", 2024-09-21T22:07:39.141 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00059461900000000001 2024-09-21T22:07:39.141 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.141 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.141 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:39.141 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.517632+0000", 2024-09-21T22:07:39.142 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8663e-05 2024-09-21T22:07:39.142 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.142 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.142 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:39.142 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.517647+0000", 2024-09-21T22:07:39.142 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4833e-05 2024-09-21T22:07:39.142 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.142 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.142 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:39.142 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.517714+0000", 2024-09-21T22:07:39.142 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.7669000000000006e-05 2024-09-21T22:07:39.142 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.142 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.143 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:39.143 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.517737+0000", 2024-09-21T22:07:39.143 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.2469000000000001e-05 2024-09-21T22:07:39.143 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.143 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.143 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:39.143 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.549370+0000", 2024-09-21T22:07:39.143 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.031633221000000003 2024-09-21T22:07:39.143 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.143 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.143 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:39.143 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.549502+0000", 2024-09-21T22:07:39.143 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013250600000000001 2024-09-21T22:07:39.143 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:39.144 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:39.144 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:39.144 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.144 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.144 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:712 2.1 e63/53 2:837ddd83:::rbd_data.126483d75f2d.000000000000000d:head v 63'10, mlcod=63'8)", 2024-09-21T22:07:39.144 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.604029+0000", 2024-09-21T22:07:39.144 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 13.513800178, 2024-09-21T22:07:39.144 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.037982371000000001, 2024-09-21T22:07:39.144 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:39.144 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:39.144 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:39.144 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:39.144 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.144 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:39.144 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.604029+0000", 2024-09-21T22:07:39.145 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:39.145 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.145 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.145 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:39.145 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.604029+0000", 2024-09-21T22:07:39.145 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:39.145 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.145 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.145 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:39.145 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.604031+0000", 2024-09-21T22:07:39.145 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8870000000000001e-06 2024-09-21T22:07:39.145 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.145 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.145 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:39.145 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.604395+0000", 2024-09-21T22:07:39.146 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00036422000000000001 2024-09-21T22:07:39.146 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.146 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.146 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:39.146 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.604424+0000", 2024-09-21T22:07:39.146 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8609e-05 2024-09-21T22:07:39.146 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.146 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.146 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:39.146 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.604437+0000", 2024-09-21T22:07:39.146 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3086e-05 2024-09-21T22:07:39.146 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.146 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.147 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:39.147 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.604509+0000", 2024-09-21T22:07:39.147 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.2448999999999997e-05 2024-09-21T22:07:39.147 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.147 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.147 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:39.147 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.604531+0000", 2024-09-21T22:07:39.147 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1602000000000001e-05 2024-09-21T22:07:39.147 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.147 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.147 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:39.147 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.641885+0000", 2024-09-21T22:07:39.147 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.037354545000000003 2024-09-21T22:07:39.147 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.147 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.148 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:39.148 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.642011+0000", 2024-09-21T22:07:39.148 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00012597300000000001 2024-09-21T22:07:39.148 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:39.148 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:39.148 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:39.148 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.148 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.148 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:718 2.1 e63/53 2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head v 63'16, mlcod=63'14)", 2024-09-21T22:07:39.148 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.759837+0000", 2024-09-21T22:07:39.148 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 13.357992492999999, 2024-09-21T22:07:39.148 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.031201353000000001, 2024-09-21T22:07:39.148 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:39.148 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:39.149 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:39.149 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:39.149 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.149 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:39.149 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.759837+0000", 2024-09-21T22:07:39.149 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:39.149 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.149 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.149 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:39.149 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.759837+0000", 2024-09-21T22:07:39.149 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:39.149 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.149 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.149 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:39.150 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.759837+0000", 2024-09-21T22:07:39.150 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.2999999999999999e-07 2024-09-21T22:07:39.150 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.150 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.150 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:39.150 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.760443+0000", 2024-09-21T22:07:39.150 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00060515600000000003 2024-09-21T22:07:39.150 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.150 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.150 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:39.150 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.760472+0000", 2024-09-21T22:07:39.150 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9416000000000001e-05 2024-09-21T22:07:39.150 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.150 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.151 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:39.151 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.760485+0000", 2024-09-21T22:07:39.151 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3359e-05 2024-09-21T22:07:39.151 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.151 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.151 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:39.151 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.760616+0000", 2024-09-21T22:07:39.151 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013029400000000001 2024-09-21T22:07:39.151 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.151 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.151 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:39.151 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.760656+0000", 2024-09-21T22:07:39.151 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.0788000000000003e-05 2024-09-21T22:07:39.151 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.152 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.152 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:39.152 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.790947+0000", 2024-09-21T22:07:39.152 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.030290480000000002 2024-09-21T22:07:39.152 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.152 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.152 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:39.152 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.791038+0000", 2024-09-21T22:07:39.152 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.0929999999999998e-05 2024-09-21T22:07:39.152 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:39.152 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:39.152 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:39.152 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.152 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.152 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:720 2.1 e63/53 2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head v 63'18, mlcod=63'16)", 2024-09-21T22:07:39.153 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.805111+0000", 2024-09-21T22:07:39.153 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 13.312718117999999, 2024-09-21T22:07:39.153 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.046082864000000001, 2024-09-21T22:07:39.153 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:39.153 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:39.153 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:39.153 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:39.153 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.153 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:39.153 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.805111+0000", 2024-09-21T22:07:39.153 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:39.153 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.153 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.153 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:39.154 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.805111+0000", 2024-09-21T22:07:39.154 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:39.154 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.154 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.154 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:39.154 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.805112+0000", 2024-09-21T22:07:39.154 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.02e-06 2024-09-21T22:07:39.154 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.154 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.154 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:39.154 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.805429+0000", 2024-09-21T22:07:39.154 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00031740799999999998 2024-09-21T22:07:39.154 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.154 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.154 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:39.155 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.805459+0000", 2024-09-21T22:07:39.155 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.955e-05 2024-09-21T22:07:39.155 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.155 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.155 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:39.155 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.805472+0000", 2024-09-21T22:07:39.155 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3533000000000001e-05 2024-09-21T22:07:39.155 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.155 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.155 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:39.155 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.805581+0000", 2024-09-21T22:07:39.155 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000108194 2024-09-21T22:07:39.155 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.155 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.155 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:39.156 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.805615+0000", 2024-09-21T22:07:39.156 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.4310000000000002e-05 2024-09-21T22:07:39.156 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.156 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.156 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:39.156 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.851146+0000", 2024-09-21T22:07:39.156 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.045531092000000002 2024-09-21T22:07:39.156 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.156 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.156 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:39.156 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.851194+0000", 2024-09-21T22:07:39.156 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.7756999999999997e-05 2024-09-21T22:07:39.156 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:39.156 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:39.156 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:39.156 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.157 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.157 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:723 2.1 e63/53 2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head v 63'20, mlcod=63'18)", 2024-09-21T22:07:39.157 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.862341+0000", 2024-09-21T22:07:39.157 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 13.255488184000001, 2024-09-21T22:07:39.157 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.036454671000000001, 2024-09-21T22:07:39.157 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:39.157 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:39.157 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:39.157 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:39.157 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.157 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:39.157 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.862341+0000", 2024-09-21T22:07:39.157 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:39.157 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.157 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.158 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:39.158 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.862341+0000", 2024-09-21T22:07:39.158 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:39.158 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.158 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.158 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:39.158 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.862342+0000", 2024-09-21T22:07:39.158 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.0299999999999997e-07 2024-09-21T22:07:39.158 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.158 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.158 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:39.158 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.862721+0000", 2024-09-21T22:07:39.158 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00037889800000000002 2024-09-21T22:07:39.158 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.159 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.159 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:39.159 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.862738+0000", 2024-09-21T22:07:39.159 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6957999999999999e-05 2024-09-21T22:07:39.159 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.159 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.159 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:39.159 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.862747+0000", 2024-09-21T22:07:39.159 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.0729999999999994e-06 2024-09-21T22:07:39.159 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.159 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.159 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:39.159 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.862830+0000", 2024-09-21T22:07:39.159 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.3113000000000002e-05 2024-09-21T22:07:39.159 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.160 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.160 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:39.160 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.862860+0000", 2024-09-21T22:07:39.160 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9980000000000001e-05 2024-09-21T22:07:39.160 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.160 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.160 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:39.160 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.898705+0000", 2024-09-21T22:07:39.160 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.035845063000000003 2024-09-21T22:07:39.160 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.160 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.160 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:39.160 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.898795+0000", 2024-09-21T22:07:39.160 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.0682999999999999e-05 2024-09-21T22:07:39.160 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:39.161 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:39.161 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:39.161 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.161 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.161 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:136 2.5 e72/37 2:b1d55633:::rbd_data.126483d75f2d.000000000000000b:head v 72'44, mlcod=72'42)", 2024-09-21T22:07:39.161 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.386273+0000", 2024-09-21T22:07:39.161 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.73155606900000003, 2024-09-21T22:07:39.161 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.033806744999999999, 2024-09-21T22:07:39.161 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:39.161 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:39.161 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:39.161 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:39.161 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.161 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:39.161 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.386273+0000", 2024-09-21T22:07:39.162 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:39.162 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.162 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.162 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:39.162 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.386273+0000", 2024-09-21T22:07:39.162 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:39.162 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.162 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.162 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:39.162 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.386274+0000", 2024-09-21T22:07:39.162 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.8299999999999996e-07 2024-09-21T22:07:39.162 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.162 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.162 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:39.162 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.387108+0000", 2024-09-21T22:07:39.163 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00083467800000000005 2024-09-21T22:07:39.163 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.163 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.163 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:39.163 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.387121+0000", 2024-09-21T22:07:39.163 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2823000000000001e-05 2024-09-21T22:07:39.163 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.163 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.163 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:39.163 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.387128+0000", 2024-09-21T22:07:39.163 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.404e-06 2024-09-21T22:07:39.163 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.163 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.163 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:39.164 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.387179+0000", 2024-09-21T22:07:39.164 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.1888000000000002e-05 2024-09-21T22:07:39.164 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.164 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.164 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:39.164 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.387201+0000", 2024-09-21T22:07:39.164 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.2019999999999999e-05 2024-09-21T22:07:39.164 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.164 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.164 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:39.164 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.419993+0000", 2024-09-21T22:07:39.164 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.032791182000000002 2024-09-21T22:07:39.164 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.164 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.165 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:39.165 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.420080+0000", 2024-09-21T22:07:39.165 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.7066999999999998e-05 2024-09-21T22:07:39.165 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:39.165 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:39.165 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:39.165 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.165 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.165 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:138 2.5 e72/37 2:b1d55633:::rbd_data.126483d75f2d.000000000000000b:head v 72'46, mlcod=72'44)", 2024-09-21T22:07:39.165 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.430104+0000", 2024-09-21T22:07:39.165 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.68772472100000004, 2024-09-21T22:07:39.165 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.034087458000000001, 2024-09-21T22:07:39.165 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:39.165 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:39.166 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:39.166 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:39.166 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.166 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:39.166 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.430104+0000", 2024-09-21T22:07:39.166 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:39.166 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.166 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.166 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:39.166 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.430104+0000", 2024-09-21T22:07:39.166 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:39.166 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.166 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.166 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:39.167 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.430105+0000", 2024-09-21T22:07:39.167 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.1680000000000001e-06 2024-09-21T22:07:39.167 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.167 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.167 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:39.167 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.430864+0000", 2024-09-21T22:07:39.167 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00075819400000000001 2024-09-21T22:07:39.167 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.167 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.167 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:39.167 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.430881+0000", 2024-09-21T22:07:39.167 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7000999999999999e-05 2024-09-21T22:07:39.167 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.167 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.168 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:39.168 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.430889+0000", 2024-09-21T22:07:39.168 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.9540000000000004e-06 2024-09-21T22:07:39.168 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.168 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.168 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:39.168 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.430959+0000", 2024-09-21T22:07:39.168 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.0858999999999994e-05 2024-09-21T22:07:39.168 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.168 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.168 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:39.168 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.430996+0000", 2024-09-21T22:07:39.168 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.6443999999999999e-05 2024-09-21T22:07:39.168 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.168 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.169 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:39.169 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.464087+0000", 2024-09-21T22:07:39.169 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.033090799999999997 2024-09-21T22:07:39.169 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.169 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.169 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:39.169 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.464192+0000", 2024-09-21T22:07:39.169 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00010503800000000001 2024-09-21T22:07:39.169 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:39.169 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:39.169 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:39.169 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.169 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.169 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:140 2.5 e72/37 2:b1d55633:::rbd_data.126483d75f2d.000000000000000b:head v 72'48, mlcod=72'46)", 2024-09-21T22:07:39.170 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.475737+0000", 2024-09-21T22:07:39.170 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.64209177299999998, 2024-09-21T22:07:39.170 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.037632025, 2024-09-21T22:07:39.170 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:39.170 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:39.170 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:39.170 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:39.170 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.170 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:39.170 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.475737+0000", 2024-09-21T22:07:39.170 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:39.170 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.170 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.170 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:39.171 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.475737+0000", 2024-09-21T22:07:39.171 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:39.171 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.171 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.171 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:39.171 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.475739+0000", 2024-09-21T22:07:39.171 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.666e-06 2024-09-21T22:07:39.171 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.171 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.171 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:39.171 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.476538+0000", 2024-09-21T22:07:39.171 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00079927399999999999 2024-09-21T22:07:39.171 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.171 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.172 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:39.172 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.476556+0000", 2024-09-21T22:07:39.172 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7666999999999999e-05 2024-09-21T22:07:39.172 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.172 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.172 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:39.172 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.476565+0000", 2024-09-21T22:07:39.172 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.0650000000000005e-06 2024-09-21T22:07:39.172 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.172 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.172 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:39.172 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.476651+0000", 2024-09-21T22:07:39.172 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.5916999999999997e-05 2024-09-21T22:07:39.172 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.172 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.173 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:39.173 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.476696+0000", 2024-09-21T22:07:39.173 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.5537000000000001e-05 2024-09-21T22:07:39.173 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.173 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.173 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:39.173 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.513300+0000", 2024-09-21T22:07:39.173 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.036603498999999998 2024-09-21T22:07:39.173 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.173 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.173 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:39.173 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.513369+0000", 2024-09-21T22:07:39.173 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.9400000000000006e-05 2024-09-21T22:07:39.173 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:39.174 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:39.174 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:39.174 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.174 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.174 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:145 2.5 e72/37 2:a70ae99f:::rbd_data.126483d75f2d.000000000000000c:head v 72'52, mlcod=72'50)", 2024-09-21T22:07:39.174 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.578171+0000", 2024-09-21T22:07:39.174 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.53965792300000004, 2024-09-21T22:07:39.174 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.037181537000000001, 2024-09-21T22:07:39.174 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:39.174 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:39.174 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:39.174 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:39.174 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.174 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:39.174 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.578171+0000", 2024-09-21T22:07:39.175 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:39.175 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.175 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.175 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:39.175 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.578171+0000", 2024-09-21T22:07:39.175 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:39.175 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.175 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.175 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:39.175 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.578172+0000", 2024-09-21T22:07:39.175 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.017e-06 2024-09-21T22:07:39.175 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.175 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.175 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:39.175 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.579000+0000", 2024-09-21T22:07:39.175 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00082799499999999999 2024-09-21T22:07:39.176 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.176 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.176 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:39.176 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.579020+0000", 2024-09-21T22:07:39.176 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.0165000000000001e-05 2024-09-21T22:07:39.176 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.176 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.176 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:39.176 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.579036+0000", 2024-09-21T22:07:39.176 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5767e-05 2024-09-21T22:07:39.176 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.176 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.176 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:39.176 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.579106+0000", 2024-09-21T22:07:39.177 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.0091000000000004e-05 2024-09-21T22:07:39.177 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.177 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.177 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:39.177 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.579131+0000", 2024-09-21T22:07:39.177 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4978e-05 2024-09-21T22:07:39.177 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.177 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.177 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:39.177 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.615249+0000", 2024-09-21T22:07:39.177 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.036118355999999997 2024-09-21T22:07:39.177 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.177 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.177 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:39.177 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.615353+0000", 2024-09-21T22:07:39.178 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000103168 2024-09-21T22:07:39.178 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:39.178 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:39.178 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:39.178 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.178 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.178 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:147 2.5 e72/37 2:a70ae99f:::rbd_data.126483d75f2d.000000000000000c:head v 72'54, mlcod=72'52)", 2024-09-21T22:07:39.178 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.627203+0000", 2024-09-21T22:07:39.178 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.49062581100000002, 2024-09-21T22:07:39.178 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.038203972000000003, 2024-09-21T22:07:39.178 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:39.178 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:39.178 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:39.178 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:39.179 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.179 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:39.179 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.627203+0000", 2024-09-21T22:07:39.179 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:39.179 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.179 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.179 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:39.179 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.627203+0000", 2024-09-21T22:07:39.179 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:39.179 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.179 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.179 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:39.179 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.627204+0000", 2024-09-21T22:07:39.179 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3069999999999999e-06 2024-09-21T22:07:39.179 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.180 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.180 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:39.180 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.628114+0000", 2024-09-21T22:07:39.180 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00090942099999999999 2024-09-21T22:07:39.180 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.180 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.180 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:39.180 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.628140+0000", 2024-09-21T22:07:39.180 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5882000000000001e-05 2024-09-21T22:07:39.180 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.180 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.180 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:39.180 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.628162+0000", 2024-09-21T22:07:39.180 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.2293000000000002e-05 2024-09-21T22:07:39.180 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.181 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.181 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:39.181 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.628250+0000", 2024-09-21T22:07:39.181 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.7973000000000006e-05 2024-09-21T22:07:39.181 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.181 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.181 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:39.181 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.628272+0000", 2024-09-21T22:07:39.181 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1682999999999999e-05 2024-09-21T22:07:39.181 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.181 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.181 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:39.181 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.665322+0000", 2024-09-21T22:07:39.181 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.037050650999999997 2024-09-21T22:07:39.182 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.182 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.182 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:39.182 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.665407+0000", 2024-09-21T22:07:39.182 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.4761999999999999e-05 2024-09-21T22:07:39.182 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:39.182 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:39.182 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:39.182 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.182 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.182 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:149 2.5 e72/37 2:a70ae99f:::rbd_data.126483d75f2d.000000000000000c:head v 72'56, mlcod=72'54)", 2024-09-21T22:07:39.182 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.690377+0000", 2024-09-21T22:07:39.182 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.42745206800000002, 2024-09-21T22:07:39.182 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.049254799000000002, 2024-09-21T22:07:39.183 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:39.183 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:39.183 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:39.183 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:39.183 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.183 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:39.183 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.690377+0000", 2024-09-21T22:07:39.183 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:39.183 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.183 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.183 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:39.183 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.690377+0000", 2024-09-21T22:07:39.183 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:39.184 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.184 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.184 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:39.184 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.690378+0000", 2024-09-21T22:07:39.184 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.344e-06 2024-09-21T22:07:39.184 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.184 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.184 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:39.184 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.691302+0000", 2024-09-21T22:07:39.184 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00092322400000000005 2024-09-21T22:07:39.184 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.184 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.184 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:39.184 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.691321+0000", 2024-09-21T22:07:39.184 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9928999999999998e-05 2024-09-21T22:07:39.185 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.185 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.185 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:39.185 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.691345+0000", 2024-09-21T22:07:39.185 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.372e-05 2024-09-21T22:07:39.185 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.185 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.185 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:39.185 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.691442+0000", 2024-09-21T22:07:39.185 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.7270999999999998e-05 2024-09-21T22:07:39.185 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.185 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.185 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:39.185 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.691512+0000", 2024-09-21T22:07:39.186 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.9183000000000005e-05 2024-09-21T22:07:39.186 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.186 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.186 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:39.186 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.739545+0000", 2024-09-21T22:07:39.186 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.048033657 2024-09-21T22:07:39.186 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.186 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.186 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:39.186 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.739632+0000", 2024-09-21T22:07:39.186 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.6471000000000006e-05 2024-09-21T22:07:39.186 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:39.186 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:39.186 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:39.186 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.187 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.187 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:156 2.1 e72/53 2:837ddd83:::rbd_data.126483d75f2d.000000000000000d:head v 72'28, mlcod=72'26)", 2024-09-21T22:07:39.187 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.824856+0000", 2024-09-21T22:07:39.187 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.29297263000000001, 2024-09-21T22:07:39.187 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.034320985999999998, 2024-09-21T22:07:39.187 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:39.187 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:39.187 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:39.187 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:39.187 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.187 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:39.187 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.824856+0000", 2024-09-21T22:07:39.187 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:39.187 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.187 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.188 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:39.188 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.824856+0000", 2024-09-21T22:07:39.188 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:39.188 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.188 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.188 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:39.188 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.824858+0000", 2024-09-21T22:07:39.188 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2979999999999999e-06 2024-09-21T22:07:39.188 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.188 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.188 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:39.188 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.825174+0000", 2024-09-21T22:07:39.188 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000316335 2024-09-21T22:07:39.188 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.189 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.189 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:39.189 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.825193+0000", 2024-09-21T22:07:39.189 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9436000000000002e-05 2024-09-21T22:07:39.189 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.189 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.189 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:39.189 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.825202+0000", 2024-09-21T22:07:39.189 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.8359999999999998e-06 2024-09-21T22:07:39.189 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.189 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.189 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:39.189 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.825286+0000", 2024-09-21T22:07:39.189 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.3683999999999994e-05 2024-09-21T22:07:39.190 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.190 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.190 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:39.190 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.825314+0000", 2024-09-21T22:07:39.190 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.7653000000000001e-05 2024-09-21T22:07:39.190 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.190 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.190 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:39.190 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.859122+0000", 2024-09-21T22:07:39.190 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.033808624000000002 2024-09-21T22:07:39.190 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.190 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.190 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:39.190 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.859177+0000", 2024-09-21T22:07:39.191 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.5120000000000001e-05 2024-09-21T22:07:39.191 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:39.191 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:39.191 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:39.191 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.191 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.191 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:158 2.1 e72/53 2:837ddd83:::rbd_data.126483d75f2d.000000000000000d:head v 72'30, mlcod=72'28)", 2024-09-21T22:07:39.191 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.869277+0000", 2024-09-21T22:07:39.191 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.24855243299999999, 2024-09-21T22:07:39.191 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.040508390999999998, 2024-09-21T22:07:39.191 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:39.191 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:39.191 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:39.191 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:39.192 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.192 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:39.192 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.869277+0000", 2024-09-21T22:07:39.192 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:39.192 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.192 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.192 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:39.192 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.869277+0000", 2024-09-21T22:07:39.192 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:39.192 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.192 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.192 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:39.192 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.869278+0000", 2024-09-21T22:07:39.192 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5269999999999999e-06 2024-09-21T22:07:39.193 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.193 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.193 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:39.193 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.869902+0000", 2024-09-21T22:07:39.193 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00062399000000000005 2024-09-21T22:07:39.193 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.193 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.193 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:39.193 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.869924+0000", 2024-09-21T22:07:39.193 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1875999999999999e-05 2024-09-21T22:07:39.193 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.193 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.193 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:39.193 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.869937+0000", 2024-09-21T22:07:39.193 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2685e-05 2024-09-21T22:07:39.194 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.194 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.194 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:39.194 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.870058+0000", 2024-09-21T22:07:39.194 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000120862 2024-09-21T22:07:39.194 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.194 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.194 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:39.194 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.870093+0000", 2024-09-21T22:07:39.194 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.5896000000000003e-05 2024-09-21T22:07:39.194 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.194 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.194 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:39.195 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.909728+0000", 2024-09-21T22:07:39.195 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.039634461000000003 2024-09-21T22:07:39.195 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.195 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.195 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:39.195 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.909785+0000", 2024-09-21T22:07:39.195 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.7093999999999999e-05 2024-09-21T22:07:39.195 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:39.195 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:39.195 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:39.195 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.195 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.195 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:161 2.1 e72/53 2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head v 72'32, mlcod=72'30)", 2024-09-21T22:07:39.195 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.924455+0000", 2024-09-21T22:07:39.195 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.19337407100000001, 2024-09-21T22:07:39.196 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.038366863000000001, 2024-09-21T22:07:39.196 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:39.196 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:39.196 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:39.196 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:39.196 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.196 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:39.196 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.924455+0000", 2024-09-21T22:07:39.196 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:39.196 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.196 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.196 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:39.196 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.924455+0000", 2024-09-21T22:07:39.196 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:39.197 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.197 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.197 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:39.197 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.924456+0000", 2024-09-21T22:07:39.197 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.003e-06 2024-09-21T22:07:39.197 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.197 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.197 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:39.197 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.924811+0000", 2024-09-21T22:07:39.197 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00035519499999999999 2024-09-21T22:07:39.197 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.197 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.197 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:39.197 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.924826+0000", 2024-09-21T22:07:39.198 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5296999999999999e-05 2024-09-21T22:07:39.198 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.198 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.198 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:39.198 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.924834+0000", 2024-09-21T22:07:39.198 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.5379999999999997e-06 2024-09-21T22:07:39.198 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.198 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.198 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:39.198 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.924910+0000", 2024-09-21T22:07:39.198 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.6488000000000004e-05 2024-09-21T22:07:39.198 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.198 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.198 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:39.199 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.924934+0000", 2024-09-21T22:07:39.199 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3598999999999999e-05 2024-09-21T22:07:39.199 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.199 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.199 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:39.199 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.962739+0000", 2024-09-21T22:07:39.199 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.037805370999999997 2024-09-21T22:07:39.199 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.199 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.199 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:39.199 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.962822+0000", 2024-09-21T22:07:39.199 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.2372000000000003e-05 2024-09-21T22:07:39.199 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:39.199 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:39.200 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:39.200 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.200 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.200 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:163 2.1 e72/53 2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head v 72'34, mlcod=72'32)", 2024-09-21T22:07:39.200 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.971683+0000", 2024-09-21T22:07:39.200 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.146145531, 2024-09-21T22:07:39.200 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.043171320999999999, 2024-09-21T22:07:39.200 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:39.200 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:39.200 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:39.200 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:39.200 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.200 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:39.200 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.971683+0000", 2024-09-21T22:07:39.200 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:39.201 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.201 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.201 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:39.201 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.971683+0000", 2024-09-21T22:07:39.201 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:39.201 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.201 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.201 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:39.201 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.971684+0000", 2024-09-21T22:07:39.201 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.6600000000000005e-07 2024-09-21T22:07:39.201 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.201 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.201 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:39.201 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.971945+0000", 2024-09-21T22:07:39.201 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00026101899999999998 2024-09-21T22:07:39.202 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.202 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.202 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:39.202 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.971959+0000", 2024-09-21T22:07:39.202 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4054e-05 2024-09-21T22:07:39.202 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.202 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.202 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:39.202 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.971969+0000", 2024-09-21T22:07:39.202 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.499e-06 2024-09-21T22:07:39.202 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.202 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.202 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:39.202 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.972037+0000", 2024-09-21T22:07:39.203 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.8448000000000006e-05 2024-09-21T22:07:39.203 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.203 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.203 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:39.203 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.972057+0000", 2024-09-21T22:07:39.203 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9817999999999999e-05 2024-09-21T22:07:39.203 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.203 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.203 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:39.203 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.014770+0000", 2024-09-21T22:07:39.203 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.042712464999999998 2024-09-21T22:07:39.203 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.203 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.203 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:39.204 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.014855+0000", 2024-09-21T22:07:39.204 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.5152000000000001e-05 2024-09-21T22:07:39.204 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:39.204 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:39.204 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:39.204 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.204 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.204 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:165 2.1 e72/53 2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head v 72'36, mlcod=72'34)", 2024-09-21T22:07:39.204 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:39.028631+0000", 2024-09-21T22:07:39.204 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 0.089198053999999999, 2024-09-21T22:07:39.204 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.033430561999999997, 2024-09-21T22:07:39.204 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:39.204 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:39.204 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:39.204 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:39.205 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.205 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:39.205 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.028631+0000", 2024-09-21T22:07:39.205 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:39.205 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.205 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.205 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:39.205 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.028631+0000", 2024-09-21T22:07:39.205 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:39.205 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.205 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.205 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:39.205 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.028633+0000", 2024-09-21T22:07:39.205 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5799999999999999e-06 2024-09-21T22:07:39.205 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.206 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.206 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:39.206 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.029209+0000", 2024-09-21T22:07:39.206 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00057670799999999999 2024-09-21T22:07:39.206 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.206 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.206 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:39.206 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.029238+0000", 2024-09-21T22:07:39.206 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8943000000000001e-05 2024-09-21T22:07:39.206 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.206 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.206 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:39.206 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.029252+0000", 2024-09-21T22:07:39.206 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3672999999999999e-05 2024-09-21T22:07:39.207 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.207 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.207 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:39.207 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.029328+0000", 2024-09-21T22:07:39.207 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.6497999999999998e-05 2024-09-21T22:07:39.207 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.207 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.207 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:39.207 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.029353+0000", 2024-09-21T22:07:39.207 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5060999999999999e-05 2024-09-21T22:07:39.207 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.207 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.207 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:39.207 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.061993+0000", 2024-09-21T22:07:39.207 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.032639286000000003 2024-09-21T22:07:39.208 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.208 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.208 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:39.208 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.062062+0000", 2024-09-21T22:07:39.208 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.8813000000000007e-05 2024-09-21T22:07:39.208 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:39.208 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:39.208 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:39.208 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:39.208 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:39.208 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:07:39.210 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T22:07:39.210 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:39.206+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:39.310 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:39.306+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:39.328 INFO:teuthology.orchestra.run.smithi039.stderr:reweighted osd.5 to 1 (10000) 2024-09-21T22:07:39.352 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-09-21T22:07:39.363 INFO:tasks.workunit.client.0.smithi157.stderr:64+0 records in 2024-09-21T22:07:39.363 INFO:tasks.workunit.client.0.smithi157.stderr:64+0 records out 2024-09-21T22:07:39.363 INFO:tasks.workunit.client.0.smithi157.stderr:67108864 bytes (67 MB, 64 MiB) copied, 3.40149 s, 19.7 MB/s 2024-09-21T22:07:39.365 INFO:tasks.workunit.client.0.smithi157.stderr:+ expect_false timeout 10 rbd bench testrbdnbd21553 --io-type write --io-size=1024 --io-total=1024 2024-09-21T22:07:39.365 INFO:tasks.workunit.client.0.smithi157.stderr:+ timeout 10 rbd bench testrbdnbd21553 --io-type write --io-size=1024 --io-total=1024 2024-09-21T22:07:39.387 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:07:39.387 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:07:39.387 INFO:teuthology.orchestra.run.smithi039.stdout: "num_ops": 0 2024-09-21T22:07:39.387 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:07:39.402 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T22:07:39.411 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:07:39.408+0000 7f2075a06640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31720) UID: 0 2024-09-21T22:07:39.437 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-21T22:07:39.434+0000 7fc90cbaa640 -1 librbd::exclusive_lock::ImageDispatch: 0x7fc8e8002060 handle_acquire_lock: failed to acquire exclusive lock: (30) Read-only file system 2024-09-21T22:07:39.438 INFO:tasks.workunit.client.0.smithi157.stderr:rbd: failed to flush: 2024-09-21T22:07:39.434+0000 7fc90cbaa640 -1 librbd::io::AioCompletion: 0x561b22658fc0 fail: (30) Read-only file system 2024-09-21T22:07:39.438 INFO:tasks.workunit.client.0.smithi157.stderr:(30) Read-only file system 2024-09-21T22:07:39.438 INFO:tasks.workunit.client.0.smithi157.stderr:bench failed: (30) Read-only file system 2024-09-21T22:07:39.445 INFO:tasks.workunit.client.0.smithi157.stderr:+ return 0 2024-09-21T22:07:39.445 INFO:tasks.workunit.client.0.smithi157.stderr:+ unmap_device /dev/nbd0 22576 2024-09-21T22:07:39.445 INFO:tasks.workunit.client.0.smithi157.stderr:+ local args=/dev/nbd0 2024-09-21T22:07:39.445 INFO:tasks.workunit.client.0.smithi157.stderr:+ local pid=22576 2024-09-21T22:07:39.445 INFO:tasks.workunit.client.0.smithi157.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd0 2024-09-21T22:07:39.445 INFO:tasks.workunit.client.0.smithi157.stderr:+ local cmd 2024-09-21T22:07:39.445 INFO:tasks.workunit.client.0.smithi157.stderr:++ id -u 2024-09-21T22:07:39.446 INFO:tasks.workunit.client.0.smithi157.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T22:07:39.446 INFO:tasks.workunit.client.0.smithi157.stderr:++ which rbd 2024-09-21T22:07:39.447 INFO:tasks.workunit.client.0.smithi157.stderr:+ cmd=/usr/bin/rbd 2024-09-21T22:07:39.447 INFO:tasks.workunit.client.0.smithi157.stderr:+ shift 2024-09-21T22:07:39.447 INFO:tasks.workunit.client.0.smithi157.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd0 2024-09-21T22:07:39.512 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:39.510+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:39.531 INFO:tasks.workunit.client.0.smithi157.stderr:+ expect_false grep '^22576\b' 2024-09-21T22:07:39.531 INFO:tasks.workunit.client.0.smithi157.stderr:+ grep '^22576\b' 2024-09-21T22:07:39.531 INFO:tasks.workunit.client.0.smithi157.stderr:+ rbd device --device-type nbd list 2024-09-21T22:07:39.562 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:07:39.562 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:07:39.562 INFO:teuthology.orchestra.run.smithi039.stdout: "complaint_time": 30, 2024-09-21T22:07:39.562 INFO:teuthology.orchestra.run.smithi039.stdout: "num_blocked_ops": 0 2024-09-21T22:07:39.562 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:07:39.577 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T22:07:39.582 INFO:tasks.workunit.client.0.smithi157.stderr:+ return 0 2024-09-21T22:07:39.583 INFO:tasks.workunit.client.0.smithi157.stderr:+ ps -C rbd-nbd 2024-09-21T22:07:39.583 INFO:tasks.workunit.client.0.smithi157.stderr:+ expect_false grep '^ *22576\b' 2024-09-21T22:07:39.583 INFO:tasks.workunit.client.0.smithi157.stderr:+ grep '^ *22576\b' 2024-09-21T22:07:39.589 INFO:tasks.workunit.client.0.smithi157.stderr:+ return 0 2024-09-21T22:07:39.589 INFO:tasks.workunit.client.0.smithi157.stderr:+ sleep 0.5 2024-09-21T22:07:39.624 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:39.622+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:39.713 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:39.710+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:39.747 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:07:39.747 INFO:teuthology.orchestra.run.smithi039.stdout: "size": 20, 2024-09-21T22:07:39.747 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 600, 2024-09-21T22:07:39.747 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [ 2024-09-21T22:07:39.747 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.747 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.3 55/54 [PushOp(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head, version: 52'2, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head@52'2, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T22:07:39.748 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.084379+0000", 2024-09-21T22:07:39.748 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 27.658378829, 2024-09-21T22:07:39.748 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.051020674000000002, 2024-09-21T22:07:39.748 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:39.748 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:39.748 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:39.748 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:39.748 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.749 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:39.749 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.084379+0000", 2024-09-21T22:07:39.749 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:39.749 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.749 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.749 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:39.749 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.084379+0000", 2024-09-21T22:07:39.749 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:39.750 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.750 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.750 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:39.750 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.084381+0000", 2024-09-21T22:07:39.750 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.73e-06 2024-09-21T22:07:39.750 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.750 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.750 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:39.750 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.085494+0000", 2024-09-21T22:07:39.750 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011131559999999999 2024-09-21T22:07:39.751 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.751 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.751 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:39.751 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.085542+0000", 2024-09-21T22:07:39.751 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8334999999999997e-05 2024-09-21T22:07:39.751 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.751 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.751 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:39.751 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.085591+0000", 2024-09-21T22:07:39.751 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.9165000000000001e-05 2024-09-21T22:07:39.752 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.752 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.752 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:39.752 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.085777+0000", 2024-09-21T22:07:39.752 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000186152 2024-09-21T22:07:39.752 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.752 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.752 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:39.752 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.085796+0000", 2024-09-21T22:07:39.752 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8612999999999999e-05 2024-09-21T22:07:39.752 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.752 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.753 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:39.753 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.135399+0000", 2024-09-21T22:07:39.753 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.049603522999999997 2024-09-21T22:07:39.753 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:39.753 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:39.753 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:39.753 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.753 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.753 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.3 55/54 [PushOp(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head, version: 52'2, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head@52'2, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:07:39.753 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.139554+0000", 2024-09-21T22:07:39.753 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 27.603203726, 2024-09-21T22:07:39.754 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.056215389999999997, 2024-09-21T22:07:39.754 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:39.754 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:39.754 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:39.754 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:39.754 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.754 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:39.754 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.139554+0000", 2024-09-21T22:07:39.754 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:39.754 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.754 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.754 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:39.754 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.139554+0000", 2024-09-21T22:07:39.755 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:39.755 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.755 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.755 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:39.755 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.139556+0000", 2024-09-21T22:07:39.755 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7999999999999999e-06 2024-09-21T22:07:39.755 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.755 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.755 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:39.755 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.140671+0000", 2024-09-21T22:07:39.755 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001115612 2024-09-21T22:07:39.755 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.755 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.755 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:39.755 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.140720+0000", 2024-09-21T22:07:39.756 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8631999999999998e-05 2024-09-21T22:07:39.756 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.756 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.756 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:39.756 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.140752+0000", 2024-09-21T22:07:39.756 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.2199000000000001e-05 2024-09-21T22:07:39.756 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.756 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.756 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:39.756 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.140900+0000", 2024-09-21T22:07:39.756 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00014786000000000001 2024-09-21T22:07:39.756 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.756 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.757 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:39.757 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.140917+0000", 2024-09-21T22:07:39.757 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6753999999999999e-05 2024-09-21T22:07:39.757 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.757 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.757 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:39.757 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.195769+0000", 2024-09-21T22:07:39.757 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.054852533000000002 2024-09-21T22:07:39.757 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:39.757 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:39.757 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:39.758 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.758 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.758 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.3 55/54 [PushOp(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head, version: 52'2, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head@52'2, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:07:39.758 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.199187+0000", 2024-09-21T22:07:39.758 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 27.543570182, 2024-09-21T22:07:39.758 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.051543975999999998, 2024-09-21T22:07:39.758 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:39.758 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:39.758 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:39.758 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:39.758 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.758 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:39.758 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.199187+0000", 2024-09-21T22:07:39.759 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:39.759 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.759 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.759 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:39.759 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.199187+0000", 2024-09-21T22:07:39.759 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:39.759 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.759 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.759 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:39.759 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.199189+0000", 2024-09-21T22:07:39.759 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7969999999999999e-06 2024-09-21T22:07:39.759 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.759 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.759 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:39.760 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.200355+0000", 2024-09-21T22:07:39.760 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011660139999999999 2024-09-21T22:07:39.760 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.760 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.760 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:39.760 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.200405+0000", 2024-09-21T22:07:39.760 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.0124000000000003e-05 2024-09-21T22:07:39.760 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.760 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.760 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:39.760 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.200457+0000", 2024-09-21T22:07:39.760 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.1177999999999997e-05 2024-09-21T22:07:39.760 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.761 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.761 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:39.761 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.200635+0000", 2024-09-21T22:07:39.761 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00017841299999999999 2024-09-21T22:07:39.761 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.761 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.761 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:39.761 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.200653+0000", 2024-09-21T22:07:39.761 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7944999999999998e-05 2024-09-21T22:07:39.761 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.761 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.761 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:39.761 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.250731+0000", 2024-09-21T22:07:39.761 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.050078505000000002 2024-09-21T22:07:39.761 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:39.762 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:39.762 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:39.762 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.762 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.762 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.3 55/54 [PushOp(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head, version: 52'3, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head@52'3, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:07:39.762 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.343754+0000", 2024-09-21T22:07:39.762 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 27.399003471, 2024-09-21T22:07:39.762 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.063560607000000005, 2024-09-21T22:07:39.762 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:39.762 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:39.762 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:39.762 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:39.762 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.763 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:39.763 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.343754+0000", 2024-09-21T22:07:39.763 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:39.763 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.763 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.763 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:39.763 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.343754+0000", 2024-09-21T22:07:39.763 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:39.763 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.763 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.763 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:39.763 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.343756+0000", 2024-09-21T22:07:39.763 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7069999999999999e-06 2024-09-21T22:07:39.763 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.764 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.764 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:39.764 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.344864+0000", 2024-09-21T22:07:39.764 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011085159999999999 2024-09-21T22:07:39.764 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.764 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.764 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:39.764 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.344914+0000", 2024-09-21T22:07:39.764 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.9462000000000002e-05 2024-09-21T22:07:39.764 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.764 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.764 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:39.764 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.344961+0000", 2024-09-21T22:07:39.765 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.7633999999999998e-05 2024-09-21T22:07:39.765 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.765 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.765 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:39.765 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.345171+0000", 2024-09-21T22:07:39.765 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000209652 2024-09-21T22:07:39.765 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.765 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.765 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:39.765 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.345193+0000", 2024-09-21T22:07:39.765 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1985999999999999e-05 2024-09-21T22:07:39.765 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.765 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.765 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:39.766 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.407315+0000", 2024-09-21T22:07:39.766 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.06212165 2024-09-21T22:07:39.766 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:39.766 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:39.766 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:39.766 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.766 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.766 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.3 55/54 [PushOp(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head, version: 52'3, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head@52'3, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:07:39.766 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.410551+0000", 2024-09-21T22:07:39.766 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 27.332206339999999, 2024-09-21T22:07:39.766 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.054517522999999998, 2024-09-21T22:07:39.766 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:39.766 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:39.766 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:39.767 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:39.767 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.767 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:39.767 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.410551+0000", 2024-09-21T22:07:39.767 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:39.767 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.767 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.767 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:39.767 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.410551+0000", 2024-09-21T22:07:39.767 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:39.767 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.767 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.767 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:39.768 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.410553+0000", 2024-09-21T22:07:39.768 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5200000000000001e-06 2024-09-21T22:07:39.768 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.768 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.768 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:39.768 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.411747+0000", 2024-09-21T22:07:39.768 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011946089999999999 2024-09-21T22:07:39.768 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.768 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.768 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:39.768 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.411796+0000", 2024-09-21T22:07:39.768 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8717999999999997e-05 2024-09-21T22:07:39.768 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.768 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.768 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:39.769 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.411844+0000", 2024-09-21T22:07:39.769 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8291999999999998e-05 2024-09-21T22:07:39.769 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.769 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.769 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:39.769 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.412066+0000", 2024-09-21T22:07:39.769 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00022115899999999999 2024-09-21T22:07:39.769 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.769 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.769 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:39.769 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.412081+0000", 2024-09-21T22:07:39.769 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5654e-05 2024-09-21T22:07:39.769 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.769 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.770 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:39.770 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.465069+0000", 2024-09-21T22:07:39.770 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.052987570999999997 2024-09-21T22:07:39.770 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:39.770 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:39.770 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:39.770 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.770 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.770 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.3 55/54 [PushOp(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head, version: 52'3, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head@52'3, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 4194304, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:07:39.770 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.468339+0000", 2024-09-21T22:07:39.770 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 27.274418676, 2024-09-21T22:07:39.770 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.064010024999999998, 2024-09-21T22:07:39.770 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:39.770 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:39.770 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:39.771 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:39.771 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.771 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:39.771 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.468339+0000", 2024-09-21T22:07:39.771 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:39.771 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.771 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.771 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:39.771 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.468339+0000", 2024-09-21T22:07:39.771 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:39.771 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.771 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.771 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:39.771 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.468340+0000", 2024-09-21T22:07:39.772 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4330000000000001e-06 2024-09-21T22:07:39.772 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.772 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.772 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:39.772 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.469484+0000", 2024-09-21T22:07:39.772 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001143535 2024-09-21T22:07:39.772 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.772 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.772 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:39.772 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.469533+0000", 2024-09-21T22:07:39.772 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8980999999999997e-05 2024-09-21T22:07:39.772 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.772 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.772 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:39.772 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.469581+0000", 2024-09-21T22:07:39.773 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8551999999999999e-05 2024-09-21T22:07:39.773 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.773 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.773 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:39.773 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.469804+0000", 2024-09-21T22:07:39.773 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00022287699999999999 2024-09-21T22:07:39.773 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.773 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.773 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:39.773 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.469829+0000", 2024-09-21T22:07:39.773 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4766e-05 2024-09-21T22:07:39.773 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.773 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.773 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:39.773 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.532349+0000", 2024-09-21T22:07:39.774 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.062519880999999999 2024-09-21T22:07:39.774 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:39.774 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:39.774 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:39.774 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.774 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.774 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4840.0:668 2.3 2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:07:39.774 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.409114+0000", 2024-09-21T22:07:39.774 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 15.33364347, 2024-09-21T22:07:39.774 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.048703758999999999, 2024-09-21T22:07:39.774 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:39.774 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:39.774 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:39.774 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-21T22:07:39.774 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4840", 2024-09-21T22:07:39.775 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:39.775 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 668 2024-09-21T22:07:39.775 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.775 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:39.775 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.775 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:39.775 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.409114+0000", 2024-09-21T22:07:39.775 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:39.775 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.775 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.775 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:39.775 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.409114+0000", 2024-09-21T22:07:39.775 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:39.775 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.776 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.776 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:39.776 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.409115+0000", 2024-09-21T22:07:39.776 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.114e-06 2024-09-21T22:07:39.776 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.776 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.776 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:39.776 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.410308+0000", 2024-09-21T22:07:39.776 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011930949999999999 2024-09-21T22:07:39.776 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.776 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.776 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:39.776 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.410325+0000", 2024-09-21T22:07:39.776 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7088e-05 2024-09-21T22:07:39.777 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.777 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.777 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:39.777 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.410334+0000", 2024-09-21T22:07:39.777 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.2549999999999997e-06 2024-09-21T22:07:39.777 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.777 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.777 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:39.777 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.410393+0000", 2024-09-21T22:07:39.777 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.9327999999999998e-05 2024-09-21T22:07:39.777 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.777 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.777 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:39.778 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.410482+0000", 2024-09-21T22:07:39.778 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.9019999999999998e-05 2024-09-21T22:07:39.778 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.778 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.778 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-21T22:07:39.778 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.410617+0000", 2024-09-21T22:07:39.778 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000135232 2024-09-21T22:07:39.778 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.778 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.778 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:39.778 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.457051+0000", 2024-09-21T22:07:39.778 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.04643415 2024-09-21T22:07:39.779 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.779 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.779 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-21T22:07:39.779 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.457675+0000", 2024-09-21T22:07:39.779 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00062332700000000002 2024-09-21T22:07:39.779 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.779 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.779 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:39.779 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.457735+0000", 2024-09-21T22:07:39.779 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.029e-05 2024-09-21T22:07:39.779 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.779 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.779 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:39.779 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.457818+0000", 2024-09-21T22:07:39.780 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.2860000000000003e-05 2024-09-21T22:07:39.780 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:39.780 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:39.780 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:39.780 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.780 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.780 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4840.0:670 2.3 2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:07:39.780 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.466980+0000", 2024-09-21T22:07:39.780 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 15.275777401999999, 2024-09-21T22:07:39.780 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.050913930000000003, 2024-09-21T22:07:39.780 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:39.780 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:39.780 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:39.780 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-21T22:07:39.781 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4840", 2024-09-21T22:07:39.781 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:39.781 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 670 2024-09-21T22:07:39.781 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.781 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:39.781 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.781 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:39.781 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.466980+0000", 2024-09-21T22:07:39.781 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:39.781 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.781 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.781 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:39.781 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.466980+0000", 2024-09-21T22:07:39.782 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:39.782 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.782 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.782 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:39.782 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.466981+0000", 2024-09-21T22:07:39.782 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0839999999999999e-06 2024-09-21T22:07:39.782 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.782 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.782 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:39.782 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.467815+0000", 2024-09-21T22:07:39.782 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00083330399999999999 2024-09-21T22:07:39.782 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.782 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.783 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:39.783 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.467832+0000", 2024-09-21T22:07:39.783 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7601e-05 2024-09-21T22:07:39.783 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.783 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.783 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:39.783 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.467840+0000", 2024-09-21T22:07:39.783 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.2220000000000002e-06 2024-09-21T22:07:39.783 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.783 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.783 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:39.783 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.467974+0000", 2024-09-21T22:07:39.783 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000133411 2024-09-21T22:07:39.784 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.784 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.784 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:39.784 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.468223+0000", 2024-09-21T22:07:39.784 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00024906400000000001 2024-09-21T22:07:39.784 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.784 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.784 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-21T22:07:39.784 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.468616+0000", 2024-09-21T22:07:39.784 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000393433 2024-09-21T22:07:39.784 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.785 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.785 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:39.785 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.517016+0000", 2024-09-21T22:07:39.785 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.048399708 2024-09-21T22:07:39.785 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.785 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.785 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-21T22:07:39.785 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.517693+0000", 2024-09-21T22:07:39.785 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00067681000000000002 2024-09-21T22:07:39.785 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.785 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.785 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:39.785 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.517789+0000", 2024-09-21T22:07:39.785 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.5778000000000004e-05 2024-09-21T22:07:39.785 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.786 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.786 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:39.786 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.517894+0000", 2024-09-21T22:07:39.786 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000105515 2024-09-21T22:07:39.786 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:39.786 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:39.786 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:39.786 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.786 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.786 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4840.0:672 2.3 2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:07:39.786 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.527127+0000", 2024-09-21T22:07:39.787 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 15.21563094, 2024-09-21T22:07:39.787 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.047345602000000001, 2024-09-21T22:07:39.787 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:39.787 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:39.787 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:39.787 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-21T22:07:39.787 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4840", 2024-09-21T22:07:39.787 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:39.787 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 672 2024-09-21T22:07:39.787 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.787 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:39.787 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.787 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:39.787 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.527127+0000", 2024-09-21T22:07:39.788 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:39.788 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.788 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.788 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:39.788 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.527127+0000", 2024-09-21T22:07:39.788 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:39.788 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.788 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.788 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:39.788 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.527128+0000", 2024-09-21T22:07:39.788 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2810000000000001e-06 2024-09-21T22:07:39.788 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.788 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.788 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:39.788 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.527962+0000", 2024-09-21T22:07:39.789 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00083445199999999996 2024-09-21T22:07:39.789 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.789 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.789 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:39.789 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.527981+0000", 2024-09-21T22:07:39.789 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9009e-05 2024-09-21T22:07:39.789 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.789 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.789 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:39.789 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.527990+0000", 2024-09-21T22:07:39.789 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.8619999999999992e-06 2024-09-21T22:07:39.789 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.789 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.789 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:39.789 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.528061+0000", 2024-09-21T22:07:39.790 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.0226000000000002e-05 2024-09-21T22:07:39.790 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.790 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.790 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:39.790 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.528156+0000", 2024-09-21T22:07:39.790 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.5724999999999996e-05 2024-09-21T22:07:39.790 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.790 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.790 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-21T22:07:39.790 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.528303+0000", 2024-09-21T22:07:39.790 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000146374 2024-09-21T22:07:39.790 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.790 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.790 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:39.791 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.573772+0000", 2024-09-21T22:07:39.791 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.045469745999999998 2024-09-21T22:07:39.791 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.791 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.791 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-21T22:07:39.791 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.574299+0000", 2024-09-21T22:07:39.791 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00052649200000000004 2024-09-21T22:07:39.791 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.791 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.791 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:39.791 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.574366+0000", 2024-09-21T22:07:39.791 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.6872000000000006e-05 2024-09-21T22:07:39.791 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.791 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.791 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:39.792 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.574472+0000", 2024-09-21T22:07:39.792 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000106563 2024-09-21T22:07:39.792 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:39.792 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:39.792 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:39.792 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.792 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.792 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4840.0:686 2.3 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:07:39.792 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.943561+0000", 2024-09-21T22:07:39.792 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 14.799196664, 2024-09-21T22:07:39.792 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.049021493999999999, 2024-09-21T22:07:39.792 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:39.792 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:39.792 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:39.792 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-21T22:07:39.793 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4840", 2024-09-21T22:07:39.793 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:39.793 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 686 2024-09-21T22:07:39.793 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.793 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:39.793 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.793 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:39.793 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.943561+0000", 2024-09-21T22:07:39.793 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:39.793 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.793 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.793 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:39.793 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.943561+0000", 2024-09-21T22:07:39.793 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:39.794 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.794 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.794 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:39.794 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.943563+0000", 2024-09-21T22:07:39.794 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9420000000000002e-06 2024-09-21T22:07:39.794 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.794 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.794 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:39.794 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.944546+0000", 2024-09-21T22:07:39.794 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00098357099999999992 2024-09-21T22:07:39.794 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.794 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.794 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:39.795 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.944565+0000", 2024-09-21T22:07:39.795 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8621e-05 2024-09-21T22:07:39.795 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.795 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.795 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:39.795 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.944575+0000", 2024-09-21T22:07:39.795 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.4639999999999995e-06 2024-09-21T22:07:39.795 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.795 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.795 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:39.795 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.944647+0000", 2024-09-21T22:07:39.795 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.2786000000000004e-05 2024-09-21T22:07:39.795 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.796 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.796 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:39.796 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.944746+0000", 2024-09-21T22:07:39.796 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.8196999999999995e-05 2024-09-21T22:07:39.796 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.796 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.796 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-21T22:07:39.796 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.944890+0000", 2024-09-21T22:07:39.796 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000144804 2024-09-21T22:07:39.796 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.796 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.796 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:39.796 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.992404+0000", 2024-09-21T22:07:39.796 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.047513211999999999 2024-09-21T22:07:39.796 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.797 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.797 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-21T22:07:39.797 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.992480+0000", 2024-09-21T22:07:39.797 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.6491999999999999e-05 2024-09-21T22:07:39.797 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.797 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.797 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:39.797 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.992521+0000", 2024-09-21T22:07:39.797 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.0608000000000002e-05 2024-09-21T22:07:39.797 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.797 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.797 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:39.797 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.992582+0000", 2024-09-21T22:07:39.797 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.1797000000000005e-05 2024-09-21T22:07:39.798 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:39.798 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:39.798 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:39.798 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.798 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.798 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4840.0:720 2.1 2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:07:39.798 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.803827+0000", 2024-09-21T22:07:39.798 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 13.938930387999999, 2024-09-21T22:07:39.798 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.049890510999999998, 2024-09-21T22:07:39.798 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:39.798 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:39.798 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:39.798 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-21T22:07:39.798 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4840", 2024-09-21T22:07:39.799 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:39.799 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 720 2024-09-21T22:07:39.799 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.799 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:39.799 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.799 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:39.799 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.803827+0000", 2024-09-21T22:07:39.799 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:39.799 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.799 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.799 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:39.799 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.803827+0000", 2024-09-21T22:07:39.799 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:39.800 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.800 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.800 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:39.800 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.803828+0000", 2024-09-21T22:07:39.800 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.569999999999999e-07 2024-09-21T22:07:39.800 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.800 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.800 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:39.800 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.804689+0000", 2024-09-21T22:07:39.800 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00086121799999999997 2024-09-21T22:07:39.800 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.800 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.800 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:39.800 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.804706+0000", 2024-09-21T22:07:39.800 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6266999999999999e-05 2024-09-21T22:07:39.801 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.801 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.801 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:39.801 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.804713+0000", 2024-09-21T22:07:39.801 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.1010000000000003e-06 2024-09-21T22:07:39.801 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.801 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.801 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:39.801 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.804781+0000", 2024-09-21T22:07:39.801 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.7871000000000001e-05 2024-09-21T22:07:39.801 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.801 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.801 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:39.801 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.804859+0000", 2024-09-21T22:07:39.802 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.7891000000000004e-05 2024-09-21T22:07:39.802 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.802 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.802 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:39.802 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.804971+0000", 2024-09-21T22:07:39.802 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00011275100000000001 2024-09-21T22:07:39.802 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.802 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.802 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:39.802 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.852704+0000", 2024-09-21T22:07:39.802 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.047732361000000001 2024-09-21T22:07:39.802 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.802 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.802 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-21T22:07:39.803 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.853446+0000", 2024-09-21T22:07:39.803 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00074258799999999997 2024-09-21T22:07:39.803 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.803 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.803 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:39.803 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.853608+0000", 2024-09-21T22:07:39.803 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00016190900000000001 2024-09-21T22:07:39.803 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.803 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.803 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:39.803 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.853718+0000", 2024-09-21T22:07:39.803 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000109597 2024-09-21T22:07:39.804 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:39.804 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:39.804 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:39.804 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.804 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.804 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4840.0:723 2.1 2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:07:39.804 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.860543+0000", 2024-09-21T22:07:39.804 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 13.882214954, 2024-09-21T22:07:39.804 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.062354957000000003, 2024-09-21T22:07:39.804 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:39.804 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:39.804 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:39.805 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-21T22:07:39.805 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4840", 2024-09-21T22:07:39.805 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:39.805 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 723 2024-09-21T22:07:39.805 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.805 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:39.805 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.805 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:39.805 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.860543+0000", 2024-09-21T22:07:39.805 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:39.806 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.806 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.806 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:39.806 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.860543+0000", 2024-09-21T22:07:39.806 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:39.806 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.806 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.806 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:39.806 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.860544+0000", 2024-09-21T22:07:39.806 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.1119999999999999e-06 2024-09-21T22:07:39.806 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.806 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.807 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:39.807 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.861400+0000", 2024-09-21T22:07:39.807 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000856393 2024-09-21T22:07:39.807 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.807 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.807 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:39.807 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.861419+0000", 2024-09-21T22:07:39.807 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8992e-05 2024-09-21T22:07:39.807 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.807 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.807 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:39.807 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.861428+0000", 2024-09-21T22:07:39.807 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.551e-06 2024-09-21T22:07:39.807 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.807 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.808 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:39.808 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.861555+0000", 2024-09-21T22:07:39.808 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000127286 2024-09-21T22:07:39.808 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.808 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.808 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:39.808 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.861816+0000", 2024-09-21T22:07:39.808 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00026100899999999998 2024-09-21T22:07:39.808 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.808 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.808 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:39.809 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.862189+0000", 2024-09-21T22:07:39.809 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00037261899999999998 2024-09-21T22:07:39.809 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.809 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.809 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:39.809 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.921514+0000", 2024-09-21T22:07:39.809 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.059325210000000003 2024-09-21T22:07:39.809 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.809 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.809 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-21T22:07:39.809 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.922721+0000", 2024-09-21T22:07:39.809 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0012066819999999999 2024-09-21T22:07:39.809 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.810 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.810 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:39.810 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.922797+0000", 2024-09-21T22:07:39.810 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.6162999999999996e-05 2024-09-21T22:07:39.810 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.810 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.810 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:39.810 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.922898+0000", 2024-09-21T22:07:39.810 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00010094000000000001 2024-09-21T22:07:39.810 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:39.810 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:39.810 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:39.810 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.811 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.811 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:725 2.0 e63/16 2:08b4f96a:::rbd_data.126483d75f2d.000000000000000f:head v 63'33, mlcod=63'31)", 2024-09-21T22:07:39.811 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.936778+0000", 2024-09-21T22:07:39.811 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 13.805979212, 2024-09-21T22:07:39.811 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.047671937999999997, 2024-09-21T22:07:39.811 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:39.811 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:39.811 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:39.811 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:39.811 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.811 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:39.811 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.936778+0000", 2024-09-21T22:07:39.811 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:39.812 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.812 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.812 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:39.812 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.936778+0000", 2024-09-21T22:07:39.812 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:39.812 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.812 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.812 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:39.812 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.936780+0000", 2024-09-21T22:07:39.812 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7799999999999999e-06 2024-09-21T22:07:39.812 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.812 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.812 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:39.812 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.938475+0000", 2024-09-21T22:07:39.812 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0016951749999999999 2024-09-21T22:07:39.813 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.813 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.813 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:39.813 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.938506+0000", 2024-09-21T22:07:39.813 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.0865999999999999e-05 2024-09-21T22:07:39.813 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.813 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.813 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:39.813 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.938520+0000", 2024-09-21T22:07:39.813 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3825999999999999e-05 2024-09-21T22:07:39.813 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.813 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.813 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:39.813 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.938624+0000", 2024-09-21T22:07:39.813 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00010391900000000001 2024-09-21T22:07:39.814 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.814 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.814 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:39.814 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.938651+0000", 2024-09-21T22:07:39.814 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.7481999999999999e-05 2024-09-21T22:07:39.814 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.814 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.814 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:39.814 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.984369+0000", 2024-09-21T22:07:39.815 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.045717595 2024-09-21T22:07:39.815 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.815 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.815 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:39.815 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.984450+0000", 2024-09-21T22:07:39.815 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.1295e-05 2024-09-21T22:07:39.815 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:39.815 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:39.815 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:39.815 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.815 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.815 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:727 2.0 e63/16 2:08b4f96a:::rbd_data.126483d75f2d.000000000000000f:head v 63'35, mlcod=63'33)", 2024-09-21T22:07:39.815 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.998090+0000", 2024-09-21T22:07:39.816 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 13.744668036, 2024-09-21T22:07:39.816 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.052205704999999998, 2024-09-21T22:07:39.816 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:39.816 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:39.816 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:39.816 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:39.816 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.816 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:39.816 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.998090+0000", 2024-09-21T22:07:39.816 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:39.816 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.816 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.816 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:39.816 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.998090+0000", 2024-09-21T22:07:39.816 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:39.817 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.817 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.817 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:39.817 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.998091+0000", 2024-09-21T22:07:39.817 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6929999999999999e-06 2024-09-21T22:07:39.817 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.817 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.817 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:39.817 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.998940+0000", 2024-09-21T22:07:39.817 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00084875200000000004 2024-09-21T22:07:39.817 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.817 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.817 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:39.818 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.998970+0000", 2024-09-21T22:07:39.818 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9709999999999998e-05 2024-09-21T22:07:39.818 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.818 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.818 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:39.818 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.998984+0000", 2024-09-21T22:07:39.818 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3739e-05 2024-09-21T22:07:39.818 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.818 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.818 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:39.818 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.999097+0000", 2024-09-21T22:07:39.818 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000113484 2024-09-21T22:07:39.819 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.819 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.819 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:39.819 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.999135+0000", 2024-09-21T22:07:39.819 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.8405999999999998e-05 2024-09-21T22:07:39.819 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.819 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.819 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:39.819 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.050245+0000", 2024-09-21T22:07:39.819 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.051109935000000002 2024-09-21T22:07:39.819 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.819 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.820 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:39.820 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.050295+0000", 2024-09-21T22:07:39.820 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.9985999999999999e-05 2024-09-21T22:07:39.820 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:39.820 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:39.820 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:39.820 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.820 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.820 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:729 2.0 e63/16 2:08b4f96a:::rbd_data.126483d75f2d.000000000000000f:head v 63'37, mlcod=63'35)", 2024-09-21T22:07:39.820 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:26.062424+0000", 2024-09-21T22:07:39.820 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 13.680333414, 2024-09-21T22:07:39.820 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.047312186999999999, 2024-09-21T22:07:39.820 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:39.821 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:39.821 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:39.821 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:39.821 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.821 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:39.821 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.062424+0000", 2024-09-21T22:07:39.821 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:39.821 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.821 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.821 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:39.821 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.062424+0000", 2024-09-21T22:07:39.821 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:39.821 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.821 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.821 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:39.822 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.062426+0000", 2024-09-21T22:07:39.822 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7129999999999999e-06 2024-09-21T22:07:39.822 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.822 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.822 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:39.822 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.063211+0000", 2024-09-21T22:07:39.822 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00078488500000000003 2024-09-21T22:07:39.822 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.822 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.822 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:39.822 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.063241+0000", 2024-09-21T22:07:39.822 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.0056000000000002e-05 2024-09-21T22:07:39.822 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.822 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.823 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:39.823 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.063255+0000", 2024-09-21T22:07:39.823 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4022999999999999e-05 2024-09-21T22:07:39.823 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.823 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.823 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:39.823 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.063384+0000", 2024-09-21T22:07:39.823 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00012920600000000001 2024-09-21T22:07:39.823 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.823 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.823 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:39.823 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.063421+0000", 2024-09-21T22:07:39.823 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.7242e-05 2024-09-21T22:07:39.824 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.824 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.824 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:39.824 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.109666+0000", 2024-09-21T22:07:39.824 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.046244446000000002 2024-09-21T22:07:39.824 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.824 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.824 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:39.824 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.109736+0000", 2024-09-21T22:07:39.824 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.0616000000000003e-05 2024-09-21T22:07:39.824 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:39.824 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:39.824 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:39.825 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.825 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.825 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:731 2.0 e63/16 2:08b4f96a:::rbd_data.126483d75f2d.000000000000000f:head v 63'39, mlcod=63'37)", 2024-09-21T22:07:39.825 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:26.135946+0000", 2024-09-21T22:07:39.825 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 13.606811907000001, 2024-09-21T22:07:39.825 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.054290359000000003, 2024-09-21T22:07:39.825 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:39.825 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:39.825 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:39.825 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:39.825 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.825 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:39.825 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.135946+0000", 2024-09-21T22:07:39.825 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:39.826 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.826 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.826 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:39.826 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.135946+0000", 2024-09-21T22:07:39.826 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:39.826 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.826 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.826 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:39.826 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.135948+0000", 2024-09-21T22:07:39.826 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7999999999999999e-06 2024-09-21T22:07:39.826 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.826 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.826 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:39.826 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.136730+0000", 2024-09-21T22:07:39.827 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00078271899999999997 2024-09-21T22:07:39.827 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.827 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.827 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:39.827 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.136760+0000", 2024-09-21T22:07:39.827 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9876e-05 2024-09-21T22:07:39.827 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.827 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.827 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:39.827 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.136774+0000", 2024-09-21T22:07:39.827 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3478999999999999e-05 2024-09-21T22:07:39.827 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.827 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.827 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:39.828 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.136944+0000", 2024-09-21T22:07:39.828 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000170626 2024-09-21T22:07:39.828 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.828 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.828 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:39.828 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.136985+0000", 2024-09-21T22:07:39.828 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.0377999999999999e-05 2024-09-21T22:07:39.828 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.828 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.828 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:39.828 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.190137+0000", 2024-09-21T22:07:39.828 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.053152438000000003 2024-09-21T22:07:39.828 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.829 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.829 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:39.829 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.190236+0000", 2024-09-21T22:07:39.829 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.9042999999999993e-05 2024-09-21T22:07:39.829 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:39.829 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:39.829 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:39.829 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.829 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.829 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:125 2.3 e71/66 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head v 71'31, mlcod=71'29)", 2024-09-21T22:07:39.829 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.066855+0000", 2024-09-21T22:07:39.829 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 1.675902472, 2024-09-21T22:07:39.829 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.059856059000000003, 2024-09-21T22:07:39.830 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:39.830 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:39.830 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:39.830 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:39.830 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.830 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:39.830 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.066855+0000", 2024-09-21T22:07:39.830 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:39.830 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.830 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.830 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:39.830 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.066855+0000", 2024-09-21T22:07:39.830 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:39.830 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.831 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.831 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:39.831 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.066857+0000", 2024-09-21T22:07:39.831 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.807e-06 2024-09-21T22:07:39.831 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.831 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.831 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:39.831 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.068365+0000", 2024-09-21T22:07:39.831 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0015077910000000001 2024-09-21T22:07:39.831 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.831 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.831 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:39.832 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.068395+0000", 2024-09-21T22:07:39.832 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.0326000000000001e-05 2024-09-21T22:07:39.832 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.832 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.832 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:39.832 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.068415+0000", 2024-09-21T22:07:39.832 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9871999999999999e-05 2024-09-21T22:07:39.832 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.833 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.833 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:39.833 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.068580+0000", 2024-09-21T22:07:39.833 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000165317 2024-09-21T22:07:39.833 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.833 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.833 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:39.833 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.068618+0000", 2024-09-21T22:07:39.833 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.7753000000000003e-05 2024-09-21T22:07:39.834 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.834 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.834 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:39.834 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.126623+0000", 2024-09-21T22:07:39.834 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.058004642000000002 2024-09-21T22:07:39.834 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.834 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.834 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:39.835 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.126711+0000", 2024-09-21T22:07:39.835 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.8551e-05 2024-09-21T22:07:39.835 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:39.835 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:39.835 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:39.835 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.835 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.836 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:127 2.3 e71/66 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head v 71'33, mlcod=71'31)", 2024-09-21T22:07:39.836 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.140864+0000", 2024-09-21T22:07:39.836 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 1.6018934439999999, 2024-09-21T22:07:39.836 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.052512380999999997, 2024-09-21T22:07:39.836 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:39.836 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:39.836 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:39.836 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:39.836 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.837 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:39.837 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.140864+0000", 2024-09-21T22:07:39.837 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:39.837 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.837 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.837 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:39.837 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.140864+0000", 2024-09-21T22:07:39.837 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:39.837 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.838 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.838 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:39.838 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.140866+0000", 2024-09-21T22:07:39.838 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.694e-06 2024-09-21T22:07:39.838 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.838 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.838 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:39.838 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.141712+0000", 2024-09-21T22:07:39.839 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00084581499999999998 2024-09-21T22:07:39.839 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.839 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.839 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:39.839 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.141742+0000", 2024-09-21T22:07:39.839 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9865999999999998e-05 2024-09-21T22:07:39.839 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.839 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.840 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:39.840 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.141761+0000", 2024-09-21T22:07:39.840 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9473000000000001e-05 2024-09-21T22:07:39.840 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.840 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.840 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:39.840 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.141919+0000", 2024-09-21T22:07:39.840 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00015758300000000001 2024-09-21T22:07:39.840 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.841 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.841 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:39.841 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.141955+0000", 2024-09-21T22:07:39.841 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.6613000000000003e-05 2024-09-21T22:07:39.841 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.841 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.841 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:39.841 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.193333+0000", 2024-09-21T22:07:39.841 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.051377260000000001 2024-09-21T22:07:39.841 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.842 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.842 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:39.842 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.193377+0000", 2024-09-21T22:07:39.842 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.4076999999999998e-05 2024-09-21T22:07:39.842 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:39.842 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:39.842 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:39.842 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.843 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.843 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:129 2.3 e71/66 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head v 71'35, mlcod=71'33)", 2024-09-21T22:07:39.843 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.209458+0000", 2024-09-21T22:07:39.843 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 1.533299695, 2024-09-21T22:07:39.843 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.056595578000000001, 2024-09-21T22:07:39.843 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:39.843 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:39.843 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:39.844 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:39.844 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.844 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:39.844 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.209458+0000", 2024-09-21T22:07:39.844 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:39.844 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.844 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.844 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:39.844 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.209458+0000", 2024-09-21T22:07:39.845 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:39.845 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.845 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.845 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:39.845 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.209460+0000", 2024-09-21T22:07:39.845 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7600000000000001e-06 2024-09-21T22:07:39.845 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.845 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.846 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:39.846 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.210317+0000", 2024-09-21T22:07:39.846 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00085771700000000003 2024-09-21T22:07:39.846 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.846 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.846 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:39.846 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.210346+0000", 2024-09-21T22:07:39.846 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8334e-05 2024-09-21T22:07:39.846 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.847 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.847 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:39.847 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.210365+0000", 2024-09-21T22:07:39.847 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9439000000000001e-05 2024-09-21T22:07:39.847 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.847 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.847 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:39.847 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.210544+0000", 2024-09-21T22:07:39.847 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00017845599999999999 2024-09-21T22:07:39.848 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.848 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.848 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:39.848 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.210584+0000", 2024-09-21T22:07:39.848 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.0423000000000002e-05 2024-09-21T22:07:39.848 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.848 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.848 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:39.849 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.265970+0000", 2024-09-21T22:07:39.849 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.055385765000000003 2024-09-21T22:07:39.849 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.849 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.849 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:39.849 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.266054+0000", 2024-09-21T22:07:39.849 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.3683999999999994e-05 2024-09-21T22:07:39.849 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:39.849 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:39.849 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:39.849 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.849 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.849 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:131 2.3 e71/66 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head v 71'37, mlcod=71'35)", 2024-09-21T22:07:39.849 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.279550+0000", 2024-09-21T22:07:39.850 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 1.463207857, 2024-09-21T22:07:39.850 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.052290419999999997, 2024-09-21T22:07:39.850 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:39.850 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:39.850 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:39.850 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:39.850 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.850 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:39.850 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.279550+0000", 2024-09-21T22:07:39.850 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:39.850 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.850 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.850 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:39.850 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.279550+0000", 2024-09-21T22:07:39.851 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:39.851 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.851 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.851 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:39.851 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.279552+0000", 2024-09-21T22:07:39.851 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9439999999999999e-06 2024-09-21T22:07:39.851 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.851 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.851 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:39.851 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.280296+0000", 2024-09-21T22:07:39.851 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00074417000000000005 2024-09-21T22:07:39.851 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.851 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.851 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:39.851 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.280324+0000", 2024-09-21T22:07:39.852 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8045999999999999e-05 2024-09-21T22:07:39.852 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.852 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.852 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:39.852 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.280345+0000", 2024-09-21T22:07:39.852 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.0849999999999999e-05 2024-09-21T22:07:39.852 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.852 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.852 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:39.852 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.280497+0000", 2024-09-21T22:07:39.852 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00015252500000000001 2024-09-21T22:07:39.852 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.852 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.852 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:39.853 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.280531+0000", 2024-09-21T22:07:39.853 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.4034e-05 2024-09-21T22:07:39.853 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.853 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.853 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:39.853 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.331769+0000", 2024-09-21T22:07:39.853 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.051237999999999999 2024-09-21T22:07:39.853 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:39.853 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:39.853 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:39.853 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.331840+0000", 2024-09-21T22:07:39.853 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.0851000000000003e-05 2024-09-21T22:07:39.854 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:39.854 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:39.854 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:39.854 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:39.854 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:39.854 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:07:39.855 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T22:07:39.856 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:39.854+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:39.888 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T22:07:39.956 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:39.952+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:40.027 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:07:40.027 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:07:40.027 INFO:teuthology.orchestra.run.smithi039.stdout: "num_ops": 0 2024-09-21T22:07:40.027 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:07:40.043 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T22:07:40.059 INFO:teuthology.orchestra.run.smithi039.stdout:osd.0: {} 2024-09-21T22:07:40.059 INFO:teuthology.orchestra.run.smithi039.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T22:07:40.060 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:40.058+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:40.076 INFO:teuthology.orchestra.run.smithi039.stdout:osd.1: {} 2024-09-21T22:07:40.077 INFO:teuthology.orchestra.run.smithi039.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T22:07:40.091 INFO:tasks.workunit.client.0.smithi157.stderr:+ DEV= 2024-09-21T22:07:40.091 INFO:tasks.workunit.client.0.smithi157.stderr:+ rbd bench testrbdnbd21553 --io-type write --io-size=1024 --io-total=1024 2024-09-21T22:07:40.093 INFO:teuthology.orchestra.run.smithi039.stdout:osd.2: {} 2024-09-21T22:07:40.093 INFO:teuthology.orchestra.run.smithi039.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T22:07:40.114 INFO:teuthology.orchestra.run.smithi039.stdout:osd.3: {} 2024-09-21T22:07:40.114 INFO:teuthology.orchestra.run.smithi039.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T22:07:40.136 INFO:teuthology.orchestra.run.smithi039.stdout:osd.4: {} 2024-09-21T22:07:40.136 INFO:teuthology.orchestra.run.smithi039.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T22:07:40.158 INFO:teuthology.orchestra.run.smithi039.stdout:osd.5: {} 2024-09-21T22:07:40.158 INFO:teuthology.orchestra.run.smithi039.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T22:07:40.158 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:40.154+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:40.161 INFO:tasks.workunit.client.0.smithi157.stdout:bench type write io_size 1024 io_threads 16 bytes 1024 pattern sequential 2024-09-21T22:07:40.161 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:40.161 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:40.164 INFO:tasks.workunit.client.0.smithi157.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-09-21T22:07:40.165 INFO:tasks.workunit.client.0.smithi157.stdout:elapsed: 0 ops: 1 ops/sec: 250.007 bytes/sec: 250 KiB/s 2024-09-21T22:07:40.178 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T22:07:40.179 INFO:tasks.workunit.client.0.smithi157.stderr:++ _sudo rbd device --device-type nbd map rbd/testrbdnbd21553 2024-09-21T22:07:40.179 INFO:tasks.workunit.client.0.smithi157.stderr:++ local cmd 2024-09-21T22:07:40.180 INFO:tasks.workunit.client.0.smithi157.stderr:+++ id -u 2024-09-21T22:07:40.181 INFO:tasks.workunit.client.0.smithi157.stderr:++ '[' 1000 -eq 0 ']' 2024-09-21T22:07:40.181 INFO:tasks.workunit.client.0.smithi157.stderr:+++ which rbd 2024-09-21T22:07:40.183 INFO:tasks.workunit.client.0.smithi157.stderr:++ cmd=/usr/bin/rbd 2024-09-21T22:07:40.183 INFO:tasks.workunit.client.0.smithi157.stderr:++ shift 2024-09-21T22:07:40.183 INFO:tasks.workunit.client.0.smithi157.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map rbd/testrbdnbd21553 2024-09-21T22:07:40.264 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:40.262+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:40.316 INFO:tasks.workunit.client.0.smithi157.stderr:+ DEV=/dev/nbd0 2024-09-21T22:07:40.316 INFO:tasks.workunit.client.0.smithi157.stderr:+ get_pid rbd 2024-09-21T22:07:40.317 INFO:tasks.workunit.client.0.smithi157.stderr:+ local pool=rbd 2024-09-21T22:07:40.317 INFO:tasks.workunit.client.0.smithi157.stderr:+ local ns= 2024-09-21T22:07:40.317 INFO:tasks.workunit.client.0.smithi157.stderr:++ rbd device --device-type nbd --format xml list 2024-09-21T22:07:40.317 INFO:tasks.workunit.client.0.smithi157.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd21553'\''][device='\''/dev/nbd0'\'']/id' 2024-09-21T22:07:40.339 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:40.340 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:40.349 INFO:teuthology.orchestra.run.smithi039.stderr:reweighted osd.0 to 1 (10000) 2024-09-21T22:07:40.354 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-21T22:07:40.357 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T22:07:40.374 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-09-21T22:07:40.374 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T22:07:40.375 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:40.370+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:40.383 INFO:tasks.workunit.client.0.smithi157.stderr:+ PID=22687 2024-09-21T22:07:40.383 INFO:tasks.workunit.client.0.smithi157.stderr:+ test -n 22687 2024-09-21T22:07:40.383 INFO:tasks.workunit.client.0.smithi157.stderr:+ ps -p 22687 -C rbd-nbd 2024-09-21T22:07:40.390 INFO:tasks.workunit.client.0.smithi157.stdout: PID TTY TIME CMD 2024-09-21T22:07:40.391 INFO:tasks.workunit.client.0.smithi157.stdout: 22687 ? 00:00:00 rbd-nbd 2024-09-21T22:07:40.391 INFO:tasks.workunit.client.0.smithi157.stderr:+ unmap_device testrbdnbd21553 22687 2024-09-21T22:07:40.391 INFO:tasks.workunit.client.0.smithi157.stderr:+ local args=testrbdnbd21553 2024-09-21T22:07:40.392 INFO:tasks.workunit.client.0.smithi157.stderr:+ local pid=22687 2024-09-21T22:07:40.392 INFO:tasks.workunit.client.0.smithi157.stderr:+ _sudo rbd device --device-type nbd unmap testrbdnbd21553 2024-09-21T22:07:40.392 INFO:tasks.workunit.client.0.smithi157.stderr:+ local cmd 2024-09-21T22:07:40.392 INFO:tasks.workunit.client.0.smithi157.stderr:++ id -u 2024-09-21T22:07:40.392 INFO:tasks.workunit.client.0.smithi157.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T22:07:40.393 INFO:tasks.workunit.client.0.smithi157.stderr:++ which rbd 2024-09-21T22:07:40.394 INFO:tasks.workunit.client.0.smithi157.stderr:+ cmd=/usr/bin/rbd 2024-09-21T22:07:40.394 INFO:tasks.workunit.client.0.smithi157.stderr:+ shift 2024-09-21T22:07:40.394 INFO:tasks.workunit.client.0.smithi157.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap testrbdnbd21553 2024-09-21T22:07:40.461 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:40.458+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:40.477 INFO:tasks.workunit.client.0.smithi157.stderr:+ rbd device --device-type nbd list 2024-09-21T22:07:40.477 INFO:tasks.workunit.client.0.smithi157.stderr:+ expect_false grep '^22687\b' 2024-09-21T22:07:40.477 INFO:tasks.workunit.client.0.smithi157.stderr:+ grep '^22687\b' 2024-09-21T22:07:40.494 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:40.494 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:40.508 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T22:07:40.527 INFO:tasks.workunit.client.0.smithi157.stderr:+ return 0 2024-09-21T22:07:40.528 INFO:tasks.workunit.client.0.smithi157.stderr:+ ps -C rbd-nbd 2024-09-21T22:07:40.528 INFO:tasks.workunit.client.0.smithi157.stderr:+ expect_false grep '^ *22687\b' 2024-09-21T22:07:40.528 INFO:tasks.workunit.client.0.smithi157.stderr:+ grep '^ *22687\b' 2024-09-21T22:07:40.534 INFO:tasks.workunit.client.0.smithi157.stderr:+ return 0 2024-09-21T22:07:40.534 INFO:tasks.workunit.client.0.smithi157.stderr:+ sleep 0.5 2024-09-21T22:07:40.562 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:07:40.560+0000 7f2075a06640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31720) UID: 0 2024-09-21T22:07:40.634 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:40.634 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:40.649 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T22:07:40.663 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:40.658+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:40.723 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T22:07:40.724 INFO:teuthology.orchestra.run.smithi039.stderr:dumped all 2024-09-21T22:07:40.742 INFO:teuthology.orchestra.run.smithi039.stdout:{"pg_ready":true,"pg_map":{"version":119,"stamp":"2024-09-21T22:07:40.348184+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":126419634,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":2341,"num_read_kb":142890,"num_write":676,"num_write_kb":256546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":35915939,"num_keys_recovered":8,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":586,"ondisk_log_size":586,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":471859200,"kb_used":408264,"kb_used_data":255188,"kb_used_omap":11,"kb_used_meta":152948,"kb_avail":471450936,"statfs":{"total":483183820800,"available":482765758464,"internally_reserved":0,"allocated":261312512,"data_stored":259765807,"data_compressed":13990,"data_compressed_allocated":73728,"data_compressed_original":1163264,"omap_allocated":11985,"internal_metadata":156619055},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":55,"apply_latency_ms":55,"commit_latency_ns":55000000,"apply_latency_ns":55000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":58720256,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":160,"num_read_kb":1130,"num_write":158,"num_write_kb":57376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.062938"},"pg_stats":[{"pgid":"1.0","version":"15'117","reported_seq":328,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T22:07:39.420226+0000","last_change":"2024-09-21T22:07:14.092075+0000","last_active":"2024-09-21T22:07:39.420226+0000","last_peered":"2024-09-21T22:07:39.420226+0000","last_clean":"2024-09-21T22:07:39.420226+0000","last_became_active":"2024-09-21T22:07:14.091129+0000","last_became_peered":"2024-09-21T22:07:14.091129+0000","last_unstale":"2024-09-21T22:07:39.420226+0000","last_undegraded":"2024-09-21T22:07:39.420226+0000","last_fullsized":"2024-09-21T22:07:39.420226+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:00.990331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:00.990331+0000","last_clean_scrub_stamp":"2024-09-21T22:06:00.990331+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:36:36.088445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"68'21","reported_seq":54,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-09-21T22:07:35.032317+0000","last_change":"2024-09-21T22:07:29.615170+0000","last_active":"2024-09-21T22:07:35.032317+0000","last_peered":"2024-09-21T22:07:35.032317+0000","last_clean":"2024-09-21T22:07:35.032317+0000","last_became_active":"2024-09-21T22:07:29.253735+0000","last_became_peered":"2024-09-21T22:07:29.253735+0000","last_unstale":"2024-09-21T22:07:35.032317+0000","last_undegraded":"2024-09-21T22:07:35.032317+0000","last_fullsized":"2024-09-21T22:07:35.032317+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:42:12.326592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":74,"num_read_kb":8200,"num_write":20,"num_write_kb":16384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":16777216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"61'2","reported_seq":51,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T22:07:39.432477+0000","last_change":"2024-09-21T22:07:28.245341+0000","last_active":"2024-09-21T22:07:39.432477+0000","last_peered":"2024-09-21T22:07:39.432477+0000","last_clean":"2024-09-21T22:07:39.432477+0000","last_became_active":"2024-09-21T22:07:28.233066+0000","last_became_peered":"2024-09-21T22:07:28.233066+0000","last_unstale":"2024-09-21T22:07:39.432477+0000","last_undegraded":"2024-09-21T22:07:39.432477+0000","last_fullsized":"2024-09-21T22:07:39.432477+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:07:16.101346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:07:16.101346+0000","last_clean_scrub_stamp":"2024-09-21T22:07:16.101346+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:00:11.717223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000947342,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"72'75","reported_seq":536,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T22:07:39.429140+0000","last_change":"2024-09-21T22:06:07.121573+0000","last_active":"2024-09-21T22:07:39.429140+0000","last_peered":"2024-09-21T22:07:39.429140+0000","last_clean":"2024-09-21T22:07:39.429140+0000","last_became_active":"2024-09-21T22:06:05.108187+0000","last_became_peered":"2024-09-21T22:06:05.108187+0000","last_unstale":"2024-09-21T22:07:39.429140+0000","last_undegraded":"2024-09-21T22:07:39.429140+0000","last_fullsized":"2024-09-21T22:07:39.429140+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T05:05:08.108341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079478100000000005,"stat_sum":{"num_bytes":25165824,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":336,"num_read_kb":35864,"num_write":74,"num_write_kb":57344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":41,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T22:07:38.340477+0000","last_change":"2024-09-21T22:07:19.214979+0000","last_active":"2024-09-21T22:07:38.340477+0000","last_peered":"2024-09-21T22:07:38.340477+0000","last_clean":"2024-09-21T22:07:38.340477+0000","last_became_active":"2024-09-21T22:07:17.152522+0000","last_became_peered":"2024-09-21T22:07:17.152522+0000","last_unstale":"2024-09-21T22:07:38.340477+0000","last_undegraded":"2024-09-21T22:07:38.340477+0000","last_fullsized":"2024-09-21T22:07:38.340477+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:07:16.101346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:07:16.101346+0000","last_clean_scrub_stamp":"2024-09-21T22:07:16.101346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:51:24.258279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036883599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"72'38","reported_seq":243,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T22:07:39.432356+0000","last_change":"2024-09-21T22:07:11.299996+0000","last_active":"2024-09-21T22:07:39.432356+0000","last_peered":"2024-09-21T22:07:39.432356+0000","last_clean":"2024-09-21T22:07:39.432356+0000","last_became_active":"2024-09-21T22:07:11.066642+0000","last_became_peered":"2024-09-21T22:07:11.066642+0000","last_unstale":"2024-09-21T22:07:39.432356+0000","last_undegraded":"2024-09-21T22:07:39.432356+0000","last_fullsized":"2024-09-21T22:07:39.432356+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:38:01.499033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010052629999999999,"stat_sum":{"num_bytes":16777216,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":68,"num_read_kb":16384,"num_write":38,"num_write_kb":32768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":43,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T22:07:39.432355+0000","last_change":"2024-09-21T22:07:19.151406+0000","last_active":"2024-09-21T22:07:39.432355+0000","last_peered":"2024-09-21T22:07:39.432355+0000","last_clean":"2024-09-21T22:07:39.432355+0000","last_became_active":"2024-09-21T22:07:17.156631+0000","last_became_peered":"2024-09-21T22:07:17.156631+0000","last_unstale":"2024-09-21T22:07:39.432355+0000","last_undegraded":"2024-09-21T22:07:39.432355+0000","last_fullsized":"2024-09-21T22:07:39.432355+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:07:16.101346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:07:16.101346+0000","last_clean_scrub_stamp":"2024-09-21T22:07:16.101346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T08:49:26.489839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00075277899999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"69'46","reported_seq":269,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-09-21T22:07:35.846004+0000","last_change":"2024-09-21T22:07:28.523773+0000","last_active":"2024-09-21T22:07:35.846004+0000","last_peered":"2024-09-21T22:07:35.846004+0000","last_clean":"2024-09-21T22:07:35.846004+0000","last_became_active":"2024-09-21T22:07:28.234283+0000","last_became_peered":"2024-09-21T22:07:28.234283+0000","last_unstale":"2024-09-21T22:07:35.846004+0000","last_undegraded":"2024-09-21T22:07:35.846004+0000","last_fullsized":"2024-09-21T22:07:35.846004+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T23:01:03.589018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388643,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":661,"num_read_kb":604,"num_write":19,"num_write_kb":8198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":8388624,"num_keys_recovered":8,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"59'1","reported_seq":43,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T22:07:39.333441+0000","last_change":"2024-09-21T22:07:19.218756+0000","last_active":"2024-09-21T22:07:39.333441+0000","last_peered":"2024-09-21T22:07:39.333441+0000","last_clean":"2024-09-21T22:07:39.333441+0000","last_became_active":"2024-09-21T22:07:17.161301+0000","last_became_peered":"2024-09-21T22:07:17.161301+0000","last_unstale":"2024-09-21T22:07:39.333441+0000","last_undegraded":"2024-09-21T22:07:39.333441+0000","last_fullsized":"2024-09-21T22:07:39.333441+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:07:16.101346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:07:16.101346+0000","last_clean_scrub_stamp":"2024-09-21T22:07:16.101346+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T09:55:09.171875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001129983,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-09-21T22:07:35.032492+0000","last_change":"2024-09-21T22:07:28.242244+0000","last_active":"2024-09-21T22:07:35.032492+0000","last_peered":"2024-09-21T22:07:35.032492+0000","last_clean":"2024-09-21T22:07:35.032492+0000","last_became_active":"2024-09-21T22:07:28.240361+0000","last_became_peered":"2024-09-21T22:07:28.240361+0000","last_unstale":"2024-09-21T22:07:35.032492+0000","last_undegraded":"2024-09-21T22:07:35.032492+0000","last_fullsized":"2024-09-21T22:07:35.032492+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:07:16.101346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:07:16.101346+0000","last_clean_scrub_stamp":"2024-09-21T22:07:16.101346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:26:46.171549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010184180000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"70'45","reported_seq":282,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T22:07:39.432505+0000","last_change":"2024-09-21T22:06:44.875768+0000","last_active":"2024-09-21T22:07:39.432505+0000","last_peered":"2024-09-21T22:07:39.432505+0000","last_clean":"2024-09-21T22:07:39.432505+0000","last_became_active":"2024-09-21T22:06:44.875201+0000","last_became_peered":"2024-09-21T22:06:44.875201+0000","last_unstale":"2024-09-21T22:07:39.432505+0000","last_undegraded":"2024-09-21T22:07:39.432505+0000","last_fullsized":"2024-09-21T22:07:39.432505+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T08:49:08.148866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00090834100000000001,"stat_sum":{"num_bytes":16777216,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":117,"num_read_kb":24610,"num_write":43,"num_write_kb":32771,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":43,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T22:07:39.432475+0000","last_change":"2024-09-21T22:07:19.151659+0000","last_active":"2024-09-21T22:07:39.432475+0000","last_peered":"2024-09-21T22:07:39.432475+0000","last_clean":"2024-09-21T22:07:39.432475+0000","last_became_active":"2024-09-21T22:07:17.155913+0000","last_became_peered":"2024-09-21T22:07:17.155913+0000","last_unstale":"2024-09-21T22:07:39.432475+0000","last_undegraded":"2024-09-21T22:07:39.432475+0000","last_fullsized":"2024-09-21T22:07:39.432475+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:07:16.101346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:07:16.101346+0000","last_clean_scrub_stamp":"2024-09-21T22:07:16.101346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T09:09:08.662264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00068604600000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"72'56","reported_seq":260,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T22:07:39.418674+0000","last_change":"2024-09-21T22:06:44.875038+0000","last_active":"2024-09-21T22:07:39.418674+0000","last_peered":"2024-09-21T22:07:39.418674+0000","last_clean":"2024-09-21T22:07:39.418674+0000","last_became_active":"2024-09-21T22:06:44.874697+0000","last_became_peered":"2024-09-21T22:06:44.874697+0000","last_unstale":"2024-09-21T22:07:39.418674+0000","last_undegraded":"2024-09-21T22:07:39.418674+0000","last_fullsized":"2024-09-21T22:07:39.418674+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T07:09:15.569760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091949900000000005,"stat_sum":{"num_bytes":12582950,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":169,"num_read_kb":16466,"num_write":72,"num_write_kb":28705,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":43,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T22:07:39.432456+0000","last_change":"2024-09-21T22:07:19.151518+0000","last_active":"2024-09-21T22:07:39.432456+0000","last_peered":"2024-09-21T22:07:39.432456+0000","last_clean":"2024-09-21T22:07:39.432456+0000","last_became_active":"2024-09-21T22:07:17.156489+0000","last_became_peered":"2024-09-21T22:07:17.156489+0000","last_unstale":"2024-09-21T22:07:39.432456+0000","last_undegraded":"2024-09-21T22:07:39.432456+0000","last_fullsized":"2024-09-21T22:07:39.432456+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:07:16.101346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:07:16.101346+0000","last_clean_scrub_stamp":"2024-09-21T22:07:16.101346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:04:16.326684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00066928400000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"70'24","reported_seq":194,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T22:07:39.432496+0000","last_change":"2024-09-21T22:06:07.133757+0000","last_active":"2024-09-21T22:07:39.432496+0000","last_peered":"2024-09-21T22:07:39.432496+0000","last_clean":"2024-09-21T22:07:39.432496+0000","last_became_active":"2024-09-21T22:06:05.109065+0000","last_became_peered":"2024-09-21T22:06:05.109065+0000","last_unstale":"2024-09-21T22:07:39.432496+0000","last_undegraded":"2024-09-21T22:07:39.432496+0000","last_fullsized":"2024-09-21T22:07:39.432496+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:31:12.654893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00088865700000000005,"stat_sum":{"num_bytes":20971520,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":46,"num_read_kb":12316,"num_write":24,"num_write_kb":28674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":41,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T22:07:38.340521+0000","last_change":"2024-09-21T22:07:19.215026+0000","last_active":"2024-09-21T22:07:38.340521+0000","last_peered":"2024-09-21T22:07:38.340521+0000","last_clean":"2024-09-21T22:07:38.340521+0000","last_became_active":"2024-09-21T22:07:17.159836+0000","last_became_peered":"2024-09-21T22:07:17.159836+0000","last_unstale":"2024-09-21T22:07:38.340521+0000","last_undegraded":"2024-09-21T22:07:38.340521+0000","last_fullsized":"2024-09-21T22:07:38.340521+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:07:16.101346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:07:16.101346+0000","last_clean_scrub_stamp":"2024-09-21T22:07:16.101346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:16:31.665427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032723700000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"73'161","reported_seq":596,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T22:07:40.139811+0000","last_change":"2024-09-21T22:06:07.122328+0000","last_active":"2024-09-21T22:07:40.139811+0000","last_peered":"2024-09-21T22:07:40.139811+0000","last_clean":"2024-09-21T22:07:40.139811+0000","last_became_active":"2024-09-21T22:06:05.220996+0000","last_became_peered":"2024-09-21T22:06:05.220996+0000","last_unstale":"2024-09-21T22:07:40.139811+0000","last_undegraded":"2024-09-21T22:07:40.139811+0000","last_fullsized":"2024-09-21T22:07:40.139811+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T23:45:49.178013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001473755,"stat_sum":{"num_bytes":25165878,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":754,"num_read_kb":28346,"num_write":225,"num_write_kb":49284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":5},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":125829247,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":2225,"num_read_kb":142790,"num_write":515,"num_write_kb":254128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":33554448,"num_keys_recovered":8,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":268476416,"data_stored":268435710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1174,"internal_metadata":0},"log_size":466,"ondisk_log_size":466,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":2361472,"data_compressed":28848,"data_compressed_allocated":147456,"data_compressed_original":2342912,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":65,"seq":279172874243,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37288,"kb_used_data":8980,"kb_used_omap":5,"kb_used_meta":28282,"kb_avail":94334552,"statfs":{"total":96636764160,"available":96598581248,"internally_reserved":0,"allocated":9195520,"data_stored":8686299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5607,"internal_metadata":28961305},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640280,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56588,"kb_used_data":25396,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94315252,"statfs":{"total":96636764160,"available":96578818048,"internally_reserved":0,"allocated":26005504,"data_stored":25478377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640283,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":111048,"kb_used_data":79856,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94260792,"statfs":{"total":96636764160,"available":96523051008,"internally_reserved":0,"allocated":81772544,"data_stored":81774383,"data_compressed":6561,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[]},{"osd":2,"up_from":0,"seq":47244640282,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640282,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":126236,"kb_used_data":95044,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94245604,"statfs":{"total":96636764160,"available":96507498496,"internally_reserved":0,"allocated":97325056,"data_stored":96786499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640283,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77104,"kb_used_data":45912,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94294736,"statfs":{"total":96636764160,"available":96557809664,"internally_reserved":0,"allocated":47013888,"data_stored":47040249,"data_compressed":7429,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":7429,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":7429,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":7429,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6561,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":46141440,"data_stored":46137382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":96477184,"data_stored":96469027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8388608,"data_stored":8388608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":83898368,"data_stored":83886172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":25174016,"data_stored":25165878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8396800,"data_stored":8388643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1174,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T22:07:40.743 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T22:07:40.764 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:40.760+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:40.821 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:40.821 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:40.835 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T22:07:40.878 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:40.876+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:40.966 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:40.962+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:41.005 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:41.005 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:41.019 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T22:07:41.035 INFO:tasks.workunit.client.0.smithi157.stderr:+ DEV= 2024-09-21T22:07:41.035 INFO:tasks.workunit.client.0.smithi157.stderr:+ rbd snap create rbd/testrbdnbd21553@snap 2024-09-21T22:07:41.067 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:41.062+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:41.093 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T22:07:41.094 INFO:teuthology.orchestra.run.smithi039.stderr:dumped all 2024-09-21T22:07:41.113 INFO:teuthology.orchestra.run.smithi039.stdout:{"pg_ready":true,"pg_map":{"version":120,"stamp":"2024-09-21T22:07:40.813220+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":126419634,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":2341,"num_read_kb":142890,"num_write":676,"num_write_kb":256546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":35915939,"num_keys_recovered":8,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":586,"ondisk_log_size":586,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":471859200,"kb_used":408264,"kb_used_data":255188,"kb_used_omap":11,"kb_used_meta":152948,"kb_avail":471450936,"statfs":{"total":483183820800,"available":482765758464,"internally_reserved":0,"allocated":261312512,"data_stored":259765807,"data_compressed":13990,"data_compressed_allocated":73728,"data_compressed_original":1163264,"omap_allocated":11985,"internal_metadata":156619055},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":55,"apply_latency_ms":55,"commit_latency_ns":55000000,"apply_latency_ns":55000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":58720256,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":160,"num_read_kb":1130,"num_write":158,"num_write_kb":57376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001285"},"pg_stats":[{"pgid":"1.0","version":"15'117","reported_seq":328,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T22:07:39.420226+0000","last_change":"2024-09-21T22:07:14.092075+0000","last_active":"2024-09-21T22:07:39.420226+0000","last_peered":"2024-09-21T22:07:39.420226+0000","last_clean":"2024-09-21T22:07:39.420226+0000","last_became_active":"2024-09-21T22:07:14.091129+0000","last_became_peered":"2024-09-21T22:07:14.091129+0000","last_unstale":"2024-09-21T22:07:39.420226+0000","last_undegraded":"2024-09-21T22:07:39.420226+0000","last_fullsized":"2024-09-21T22:07:39.420226+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:00.990331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:00.990331+0000","last_clean_scrub_stamp":"2024-09-21T22:06:00.990331+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:36:36.088445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"68'21","reported_seq":54,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-09-21T22:07:35.032317+0000","last_change":"2024-09-21T22:07:29.615170+0000","last_active":"2024-09-21T22:07:35.032317+0000","last_peered":"2024-09-21T22:07:35.032317+0000","last_clean":"2024-09-21T22:07:35.032317+0000","last_became_active":"2024-09-21T22:07:29.253735+0000","last_became_peered":"2024-09-21T22:07:29.253735+0000","last_unstale":"2024-09-21T22:07:35.032317+0000","last_undegraded":"2024-09-21T22:07:35.032317+0000","last_fullsized":"2024-09-21T22:07:35.032317+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:42:12.326592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":74,"num_read_kb":8200,"num_write":20,"num_write_kb":16384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":16777216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"61'2","reported_seq":51,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T22:07:39.432477+0000","last_change":"2024-09-21T22:07:28.245341+0000","last_active":"2024-09-21T22:07:39.432477+0000","last_peered":"2024-09-21T22:07:39.432477+0000","last_clean":"2024-09-21T22:07:39.432477+0000","last_became_active":"2024-09-21T22:07:28.233066+0000","last_became_peered":"2024-09-21T22:07:28.233066+0000","last_unstale":"2024-09-21T22:07:39.432477+0000","last_undegraded":"2024-09-21T22:07:39.432477+0000","last_fullsized":"2024-09-21T22:07:39.432477+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:07:16.101346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:07:16.101346+0000","last_clean_scrub_stamp":"2024-09-21T22:07:16.101346+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:00:11.717223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000947342,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"72'75","reported_seq":536,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T22:07:39.429140+0000","last_change":"2024-09-21T22:06:07.121573+0000","last_active":"2024-09-21T22:07:39.429140+0000","last_peered":"2024-09-21T22:07:39.429140+0000","last_clean":"2024-09-21T22:07:39.429140+0000","last_became_active":"2024-09-21T22:06:05.108187+0000","last_became_peered":"2024-09-21T22:06:05.108187+0000","last_unstale":"2024-09-21T22:07:39.429140+0000","last_undegraded":"2024-09-21T22:07:39.429140+0000","last_fullsized":"2024-09-21T22:07:39.429140+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T05:05:08.108341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079478100000000005,"stat_sum":{"num_bytes":25165824,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":336,"num_read_kb":35864,"num_write":74,"num_write_kb":57344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":41,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T22:07:38.340477+0000","last_change":"2024-09-21T22:07:19.214979+0000","last_active":"2024-09-21T22:07:38.340477+0000","last_peered":"2024-09-21T22:07:38.340477+0000","last_clean":"2024-09-21T22:07:38.340477+0000","last_became_active":"2024-09-21T22:07:17.152522+0000","last_became_peered":"2024-09-21T22:07:17.152522+0000","last_unstale":"2024-09-21T22:07:38.340477+0000","last_undegraded":"2024-09-21T22:07:38.340477+0000","last_fullsized":"2024-09-21T22:07:38.340477+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:07:16.101346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:07:16.101346+0000","last_clean_scrub_stamp":"2024-09-21T22:07:16.101346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:51:24.258279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036883599999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"72'38","reported_seq":243,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T22:07:39.432356+0000","last_change":"2024-09-21T22:07:11.299996+0000","last_active":"2024-09-21T22:07:39.432356+0000","last_peered":"2024-09-21T22:07:39.432356+0000","last_clean":"2024-09-21T22:07:39.432356+0000","last_became_active":"2024-09-21T22:07:11.066642+0000","last_became_peered":"2024-09-21T22:07:11.066642+0000","last_unstale":"2024-09-21T22:07:39.432356+0000","last_undegraded":"2024-09-21T22:07:39.432356+0000","last_fullsized":"2024-09-21T22:07:39.432356+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:38:01.499033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010052629999999999,"stat_sum":{"num_bytes":16777216,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":68,"num_read_kb":16384,"num_write":38,"num_write_kb":32768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":8388608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":43,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T22:07:39.432355+0000","last_change":"2024-09-21T22:07:19.151406+0000","last_active":"2024-09-21T22:07:39.432355+0000","last_peered":"2024-09-21T22:07:39.432355+0000","last_clean":"2024-09-21T22:07:39.432355+0000","last_became_active":"2024-09-21T22:07:17.156631+0000","last_became_peered":"2024-09-21T22:07:17.156631+0000","last_unstale":"2024-09-21T22:07:39.432355+0000","last_undegraded":"2024-09-21T22:07:39.432355+0000","last_fullsized":"2024-09-21T22:07:39.432355+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:07:16.101346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:07:16.101346+0000","last_clean_scrub_stamp":"2024-09-21T22:07:16.101346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T08:49:26.489839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00075277899999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"69'46","reported_seq":269,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-09-21T22:07:35.846004+0000","last_change":"2024-09-21T22:07:28.523773+0000","last_active":"2024-09-21T22:07:35.846004+0000","last_peered":"2024-09-21T22:07:35.846004+0000","last_clean":"2024-09-21T22:07:35.846004+0000","last_became_active":"2024-09-21T22:07:28.234283+0000","last_became_peered":"2024-09-21T22:07:28.234283+0000","last_unstale":"2024-09-21T22:07:35.846004+0000","last_undegraded":"2024-09-21T22:07:35.846004+0000","last_fullsized":"2024-09-21T22:07:35.846004+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T23:01:03.589018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388643,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":661,"num_read_kb":604,"num_write":19,"num_write_kb":8198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":8388624,"num_keys_recovered":8,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"59'1","reported_seq":43,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T22:07:39.333441+0000","last_change":"2024-09-21T22:07:19.218756+0000","last_active":"2024-09-21T22:07:39.333441+0000","last_peered":"2024-09-21T22:07:39.333441+0000","last_clean":"2024-09-21T22:07:39.333441+0000","last_became_active":"2024-09-21T22:07:17.161301+0000","last_became_peered":"2024-09-21T22:07:17.161301+0000","last_unstale":"2024-09-21T22:07:39.333441+0000","last_undegraded":"2024-09-21T22:07:39.333441+0000","last_fullsized":"2024-09-21T22:07:39.333441+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:07:16.101346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:07:16.101346+0000","last_clean_scrub_stamp":"2024-09-21T22:07:16.101346+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T09:55:09.171875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001129983,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-09-21T22:07:35.032492+0000","last_change":"2024-09-21T22:07:28.242244+0000","last_active":"2024-09-21T22:07:35.032492+0000","last_peered":"2024-09-21T22:07:35.032492+0000","last_clean":"2024-09-21T22:07:35.032492+0000","last_became_active":"2024-09-21T22:07:28.240361+0000","last_became_peered":"2024-09-21T22:07:28.240361+0000","last_unstale":"2024-09-21T22:07:35.032492+0000","last_undegraded":"2024-09-21T22:07:35.032492+0000","last_fullsized":"2024-09-21T22:07:35.032492+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:07:16.101346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:07:16.101346+0000","last_clean_scrub_stamp":"2024-09-21T22:07:16.101346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:26:46.171549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010184180000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"70'45","reported_seq":282,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T22:07:39.432505+0000","last_change":"2024-09-21T22:06:44.875768+0000","last_active":"2024-09-21T22:07:39.432505+0000","last_peered":"2024-09-21T22:07:39.432505+0000","last_clean":"2024-09-21T22:07:39.432505+0000","last_became_active":"2024-09-21T22:06:44.875201+0000","last_became_peered":"2024-09-21T22:06:44.875201+0000","last_unstale":"2024-09-21T22:07:39.432505+0000","last_undegraded":"2024-09-21T22:07:39.432505+0000","last_fullsized":"2024-09-21T22:07:39.432505+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T08:49:08.148866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00090834100000000001,"stat_sum":{"num_bytes":16777216,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":117,"num_read_kb":24610,"num_write":43,"num_write_kb":32771,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":43,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T22:07:39.432475+0000","last_change":"2024-09-21T22:07:19.151659+0000","last_active":"2024-09-21T22:07:39.432475+0000","last_peered":"2024-09-21T22:07:39.432475+0000","last_clean":"2024-09-21T22:07:39.432475+0000","last_became_active":"2024-09-21T22:07:17.155913+0000","last_became_peered":"2024-09-21T22:07:17.155913+0000","last_unstale":"2024-09-21T22:07:39.432475+0000","last_undegraded":"2024-09-21T22:07:39.432475+0000","last_fullsized":"2024-09-21T22:07:39.432475+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:07:16.101346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:07:16.101346+0000","last_clean_scrub_stamp":"2024-09-21T22:07:16.101346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T09:09:08.662264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00068604600000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"72'56","reported_seq":260,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T22:07:39.418674+0000","last_change":"2024-09-21T22:06:44.875038+0000","last_active":"2024-09-21T22:07:39.418674+0000","last_peered":"2024-09-21T22:07:39.418674+0000","last_clean":"2024-09-21T22:07:39.418674+0000","last_became_active":"2024-09-21T22:06:44.874697+0000","last_became_peered":"2024-09-21T22:06:44.874697+0000","last_unstale":"2024-09-21T22:07:39.418674+0000","last_undegraded":"2024-09-21T22:07:39.418674+0000","last_fullsized":"2024-09-21T22:07:39.418674+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T07:09:15.569760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091949900000000005,"stat_sum":{"num_bytes":12582950,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":169,"num_read_kb":16466,"num_write":72,"num_write_kb":28705,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":43,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T22:07:39.432456+0000","last_change":"2024-09-21T22:07:19.151518+0000","last_active":"2024-09-21T22:07:39.432456+0000","last_peered":"2024-09-21T22:07:39.432456+0000","last_clean":"2024-09-21T22:07:39.432456+0000","last_became_active":"2024-09-21T22:07:17.156489+0000","last_became_peered":"2024-09-21T22:07:17.156489+0000","last_unstale":"2024-09-21T22:07:39.432456+0000","last_undegraded":"2024-09-21T22:07:39.432456+0000","last_fullsized":"2024-09-21T22:07:39.432456+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:07:16.101346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:07:16.101346+0000","last_clean_scrub_stamp":"2024-09-21T22:07:16.101346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:04:16.326684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00066928400000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"70'24","reported_seq":194,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T22:07:39.432496+0000","last_change":"2024-09-21T22:06:07.133757+0000","last_active":"2024-09-21T22:07:39.432496+0000","last_peered":"2024-09-21T22:07:39.432496+0000","last_clean":"2024-09-21T22:07:39.432496+0000","last_became_active":"2024-09-21T22:06:05.109065+0000","last_became_peered":"2024-09-21T22:06:05.109065+0000","last_unstale":"2024-09-21T22:07:39.432496+0000","last_undegraded":"2024-09-21T22:07:39.432496+0000","last_fullsized":"2024-09-21T22:07:39.432496+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:31:12.654893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00088865700000000005,"stat_sum":{"num_bytes":20971520,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":46,"num_read_kb":12316,"num_write":24,"num_write_kb":28674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":41,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-09-21T22:07:38.340521+0000","last_change":"2024-09-21T22:07:19.215026+0000","last_active":"2024-09-21T22:07:38.340521+0000","last_peered":"2024-09-21T22:07:38.340521+0000","last_clean":"2024-09-21T22:07:38.340521+0000","last_became_active":"2024-09-21T22:07:17.159836+0000","last_became_peered":"2024-09-21T22:07:17.159836+0000","last_unstale":"2024-09-21T22:07:38.340521+0000","last_undegraded":"2024-09-21T22:07:38.340521+0000","last_fullsized":"2024-09-21T22:07:38.340521+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:07:16.101346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:07:16.101346+0000","last_clean_scrub_stamp":"2024-09-21T22:07:16.101346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:16:31.665427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032723700000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"73'161","reported_seq":596,"reported_epoch":73,"state":"active+clean","last_fresh":"2024-09-21T22:07:40.139811+0000","last_change":"2024-09-21T22:06:07.122328+0000","last_active":"2024-09-21T22:07:40.139811+0000","last_peered":"2024-09-21T22:07:40.139811+0000","last_clean":"2024-09-21T22:07:40.139811+0000","last_became_active":"2024-09-21T22:06:05.220996+0000","last_became_peered":"2024-09-21T22:06:05.220996+0000","last_unstale":"2024-09-21T22:07:40.139811+0000","last_undegraded":"2024-09-21T22:07:40.139811+0000","last_fullsized":"2024-09-21T22:07:40.139811+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T23:45:49.178013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001473755,"stat_sum":{"num_bytes":25165878,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":754,"num_read_kb":28346,"num_write":225,"num_write_kb":49284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":5},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":125829247,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":2225,"num_read_kb":142790,"num_write":515,"num_write_kb":254128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":33554448,"num_keys_recovered":8,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":268476416,"data_stored":268435710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1174,"internal_metadata":0},"log_size":466,"ondisk_log_size":466,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":2361472,"data_compressed":28848,"data_compressed_allocated":147456,"data_compressed_original":2342912,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":65,"seq":279172874243,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37288,"kb_used_data":8980,"kb_used_omap":5,"kb_used_meta":28282,"kb_avail":94334552,"statfs":{"total":96636764160,"available":96598581248,"internally_reserved":0,"allocated":9195520,"data_stored":8686299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5607,"internal_metadata":28961305},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640280,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56588,"kb_used_data":25396,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94315252,"statfs":{"total":96636764160,"available":96578818048,"internally_reserved":0,"allocated":26005504,"data_stored":25478377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640283,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":111048,"kb_used_data":79856,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94260792,"statfs":{"total":96636764160,"available":96523051008,"internally_reserved":0,"allocated":81772544,"data_stored":81774383,"data_compressed":6561,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[]},{"osd":2,"up_from":0,"seq":47244640282,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640282,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":126236,"kb_used_data":95044,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94245604,"statfs":{"total":96636764160,"available":96507498496,"internally_reserved":0,"allocated":97325056,"data_stored":96786499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640283,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77104,"kb_used_data":45912,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94294736,"statfs":{"total":96636764160,"available":96557809664,"internally_reserved":0,"allocated":47013888,"data_stored":47040249,"data_compressed":7429,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":7429,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":7429,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":7429,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6561,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":46141440,"data_stored":46137382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":96477184,"data_stored":96469027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8388608,"data_stored":8388608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":83898368,"data_stored":83886172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":25174016,"data_stored":25165878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8396800,"data_stored":8388643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1174,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T22:07:41.114 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-09-21T22:07:41.114 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T22:07:41.114 INFO:tasks.thrashosds.thrasher:inject_pause on osd.0 2024-09-21T22:07:41.115 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-21T22:07:41.115 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-21T22:07:41.115 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok config set filestore_inject_stall 3 2024-09-21T22:07:41.152 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:41.152 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:41.167 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T22:07:41.167 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:41.164+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:41.269 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:41.266+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:41.282 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:07:41.283 INFO:teuthology.orchestra.run.smithi039.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-21T22:07:41.283 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:07:41.337 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:41.337 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:41.346 INFO:teuthology.orchestra.run.smithi039.stderr:nodeep-scrub is set 2024-09-21T22:07:41.352 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T22:07:41.370 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:41.366+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:41.372 INFO:tasks.workunit.client.0.smithi157.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-09-21T22:07:41.388 INFO:tasks.workunit.client.0.smithi157.stderr:++ _sudo rbd device --device-type nbd map rbd/testrbdnbd21553@snap 2024-09-21T22:07:41.388 INFO:tasks.workunit.client.0.smithi157.stderr:++ local cmd 2024-09-21T22:07:41.389 INFO:tasks.workunit.client.0.smithi157.stderr:+++ id -u 2024-09-21T22:07:41.390 INFO:tasks.workunit.client.0.smithi157.stderr:++ '[' 1000 -eq 0 ']' 2024-09-21T22:07:41.391 INFO:tasks.workunit.client.0.smithi157.stderr:+++ which rbd 2024-09-21T22:07:41.393 INFO:tasks.workunit.client.0.smithi157.stderr:++ cmd=/usr/bin/rbd 2024-09-21T22:07:41.393 INFO:tasks.workunit.client.0.smithi157.stderr:++ shift 2024-09-21T22:07:41.393 INFO:tasks.workunit.client.0.smithi157.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map rbd/testrbdnbd21553@snap 2024-09-21T22:07:41.471 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:41.468+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:41.481 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:41.481 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:41.497 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T22:07:41.536 INFO:tasks.workunit.client.0.smithi157.stderr:+ DEV=/dev/nbd0 2024-09-21T22:07:41.537 INFO:tasks.workunit.client.0.smithi157.stderr:+ get_pid rbd 2024-09-21T22:07:41.537 INFO:tasks.workunit.client.0.smithi157.stderr:+ local pool=rbd 2024-09-21T22:07:41.537 INFO:tasks.workunit.client.0.smithi157.stderr:+ local ns= 2024-09-21T22:07:41.537 INFO:tasks.workunit.client.0.smithi157.stderr:++ rbd device --device-type nbd --format xml list 2024-09-21T22:07:41.537 INFO:tasks.workunit.client.0.smithi157.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd21553'\''][device='\''/dev/nbd0'\'']/id' 2024-09-21T22:07:41.572 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:41.570+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:41.603 INFO:tasks.workunit.client.0.smithi157.stderr:+ PID=22784 2024-09-21T22:07:41.604 INFO:tasks.workunit.client.0.smithi157.stderr:+ test -n 22784 2024-09-21T22:07:41.604 INFO:tasks.workunit.client.0.smithi157.stderr:+ ps -p 22784 -C rbd-nbd 2024-09-21T22:07:41.610 INFO:tasks.workunit.client.0.smithi157.stdout: PID TTY TIME CMD 2024-09-21T22:07:41.610 INFO:tasks.workunit.client.0.smithi157.stdout: 22784 ? 00:00:00 rbd-nbd 2024-09-21T22:07:41.610 INFO:tasks.workunit.client.0.smithi157.stderr:+ unmap_device testrbdnbd21553@snap 22784 2024-09-21T22:07:41.611 INFO:tasks.workunit.client.0.smithi157.stderr:+ local args=testrbdnbd21553@snap 2024-09-21T22:07:41.611 INFO:tasks.workunit.client.0.smithi157.stderr:+ local pid=22784 2024-09-21T22:07:41.611 INFO:tasks.workunit.client.0.smithi157.stderr:+ _sudo rbd device --device-type nbd unmap testrbdnbd21553@snap 2024-09-21T22:07:41.611 INFO:tasks.workunit.client.0.smithi157.stderr:+ local cmd 2024-09-21T22:07:41.611 INFO:tasks.workunit.client.0.smithi157.stderr:++ id -u 2024-09-21T22:07:41.612 INFO:tasks.workunit.client.0.smithi157.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T22:07:41.614 INFO:tasks.workunit.client.0.smithi157.stderr:++ which rbd 2024-09-21T22:07:41.614 INFO:tasks.workunit.client.0.smithi157.stderr:+ cmd=/usr/bin/rbd 2024-09-21T22:07:41.615 INFO:tasks.workunit.client.0.smithi157.stderr:+ shift 2024-09-21T22:07:41.615 INFO:tasks.workunit.client.0.smithi157.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap testrbdnbd21553@snap 2024-09-21T22:07:41.669 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:41.669 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:41.672 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:41.670+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:41.683 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T22:07:41.700 INFO:tasks.workunit.client.0.smithi157.stderr:+ rbd device --device-type nbd list 2024-09-21T22:07:41.701 INFO:tasks.workunit.client.0.smithi157.stderr:+ expect_false grep '^22784\b' 2024-09-21T22:07:41.701 INFO:tasks.workunit.client.0.smithi157.stderr:+ grep '^22784\b' 2024-09-21T22:07:41.752 INFO:tasks.workunit.client.0.smithi157.stderr:+ return 0 2024-09-21T22:07:41.753 INFO:tasks.workunit.client.0.smithi157.stderr:+ ps -C rbd-nbd 2024-09-21T22:07:41.753 INFO:tasks.workunit.client.0.smithi157.stderr:+ expect_false grep '^ *22784\b' 2024-09-21T22:07:41.753 INFO:tasks.workunit.client.0.smithi157.stderr:+ grep '^ *22784\b' 2024-09-21T22:07:41.759 INFO:tasks.workunit.client.0.smithi157.stderr:+ return 0 2024-09-21T22:07:41.759 INFO:tasks.workunit.client.0.smithi157.stderr:+ sleep 0.5 2024-09-21T22:07:41.774 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:41.770+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:41.803 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:41.803 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:41.817 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T22:07:41.874 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:41.870+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:41.953 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:41.953 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:41.967 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T22:07:42.008 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:42.006+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:42.075 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:42.076+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:42.137 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:42.137 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:42.150 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T22:07:42.176 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:42.176+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:42.180 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T22:07:42.260 INFO:tasks.workunit.client.0.smithi157.stderr:+ DEV= 2024-09-21T22:07:42.261 INFO:tasks.workunit.client.0.smithi157.stderr:++ rbd snap ls rbd/testrbdnbd21553 2024-09-21T22:07:42.261 INFO:tasks.workunit.client.0.smithi157.stderr:++ awk '$2 == "snap" {print $1}' 2024-09-21T22:07:42.270 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:42.271 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:42.277 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:42.276+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:42.286 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T22:07:42.324 INFO:tasks.workunit.client.0.smithi157.stderr:+ SNAPID=3 2024-09-21T22:07:42.325 INFO:tasks.workunit.client.0.smithi157.stderr:++ _sudo rbd device --device-type nbd map --snap-id 3 rbd/testrbdnbd21553 2024-09-21T22:07:42.325 INFO:tasks.workunit.client.0.smithi157.stderr:++ local cmd 2024-09-21T22:07:42.325 INFO:tasks.workunit.client.0.smithi157.stderr:+++ id -u 2024-09-21T22:07:42.326 INFO:tasks.workunit.client.0.smithi157.stderr:++ '[' 1000 -eq 0 ']' 2024-09-21T22:07:42.326 INFO:tasks.workunit.client.0.smithi157.stderr:+++ which rbd 2024-09-21T22:07:42.328 INFO:tasks.workunit.client.0.smithi157.stderr:++ cmd=/usr/bin/rbd 2024-09-21T22:07:42.328 INFO:tasks.workunit.client.0.smithi157.stderr:++ shift 2024-09-21T22:07:42.328 INFO:tasks.workunit.client.0.smithi157.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map --snap-id 3 rbd/testrbdnbd21553 2024-09-21T22:07:42.350 INFO:teuthology.orchestra.run.smithi039.stdout:osd.0: {} 2024-09-21T22:07:42.350 INFO:teuthology.orchestra.run.smithi039.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T22:07:42.367 INFO:teuthology.orchestra.run.smithi039.stdout:osd.1: {} 2024-09-21T22:07:42.368 INFO:teuthology.orchestra.run.smithi039.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T22:07:42.378 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:42.374+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:42.385 INFO:teuthology.orchestra.run.smithi039.stdout:osd.2: {} 2024-09-21T22:07:42.386 INFO:teuthology.orchestra.run.smithi039.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T22:07:42.407 INFO:teuthology.orchestra.run.smithi039.stdout:osd.3: {} 2024-09-21T22:07:42.407 INFO:teuthology.orchestra.run.smithi039.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T22:07:42.421 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:07:42.422 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:07:42.422 INFO:teuthology.orchestra.run.smithi039.stdout: "num_ops": 0 2024-09-21T22:07:42.422 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:07:42.427 INFO:teuthology.orchestra.run.smithi039.stdout:osd.4: {} 2024-09-21T22:07:42.428 INFO:teuthology.orchestra.run.smithi039.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T22:07:42.436 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T22:07:42.450 INFO:teuthology.orchestra.run.smithi039.stdout:osd.5: {} 2024-09-21T22:07:42.450 INFO:teuthology.orchestra.run.smithi039.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T22:07:42.516 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:42.514+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:42.565 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:07:42.565 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:07:42.565 INFO:teuthology.orchestra.run.smithi039.stdout: "complaint_time": 30, 2024-09-21T22:07:42.565 INFO:teuthology.orchestra.run.smithi039.stdout: "num_blocked_ops": 0 2024-09-21T22:07:42.565 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:07:42.579 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T22:07:42.580 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:42.574+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:42.679 INFO:tasks.workunit.client.0.smithi157.stderr:+ DEV=/dev/nbd0 2024-09-21T22:07:42.679 INFO:tasks.workunit.client.0.smithi157.stderr:+ get_pid rbd 2024-09-21T22:07:42.679 INFO:tasks.workunit.client.0.smithi157.stderr:+ local pool=rbd 2024-09-21T22:07:42.679 INFO:tasks.workunit.client.0.smithi157.stderr:+ local ns= 2024-09-21T22:07:42.680 INFO:tasks.workunit.client.0.smithi157.stderr:++ rbd device --device-type nbd --format xml list 2024-09-21T22:07:42.680 INFO:tasks.workunit.client.0.smithi157.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd21553'\''][device='\''/dev/nbd0'\'']/id' 2024-09-21T22:07:42.680 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:42.678+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:42.751 INFO:tasks.workunit.client.0.smithi157.stderr:+ PID=22883 2024-09-21T22:07:42.752 INFO:tasks.workunit.client.0.smithi157.stderr:+ test -n 22883 2024-09-21T22:07:42.752 INFO:tasks.workunit.client.0.smithi157.stderr:+ ps -p 22883 -C rbd-nbd 2024-09-21T22:07:42.756 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:07:42.756 INFO:teuthology.orchestra.run.smithi039.stdout: "size": 20, 2024-09-21T22:07:42.756 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 600, 2024-09-21T22:07:42.756 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [ 2024-09-21T22:07:42.757 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.757 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.3 55/54 [PushOp(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head, version: 52'2, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head@52'2, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T22:07:42.757 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.084379+0000", 2024-09-21T22:07:42.757 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 30.668004639999999, 2024-09-21T22:07:42.757 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.051020674000000002, 2024-09-21T22:07:42.757 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:42.757 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:42.757 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:42.757 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:42.757 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.757 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:42.757 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.084379+0000", 2024-09-21T22:07:42.757 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:42.758 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.758 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.758 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:42.758 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.084379+0000", 2024-09-21T22:07:42.758 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:42.758 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.758 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.758 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:42.758 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.084381+0000", 2024-09-21T22:07:42.758 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.73e-06 2024-09-21T22:07:42.758 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.758 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.758 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:42.758 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.085494+0000", 2024-09-21T22:07:42.759 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011131559999999999 2024-09-21T22:07:42.759 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.759 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.759 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:42.759 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.085542+0000", 2024-09-21T22:07:42.759 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8334999999999997e-05 2024-09-21T22:07:42.759 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.759 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.759 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:42.759 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.085591+0000", 2024-09-21T22:07:42.759 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.9165000000000001e-05 2024-09-21T22:07:42.760 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.760 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.760 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:42.760 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.085777+0000", 2024-09-21T22:07:42.760 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000186152 2024-09-21T22:07:42.760 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.760 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.760 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:42.760 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.085796+0000", 2024-09-21T22:07:42.760 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8612999999999999e-05 2024-09-21T22:07:42.760 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.760 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.760 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:42.760 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.135399+0000", 2024-09-21T22:07:42.761 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.049603522999999997 2024-09-21T22:07:42.761 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:42.761 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:42.761 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:42.761 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.761 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.761 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.3 55/54 [PushOp(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head, version: 52'2, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head@52'2, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:07:42.761 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.139554+0000", 2024-09-21T22:07:42.761 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 30.612829537, 2024-09-21T22:07:42.761 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.056215389999999997, 2024-09-21T22:07:42.761 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:42.761 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:42.761 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:42.761 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:42.762 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.762 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:42.762 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.139554+0000", 2024-09-21T22:07:42.762 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:42.762 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.762 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.762 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:42.762 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.139554+0000", 2024-09-21T22:07:42.762 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:42.762 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.762 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.762 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:42.762 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.139556+0000", 2024-09-21T22:07:42.762 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7999999999999999e-06 2024-09-21T22:07:42.762 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.763 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.763 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:42.763 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.140671+0000", 2024-09-21T22:07:42.763 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001115612 2024-09-21T22:07:42.763 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.763 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.763 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:42.763 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.140720+0000", 2024-09-21T22:07:42.763 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8631999999999998e-05 2024-09-21T22:07:42.763 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.763 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.763 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:42.763 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.140752+0000", 2024-09-21T22:07:42.763 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.2199000000000001e-05 2024-09-21T22:07:42.764 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.764 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.764 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:42.764 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.140900+0000", 2024-09-21T22:07:42.764 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00014786000000000001 2024-09-21T22:07:42.764 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.764 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.764 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:42.764 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.140917+0000", 2024-09-21T22:07:42.764 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6753999999999999e-05 2024-09-21T22:07:42.764 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.764 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.764 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:42.764 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.195769+0000", 2024-09-21T22:07:42.765 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.054852533000000002 2024-09-21T22:07:42.765 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:42.765 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:42.765 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:42.765 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.765 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.765 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.3 55/54 [PushOp(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head, version: 52'2, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head@52'2, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:07:42.765 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.199187+0000", 2024-09-21T22:07:42.765 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 30.553195992999999, 2024-09-21T22:07:42.765 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.051543975999999998, 2024-09-21T22:07:42.765 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:42.765 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:42.765 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:42.765 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:42.766 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.766 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:42.766 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.199187+0000", 2024-09-21T22:07:42.766 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:42.766 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.766 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.766 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:42.766 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.199187+0000", 2024-09-21T22:07:42.766 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:42.766 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.766 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.766 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:42.766 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.199189+0000", 2024-09-21T22:07:42.766 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7969999999999999e-06 2024-09-21T22:07:42.766 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.767 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.767 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:42.767 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.200355+0000", 2024-09-21T22:07:42.767 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011660139999999999 2024-09-21T22:07:42.767 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.767 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.767 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:42.767 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.200405+0000", 2024-09-21T22:07:42.767 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.0124000000000003e-05 2024-09-21T22:07:42.767 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.767 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.767 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:42.767 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.200457+0000", 2024-09-21T22:07:42.767 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.1177999999999997e-05 2024-09-21T22:07:42.768 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.768 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.768 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:42.768 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.200635+0000", 2024-09-21T22:07:42.768 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00017841299999999999 2024-09-21T22:07:42.768 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.768 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.768 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:42.768 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.200653+0000", 2024-09-21T22:07:42.768 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7944999999999998e-05 2024-09-21T22:07:42.768 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.768 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.769 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:42.769 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.250731+0000", 2024-09-21T22:07:42.769 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.050078505000000002 2024-09-21T22:07:42.769 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:42.769 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:42.769 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:42.769 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.769 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.769 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.3 55/54 [PushOp(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head, version: 52'3, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head@52'3, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:07:42.769 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.343754+0000", 2024-09-21T22:07:42.770 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 30.408629282, 2024-09-21T22:07:42.770 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.063560607000000005, 2024-09-21T22:07:42.770 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:42.770 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:42.770 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:42.771 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:42.771 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.771 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:42.771 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.343754+0000", 2024-09-21T22:07:42.771 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:42.772 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.772 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.772 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:42.772 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.343754+0000", 2024-09-21T22:07:42.772 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:42.772 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.773 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.773 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:42.773 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.343756+0000", 2024-09-21T22:07:42.773 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7069999999999999e-06 2024-09-21T22:07:42.773 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.773 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.774 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:42.774 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.344864+0000", 2024-09-21T22:07:42.774 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011085159999999999 2024-09-21T22:07:42.774 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.774 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.774 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:42.775 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.344914+0000", 2024-09-21T22:07:42.775 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.9462000000000002e-05 2024-09-21T22:07:42.775 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.775 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.775 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:42.775 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.344961+0000", 2024-09-21T22:07:42.775 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.7633999999999998e-05 2024-09-21T22:07:42.776 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.776 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.776 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:42.776 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.345171+0000", 2024-09-21T22:07:42.776 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000209652 2024-09-21T22:07:42.776 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.776 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.777 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:42.777 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.345193+0000", 2024-09-21T22:07:42.777 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1985999999999999e-05 2024-09-21T22:07:42.777 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.777 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.777 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:42.777 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.407315+0000", 2024-09-21T22:07:42.777 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.06212165 2024-09-21T22:07:42.778 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:42.778 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:42.778 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:42.778 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.778 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.778 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.3 55/54 [PushOp(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head, version: 52'3, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head@52'3, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:07:42.778 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.410551+0000", 2024-09-21T22:07:42.779 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 30.341832150999998, 2024-09-21T22:07:42.779 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.054517522999999998, 2024-09-21T22:07:42.779 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:42.779 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:42.779 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:42.779 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:42.780 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.780 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:42.780 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.410551+0000", 2024-09-21T22:07:42.780 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:42.780 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.780 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.780 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:42.780 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.410551+0000", 2024-09-21T22:07:42.780 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:42.780 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.780 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.781 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:42.781 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.410553+0000", 2024-09-21T22:07:42.781 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5200000000000001e-06 2024-09-21T22:07:42.781 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.781 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.781 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:42.781 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.411747+0000", 2024-09-21T22:07:42.781 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011946089999999999 2024-09-21T22:07:42.781 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.781 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.781 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:42.781 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.411796+0000", 2024-09-21T22:07:42.781 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8717999999999997e-05 2024-09-21T22:07:42.781 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.782 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.782 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:42.782 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.411844+0000", 2024-09-21T22:07:42.782 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8291999999999998e-05 2024-09-21T22:07:42.782 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.782 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.782 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:42.782 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.412066+0000", 2024-09-21T22:07:42.782 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00022115899999999999 2024-09-21T22:07:42.782 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.782 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.783 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:42.783 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.412081+0000", 2024-09-21T22:07:42.783 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5654e-05 2024-09-21T22:07:42.783 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.783 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.783 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:42.783 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.465069+0000", 2024-09-21T22:07:42.783 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.052987570999999997 2024-09-21T22:07:42.783 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:42.784 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:42.784 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:42.784 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.784 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.784 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.3 55/54 [PushOp(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head, version: 52'3, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head@52'3, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 4194304, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:07:42.784 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.468339+0000", 2024-09-21T22:07:42.785 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 30.284044486999999, 2024-09-21T22:07:42.785 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.064010024999999998, 2024-09-21T22:07:42.785 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:42.785 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:42.785 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:42.785 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:42.785 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.785 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:42.785 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.468339+0000", 2024-09-21T22:07:42.785 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:42.785 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.786 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.786 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:42.786 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.468339+0000", 2024-09-21T22:07:42.786 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:42.786 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.786 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.786 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:42.786 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.468340+0000", 2024-09-21T22:07:42.786 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4330000000000001e-06 2024-09-21T22:07:42.786 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.786 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.787 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:42.787 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.469484+0000", 2024-09-21T22:07:42.787 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001143535 2024-09-21T22:07:42.787 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.787 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.787 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:42.787 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.469533+0000", 2024-09-21T22:07:42.787 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8980999999999997e-05 2024-09-21T22:07:42.787 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.788 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.788 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:42.788 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.469581+0000", 2024-09-21T22:07:42.788 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8551999999999999e-05 2024-09-21T22:07:42.788 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.788 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.788 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:42.788 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.469804+0000", 2024-09-21T22:07:42.788 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00022287699999999999 2024-09-21T22:07:42.788 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.788 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.788 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:42.789 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.469829+0000", 2024-09-21T22:07:42.789 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4766e-05 2024-09-21T22:07:42.789 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.789 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.789 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:42.789 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.532349+0000", 2024-09-21T22:07:42.789 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.062519880999999999 2024-09-21T22:07:42.789 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:42.789 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:42.789 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:42.789 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.789 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.789 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4840.0:668 2.3 2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:07:42.790 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.409114+0000", 2024-09-21T22:07:42.790 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 18.343269281000001, 2024-09-21T22:07:42.790 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.048703758999999999, 2024-09-21T22:07:42.790 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:42.790 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:42.790 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:42.790 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-21T22:07:42.790 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4840", 2024-09-21T22:07:42.790 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:42.790 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 668 2024-09-21T22:07:42.790 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.790 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:42.791 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.791 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:42.791 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.409114+0000", 2024-09-21T22:07:42.791 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:42.791 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.791 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.791 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:42.791 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.409114+0000", 2024-09-21T22:07:42.792 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:42.792 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.792 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.792 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:42.792 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.409115+0000", 2024-09-21T22:07:42.792 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.114e-06 2024-09-21T22:07:42.793 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.793 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.793 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:42.793 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.410308+0000", 2024-09-21T22:07:42.793 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011930949999999999 2024-09-21T22:07:42.793 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.794 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.794 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:42.794 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.410325+0000", 2024-09-21T22:07:42.794 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7088e-05 2024-09-21T22:07:42.794 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.795 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.795 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:42.795 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.410334+0000", 2024-09-21T22:07:42.795 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.2549999999999997e-06 2024-09-21T22:07:42.795 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.795 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.796 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:42.796 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.410393+0000", 2024-09-21T22:07:42.796 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.9327999999999998e-05 2024-09-21T22:07:42.796 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.796 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.796 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:42.797 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.410482+0000", 2024-09-21T22:07:42.797 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.9019999999999998e-05 2024-09-21T22:07:42.797 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.797 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.797 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-21T22:07:42.798 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.410617+0000", 2024-09-21T22:07:42.798 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000135232 2024-09-21T22:07:42.798 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.798 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.798 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:42.798 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.457051+0000", 2024-09-21T22:07:42.799 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.04643415 2024-09-21T22:07:42.799 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.799 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.799 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-21T22:07:42.800 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.457675+0000", 2024-09-21T22:07:42.800 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00062332700000000002 2024-09-21T22:07:42.800 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.800 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.800 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:42.800 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.457735+0000", 2024-09-21T22:07:42.801 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.029e-05 2024-09-21T22:07:42.801 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.801 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.801 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:42.801 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.457818+0000", 2024-09-21T22:07:42.801 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.2860000000000003e-05 2024-09-21T22:07:42.801 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:42.802 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:42.802 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:42.802 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.802 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.802 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4840.0:670 2.3 2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:07:42.802 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.466980+0000", 2024-09-21T22:07:42.803 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 18.285403212999999, 2024-09-21T22:07:42.803 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.050913930000000003, 2024-09-21T22:07:42.803 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:42.803 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:42.803 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:42.803 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-21T22:07:42.804 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4840", 2024-09-21T22:07:42.804 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:42.804 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 670 2024-09-21T22:07:42.804 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.804 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:42.804 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.804 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:42.805 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.466980+0000", 2024-09-21T22:07:42.805 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:42.805 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.805 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.805 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:42.805 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.466980+0000", 2024-09-21T22:07:42.805 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:42.805 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.806 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.806 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:42.806 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.466981+0000", 2024-09-21T22:07:42.806 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0839999999999999e-06 2024-09-21T22:07:42.806 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.806 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.806 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:42.807 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.467815+0000", 2024-09-21T22:07:42.807 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00083330399999999999 2024-09-21T22:07:42.807 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.807 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.807 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:42.807 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.467832+0000", 2024-09-21T22:07:42.807 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7601e-05 2024-09-21T22:07:42.807 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.807 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.808 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:42.808 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.467840+0000", 2024-09-21T22:07:42.808 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.2220000000000002e-06 2024-09-21T22:07:42.808 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.808 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.808 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:42.808 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.467974+0000", 2024-09-21T22:07:42.808 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000133411 2024-09-21T22:07:42.808 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.808 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.808 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:42.808 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.468223+0000", 2024-09-21T22:07:42.808 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00024906400000000001 2024-09-21T22:07:42.808 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.808 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.809 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-21T22:07:42.809 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.468616+0000", 2024-09-21T22:07:42.809 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000393433 2024-09-21T22:07:42.809 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.809 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.809 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:42.809 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.517016+0000", 2024-09-21T22:07:42.809 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.048399708 2024-09-21T22:07:42.809 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.809 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.809 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-21T22:07:42.809 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.517693+0000", 2024-09-21T22:07:42.809 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00067681000000000002 2024-09-21T22:07:42.809 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.810 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.810 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:42.810 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.517789+0000", 2024-09-21T22:07:42.810 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.5778000000000004e-05 2024-09-21T22:07:42.810 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.810 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.810 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:42.811 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.517894+0000", 2024-09-21T22:07:42.811 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000105515 2024-09-21T22:07:42.811 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:42.811 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:42.811 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:42.811 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.811 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.811 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4840.0:672 2.3 2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:07:42.811 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.527127+0000", 2024-09-21T22:07:42.812 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 18.225256751, 2024-09-21T22:07:42.812 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.047345602000000001, 2024-09-21T22:07:42.812 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:42.812 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:42.812 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:42.812 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-21T22:07:42.812 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4840", 2024-09-21T22:07:42.812 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:42.812 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 672 2024-09-21T22:07:42.812 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.812 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:42.813 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.813 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:42.813 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.527127+0000", 2024-09-21T22:07:42.813 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:42.813 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.813 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.813 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:42.813 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.527127+0000", 2024-09-21T22:07:42.813 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:42.813 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.813 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.813 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:42.813 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.527128+0000", 2024-09-21T22:07:42.814 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2810000000000001e-06 2024-09-21T22:07:42.814 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.814 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.814 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:42.814 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.527962+0000", 2024-09-21T22:07:42.814 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00083445199999999996 2024-09-21T22:07:42.814 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.814 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.814 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:42.814 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.527981+0000", 2024-09-21T22:07:42.815 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9009e-05 2024-09-21T22:07:42.815 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.815 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.815 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:42.815 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.527990+0000", 2024-09-21T22:07:42.815 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.8619999999999992e-06 2024-09-21T22:07:42.815 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.815 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.815 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:42.815 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.528061+0000", 2024-09-21T22:07:42.816 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.0226000000000002e-05 2024-09-21T22:07:42.816 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.816 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.816 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:42.816 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.528156+0000", 2024-09-21T22:07:42.816 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.5724999999999996e-05 2024-09-21T22:07:42.816 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.816 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.816 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-21T22:07:42.816 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.528303+0000", 2024-09-21T22:07:42.817 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000146374 2024-09-21T22:07:42.817 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.817 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.817 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:42.817 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.573772+0000", 2024-09-21T22:07:42.817 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.045469745999999998 2024-09-21T22:07:42.817 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.817 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.817 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-21T22:07:42.817 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.574299+0000", 2024-09-21T22:07:42.817 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00052649200000000004 2024-09-21T22:07:42.817 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.818 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.818 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:42.818 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.574366+0000", 2024-09-21T22:07:42.818 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.6872000000000006e-05 2024-09-21T22:07:42.818 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.818 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.818 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:42.818 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.574472+0000", 2024-09-21T22:07:42.818 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000106563 2024-09-21T22:07:42.819 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:42.819 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:42.819 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:42.819 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.819 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.819 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4840.0:686 2.3 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:07:42.819 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.943561+0000", 2024-09-21T22:07:42.819 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 17.808822474999999, 2024-09-21T22:07:42.819 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.049021493999999999, 2024-09-21T22:07:42.819 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:42.819 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:42.819 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:42.819 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-21T22:07:42.820 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4840", 2024-09-21T22:07:42.820 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:42.820 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 686 2024-09-21T22:07:42.820 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.820 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:42.820 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.820 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:42.820 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.943561+0000", 2024-09-21T22:07:42.820 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:42.820 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.820 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.820 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:42.820 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.943561+0000", 2024-09-21T22:07:42.821 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:42.821 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.821 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.821 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:42.821 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.943563+0000", 2024-09-21T22:07:42.821 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9420000000000002e-06 2024-09-21T22:07:42.821 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.821 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.821 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:42.821 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.944546+0000", 2024-09-21T22:07:42.821 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00098357099999999992 2024-09-21T22:07:42.821 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.821 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.821 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:42.822 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.944565+0000", 2024-09-21T22:07:42.822 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8621e-05 2024-09-21T22:07:42.822 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.822 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.822 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:42.822 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.944575+0000", 2024-09-21T22:07:42.822 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.4639999999999995e-06 2024-09-21T22:07:42.823 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.823 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.823 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:42.823 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.944647+0000", 2024-09-21T22:07:42.823 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.2786000000000004e-05 2024-09-21T22:07:42.823 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.823 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.823 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:42.824 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.944746+0000", 2024-09-21T22:07:42.824 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.8196999999999995e-05 2024-09-21T22:07:42.824 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.824 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.824 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-21T22:07:42.824 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.944890+0000", 2024-09-21T22:07:42.824 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000144804 2024-09-21T22:07:42.825 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.825 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.825 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:42.825 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.992404+0000", 2024-09-21T22:07:42.825 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.047513211999999999 2024-09-21T22:07:42.825 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.825 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.826 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-21T22:07:42.826 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.992480+0000", 2024-09-21T22:07:42.826 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.6491999999999999e-05 2024-09-21T22:07:42.826 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.826 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.826 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:42.826 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.992521+0000", 2024-09-21T22:07:42.826 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.0608000000000002e-05 2024-09-21T22:07:42.827 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.827 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.827 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:42.827 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.992582+0000", 2024-09-21T22:07:42.827 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.1797000000000005e-05 2024-09-21T22:07:42.827 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:42.827 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:42.827 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:42.828 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.828 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.828 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4840.0:720 2.1 2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:07:42.828 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.803827+0000", 2024-09-21T22:07:42.828 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 16.948556198999999, 2024-09-21T22:07:42.828 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.049890510999999998, 2024-09-21T22:07:42.828 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:42.829 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:42.829 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:42.829 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-21T22:07:42.829 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4840", 2024-09-21T22:07:42.829 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:42.829 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 720 2024-09-21T22:07:42.829 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.830 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:42.830 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.830 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:42.830 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.803827+0000", 2024-09-21T22:07:42.830 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:42.830 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.830 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.831 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:42.831 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.803827+0000", 2024-09-21T22:07:42.831 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:42.831 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.831 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.831 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:42.831 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.803828+0000", 2024-09-21T22:07:42.832 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.569999999999999e-07 2024-09-21T22:07:42.832 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.832 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.832 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:42.832 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.804689+0000", 2024-09-21T22:07:42.832 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00086121799999999997 2024-09-21T22:07:42.832 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.833 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.833 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:42.833 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.804706+0000", 2024-09-21T22:07:42.833 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6266999999999999e-05 2024-09-21T22:07:42.833 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.833 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.833 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:42.833 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.804713+0000", 2024-09-21T22:07:42.833 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.1010000000000003e-06 2024-09-21T22:07:42.833 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.834 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.834 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:42.834 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.804781+0000", 2024-09-21T22:07:42.834 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.7871000000000001e-05 2024-09-21T22:07:42.834 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.834 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.834 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:42.834 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.804859+0000", 2024-09-21T22:07:42.835 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.7891000000000004e-05 2024-09-21T22:07:42.835 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.835 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.835 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:42.835 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.804971+0000", 2024-09-21T22:07:42.835 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00011275100000000001 2024-09-21T22:07:42.835 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.835 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.835 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:42.835 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.852704+0000", 2024-09-21T22:07:42.836 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.047732361000000001 2024-09-21T22:07:42.836 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.836 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.836 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-21T22:07:42.836 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.853446+0000", 2024-09-21T22:07:42.836 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00074258799999999997 2024-09-21T22:07:42.836 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.836 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.836 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:42.836 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.853608+0000", 2024-09-21T22:07:42.836 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00016190900000000001 2024-09-21T22:07:42.836 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.836 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.836 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:42.837 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.853718+0000", 2024-09-21T22:07:42.837 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000109597 2024-09-21T22:07:42.837 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:42.837 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:42.837 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:42.837 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.837 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.837 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4840.0:723 2.1 2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:07:42.837 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.860543+0000", 2024-09-21T22:07:42.837 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 16.891840765000001, 2024-09-21T22:07:42.837 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.062354957000000003, 2024-09-21T22:07:42.837 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:42.837 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:42.837 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:42.838 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-21T22:07:42.838 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4840", 2024-09-21T22:07:42.838 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:42.838 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 723 2024-09-21T22:07:42.838 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.838 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:42.838 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.838 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:42.838 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.860543+0000", 2024-09-21T22:07:42.838 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:42.838 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.838 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.839 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:42.839 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.860543+0000", 2024-09-21T22:07:42.839 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:42.839 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.839 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.839 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:42.839 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.860544+0000", 2024-09-21T22:07:42.839 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.1119999999999999e-06 2024-09-21T22:07:42.839 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.839 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.839 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:42.839 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.861400+0000", 2024-09-21T22:07:42.839 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000856393 2024-09-21T22:07:42.840 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.840 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.840 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:42.840 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.861419+0000", 2024-09-21T22:07:42.840 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8992e-05 2024-09-21T22:07:42.840 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.840 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.840 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:42.840 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.861428+0000", 2024-09-21T22:07:42.840 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.551e-06 2024-09-21T22:07:42.840 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.840 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.840 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:42.840 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.861555+0000", 2024-09-21T22:07:42.841 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000127286 2024-09-21T22:07:42.841 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.841 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.841 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:42.841 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.861816+0000", 2024-09-21T22:07:42.841 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00026100899999999998 2024-09-21T22:07:42.841 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.841 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.841 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:42.841 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.862189+0000", 2024-09-21T22:07:42.841 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00037261899999999998 2024-09-21T22:07:42.841 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.842 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.842 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:42.842 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.921514+0000", 2024-09-21T22:07:42.842 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.059325210000000003 2024-09-21T22:07:42.842 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.842 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.842 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-21T22:07:42.842 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.922721+0000", 2024-09-21T22:07:42.842 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0012066819999999999 2024-09-21T22:07:42.842 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.842 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.842 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:42.843 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.922797+0000", 2024-09-21T22:07:42.843 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.6162999999999996e-05 2024-09-21T22:07:42.843 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.843 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.843 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:42.843 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.922898+0000", 2024-09-21T22:07:42.843 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00010094000000000001 2024-09-21T22:07:42.843 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:42.843 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:42.843 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:42.843 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.844 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.844 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:725 2.0 e63/16 2:08b4f96a:::rbd_data.126483d75f2d.000000000000000f:head v 63'33, mlcod=63'31)", 2024-09-21T22:07:42.844 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.936778+0000", 2024-09-21T22:07:42.844 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 16.815605023, 2024-09-21T22:07:42.844 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.047671937999999997, 2024-09-21T22:07:42.844 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:42.844 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:42.844 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:42.844 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:42.845 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.845 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:42.845 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.936778+0000", 2024-09-21T22:07:42.845 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:42.845 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.845 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.845 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:42.845 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.936778+0000", 2024-09-21T22:07:42.845 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:42.845 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.845 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.845 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:42.846 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.936780+0000", 2024-09-21T22:07:42.846 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7799999999999999e-06 2024-09-21T22:07:42.846 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.846 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.846 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:42.846 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.938475+0000", 2024-09-21T22:07:42.846 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0016951749999999999 2024-09-21T22:07:42.846 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.846 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.846 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:42.847 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.938506+0000", 2024-09-21T22:07:42.847 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.0865999999999999e-05 2024-09-21T22:07:42.847 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.847 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.847 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:42.847 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.938520+0000", 2024-09-21T22:07:42.847 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3825999999999999e-05 2024-09-21T22:07:42.847 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.847 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.847 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:42.847 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.938624+0000", 2024-09-21T22:07:42.847 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00010391900000000001 2024-09-21T22:07:42.847 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.847 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.848 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:42.848 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.938651+0000", 2024-09-21T22:07:42.848 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.7481999999999999e-05 2024-09-21T22:07:42.848 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.848 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.848 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:42.848 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.984369+0000", 2024-09-21T22:07:42.848 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.045717595 2024-09-21T22:07:42.848 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.848 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.848 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:42.848 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.984450+0000", 2024-09-21T22:07:42.848 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.1295e-05 2024-09-21T22:07:42.849 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:42.849 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:42.849 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:42.849 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.849 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.849 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:727 2.0 e63/16 2:08b4f96a:::rbd_data.126483d75f2d.000000000000000f:head v 63'35, mlcod=63'33)", 2024-09-21T22:07:42.849 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.998090+0000", 2024-09-21T22:07:42.849 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 16.754293847, 2024-09-21T22:07:42.849 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.052205704999999998, 2024-09-21T22:07:42.850 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:42.850 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:42.850 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:42.850 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:42.850 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.850 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:42.850 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.998090+0000", 2024-09-21T22:07:42.850 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:42.850 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.850 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.850 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:42.851 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.998090+0000", 2024-09-21T22:07:42.851 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:42.851 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.851 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.851 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:42.851 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.998091+0000", 2024-09-21T22:07:42.851 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6929999999999999e-06 2024-09-21T22:07:42.851 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.851 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.851 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:42.851 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.998940+0000", 2024-09-21T22:07:42.851 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00084875200000000004 2024-09-21T22:07:42.852 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.852 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.852 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:42.852 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.998970+0000", 2024-09-21T22:07:42.852 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9709999999999998e-05 2024-09-21T22:07:42.852 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.852 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.852 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:42.852 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.998984+0000", 2024-09-21T22:07:42.852 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3739e-05 2024-09-21T22:07:42.852 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.853 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.853 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:42.853 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.999097+0000", 2024-09-21T22:07:42.853 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000113484 2024-09-21T22:07:42.853 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.853 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.853 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:42.853 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.999135+0000", 2024-09-21T22:07:42.853 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.8405999999999998e-05 2024-09-21T22:07:42.853 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.853 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.853 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:42.854 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.050245+0000", 2024-09-21T22:07:42.854 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.051109935000000002 2024-09-21T22:07:42.854 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.854 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.854 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:42.854 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.050295+0000", 2024-09-21T22:07:42.854 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.9985999999999999e-05 2024-09-21T22:07:42.854 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:42.854 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:42.854 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:42.854 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.855 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.855 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:729 2.0 e63/16 2:08b4f96a:::rbd_data.126483d75f2d.000000000000000f:head v 63'37, mlcod=63'35)", 2024-09-21T22:07:42.855 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:26.062424+0000", 2024-09-21T22:07:42.855 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 16.689959224999999, 2024-09-21T22:07:42.855 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.047312186999999999, 2024-09-21T22:07:42.855 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:42.855 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:42.855 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:42.855 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:42.855 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.856 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:42.856 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.062424+0000", 2024-09-21T22:07:42.856 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:42.856 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.856 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.856 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:42.856 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.062424+0000", 2024-09-21T22:07:42.856 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:42.857 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.857 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.857 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:42.857 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.062426+0000", 2024-09-21T22:07:42.857 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7129999999999999e-06 2024-09-21T22:07:42.857 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.857 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.857 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:42.857 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.063211+0000", 2024-09-21T22:07:42.857 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00078488500000000003 2024-09-21T22:07:42.857 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.858 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.858 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:42.858 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.063241+0000", 2024-09-21T22:07:42.858 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.0056000000000002e-05 2024-09-21T22:07:42.858 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.858 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.858 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:42.858 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.063255+0000", 2024-09-21T22:07:42.858 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4022999999999999e-05 2024-09-21T22:07:42.858 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.858 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.859 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:42.859 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.063384+0000", 2024-09-21T22:07:42.859 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00012920600000000001 2024-09-21T22:07:42.859 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.859 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.859 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:42.859 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.063421+0000", 2024-09-21T22:07:42.859 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.7242e-05 2024-09-21T22:07:42.859 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.859 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.859 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:42.859 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.109666+0000", 2024-09-21T22:07:42.859 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.046244446000000002 2024-09-21T22:07:42.860 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.860 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.860 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:42.860 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.109736+0000", 2024-09-21T22:07:42.860 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.0616000000000003e-05 2024-09-21T22:07:42.860 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:42.860 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:42.860 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:42.860 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.860 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.860 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:731 2.0 e63/16 2:08b4f96a:::rbd_data.126483d75f2d.000000000000000f:head v 63'39, mlcod=63'37)", 2024-09-21T22:07:42.861 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:26.135946+0000", 2024-09-21T22:07:42.861 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 16.616437718, 2024-09-21T22:07:42.861 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.054290359000000003, 2024-09-21T22:07:42.861 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:42.861 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:42.861 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:42.861 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:42.861 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.861 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:42.861 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.135946+0000", 2024-09-21T22:07:42.861 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:42.862 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.862 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.862 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:42.862 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.135946+0000", 2024-09-21T22:07:42.862 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:42.862 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.862 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.862 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:42.862 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.135948+0000", 2024-09-21T22:07:42.862 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7999999999999999e-06 2024-09-21T22:07:42.862 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.862 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.862 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:42.862 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.136730+0000", 2024-09-21T22:07:42.863 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00078271899999999997 2024-09-21T22:07:42.863 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.863 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.863 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:42.863 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.136760+0000", 2024-09-21T22:07:42.863 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9876e-05 2024-09-21T22:07:42.863 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.863 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.863 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:42.863 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.136774+0000", 2024-09-21T22:07:42.863 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3478999999999999e-05 2024-09-21T22:07:42.863 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.863 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.863 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:42.863 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.136944+0000", 2024-09-21T22:07:42.864 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000170626 2024-09-21T22:07:42.864 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.864 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.864 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:42.864 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.136985+0000", 2024-09-21T22:07:42.864 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.0377999999999999e-05 2024-09-21T22:07:42.864 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.864 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.864 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:42.864 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.190137+0000", 2024-09-21T22:07:42.864 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.053152438000000003 2024-09-21T22:07:42.864 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.864 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.864 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:42.864 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.190236+0000", 2024-09-21T22:07:42.864 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.9042999999999993e-05 2024-09-21T22:07:42.865 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:42.865 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:42.865 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:42.865 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.865 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.865 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:125 2.3 e71/66 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head v 71'31, mlcod=71'29)", 2024-09-21T22:07:42.865 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.066855+0000", 2024-09-21T22:07:42.865 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 4.685528283, 2024-09-21T22:07:42.865 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.059856059000000003, 2024-09-21T22:07:42.865 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:42.865 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:42.865 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:42.865 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:42.865 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.865 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:42.865 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.066855+0000", 2024-09-21T22:07:42.866 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:42.866 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.866 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.866 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:42.866 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.066855+0000", 2024-09-21T22:07:42.866 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:42.866 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.866 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.866 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:42.866 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.066857+0000", 2024-09-21T22:07:42.866 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.807e-06 2024-09-21T22:07:42.866 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.866 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.866 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:42.866 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.068365+0000", 2024-09-21T22:07:42.866 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0015077910000000001 2024-09-21T22:07:42.867 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.867 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.867 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:42.867 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.068395+0000", 2024-09-21T22:07:42.867 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.0326000000000001e-05 2024-09-21T22:07:42.867 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.867 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.867 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:42.867 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.068415+0000", 2024-09-21T22:07:42.868 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9871999999999999e-05 2024-09-21T22:07:42.868 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.868 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.868 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:42.868 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.068580+0000", 2024-09-21T22:07:42.868 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000165317 2024-09-21T22:07:42.868 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.868 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.868 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:42.868 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.068618+0000", 2024-09-21T22:07:42.868 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.7753000000000003e-05 2024-09-21T22:07:42.868 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.868 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.869 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:42.869 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.126623+0000", 2024-09-21T22:07:42.869 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.058004642000000002 2024-09-21T22:07:42.869 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.869 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.869 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:42.869 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.126711+0000", 2024-09-21T22:07:42.869 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.8551e-05 2024-09-21T22:07:42.869 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:42.869 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:42.869 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:42.869 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.869 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.869 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:127 2.3 e71/66 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head v 71'33, mlcod=71'31)", 2024-09-21T22:07:42.870 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.140864+0000", 2024-09-21T22:07:42.870 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 4.6115192550000002, 2024-09-21T22:07:42.870 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.052512380999999997, 2024-09-21T22:07:42.870 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:42.870 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:42.870 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:42.870 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:42.870 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.870 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:42.870 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.140864+0000", 2024-09-21T22:07:42.871 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:42.871 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.871 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.871 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:42.871 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.140864+0000", 2024-09-21T22:07:42.871 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:42.871 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.871 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.871 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:42.871 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.140866+0000", 2024-09-21T22:07:42.871 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.694e-06 2024-09-21T22:07:42.872 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.872 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.872 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:42.872 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.141712+0000", 2024-09-21T22:07:42.872 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00084581499999999998 2024-09-21T22:07:42.872 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.872 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.872 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:42.872 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.141742+0000", 2024-09-21T22:07:42.872 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9865999999999998e-05 2024-09-21T22:07:42.872 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.873 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.873 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:42.873 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.141761+0000", 2024-09-21T22:07:42.873 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9473000000000001e-05 2024-09-21T22:07:42.873 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.873 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.873 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:42.873 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.141919+0000", 2024-09-21T22:07:42.873 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00015758300000000001 2024-09-21T22:07:42.873 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.873 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.873 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:42.873 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.141955+0000", 2024-09-21T22:07:42.873 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.6613000000000003e-05 2024-09-21T22:07:42.873 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.874 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.874 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:42.874 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.193333+0000", 2024-09-21T22:07:42.874 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.051377260000000001 2024-09-21T22:07:42.874 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.874 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.874 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:42.874 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.193377+0000", 2024-09-21T22:07:42.874 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.4076999999999998e-05 2024-09-21T22:07:42.874 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:42.874 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:42.874 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:42.874 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.875 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.875 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:129 2.3 e71/66 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head v 71'35, mlcod=71'33)", 2024-09-21T22:07:42.875 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.209458+0000", 2024-09-21T22:07:42.875 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 4.5429255059999996, 2024-09-21T22:07:42.875 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.056595578000000001, 2024-09-21T22:07:42.875 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:42.875 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:42.875 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:42.875 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:42.875 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.875 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:42.875 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.209458+0000", 2024-09-21T22:07:42.875 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:42.876 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.876 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.876 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:42.876 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.209458+0000", 2024-09-21T22:07:42.876 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:42.876 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.876 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.876 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:42.876 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.209460+0000", 2024-09-21T22:07:42.876 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7600000000000001e-06 2024-09-21T22:07:42.876 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.876 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.876 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:42.876 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.210317+0000", 2024-09-21T22:07:42.877 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00085771700000000003 2024-09-21T22:07:42.877 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.877 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.877 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:42.877 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.210346+0000", 2024-09-21T22:07:42.877 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8334e-05 2024-09-21T22:07:42.877 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.877 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.878 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:42.878 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.210365+0000", 2024-09-21T22:07:42.878 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9439000000000001e-05 2024-09-21T22:07:42.878 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.878 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.878 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:42.878 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.210544+0000", 2024-09-21T22:07:42.878 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00017845599999999999 2024-09-21T22:07:42.878 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.878 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.879 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:42.879 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.210584+0000", 2024-09-21T22:07:42.879 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.0423000000000002e-05 2024-09-21T22:07:42.879 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.879 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.879 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:42.879 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.265970+0000", 2024-09-21T22:07:42.879 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.055385765000000003 2024-09-21T22:07:42.879 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.879 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.879 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:42.880 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.266054+0000", 2024-09-21T22:07:42.880 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.3683999999999994e-05 2024-09-21T22:07:42.880 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:42.880 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:42.880 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:42.880 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.880 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.880 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:131 2.3 e71/66 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head v 71'37, mlcod=71'35)", 2024-09-21T22:07:42.880 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.279550+0000", 2024-09-21T22:07:42.880 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 4.4728336679999998, 2024-09-21T22:07:42.880 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.052290419999999997, 2024-09-21T22:07:42.880 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:42.881 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:42.881 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:42.881 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:42.881 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.881 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:42.881 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.279550+0000", 2024-09-21T22:07:42.881 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:42.881 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.881 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.881 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:42.881 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.279550+0000", 2024-09-21T22:07:42.881 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:42.882 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.882 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.882 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:42.882 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.279552+0000", 2024-09-21T22:07:42.882 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9439999999999999e-06 2024-09-21T22:07:42.882 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.882 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.882 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:42.882 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.280296+0000", 2024-09-21T22:07:42.882 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00074417000000000005 2024-09-21T22:07:42.882 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.883 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.883 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:42.883 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.280324+0000", 2024-09-21T22:07:42.883 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8045999999999999e-05 2024-09-21T22:07:42.883 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.883 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.883 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:42.883 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.280345+0000", 2024-09-21T22:07:42.883 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.0849999999999999e-05 2024-09-21T22:07:42.883 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.883 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.884 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:42.884 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.280497+0000", 2024-09-21T22:07:42.884 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00015252500000000001 2024-09-21T22:07:42.884 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.884 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.884 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:42.884 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.280531+0000", 2024-09-21T22:07:42.884 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.4034e-05 2024-09-21T22:07:42.884 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.884 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.884 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:42.885 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.331769+0000", 2024-09-21T22:07:42.885 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.051237999999999999 2024-09-21T22:07:42.885 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:42.885 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:42.885 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:42.885 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.331840+0000", 2024-09-21T22:07:42.885 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.0851000000000003e-05 2024-09-21T22:07:42.885 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:42.885 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:42.886 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:42.886 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:42.886 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:42.886 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:07:42.887 INFO:tasks.workunit.client.0.smithi157.stdout: PID TTY TIME CMD 2024-09-21T22:07:42.887 INFO:tasks.workunit.client.0.smithi157.stdout: 22883 ? 00:00:00 rbd-nbd 2024-09-21T22:07:42.889 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T22:07:42.890 INFO:tasks.workunit.client.0.smithi157.stderr:+ unmap_device '--snap-id 3 testrbdnbd21553' 22883 2024-09-21T22:07:42.890 INFO:tasks.workunit.client.0.smithi157.stderr:+ local 'args=--snap-id 3 testrbdnbd21553' 2024-09-21T22:07:42.890 INFO:tasks.workunit.client.0.smithi157.stderr:+ local pid=22883 2024-09-21T22:07:42.890 INFO:tasks.workunit.client.0.smithi157.stderr:+ _sudo rbd device --device-type nbd unmap --snap-id 3 testrbdnbd21553 2024-09-21T22:07:42.890 INFO:tasks.workunit.client.0.smithi157.stderr:+ local cmd 2024-09-21T22:07:42.890 INFO:tasks.workunit.client.0.smithi157.stderr:++ id -u 2024-09-21T22:07:42.890 INFO:tasks.workunit.client.0.smithi157.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T22:07:42.890 INFO:tasks.workunit.client.0.smithi157.stderr:++ which rbd 2024-09-21T22:07:42.890 INFO:tasks.workunit.client.0.smithi157.stderr:+ cmd=/usr/bin/rbd 2024-09-21T22:07:42.890 INFO:tasks.workunit.client.0.smithi157.stderr:+ shift 2024-09-21T22:07:42.891 INFO:tasks.workunit.client.0.smithi157.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap --snap-id 3 testrbdnbd21553 2024-09-21T22:07:42.891 INFO:tasks.workunit.client.0.smithi157.stderr:+ rbd device --device-type nbd list 2024-09-21T22:07:42.891 INFO:tasks.workunit.client.0.smithi157.stderr:+ expect_false grep '^22883\b' 2024-09-21T22:07:42.891 INFO:tasks.workunit.client.0.smithi157.stderr:+ grep '^22883\b' 2024-09-21T22:07:42.891 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:42.884+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:42.908 INFO:tasks.workunit.client.0.smithi157.stderr:+ return 0 2024-09-21T22:07:42.908 INFO:tasks.workunit.client.0.smithi157.stderr:+ ps -C rbd-nbd 2024-09-21T22:07:42.909 INFO:tasks.workunit.client.0.smithi157.stderr:+ expect_false grep '^ *22883\b' 2024-09-21T22:07:42.909 INFO:tasks.workunit.client.0.smithi157.stderr:+ grep '^ *22883\b' 2024-09-21T22:07:42.914 INFO:tasks.workunit.client.0.smithi157.stderr:+ return 0 2024-09-21T22:07:42.915 INFO:tasks.workunit.client.0.smithi157.stderr:+ sleep 0.5 2024-09-21T22:07:42.988 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:07:42.984+0000 7f2075a06640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31720) UID: 0 2024-09-21T22:07:43.064 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:07:43.064 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:07:43.064 INFO:teuthology.orchestra.run.smithi039.stdout: "num_ops": 0 2024-09-21T22:07:43.064 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:07:43.079 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T22:07:43.124 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:43.122+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:43.190 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:43.186+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:43.199 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:07:43.199 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:07:43.199 INFO:teuthology.orchestra.run.smithi039.stdout: "complaint_time": 30, 2024-09-21T22:07:43.200 INFO:teuthology.orchestra.run.smithi039.stdout: "num_blocked_ops": 0 2024-09-21T22:07:43.200 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:07:43.214 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T22:07:43.291 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:43.288+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:43.374 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-21T22:07:43.386 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:07:43.387 INFO:teuthology.orchestra.run.smithi039.stdout: "size": 20, 2024-09-21T22:07:43.387 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 600, 2024-09-21T22:07:43.387 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [ 2024-09-21T22:07:43.388 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.388 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPull(2.3 e55/54 cost 1048576)", 2024-09-21T22:07:43.388 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.081116+0000", 2024-09-21T22:07:43.388 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 31.302932389999999, 2024-09-21T22:07:43.388 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0030214159999999999, 2024-09-21T22:07:43.389 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:43.389 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:43.389 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-21T22:07:43.389 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:43.389 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.390 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:43.390 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.081116+0000", 2024-09-21T22:07:43.390 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:43.390 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.390 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.391 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:43.391 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.081116+0000", 2024-09-21T22:07:43.391 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:43.391 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.391 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.392 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:43.392 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.081118+0000", 2024-09-21T22:07:43.392 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.545e-06 2024-09-21T22:07:43.392 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.392 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.393 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:43.393 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.081133+0000", 2024-09-21T22:07:43.393 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5642e-05 2024-09-21T22:07:43.393 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.393 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.394 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:43.394 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.081155+0000", 2024-09-21T22:07:43.394 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1858e-05 2024-09-21T22:07:43.394 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.394 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.395 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:43.395 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.081171+0000", 2024-09-21T22:07:43.395 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5804e-05 2024-09-21T22:07:43.395 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.395 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.396 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:43.396 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.081291+0000", 2024-09-21T22:07:43.396 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000120553 2024-09-21T22:07:43.396 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.396 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.396 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:43.397 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.084137+0000", 2024-09-21T22:07:43.397 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.002846014 2024-09-21T22:07:43.397 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:43.397 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:43.397 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:43.398 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.398 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.398 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPull(2.3 e55/54 cost 1048576)", 2024-09-21T22:07:43.398 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.136357+0000", 2024-09-21T22:07:43.398 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 31.247691324000002, 2024-09-21T22:07:43.399 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.002843904, 2024-09-21T22:07:43.399 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:43.399 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:43.399 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-21T22:07:43.399 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:43.399 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.400 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:43.400 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.136357+0000", 2024-09-21T22:07:43.400 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:43.400 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.400 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.400 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:43.401 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.136357+0000", 2024-09-21T22:07:43.401 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:43.401 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.401 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.401 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:43.401 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.136359+0000", 2024-09-21T22:07:43.402 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.0669999999999999e-06 2024-09-21T22:07:43.402 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.402 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.402 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:43.402 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.136381+0000", 2024-09-21T22:07:43.403 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.2104999999999999e-05 2024-09-21T22:07:43.403 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.403 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.403 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:43.403 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.136412+0000", 2024-09-21T22:07:43.403 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.0602999999999998e-05 2024-09-21T22:07:43.404 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.404 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.404 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:43.404 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.136427+0000", 2024-09-21T22:07:43.404 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4832e-05 2024-09-21T22:07:43.405 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.405 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.405 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:43.405 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.136569+0000", 2024-09-21T22:07:43.405 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000142702 2024-09-21T22:07:43.405 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.406 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.406 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:43.406 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.139201+0000", 2024-09-21T22:07:43.406 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.002631595 2024-09-21T22:07:43.407 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:43.407 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:43.407 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:43.407 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.407 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.407 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:667 2.3 e63/54 2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head v 63'4, mlcod=0'0)", 2024-09-21T22:07:43.408 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.402454+0000", 2024-09-21T22:07:43.408 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 18.981594806, 2024-09-21T22:07:43.408 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0058375959999999996, 2024-09-21T22:07:43.408 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:43.408 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:43.409 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:43.409 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:43.409 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.409 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:43.410 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.402454+0000", 2024-09-21T22:07:43.410 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:43.410 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.410 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.410 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:43.411 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.402454+0000", 2024-09-21T22:07:43.411 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:43.411 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.411 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.411 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:43.411 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.402456+0000", 2024-09-21T22:07:43.412 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1179999999999999e-06 2024-09-21T22:07:43.412 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.412 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.412 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:43.412 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.402618+0000", 2024-09-21T22:07:43.413 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00016243 2024-09-21T22:07:43.413 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.413 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.413 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:43.413 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.402643+0000", 2024-09-21T22:07:43.414 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4638999999999999e-05 2024-09-21T22:07:43.414 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.414 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.414 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:43.414 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.402658+0000", 2024-09-21T22:07:43.415 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5211000000000001e-05 2024-09-21T22:07:43.415 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.415 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.415 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:43.415 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.402756+0000", 2024-09-21T22:07:43.416 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.7880000000000005e-05 2024-09-21T22:07:43.416 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.416 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.416 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:43.417 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.402779+0000", 2024-09-21T22:07:43.417 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3524999999999999e-05 2024-09-21T22:07:43.417 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.417 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.417 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:43.417 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.408233+0000", 2024-09-21T22:07:43.418 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0054534700000000002 2024-09-21T22:07:43.418 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.418 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.418 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:43.418 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.408291+0000", 2024-09-21T22:07:43.419 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.8322999999999997e-05 2024-09-21T22:07:43.419 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:43.419 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:43.419 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:43.419 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.420 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.420 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:668 2.3 e63/54 2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head v 63'5, mlcod=52'3)", 2024-09-21T22:07:43.420 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.410724+0000", 2024-09-21T22:07:43.420 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 18.973324388999998, 2024-09-21T22:07:43.420 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.031657809000000002, 2024-09-21T22:07:43.420 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:43.421 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:43.421 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:43.421 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:43.421 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.421 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:43.421 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.410724+0000", 2024-09-21T22:07:43.422 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:43.422 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.422 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.422 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:43.422 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.410724+0000", 2024-09-21T22:07:43.422 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:43.423 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.423 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.423 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:43.423 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.410725+0000", 2024-09-21T22:07:43.423 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.4300000000000001e-07 2024-09-21T22:07:43.423 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.423 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.424 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:43.424 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.411436+0000", 2024-09-21T22:07:43.424 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00071118600000000004 2024-09-21T22:07:43.424 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.424 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.424 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:43.425 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.411455+0000", 2024-09-21T22:07:43.425 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8357999999999999e-05 2024-09-21T22:07:43.425 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.425 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.425 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:43.425 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.411470+0000", 2024-09-21T22:07:43.426 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5753999999999999e-05 2024-09-21T22:07:43.426 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.426 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.426 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:43.426 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.411537+0000", 2024-09-21T22:07:43.426 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.6766000000000004e-05 2024-09-21T22:07:43.427 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.427 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.427 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:43.427 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.411571+0000", 2024-09-21T22:07:43.427 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.3555e-05 2024-09-21T22:07:43.427 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.428 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.428 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:43.428 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.442317+0000", 2024-09-21T22:07:43.428 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.030746393 2024-09-21T22:07:43.428 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.428 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.429 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:43.429 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.442382+0000", 2024-09-21T22:07:43.429 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.4853999999999999e-05 2024-09-21T22:07:43.429 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:43.429 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:43.430 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:43.430 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.430 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.430 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:669 2.3 e63/54 2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head v 63'6, mlcod=63'4)", 2024-09-21T22:07:43.430 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.459924+0000", 2024-09-21T22:07:43.431 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 18.924124831, 2024-09-21T22:07:43.431 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0062657900000000002, 2024-09-21T22:07:43.431 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:43.431 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:43.431 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:43.431 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:43.432 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.432 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:43.432 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.459924+0000", 2024-09-21T22:07:43.432 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:43.433 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.433 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.433 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:43.433 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.459924+0000", 2024-09-21T22:07:43.433 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:43.434 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.434 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.434 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:43.434 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.459925+0000", 2024-09-21T22:07:43.434 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8169999999999999e-06 2024-09-21T22:07:43.435 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.435 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.435 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:43.435 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.459983+0000", 2024-09-21T22:07:43.435 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.7567e-05 2024-09-21T22:07:43.436 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.436 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.436 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:43.436 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.460011+0000", 2024-09-21T22:07:43.436 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8455000000000001e-05 2024-09-21T22:07:43.437 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.437 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.437 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:43.437 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.460021+0000", 2024-09-21T22:07:43.438 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.9410000000000002e-06 2024-09-21T22:07:43.438 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.438 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.438 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:43.438 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.460093+0000", 2024-09-21T22:07:43.439 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.1231999999999998e-05 2024-09-21T22:07:43.439 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.439 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.439 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:43.439 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.460124+0000", 2024-09-21T22:07:43.440 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.1335999999999999e-05 2024-09-21T22:07:43.440 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.440 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.440 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:43.440 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.466130+0000", 2024-09-21T22:07:43.441 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0060055519999999999 2024-09-21T22:07:43.441 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.441 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.441 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:43.441 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.466189+0000", 2024-09-21T22:07:43.442 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.9889999999999997e-05 2024-09-21T22:07:43.442 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:43.442 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:43.442 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:43.442 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.442 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.443 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:670 2.3 e63/54 2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head v 63'7, mlcod=63'5)", 2024-09-21T22:07:43.443 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.468764+0000", 2024-09-21T22:07:43.443 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 18.915284849999999, 2024-09-21T22:07:43.443 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.037963433999999997, 2024-09-21T22:07:43.443 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:43.444 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:43.444 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:43.444 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:43.444 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.444 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:43.445 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.468764+0000", 2024-09-21T22:07:43.445 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:43.445 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.445 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.445 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:43.446 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.468764+0000", 2024-09-21T22:07:43.446 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:43.446 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.446 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.446 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:43.447 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.468765+0000", 2024-09-21T22:07:43.447 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.9699999999999994e-07 2024-09-21T22:07:43.447 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.447 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.447 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:43.447 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.469295+0000", 2024-09-21T22:07:43.448 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00053003600000000001 2024-09-21T22:07:43.448 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.448 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.448 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:43.448 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.469311+0000", 2024-09-21T22:07:43.448 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6841999999999999e-05 2024-09-21T22:07:43.449 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.449 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.449 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:43.449 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.469324+0000", 2024-09-21T22:07:43.449 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2373e-05 2024-09-21T22:07:43.449 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.450 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.450 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:43.450 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.469391+0000", 2024-09-21T22:07:43.450 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.6880999999999999e-05 2024-09-21T22:07:43.450 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.450 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.451 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:43.451 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.469419+0000", 2024-09-21T22:07:43.451 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8742999999999999e-05 2024-09-21T22:07:43.451 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.451 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.451 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:43.452 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.506662+0000", 2024-09-21T22:07:43.452 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.037242994000000001 2024-09-21T22:07:43.452 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.452 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.452 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:43.452 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.506727+0000", 2024-09-21T22:07:43.452 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.4567999999999996e-05 2024-09-21T22:07:43.453 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:43.453 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:43.453 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:43.453 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.453 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.453 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:671 2.3 e63/54 2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head v 63'8, mlcod=63'6)", 2024-09-21T22:07:43.454 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.519938+0000", 2024-09-21T22:07:43.454 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 18.864110311000001, 2024-09-21T22:07:43.454 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0058323419999999999, 2024-09-21T22:07:43.454 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:43.454 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:43.454 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:43.455 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:43.455 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.455 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:43.455 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.519938+0000", 2024-09-21T22:07:43.455 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:43.455 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.456 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.456 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:43.456 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.519938+0000", 2024-09-21T22:07:43.456 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:43.456 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.456 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.457 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:43.457 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.519940+0000", 2024-09-21T22:07:43.457 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.666e-06 2024-09-21T22:07:43.457 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.457 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.458 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:43.458 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.520031+0000", 2024-09-21T22:07:43.458 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.1051999999999995e-05 2024-09-21T22:07:43.458 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.458 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.459 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:43.459 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.520056+0000", 2024-09-21T22:07:43.459 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5459000000000001e-05 2024-09-21T22:07:43.459 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.459 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.459 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:43.459 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.520072+0000", 2024-09-21T22:07:43.459 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5801e-05 2024-09-21T22:07:43.459 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.459 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.460 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:43.460 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.520178+0000", 2024-09-21T22:07:43.460 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000105388 2024-09-21T22:07:43.460 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.460 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.460 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:43.460 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.520204+0000", 2024-09-21T22:07:43.460 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.6208e-05 2024-09-21T22:07:43.460 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.460 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.460 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:43.460 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.525735+0000", 2024-09-21T22:07:43.461 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0055308520000000002 2024-09-21T22:07:43.461 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.461 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.461 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:43.461 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.525770+0000", 2024-09-21T22:07:43.461 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.5916e-05 2024-09-21T22:07:43.461 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:43.461 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:43.461 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:43.461 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.461 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.461 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:672 2.3 e63/54 2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head v 63'9, mlcod=63'7)", 2024-09-21T22:07:43.461 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.528432+0000", 2024-09-21T22:07:43.461 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 18.855616935, 2024-09-21T22:07:43.462 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.029763412999999999, 2024-09-21T22:07:43.462 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:43.462 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:43.462 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:43.462 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:43.462 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.462 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:43.462 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.528432+0000", 2024-09-21T22:07:43.462 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:43.462 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.463 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.463 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:43.463 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.528432+0000", 2024-09-21T22:07:43.463 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:43.463 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.463 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.463 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:43.463 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.528433+0000", 2024-09-21T22:07:43.463 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7179999999999999e-06 2024-09-21T22:07:43.463 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.463 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.463 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:43.463 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.528926+0000", 2024-09-21T22:07:43.463 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00049238500000000002 2024-09-21T22:07:43.464 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.464 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.464 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:43.464 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.528955+0000", 2024-09-21T22:07:43.464 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9037e-05 2024-09-21T22:07:43.464 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.464 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.464 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:43.464 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.528971+0000", 2024-09-21T22:07:43.464 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6622000000000001e-05 2024-09-21T22:07:43.464 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.465 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.465 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:43.465 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.529047+0000", 2024-09-21T22:07:43.465 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.5246000000000004e-05 2024-09-21T22:07:43.465 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.465 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.465 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:43.465 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.529074+0000", 2024-09-21T22:07:43.465 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.7532e-05 2024-09-21T22:07:43.465 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.465 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.465 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:43.466 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.558149+0000", 2024-09-21T22:07:43.466 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.029075192999999999 2024-09-21T22:07:43.466 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.466 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.466 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:43.466 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.558195+0000", 2024-09-21T22:07:43.466 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.5680000000000003e-05 2024-09-21T22:07:43.466 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:43.466 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:43.466 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:43.467 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.467 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.467 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:673 2.3 e63/54 2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head v 63'10, mlcod=63'8)", 2024-09-21T22:07:43.467 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.576513+0000", 2024-09-21T22:07:43.467 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 18.8075355, 2024-09-21T22:07:43.467 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.003674666, 2024-09-21T22:07:43.467 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:43.467 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:43.467 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:43.467 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:43.467 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.467 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:43.468 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.576513+0000", 2024-09-21T22:07:43.468 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:43.468 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.468 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.468 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:43.468 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.576513+0000", 2024-09-21T22:07:43.468 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:43.468 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.468 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.468 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:43.468 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.576514+0000", 2024-09-21T22:07:43.468 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.1299999999999998e-07 2024-09-21T22:07:43.469 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.469 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.469 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:43.469 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.576567+0000", 2024-09-21T22:07:43.469 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.2710000000000002e-05 2024-09-21T22:07:43.469 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.469 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.469 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:43.469 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.576579+0000", 2024-09-21T22:07:43.469 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2035000000000001e-05 2024-09-21T22:07:43.469 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.470 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.470 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:43.470 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.576585+0000", 2024-09-21T22:07:43.470 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.6050000000000003e-06 2024-09-21T22:07:43.470 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.470 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.470 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:43.470 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.576638+0000", 2024-09-21T22:07:43.470 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.2726999999999998e-05 2024-09-21T22:07:43.470 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.471 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.471 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:43.471 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.576656+0000", 2024-09-21T22:07:43.471 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8084000000000002e-05 2024-09-21T22:07:43.471 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.471 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.471 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:43.471 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.580156+0000", 2024-09-21T22:07:43.471 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0035002700000000002 2024-09-21T22:07:43.471 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.471 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.471 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:43.472 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.580188+0000", 2024-09-21T22:07:43.472 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.1322000000000003e-05 2024-09-21T22:07:43.472 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:43.472 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:43.472 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:43.472 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.472 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.472 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:674 2.3 e63/54 2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head v 63'11, mlcod=63'9)", 2024-09-21T22:07:43.472 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.584683+0000", 2024-09-21T22:07:43.472 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 18.799365058999999, 2024-09-21T22:07:43.472 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.028028352999999999, 2024-09-21T22:07:43.473 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:43.473 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:43.473 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:43.473 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:43.473 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.473 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:43.473 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.584683+0000", 2024-09-21T22:07:43.473 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:43.473 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.473 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.473 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:43.473 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.584683+0000", 2024-09-21T22:07:43.474 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:43.474 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.474 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.474 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:43.474 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.584684+0000", 2024-09-21T22:07:43.474 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.4600000000000004e-07 2024-09-21T22:07:43.474 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.474 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.474 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:43.474 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.585041+0000", 2024-09-21T22:07:43.474 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000356752 2024-09-21T22:07:43.474 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.475 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.475 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:43.475 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.585054+0000", 2024-09-21T22:07:43.475 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3387000000000001e-05 2024-09-21T22:07:43.475 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.475 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.475 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:43.475 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.585065+0000", 2024-09-21T22:07:43.475 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.1124999999999999e-05 2024-09-21T22:07:43.475 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.475 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.476 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:43.476 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.585142+0000", 2024-09-21T22:07:43.476 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.6743999999999996e-05 2024-09-21T22:07:43.476 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.476 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.476 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:43.476 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.585170+0000", 2024-09-21T22:07:43.476 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.7569999999999999e-05 2024-09-21T22:07:43.476 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.476 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.476 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:43.476 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.612637+0000", 2024-09-21T22:07:43.477 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.027467803999999998 2024-09-21T22:07:43.477 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.477 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.477 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:43.477 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.612712+0000", 2024-09-21T22:07:43.477 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.4325000000000004e-05 2024-09-21T22:07:43.477 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:43.477 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:43.477 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:43.477 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.477 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.477 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:683 2.3 e63/54 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head v 63'12, mlcod=63'10)", 2024-09-21T22:07:43.477 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.888242+0000", 2024-09-21T22:07:43.477 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 18.495806440999999, 2024-09-21T22:07:43.478 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0085158290000000008, 2024-09-21T22:07:43.478 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:43.478 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:43.478 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:43.478 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:43.478 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.478 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:43.478 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.888242+0000", 2024-09-21T22:07:43.478 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:43.478 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.478 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.478 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:43.478 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.888242+0000", 2024-09-21T22:07:43.478 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:43.479 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.479 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.479 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:43.479 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.888244+0000", 2024-09-21T22:07:43.479 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.553e-06 2024-09-21T22:07:43.479 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.479 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.479 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:43.479 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.888295+0000", 2024-09-21T22:07:43.479 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.1916000000000002e-05 2024-09-21T22:07:43.479 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.479 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.479 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:43.479 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.888319+0000", 2024-09-21T22:07:43.480 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3139999999999999e-05 2024-09-21T22:07:43.480 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.480 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.480 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:43.480 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.888332+0000", 2024-09-21T22:07:43.480 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2993000000000001e-05 2024-09-21T22:07:43.480 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.480 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.480 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:43.480 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.888441+0000", 2024-09-21T22:07:43.480 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000109039 2024-09-21T22:07:43.480 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.480 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.480 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:43.480 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.888481+0000", 2024-09-21T22:07:43.481 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.9950000000000002e-05 2024-09-21T22:07:43.481 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.481 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.481 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:43.481 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.896688+0000", 2024-09-21T22:07:43.481 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0082078729999999992 2024-09-21T22:07:43.481 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.481 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.481 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:43.481 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.896758+0000", 2024-09-21T22:07:43.481 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.9364999999999997e-05 2024-09-21T22:07:43.481 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:43.481 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:43.481 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:43.482 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.482 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.482 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:684 2.3 e63/54 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head v 63'13, mlcod=63'11)", 2024-09-21T22:07:43.482 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.899243+0000", 2024-09-21T22:07:43.482 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 18.484805664, 2024-09-21T22:07:43.482 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.033436192000000003, 2024-09-21T22:07:43.482 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:43.482 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:43.482 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:43.482 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:43.482 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.482 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:43.482 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.899243+0000", 2024-09-21T22:07:43.482 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:43.483 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.483 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.483 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:43.483 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.899243+0000", 2024-09-21T22:07:43.483 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:43.483 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.483 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.483 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:43.483 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.899244+0000", 2024-09-21T22:07:43.483 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.705e-06 2024-09-21T22:07:43.483 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.483 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.483 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:43.483 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.899604+0000", 2024-09-21T22:07:43.483 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00035930599999999999 2024-09-21T22:07:43.484 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.484 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.484 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:43.484 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.899622+0000", 2024-09-21T22:07:43.484 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8487000000000001e-05 2024-09-21T22:07:43.484 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.484 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.484 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:43.484 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.899634+0000", 2024-09-21T22:07:43.484 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2116999999999999e-05 2024-09-21T22:07:43.484 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.484 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.484 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:43.484 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.899721+0000", 2024-09-21T22:07:43.485 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.6718999999999994e-05 2024-09-21T22:07:43.485 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.485 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.485 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:43.485 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.899755+0000", 2024-09-21T22:07:43.485 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.4292000000000003e-05 2024-09-21T22:07:43.485 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.485 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.485 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:43.485 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.932545+0000", 2024-09-21T22:07:43.485 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.03278959 2024-09-21T22:07:43.485 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.485 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.485 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:43.485 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.932679+0000", 2024-09-21T22:07:43.486 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013397600000000001 2024-09-21T22:07:43.486 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:43.486 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:43.486 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:43.486 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.486 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.486 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:685 2.3 e63/54 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head v 63'14, mlcod=63'12)", 2024-09-21T22:07:43.486 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.935246+0000", 2024-09-21T22:07:43.486 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 18.448802827000002, 2024-09-21T22:07:43.486 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0073590590000000003, 2024-09-21T22:07:43.486 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:43.486 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:43.486 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:43.487 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:43.487 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.487 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:43.487 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.935246+0000", 2024-09-21T22:07:43.487 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:43.487 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.487 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.487 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:43.487 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.935246+0000", 2024-09-21T22:07:43.487 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:43.487 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.487 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.488 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:43.488 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.935247+0000", 2024-09-21T22:07:43.488 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4470000000000001e-06 2024-09-21T22:07:43.488 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.488 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.488 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:43.488 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.935295+0000", 2024-09-21T22:07:43.488 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.7963000000000001e-05 2024-09-21T22:07:43.488 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.488 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.488 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:43.488 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.935313+0000", 2024-09-21T22:07:43.488 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8317000000000001e-05 2024-09-21T22:07:43.489 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.489 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.489 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:43.489 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.935352+0000", 2024-09-21T22:07:43.489 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.8552e-05 2024-09-21T22:07:43.489 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.489 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.489 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:43.489 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.935438+0000", 2024-09-21T22:07:43.489 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.6405999999999999e-05 2024-09-21T22:07:43.489 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.489 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.490 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:43.490 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.935471+0000", 2024-09-21T22:07:43.490 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.3142999999999998e-05 2024-09-21T22:07:43.490 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.490 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.490 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:43.490 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.942552+0000", 2024-09-21T22:07:43.490 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0070809510000000003 2024-09-21T22:07:43.490 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.490 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.490 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:43.490 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.942605+0000", 2024-09-21T22:07:43.491 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.2280000000000001e-05 2024-09-21T22:07:43.491 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:43.491 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:43.491 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:43.491 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.491 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.491 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:686 2.3 e63/54 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head v 63'15, mlcod=63'13)", 2024-09-21T22:07:43.491 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.945043+0000", 2024-09-21T22:07:43.491 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 18.439005514000002, 2024-09-21T22:07:43.491 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.04716302, 2024-09-21T22:07:43.491 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:43.492 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:43.492 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:43.492 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:43.492 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.492 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:43.492 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.945043+0000", 2024-09-21T22:07:43.492 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:43.492 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.492 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.492 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:43.492 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.945043+0000", 2024-09-21T22:07:43.492 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:43.493 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.493 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.493 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:43.493 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.945045+0000", 2024-09-21T22:07:43.493 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8700000000000001e-06 2024-09-21T22:07:43.493 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.493 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.493 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:43.493 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.945899+0000", 2024-09-21T22:07:43.493 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00085401900000000002 2024-09-21T22:07:43.493 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.493 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.494 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:43.494 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.945928+0000", 2024-09-21T22:07:43.494 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9193999999999999e-05 2024-09-21T22:07:43.494 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.494 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.494 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:43.494 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.945948+0000", 2024-09-21T22:07:43.494 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9876000000000001e-05 2024-09-21T22:07:43.494 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.494 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.494 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:43.495 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.946060+0000", 2024-09-21T22:07:43.495 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00011175099999999999 2024-09-21T22:07:43.495 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.495 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.495 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:43.495 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.946103+0000", 2024-09-21T22:07:43.495 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.3090999999999998e-05 2024-09-21T22:07:43.495 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.495 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.495 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:43.495 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.992151+0000", 2024-09-21T22:07:43.496 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.046047948999999998 2024-09-21T22:07:43.496 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.496 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.496 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:43.496 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.992206+0000", 2024-09-21T22:07:43.496 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.5269999999999998e-05 2024-09-21T22:07:43.496 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:43.496 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:43.496 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:43.496 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.496 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.496 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:687 2.3 e63/54 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head v 63'16, mlcod=63'14)", 2024-09-21T22:07:43.497 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.994473+0000", 2024-09-21T22:07:43.497 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 18.389575891, 2024-09-21T22:07:43.497 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0054660990000000003, 2024-09-21T22:07:43.497 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:43.497 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:43.497 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:43.497 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:43.497 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.497 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:43.497 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.994473+0000", 2024-09-21T22:07:43.497 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:43.498 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.498 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.498 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:43.498 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.994473+0000", 2024-09-21T22:07:43.498 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:43.498 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.498 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.498 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:43.498 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.994474+0000", 2024-09-21T22:07:43.498 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.463e-06 2024-09-21T22:07:43.498 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.499 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.499 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:43.499 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.994546+0000", 2024-09-21T22:07:43.499 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.2260000000000003e-05 2024-09-21T22:07:43.499 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.499 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.499 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:43.499 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.994564+0000", 2024-09-21T22:07:43.499 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7983999999999999e-05 2024-09-21T22:07:43.499 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.499 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.499 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:43.499 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.994573+0000", 2024-09-21T22:07:43.500 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.4710000000000007e-06 2024-09-21T22:07:43.500 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.500 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.500 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:43.500 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.994634+0000", 2024-09-21T22:07:43.500 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.0872000000000003e-05 2024-09-21T22:07:43.500 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.500 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.500 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:43.500 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.994658+0000", 2024-09-21T22:07:43.500 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4573e-05 2024-09-21T22:07:43.500 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.501 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.501 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:43.501 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.999894+0000", 2024-09-21T22:07:43.501 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0052362390000000002 2024-09-21T22:07:43.501 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.501 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.501 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:43.501 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.999939+0000", 2024-09-21T22:07:43.501 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.4236999999999997e-05 2024-09-21T22:07:43.501 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:43.501 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:43.501 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:43.502 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.502 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.502 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:688 2.3 e63/54 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head v 63'17, mlcod=63'15)", 2024-09-21T22:07:43.502 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.001812+0000", 2024-09-21T22:07:43.502 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 18.382236809999998, 2024-09-21T22:07:43.502 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.028785011999999999, 2024-09-21T22:07:43.502 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:43.502 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:43.502 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:43.502 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:43.502 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.502 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:43.503 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.001812+0000", 2024-09-21T22:07:43.503 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:43.503 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.503 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.503 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:43.503 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.001812+0000", 2024-09-21T22:07:43.503 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:43.503 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.503 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.503 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:43.503 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.001813+0000", 2024-09-21T22:07:43.503 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2699999999999999e-06 2024-09-21T22:07:43.504 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.504 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.504 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:43.504 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.002252+0000", 2024-09-21T22:07:43.504 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00043878599999999998 2024-09-21T22:07:43.504 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.504 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.504 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:43.504 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.002271+0000", 2024-09-21T22:07:43.504 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9346000000000001e-05 2024-09-21T22:07:43.504 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.504 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.504 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:43.504 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.002282+0000", 2024-09-21T22:07:43.505 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0951000000000001e-05 2024-09-21T22:07:43.505 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.505 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.505 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:43.505 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.002398+0000", 2024-09-21T22:07:43.505 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000116412 2024-09-21T22:07:43.505 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.505 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.505 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:43.505 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.002425+0000", 2024-09-21T22:07:43.505 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.6291999999999998e-05 2024-09-21T22:07:43.505 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.505 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.505 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:43.506 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.030528+0000", 2024-09-21T22:07:43.506 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.028103390999999998 2024-09-21T22:07:43.506 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.506 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.506 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:43.506 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.030597+0000", 2024-09-21T22:07:43.506 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.8564000000000003e-05 2024-09-21T22:07:43.506 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:43.506 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:43.506 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:43.506 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.506 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.506 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:689 2.3 e63/54 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head v 63'18, mlcod=63'16)", 2024-09-21T22:07:43.506 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.032891+0000", 2024-09-21T22:07:43.507 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 18.351157228000002, 2024-09-21T22:07:43.507 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.005061032, 2024-09-21T22:07:43.507 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:43.507 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:43.507 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:43.507 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:43.507 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.507 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:43.507 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.032891+0000", 2024-09-21T22:07:43.507 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:43.507 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.507 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.507 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:43.507 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.032891+0000", 2024-09-21T22:07:43.508 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:43.508 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.508 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.508 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:43.508 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.032893+0000", 2024-09-21T22:07:43.508 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3030000000000001e-06 2024-09-21T22:07:43.508 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.508 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.508 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:43.508 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.032952+0000", 2024-09-21T22:07:43.508 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.9070000000000002e-05 2024-09-21T22:07:43.508 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.508 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.508 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:43.508 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.032969+0000", 2024-09-21T22:07:43.509 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7839e-05 2024-09-21T22:07:43.509 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.509 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.509 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:43.509 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.032979+0000", 2024-09-21T22:07:43.509 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.5370000000000003e-06 2024-09-21T22:07:43.509 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.509 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.509 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:43.509 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.033041+0000", 2024-09-21T22:07:43.509 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.1562000000000005e-05 2024-09-21T22:07:43.509 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.509 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.509 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:43.510 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.033068+0000", 2024-09-21T22:07:43.510 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.7226999999999999e-05 2024-09-21T22:07:43.510 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.510 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.510 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:43.510 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.037908+0000", 2024-09-21T22:07:43.510 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0048401770000000002 2024-09-21T22:07:43.510 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.510 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.510 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:43.510 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.037952+0000", 2024-09-21T22:07:43.510 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.4317000000000003e-05 2024-09-21T22:07:43.510 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:43.510 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:43.511 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:43.511 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.511 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.511 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:690 2.3 e63/54 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head v 63'19, mlcod=63'17)", 2024-09-21T22:07:43.511 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.039813+0000", 2024-09-21T22:07:43.511 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 18.344235010999999, 2024-09-21T22:07:43.511 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.035351891000000003, 2024-09-21T22:07:43.511 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:43.511 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:43.511 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:43.511 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:43.511 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.511 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:43.512 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.039813+0000", 2024-09-21T22:07:43.512 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:43.512 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.512 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.512 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:43.512 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.039813+0000", 2024-09-21T22:07:43.512 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:43.512 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.512 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.512 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:43.512 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.039814+0000", 2024-09-21T22:07:43.512 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.5300000000000002e-07 2024-09-21T22:07:43.512 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.512 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.513 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:43.513 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.040166+0000", 2024-09-21T22:07:43.513 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00035177 2024-09-21T22:07:43.513 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.513 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.513 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:43.513 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.040185+0000", 2024-09-21T22:07:43.513 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9000000000000001e-05 2024-09-21T22:07:43.513 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.513 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.513 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:43.513 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.040195+0000", 2024-09-21T22:07:43.514 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0254999999999999e-05 2024-09-21T22:07:43.514 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.514 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.514 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:43.514 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.040262+0000", 2024-09-21T22:07:43.514 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.6198000000000006e-05 2024-09-21T22:07:43.514 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.514 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.514 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:43.514 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.040288+0000", 2024-09-21T22:07:43.514 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5931999999999998e-05 2024-09-21T22:07:43.514 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.515 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.515 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:43.515 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.075106+0000", 2024-09-21T22:07:43.515 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.034818546999999998 2024-09-21T22:07:43.515 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.515 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.515 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:43.515 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.075165+0000", 2024-09-21T22:07:43.515 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.9236e-05 2024-09-21T22:07:43.515 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:43.515 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:43.515 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:43.516 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.516 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.516 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPull(2.3 e67/66 cost 1048576)", 2024-09-21T22:07:43.516 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:29.414957+0000", 2024-09-21T22:07:43.516 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 13.969091632, 2024-09-21T22:07:43.516 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0032647269999999998, 2024-09-21T22:07:43.516 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:43.516 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:43.516 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-21T22:07:43.516 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:43.516 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.516 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:43.516 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:29.414957+0000", 2024-09-21T22:07:43.517 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:43.517 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.517 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.517 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:43.517 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:29.414957+0000", 2024-09-21T22:07:43.517 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:43.517 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.517 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.517 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:43.517 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:29.414959+0000", 2024-09-21T22:07:43.517 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.396e-06 2024-09-21T22:07:43.518 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.518 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.518 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:43.518 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:29.414986+0000", 2024-09-21T22:07:43.518 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.72e-05 2024-09-21T22:07:43.518 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.518 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.518 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:43.518 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:29.415018+0000", 2024-09-21T22:07:43.518 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.2079000000000002e-05 2024-09-21T22:07:43.518 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.519 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.519 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:43.519 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:29.415035+0000", 2024-09-21T22:07:43.519 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6316000000000001e-05 2024-09-21T22:07:43.519 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.519 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.519 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:43.519 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:29.415201+0000", 2024-09-21T22:07:43.519 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00016583900000000001 2024-09-21T22:07:43.519 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.519 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.519 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:43.519 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:29.418222+0000", 2024-09-21T22:07:43.520 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0030208969999999998 2024-09-21T22:07:43.520 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:43.520 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:43.520 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:43.520 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.520 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.520 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPull(2.3 e67/66 cost 1048576)", 2024-09-21T22:07:43.520 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:29.486922+0000", 2024-09-21T22:07:43.520 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 13.897126695000001, 2024-09-21T22:07:43.520 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0029880940000000002, 2024-09-21T22:07:43.520 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:43.520 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:43.520 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-21T22:07:43.520 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:43.520 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.521 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:43.521 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:29.486922+0000", 2024-09-21T22:07:43.521 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:43.521 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.521 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.521 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:43.521 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:29.486922+0000", 2024-09-21T22:07:43.521 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:43.521 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.521 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.521 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:43.521 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:29.486924+0000", 2024-09-21T22:07:43.521 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.897e-06 2024-09-21T22:07:43.521 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.521 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.522 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:43.522 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:29.486975+0000", 2024-09-21T22:07:43.522 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.1275e-05 2024-09-21T22:07:43.522 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.522 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.522 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:43.522 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:29.487002+0000", 2024-09-21T22:07:43.522 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.7469000000000001e-05 2024-09-21T22:07:43.522 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.522 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.522 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:43.523 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:29.487016+0000", 2024-09-21T22:07:43.523 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3587000000000001e-05 2024-09-21T22:07:43.523 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.523 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.523 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:43.523 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:29.487158+0000", 2024-09-21T22:07:43.523 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00014215899999999999 2024-09-21T22:07:43.523 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:43.523 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:43.523 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:43.523 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:29.489910+0000", 2024-09-21T22:07:43.523 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0027517069999999999 2024-09-21T22:07:43.523 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:43.524 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:43.524 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:43.524 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:43.524 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:43.524 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:07:43.525 INFO:tasks.workunit.client.0.smithi157.stderr:+ DEV= 2024-09-21T22:07:43.526 INFO:tasks.workunit.client.0.smithi157.stderr:+ rbd snap create rbd/ns/testrbdnbd21553@snap 2024-09-21T22:07:43.526 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T22:07:43.527 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:43.522+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:43.626 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:43.622+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:43.661 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:07:43.661 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:07:43.661 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-09-21T22:07:43.661 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:07:43.676 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T22:07:43.728 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:43.726+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:43.801 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:07:43.801 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:07:43.801 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-21T22:07:43.801 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-21T22:07:43.801 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:07:43.815 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T22:07:43.862 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:43.860+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:43.929 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:07:43.924+0000 7f2075a06640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31720) UID: 0 2024-09-21T22:07:43.936 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:07:43.936 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-21T22:07:43.936 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-21T22:07:43.936 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-21T22:07:43.936 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.936 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:07:43.936 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.036355+0000", 2024-09-21T22:07:43.936 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 101.898901088, 2024-09-21T22:07:43.937 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.062441934999999997, 2024-09-21T22:07:43.937 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:43.937 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:43.937 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:43.937 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:43.937 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:07:43.937 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:07:43.937 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1 2024-09-21T22:07:43.937 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.937 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:43.937 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.938 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:43.938 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036355+0000", 2024-09-21T22:07:43.938 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:43.938 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.938 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.938 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:43.938 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036355+0000", 2024-09-21T22:07:43.938 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:43.938 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.938 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.938 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:43.938 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036357+0000", 2024-09-21T22:07:43.938 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.446e-06 2024-09-21T22:07:43.939 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.939 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.939 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:43.939 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036373+0000", 2024-09-21T22:07:43.939 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5447e-05 2024-09-21T22:07:43.939 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.939 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.939 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:43.939 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036399+0000", 2024-09-21T22:07:43.939 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6423000000000001e-05 2024-09-21T22:07:43.939 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.939 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.939 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:43.940 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036420+0000", 2024-09-21T22:07:43.940 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1129999999999999e-05 2024-09-21T22:07:43.940 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.940 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.940 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:43.940 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036557+0000", 2024-09-21T22:07:43.940 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013686899999999999 2024-09-21T22:07:43.940 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.940 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.940 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-21T22:07:43.940 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036585+0000", 2024-09-21T22:07:43.940 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8362999999999999e-05 2024-09-21T22:07:43.940 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.941 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.941 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:43.941 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.095068+0000", 2024-09-21T22:07:43.941 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058482972000000001 2024-09-21T22:07:43.941 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.941 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.941 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:43.941 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.095516+0000", 2024-09-21T22:07:43.941 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00044802999999999998 2024-09-21T22:07:43.941 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.941 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.941 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:43.941 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.095869+0000", 2024-09-21T22:07:43.941 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000352104 2024-09-21T22:07:43.942 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.942 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.942 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:43.942 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.097055+0000", 2024-09-21T22:07:43.942 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011865599999999999 2024-09-21T22:07:43.942 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.942 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.942 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:43.942 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.098637+0000", 2024-09-21T22:07:43.942 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001581458 2024-09-21T22:07:43.942 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.942 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.942 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:43.943 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.098728+0000", 2024-09-21T22:07:43.943 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1043000000000002e-05 2024-09-21T22:07:43.943 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.943 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.943 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:43.943 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.098797+0000", 2024-09-21T22:07:43.943 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9090000000000004e-05 2024-09-21T22:07:43.943 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:43.943 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:43.943 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:43.944 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.944 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.944 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4840.0:678 2.7 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:07:43.944 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.678289+0000", 2024-09-21T22:07:43.944 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 19.256967032999999, 2024-09-21T22:07:43.944 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.055089659999999999, 2024-09-21T22:07:43.944 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:43.944 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:43.944 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:43.944 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:43.944 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4840", 2024-09-21T22:07:43.945 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:43.945 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 678 2024-09-21T22:07:43.945 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.945 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:43.945 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.945 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:43.945 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.678289+0000", 2024-09-21T22:07:43.945 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:43.945 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.945 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.945 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:43.945 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.678289+0000", 2024-09-21T22:07:43.945 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:43.946 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.946 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.946 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:43.946 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.678291+0000", 2024-09-21T22:07:43.946 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7400000000000001e-06 2024-09-21T22:07:43.946 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.946 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.946 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:43.946 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.679129+0000", 2024-09-21T22:07:43.946 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00083801700000000004 2024-09-21T22:07:43.946 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.947 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.947 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:43.947 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.679161+0000", 2024-09-21T22:07:43.947 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2398999999999999e-05 2024-09-21T22:07:43.947 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.947 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.947 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:43.947 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.679176+0000", 2024-09-21T22:07:43.947 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5396999999999998e-05 2024-09-21T22:07:43.947 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.947 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.948 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:43.948 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.679256+0000", 2024-09-21T22:07:43.948 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9271999999999996e-05 2024-09-21T22:07:43.948 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.948 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.948 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:43.948 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.679333+0000", 2024-09-21T22:07:43.948 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7682999999999995e-05 2024-09-21T22:07:43.948 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.949 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.949 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-09-21T22:07:43.949 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.679457+0000", 2024-09-21T22:07:43.949 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012327900000000001 2024-09-21T22:07:43.949 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.949 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.949 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:43.949 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.713209+0000", 2024-09-21T22:07:43.949 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.033752058000000001 2024-09-21T22:07:43.949 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.950 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.950 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:43.950 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.733243+0000", 2024-09-21T22:07:43.950 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020034337999999999 2024-09-21T22:07:43.950 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.950 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.950 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:43.950 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.733320+0000", 2024-09-21T22:07:43.950 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7354999999999994e-05 2024-09-21T22:07:43.950 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.950 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.950 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:43.951 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.733378+0000", 2024-09-21T22:07:43.951 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8121999999999997e-05 2024-09-21T22:07:43.951 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:43.951 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:43.951 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:43.951 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.951 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.951 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4840.0:680 2.7 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:07:43.951 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.744506+0000", 2024-09-21T22:07:43.952 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 19.190750233999999, 2024-09-21T22:07:43.952 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.063865055000000004, 2024-09-21T22:07:43.952 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:43.952 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:43.952 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:43.952 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:43.952 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4840", 2024-09-21T22:07:43.952 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:43.952 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 680 2024-09-21T22:07:43.952 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.952 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:43.952 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.952 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:43.953 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.744506+0000", 2024-09-21T22:07:43.953 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:43.953 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.953 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.953 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:43.953 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.744506+0000", 2024-09-21T22:07:43.953 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:43.953 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.954 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.954 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:43.954 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.744508+0000", 2024-09-21T22:07:43.954 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.103e-06 2024-09-21T22:07:43.954 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.954 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.954 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:43.954 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.745371+0000", 2024-09-21T22:07:43.954 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00086322199999999997 2024-09-21T22:07:43.954 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.954 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.955 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:43.955 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.745401+0000", 2024-09-21T22:07:43.955 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9955999999999999e-05 2024-09-21T22:07:43.955 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.955 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.955 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:43.955 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.745416+0000", 2024-09-21T22:07:43.955 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4873e-05 2024-09-21T22:07:43.956 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.956 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.956 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:43.956 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.745509+0000", 2024-09-21T22:07:43.956 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3171000000000006e-05 2024-09-21T22:07:43.956 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.956 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.956 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:43.956 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.745632+0000", 2024-09-21T22:07:43.956 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012272699999999999 2024-09-21T22:07:43.956 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.957 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.957 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-09-21T22:07:43.957 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.745789+0000", 2024-09-21T22:07:43.957 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015781399999999999 2024-09-21T22:07:43.957 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.957 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.957 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:43.957 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.785494+0000", 2024-09-21T22:07:43.958 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.039704401 2024-09-21T22:07:43.958 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.958 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.958 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:43.958 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.808090+0000", 2024-09-21T22:07:43.958 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022596207 2024-09-21T22:07:43.958 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.958 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.958 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:43.958 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.808246+0000", 2024-09-21T22:07:43.958 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015567200000000001 2024-09-21T22:07:43.959 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.959 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.959 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:43.959 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.808371+0000", 2024-09-21T22:07:43.959 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012490900000000001 2024-09-21T22:07:43.959 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:43.959 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:43.960 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:43.960 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.960 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.960 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4840.0:682 2.7 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:07:43.960 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.822415+0000", 2024-09-21T22:07:43.960 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 19.112840671000001, 2024-09-21T22:07:43.960 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.065044633000000004, 2024-09-21T22:07:43.960 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:43.960 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:43.961 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:43.961 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:43.961 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4840", 2024-09-21T22:07:43.961 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:43.961 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 682 2024-09-21T22:07:43.961 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.961 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:43.962 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.962 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:43.962 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.822415+0000", 2024-09-21T22:07:43.962 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:43.962 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.962 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.962 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:43.962 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.822415+0000", 2024-09-21T22:07:43.962 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:43.963 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.963 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.963 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:43.963 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.822417+0000", 2024-09-21T22:07:43.963 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8029999999999999e-06 2024-09-21T22:07:43.963 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.963 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.963 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:43.963 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.823308+0000", 2024-09-21T22:07:43.963 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00089146500000000003 2024-09-21T22:07:43.963 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.963 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.964 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:43.964 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.823339+0000", 2024-09-21T22:07:43.964 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0493000000000001e-05 2024-09-21T22:07:43.964 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.964 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.964 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:43.964 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.823354+0000", 2024-09-21T22:07:43.964 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5472999999999999e-05 2024-09-21T22:07:43.964 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.964 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.964 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:43.964 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.823476+0000", 2024-09-21T22:07:43.964 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000121209 2024-09-21T22:07:43.965 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.965 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.965 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:43.965 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.823650+0000", 2024-09-21T22:07:43.965 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017394599999999999 2024-09-21T22:07:43.965 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.965 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.965 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-09-21T22:07:43.965 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.823873+0000", 2024-09-21T22:07:43.965 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022314899999999999 2024-09-21T22:07:43.965 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.965 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.965 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:43.965 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.884396+0000", 2024-09-21T22:07:43.966 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.060523674999999999 2024-09-21T22:07:43.966 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.966 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.966 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:43.966 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.887238+0000", 2024-09-21T22:07:43.966 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028420450000000001 2024-09-21T22:07:43.966 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.966 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.966 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:43.966 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.887369+0000", 2024-09-21T22:07:43.966 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013010699999999999 2024-09-21T22:07:43.966 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.966 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.967 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:43.967 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.887460+0000", 2024-09-21T22:07:43.967 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1267999999999994e-05 2024-09-21T22:07:43.967 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:43.967 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:43.967 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:43.967 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.967 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.968 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4840.0:692 2.5 2:b1d55633:::rbd_data.126483d75f2d.000000000000000b:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:07:43.968 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:25.091048+0000", 2024-09-21T22:07:43.968 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 18.844208136999999, 2024-09-21T22:07:43.968 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.060917092999999999, 2024-09-21T22:07:43.968 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:43.968 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:43.968 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:43.968 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:43.968 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4840", 2024-09-21T22:07:43.968 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:43.968 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 692 2024-09-21T22:07:43.968 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.968 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:43.968 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.968 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:43.969 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.091048+0000", 2024-09-21T22:07:43.969 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:43.969 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.969 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.969 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:43.969 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.091048+0000", 2024-09-21T22:07:43.969 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:43.969 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.969 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.969 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:43.969 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.091050+0000", 2024-09-21T22:07:43.969 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.03e-06 2024-09-21T22:07:43.969 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.970 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.970 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:43.970 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.091989+0000", 2024-09-21T22:07:43.970 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00093932100000000001 2024-09-21T22:07:43.970 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.970 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.970 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:43.970 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.092023+0000", 2024-09-21T22:07:43.971 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4310000000000002e-05 2024-09-21T22:07:43.971 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.971 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.971 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:43.971 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.092040+0000", 2024-09-21T22:07:43.971 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6426000000000002e-05 2024-09-21T22:07:43.971 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.971 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.971 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:43.971 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.092175+0000", 2024-09-21T22:07:43.971 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013500299999999999 2024-09-21T22:07:43.971 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.972 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.972 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:43.972 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.092307+0000", 2024-09-21T22:07:43.972 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000132562 2024-09-21T22:07:43.972 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.972 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.972 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:43.972 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.092499+0000", 2024-09-21T22:07:43.972 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000191972 2024-09-21T22:07:43.972 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.972 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.972 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:43.973 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.151065+0000", 2024-09-21T22:07:43.973 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058565514999999999 2024-09-21T22:07:43.973 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.973 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.973 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:43.973 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.151844+0000", 2024-09-21T22:07:43.973 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00077930800000000004 2024-09-21T22:07:43.973 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.973 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.973 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:43.973 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.151916+0000", 2024-09-21T22:07:43.973 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1921999999999993e-05 2024-09-21T22:07:43.974 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.974 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.974 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:43.974 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.151965+0000", 2024-09-21T22:07:43.974 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8724000000000003e-05 2024-09-21T22:07:43.974 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:43.974 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:43.974 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:43.974 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.974 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.974 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4840.0:694 2.5 2:b1d55633:::rbd_data.126483d75f2d.000000000000000b:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:07:43.974 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:25.162829+0000", 2024-09-21T22:07:43.974 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 18.772426803999998, 2024-09-21T22:07:43.975 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.045653405000000001, 2024-09-21T22:07:43.975 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:43.975 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:43.975 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:43.975 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:43.975 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4840", 2024-09-21T22:07:43.975 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:43.975 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 694 2024-09-21T22:07:43.975 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.975 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:43.975 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.975 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:43.976 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.162829+0000", 2024-09-21T22:07:43.976 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:43.976 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.976 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.976 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:43.976 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.162829+0000", 2024-09-21T22:07:43.976 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:43.976 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.976 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.976 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:43.976 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.162831+0000", 2024-09-21T22:07:43.977 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7799999999999999e-06 2024-09-21T22:07:43.977 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.977 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.977 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:43.977 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.163904+0000", 2024-09-21T22:07:43.977 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001073673 2024-09-21T22:07:43.977 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.977 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.977 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:43.977 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.163937+0000", 2024-09-21T22:07:43.977 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2100000000000001e-05 2024-09-21T22:07:43.978 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.978 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.978 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:43.978 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.163952+0000", 2024-09-21T22:07:43.978 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5517000000000001e-05 2024-09-21T22:07:43.978 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.978 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.978 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:43.978 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.164060+0000", 2024-09-21T22:07:43.978 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000107909 2024-09-21T22:07:43.978 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.979 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.979 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:43.979 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.164162+0000", 2024-09-21T22:07:43.979 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000101619 2024-09-21T22:07:43.979 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.979 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.979 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:43.979 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.164318+0000", 2024-09-21T22:07:43.979 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015683599999999999 2024-09-21T22:07:43.979 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.979 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.979 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:43.980 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.207369+0000", 2024-09-21T22:07:43.980 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.043050401000000002 2024-09-21T22:07:43.980 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.980 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.980 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:43.980 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.208256+0000", 2024-09-21T22:07:43.980 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00088747800000000003 2024-09-21T22:07:43.980 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.980 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.980 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:43.980 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.208350+0000", 2024-09-21T22:07:43.980 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4129000000000006e-05 2024-09-21T22:07:43.981 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.981 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.981 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:43.981 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.208482+0000", 2024-09-21T22:07:43.981 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000131963 2024-09-21T22:07:43.981 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:43.981 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:43.981 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:43.981 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.982 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.982 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4840.0:696 2.5 2:b1d55633:::rbd_data.126483d75f2d.000000000000000b:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:07:43.982 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:25.218641+0000", 2024-09-21T22:07:43.982 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 18.716614869000001, 2024-09-21T22:07:43.982 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058363669999999999, 2024-09-21T22:07:43.982 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:43.982 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:43.982 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:43.982 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:43.982 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4840", 2024-09-21T22:07:43.983 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:43.983 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 696 2024-09-21T22:07:43.983 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.983 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:43.983 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.983 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:43.983 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.218641+0000", 2024-09-21T22:07:43.983 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:43.983 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.983 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.983 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:43.983 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.218641+0000", 2024-09-21T22:07:43.983 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:43.983 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.984 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.984 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:43.984 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.218642+0000", 2024-09-21T22:07:43.984 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.266e-06 2024-09-21T22:07:43.984 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.984 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.984 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:43.984 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.219432+0000", 2024-09-21T22:07:43.984 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00078986300000000002 2024-09-21T22:07:43.984 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.984 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.984 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:43.984 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.219450+0000", 2024-09-21T22:07:43.985 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7526000000000001e-05 2024-09-21T22:07:43.985 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.985 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.985 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:43.985 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.219459+0000", 2024-09-21T22:07:43.985 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3000000000000007e-06 2024-09-21T22:07:43.985 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.985 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.985 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:43.985 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.219655+0000", 2024-09-21T22:07:43.985 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019599099999999999 2024-09-21T22:07:43.985 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.985 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.985 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:43.986 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.219917+0000", 2024-09-21T22:07:43.986 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026198200000000001 2024-09-21T22:07:43.986 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.986 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.986 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:43.986 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.220198+0000", 2024-09-21T22:07:43.986 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00028154500000000002 2024-09-21T22:07:43.986 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.986 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.986 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:43.986 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.268856+0000", 2024-09-21T22:07:43.986 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.048657687999999998 2024-09-21T22:07:43.987 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.987 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.987 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:43.987 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.276903+0000", 2024-09-21T22:07:43.987 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0080472249999999999 2024-09-21T22:07:43.987 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.987 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.987 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:43.987 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.276958+0000", 2024-09-21T22:07:43.987 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4184999999999997e-05 2024-09-21T22:07:43.987 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.987 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.987 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:43.988 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.277005+0000", 2024-09-21T22:07:43.988 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7098999999999997e-05 2024-09-21T22:07:43.988 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:43.988 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:43.988 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:43.988 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.988 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.988 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4840.0:698 2.5 2:b1d55633:::rbd_data.126483d75f2d.000000000000000b:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:07:43.988 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:25.289380+0000", 2024-09-21T22:07:43.988 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 18.645876321999999, 2024-09-21T22:07:43.988 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.055691072000000001, 2024-09-21T22:07:43.988 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:43.989 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:43.989 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:43.989 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:43.989 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4840", 2024-09-21T22:07:43.989 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:43.989 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 698 2024-09-21T22:07:43.989 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.989 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:43.989 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.989 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:43.989 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.289380+0000", 2024-09-21T22:07:43.989 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:43.990 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.990 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.990 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:43.990 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.289380+0000", 2024-09-21T22:07:43.990 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:43.990 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.990 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.990 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:43.990 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.289381+0000", 2024-09-21T22:07:43.991 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3540000000000001e-06 2024-09-21T22:07:43.991 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.991 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.991 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:43.991 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.290221+0000", 2024-09-21T22:07:43.991 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00083999900000000002 2024-09-21T22:07:43.991 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.991 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.991 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:43.991 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.290238+0000", 2024-09-21T22:07:43.991 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7130999999999999e-05 2024-09-21T22:07:43.991 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.992 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.992 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:43.992 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.290247+0000", 2024-09-21T22:07:43.992 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6500000000000002e-06 2024-09-21T22:07:43.992 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.992 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.992 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:43.992 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.290320+0000", 2024-09-21T22:07:43.992 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3530999999999998e-05 2024-09-21T22:07:43.992 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.993 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.993 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:43.993 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.290433+0000", 2024-09-21T22:07:43.993 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000112744 2024-09-21T22:07:43.993 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.993 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.993 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:43.993 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.290605+0000", 2024-09-21T22:07:43.993 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017253700000000001 2024-09-21T22:07:43.993 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.993 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.994 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:43.994 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.320545+0000", 2024-09-21T22:07:43.994 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029939582999999999 2024-09-21T22:07:43.994 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.994 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.994 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:43.994 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.344991+0000", 2024-09-21T22:07:43.994 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024445956000000001 2024-09-21T22:07:43.994 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.994 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.995 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:43.995 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.345034+0000", 2024-09-21T22:07:43.995 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2806000000000003e-05 2024-09-21T22:07:43.995 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.995 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.995 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:43.995 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.345071+0000", 2024-09-21T22:07:43.995 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6780999999999999e-05 2024-09-21T22:07:43.995 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:43.995 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:43.995 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:43.995 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.995 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.996 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4840.0:725 2.0 2:08b4f96a:::rbd_data.126483d75f2d.000000000000000f:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:07:43.996 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:25.936440+0000", 2024-09-21T22:07:43.996 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 17.998815824000001, 2024-09-21T22:07:43.996 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.049962899999999998, 2024-09-21T22:07:43.996 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:43.996 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:43.996 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:43.996 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:43.996 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4840", 2024-09-21T22:07:43.996 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:43.996 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 725 2024-09-21T22:07:43.997 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.997 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:43.997 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.997 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:43.997 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.936440+0000", 2024-09-21T22:07:43.997 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:43.997 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.997 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.997 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:43.997 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.936440+0000", 2024-09-21T22:07:43.997 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:43.997 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.998 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.998 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:43.998 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.936442+0000", 2024-09-21T22:07:43.998 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0360000000000001e-06 2024-09-21T22:07:43.998 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.998 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.998 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:43.998 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.937560+0000", 2024-09-21T22:07:43.998 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011176300000000001 2024-09-21T22:07:43.998 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.998 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.998 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:43.998 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.937592+0000", 2024-09-21T22:07:43.998 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2743000000000001e-05 2024-09-21T22:07:43.999 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.999 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.999 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:43.999 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.937608+0000", 2024-09-21T22:07:43.999 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5477000000000001e-05 2024-09-21T22:07:43.999 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.999 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.999 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:43.999 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.937695+0000", 2024-09-21T22:07:43.999 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7220999999999997e-05 2024-09-21T22:07:43.999 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:43.999 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:43.999 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:43.999 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.937792+0000", 2024-09-21T22:07:43.999 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7261000000000003e-05 2024-09-21T22:07:44.000 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.000 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.000 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-09-21T22:07:44.000 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.937934+0000", 2024-09-21T22:07:44.000 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000141376 2024-09-21T22:07:44.000 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.000 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.000 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:44.000 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.975222+0000", 2024-09-21T22:07:44.000 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.037287995999999997 2024-09-21T22:07:44.000 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.000 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.000 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:44.001 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.986285+0000", 2024-09-21T22:07:44.001 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011063329 2024-09-21T22:07:44.001 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.001 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.001 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:44.001 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.986348+0000", 2024-09-21T22:07:44.001 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3330000000000005e-05 2024-09-21T22:07:44.001 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.001 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.001 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:44.001 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.986403+0000", 2024-09-21T22:07:44.001 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4500999999999999e-05 2024-09-21T22:07:44.001 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:44.002 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:44.002 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:44.002 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.002 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.002 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4840.0:727 2.0 2:08b4f96a:::rbd_data.126483d75f2d.000000000000000f:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:07:44.002 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:25.998176+0000", 2024-09-21T22:07:44.002 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 17.937079626999999, 2024-09-21T22:07:44.002 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.054236470000000002, 2024-09-21T22:07:44.002 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:44.002 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:44.002 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:44.003 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:44.003 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4840", 2024-09-21T22:07:44.003 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:44.003 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 727 2024-09-21T22:07:44.003 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.003 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:44.003 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.003 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:44.003 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.998176+0000", 2024-09-21T22:07:44.004 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:44.004 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.004 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.004 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:44.004 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.998176+0000", 2024-09-21T22:07:44.004 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:44.004 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.004 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.004 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:44.004 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.998179+0000", 2024-09-21T22:07:44.004 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3870000000000002e-06 2024-09-21T22:07:44.004 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.005 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.005 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:44.005 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.999060+0000", 2024-09-21T22:07:44.005 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00088148800000000004 2024-09-21T22:07:44.005 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.005 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.005 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:44.005 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.999094+0000", 2024-09-21T22:07:44.005 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3779000000000003e-05 2024-09-21T22:07:44.005 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.006 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.006 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:44.006 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.999111+0000", 2024-09-21T22:07:44.006 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6990000000000002e-05 2024-09-21T22:07:44.006 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.006 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.006 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:44.006 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.999190+0000", 2024-09-21T22:07:44.006 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9354e-05 2024-09-21T22:07:44.006 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.006 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.006 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:44.007 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.999251+0000", 2024-09-21T22:07:44.007 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0733000000000003e-05 2024-09-21T22:07:44.007 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.007 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.007 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-09-21T22:07:44.007 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.999342+0000", 2024-09-21T22:07:44.007 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1376000000000001e-05 2024-09-21T22:07:44.007 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.007 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.007 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:44.007 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.032392+0000", 2024-09-21T22:07:44.008 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.033049626999999998 2024-09-21T22:07:44.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.008 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:44.008 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.052249+0000", 2024-09-21T22:07:44.008 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019856944000000001 2024-09-21T22:07:44.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.008 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:44.008 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.052342+0000", 2024-09-21T22:07:44.008 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2650000000000002e-05 2024-09-21T22:07:44.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.009 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.009 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:44.009 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.052413+0000", 2024-09-21T22:07:44.009 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1142000000000004e-05 2024-09-21T22:07:44.009 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:44.009 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:44.009 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:44.009 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.009 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.009 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4840.0:729 2.0 2:08b4f96a:::rbd_data.126483d75f2d.000000000000000f:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:07:44.009 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:26.062012+0000", 2024-09-21T22:07:44.010 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 17.873243636000002, 2024-09-21T22:07:44.010 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.060656823999999998, 2024-09-21T22:07:44.010 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:44.010 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:44.010 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:44.010 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:44.010 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4840", 2024-09-21T22:07:44.010 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:44.010 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 729 2024-09-21T22:07:44.010 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.010 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:44.010 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.010 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:44.011 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.062012+0000", 2024-09-21T22:07:44.011 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:44.011 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.011 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.011 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:44.011 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.062012+0000", 2024-09-21T22:07:44.011 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:44.011 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.011 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.011 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:44.011 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.062013+0000", 2024-09-21T22:07:44.011 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.243e-06 2024-09-21T22:07:44.011 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.012 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:44.012 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.062827+0000", 2024-09-21T22:07:44.012 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00081374799999999999 2024-09-21T22:07:44.012 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.012 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:44.012 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.062845+0000", 2024-09-21T22:07:44.012 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.808e-05 2024-09-21T22:07:44.012 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.012 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:44.012 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.062854+0000", 2024-09-21T22:07:44.012 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1039999999999996e-06 2024-09-21T22:07:44.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.013 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.013 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:44.013 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.062970+0000", 2024-09-21T22:07:44.013 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011601 2024-09-21T22:07:44.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.013 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.013 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:44.013 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.063230+0000", 2024-09-21T22:07:44.013 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025918799999999999 2024-09-21T22:07:44.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.013 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.013 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-09-21T22:07:44.013 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.063641+0000", 2024-09-21T22:07:44.014 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00041152799999999998 2024-09-21T22:07:44.014 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.014 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.014 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:44.014 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.122015+0000", 2024-09-21T22:07:44.014 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058374246999999997 2024-09-21T22:07:44.014 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.014 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.014 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:44.014 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.122451+0000", 2024-09-21T22:07:44.014 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000435771 2024-09-21T22:07:44.014 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.015 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.015 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:44.015 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.122576+0000", 2024-09-21T22:07:44.015 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012469099999999999 2024-09-21T22:07:44.015 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.015 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.015 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:44.015 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.122669+0000", 2024-09-21T22:07:44.015 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3214000000000006e-05 2024-09-21T22:07:44.015 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:44.015 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:44.015 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:44.016 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.016 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.016 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4840.0:731 2.0 2:08b4f96a:::rbd_data.126483d75f2d.000000000000000f:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:07:44.016 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:26.135603+0000", 2024-09-21T22:07:44.016 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 17.799653136, 2024-09-21T22:07:44.016 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.056838474999999999, 2024-09-21T22:07:44.016 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:44.016 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:44.016 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:44.016 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:44.017 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4840", 2024-09-21T22:07:44.017 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:44.017 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 731 2024-09-21T22:07:44.017 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.017 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:44.017 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.017 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:44.017 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.135603+0000", 2024-09-21T22:07:44.017 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:44.017 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.017 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.018 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:44.018 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.135603+0000", 2024-09-21T22:07:44.018 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:44.018 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.018 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.018 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:44.018 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.135605+0000", 2024-09-21T22:07:44.018 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9700000000000002e-06 2024-09-21T22:07:44.019 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.019 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.019 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:44.019 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.136554+0000", 2024-09-21T22:07:44.019 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00094886700000000005 2024-09-21T22:07:44.019 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.019 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.019 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:44.019 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.136582+0000", 2024-09-21T22:07:44.020 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8393000000000001e-05 2024-09-21T22:07:44.020 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.020 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.020 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:44.020 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.136596+0000", 2024-09-21T22:07:44.020 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.43e-05 2024-09-21T22:07:44.020 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.020 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.020 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:44.020 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.136691+0000", 2024-09-21T22:07:44.021 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4843e-05 2024-09-21T22:07:44.021 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.021 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.021 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:44.021 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.136873+0000", 2024-09-21T22:07:44.021 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018237099999999999 2024-09-21T22:07:44.021 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.021 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.021 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-09-21T22:07:44.021 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.137140+0000", 2024-09-21T22:07:44.021 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026642700000000002 2024-09-21T22:07:44.022 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.022 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.022 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:44.022 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.180580+0000", 2024-09-21T22:07:44.022 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.043440078 2024-09-21T22:07:44.022 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.022 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.022 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:44.022 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.192175+0000", 2024-09-21T22:07:44.022 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011594959 2024-09-21T22:07:44.023 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.023 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.023 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:44.023 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.192353+0000", 2024-09-21T22:07:44.023 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017850200000000001 2024-09-21T22:07:44.023 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.023 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.023 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:44.023 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.192441+0000", 2024-09-21T22:07:44.023 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7764999999999997e-05 2024-09-21T22:07:44.023 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:44.023 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:44.023 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:44.023 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.023 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.024 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5008.0:116 2.7 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-21T22:07:44.024 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:37.866431+0000", 2024-09-21T22:07:44.024 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 6.0688248639999998, 2024-09-21T22:07:44.024 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.046674751, 2024-09-21T22:07:44.024 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:44.024 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:44.024 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:44.024 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:44.024 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5008", 2024-09-21T22:07:44.024 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/1400114666", 2024-09-21T22:07:44.025 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 116 2024-09-21T22:07:44.025 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.025 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:44.025 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.025 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:44.025 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.866431+0000", 2024-09-21T22:07:44.025 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:44.025 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.025 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.025 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:44.025 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.866431+0000", 2024-09-21T22:07:44.025 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:44.025 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.025 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.026 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:44.026 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.866433+0000", 2024-09-21T22:07:44.026 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.592e-06 2024-09-21T22:07:44.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.026 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.026 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:44.026 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.867223+0000", 2024-09-21T22:07:44.026 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00079020900000000005 2024-09-21T22:07:44.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.026 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.026 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:44.026 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.867243+0000", 2024-09-21T22:07:44.026 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0333e-05 2024-09-21T22:07:44.027 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.027 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.027 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:44.027 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.867254+0000", 2024-09-21T22:07:44.027 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0558e-05 2024-09-21T22:07:44.027 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.027 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.027 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:44.027 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.867324+0000", 2024-09-21T22:07:44.027 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0290999999999995e-05 2024-09-21T22:07:44.027 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.027 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.027 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:44.027 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.867398+0000", 2024-09-21T22:07:44.028 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4509000000000001e-05 2024-09-21T22:07:44.028 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.028 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.028 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-09-21T22:07:44.028 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.867514+0000", 2024-09-21T22:07:44.028 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011581799999999999 2024-09-21T22:07:44.028 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.028 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.028 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:44.028 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.899084+0000", 2024-09-21T22:07:44.028 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.031569901999999997 2024-09-21T22:07:44.028 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.029 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.029 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:44.029 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.913011+0000", 2024-09-21T22:07:44.029 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013926945 2024-09-21T22:07:44.029 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.029 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.029 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:44.029 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.913063+0000", 2024-09-21T22:07:44.029 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2352999999999998e-05 2024-09-21T22:07:44.029 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.029 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.029 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:44.029 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.913106+0000", 2024-09-21T22:07:44.029 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2240999999999998e-05 2024-09-21T22:07:44.029 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:44.030 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:44.030 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:44.030 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.030 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.030 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5008.0:143 2.5 2:a70ae99f:::rbd_data.126483d75f2d.000000000000000c:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e72)", 2024-09-21T22:07:44.030 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:38.527661+0000", 2024-09-21T22:07:44.030 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 5.4075950539999997, 2024-09-21T22:07:44.030 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.042001680999999999, 2024-09-21T22:07:44.030 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:44.030 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:44.030 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:44.031 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:44.031 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5008", 2024-09-21T22:07:44.031 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/1400114666", 2024-09-21T22:07:44.031 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 143 2024-09-21T22:07:44.031 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.031 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:44.031 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.031 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:44.031 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.527661+0000", 2024-09-21T22:07:44.031 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:44.031 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.031 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.031 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:44.032 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.527661+0000", 2024-09-21T22:07:44.032 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:44.032 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.032 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.032 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:44.032 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.527662+0000", 2024-09-21T22:07:44.032 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1540000000000001e-06 2024-09-21T22:07:44.032 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.032 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.032 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:44.032 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.528503+0000", 2024-09-21T22:07:44.032 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00084101800000000002 2024-09-21T22:07:44.032 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.033 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.033 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:44.033 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.528523+0000", 2024-09-21T22:07:44.033 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0322e-05 2024-09-21T22:07:44.033 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.033 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.033 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:44.033 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.528534+0000", 2024-09-21T22:07:44.033 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0564000000000001e-05 2024-09-21T22:07:44.033 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.033 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.033 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:44.034 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.528636+0000", 2024-09-21T22:07:44.034 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010234300000000001 2024-09-21T22:07:44.034 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.034 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.034 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:44.034 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.528736+0000", 2024-09-21T22:07:44.034 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9910999999999999e-05 2024-09-21T22:07:44.034 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.034 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.034 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:44.034 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.528896+0000", 2024-09-21T22:07:44.034 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016024 2024-09-21T22:07:44.034 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.035 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:44.035 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.566383+0000", 2024-09-21T22:07:44.035 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.037487174999999998 2024-09-21T22:07:44.035 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.035 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:44.035 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.569570+0000", 2024-09-21T22:07:44.035 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0031864269999999999 2024-09-21T22:07:44.035 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.036 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:44.036 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.569620+0000", 2024-09-21T22:07:44.036 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0321000000000002e-05 2024-09-21T22:07:44.036 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.036 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.036 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:44.036 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.569662+0000", 2024-09-21T22:07:44.036 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2206000000000002e-05 2024-09-21T22:07:44.036 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:44.036 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:44.036 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:44.036 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.037 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.037 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5008.0:147 2.5 2:a70ae99f:::rbd_data.126483d75f2d.000000000000000c:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e72)", 2024-09-21T22:07:44.037 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:38.627134+0000", 2024-09-21T22:07:44.037 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 5.3081219669999999, 2024-09-21T22:07:44.037 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.052909400000000002, 2024-09-21T22:07:44.037 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:44.037 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:44.037 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:44.037 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:44.037 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5008", 2024-09-21T22:07:44.037 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/1400114666", 2024-09-21T22:07:44.038 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 147 2024-09-21T22:07:44.038 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.038 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:44.038 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.038 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:44.038 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.627134+0000", 2024-09-21T22:07:44.038 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:44.038 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.038 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.038 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:44.038 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.627134+0000", 2024-09-21T22:07:44.039 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:44.039 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.039 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.039 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:44.039 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.627136+0000", 2024-09-21T22:07:44.039 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2129999999999998e-06 2024-09-21T22:07:44.039 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.039 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.039 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:44.039 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.628233+0000", 2024-09-21T22:07:44.040 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001097223 2024-09-21T22:07:44.040 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.040 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.040 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:44.040 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.628266+0000", 2024-09-21T22:07:44.040 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2382999999999998e-05 2024-09-21T22:07:44.040 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.040 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.040 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:44.041 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.628281+0000", 2024-09-21T22:07:44.041 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5417000000000002e-05 2024-09-21T22:07:44.041 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.041 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.041 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:44.041 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.628412+0000", 2024-09-21T22:07:44.041 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013106700000000001 2024-09-21T22:07:44.041 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.041 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.041 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:44.041 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.628529+0000", 2024-09-21T22:07:44.042 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000117269 2024-09-21T22:07:44.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.042 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.042 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:44.042 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.628704+0000", 2024-09-21T22:07:44.042 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017447599999999999 2024-09-21T22:07:44.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.042 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.042 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:44.042 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.676772+0000", 2024-09-21T22:07:44.043 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.048068196000000001 2024-09-21T22:07:44.043 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.043 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.043 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:44.043 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.679874+0000", 2024-09-21T22:07:44.043 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0031019770000000001 2024-09-21T22:07:44.043 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.043 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.043 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:44.043 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.679974+0000", 2024-09-21T22:07:44.043 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9579000000000003e-05 2024-09-21T22:07:44.044 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.044 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.044 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:44.044 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.680043+0000", 2024-09-21T22:07:44.044 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9599999999999998e-05 2024-09-21T22:07:44.044 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:44.044 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:44.044 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:44.044 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.044 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.044 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5008.0:149 2.5 2:a70ae99f:::rbd_data.126483d75f2d.000000000000000c:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e72)", 2024-09-21T22:07:44.045 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:38.690581+0000", 2024-09-21T22:07:44.045 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 5.2446750350000002, 2024-09-21T22:07:44.045 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.051360648000000002, 2024-09-21T22:07:44.045 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:44.045 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:44.045 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:44.045 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:44.045 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5008", 2024-09-21T22:07:44.045 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/1400114666", 2024-09-21T22:07:44.045 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 149 2024-09-21T22:07:44.045 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.045 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:44.046 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.046 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:44.046 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.690581+0000", 2024-09-21T22:07:44.046 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:44.046 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.046 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.046 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:44.046 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.690581+0000", 2024-09-21T22:07:44.046 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:44.046 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.046 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.047 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:44.047 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.690583+0000", 2024-09-21T22:07:44.047 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3e-06 2024-09-21T22:07:44.047 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.047 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.047 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:44.047 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.691373+0000", 2024-09-21T22:07:44.047 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00078976800000000005 2024-09-21T22:07:44.047 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.047 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.047 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:44.048 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.691406+0000", 2024-09-21T22:07:44.048 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2787000000000003e-05 2024-09-21T22:07:44.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.048 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:44.048 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.691422+0000", 2024-09-21T22:07:44.048 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6443000000000002e-05 2024-09-21T22:07:44.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.048 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:44.048 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.691597+0000", 2024-09-21T22:07:44.048 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017483599999999999 2024-09-21T22:07:44.049 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.049 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.049 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:44.049 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.691737+0000", 2024-09-21T22:07:44.049 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014013900000000001 2024-09-21T22:07:44.049 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.049 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.049 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:44.049 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.691914+0000", 2024-09-21T22:07:44.049 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017682 2024-09-21T22:07:44.049 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.049 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.049 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:44.050 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.732757+0000", 2024-09-21T22:07:44.050 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.040843438000000003 2024-09-21T22:07:44.050 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.050 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.050 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:44.050 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.741844+0000", 2024-09-21T22:07:44.050 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0090864139999999993 2024-09-21T22:07:44.050 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.050 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.050 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:44.050 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.741898+0000", 2024-09-21T22:07:44.050 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4336000000000003e-05 2024-09-21T22:07:44.050 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.051 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.051 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:44.051 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.741941+0000", 2024-09-21T22:07:44.051 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3367e-05 2024-09-21T22:07:44.051 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:44.051 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:44.051 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:44.051 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.051 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.051 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5008.0:170 2.0 2:08b4f96a:::rbd_data.126483d75f2d.000000000000000f:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e72)", 2024-09-21T22:07:44.051 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:39.132043+0000", 2024-09-21T22:07:44.051 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.8032131099999997, 2024-09-21T22:07:44.052 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.050566569999999998, 2024-09-21T22:07:44.052 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:44.052 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:44.052 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:44.052 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:44.052 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5008", 2024-09-21T22:07:44.052 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/1400114666", 2024-09-21T22:07:44.052 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 170 2024-09-21T22:07:44.052 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.052 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:44.052 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.052 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:44.052 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.132043+0000", 2024-09-21T22:07:44.052 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:44.053 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.053 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.053 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:44.053 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.132043+0000", 2024-09-21T22:07:44.053 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:44.053 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.053 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.053 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:44.053 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.132065+0000", 2024-09-21T22:07:44.053 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1999999999999999e-05 2024-09-21T22:07:44.053 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.053 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.054 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:44.054 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.132972+0000", 2024-09-21T22:07:44.054 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00090684399999999997 2024-09-21T22:07:44.054 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.054 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.054 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:44.054 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.133002+0000", 2024-09-21T22:07:44.054 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0936000000000003e-05 2024-09-21T22:07:44.054 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.054 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.054 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:44.054 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.133019+0000", 2024-09-21T22:07:44.055 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6004000000000001e-05 2024-09-21T22:07:44.055 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.055 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.055 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:44.055 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.133170+0000", 2024-09-21T22:07:44.055 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015154099999999999 2024-09-21T22:07:44.055 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.055 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.055 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:44.055 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.133269+0000", 2024-09-21T22:07:44.055 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8876999999999995e-05 2024-09-21T22:07:44.056 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.056 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.056 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-09-21T22:07:44.056 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.133413+0000", 2024-09-21T22:07:44.056 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000143931 2024-09-21T22:07:44.056 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.056 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.056 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:44.056 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.181524+0000", 2024-09-21T22:07:44.056 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.048111196000000002 2024-09-21T22:07:44.056 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.056 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.057 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:44.057 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.182494+0000", 2024-09-21T22:07:44.057 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00096976799999999998 2024-09-21T22:07:44.057 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.057 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.057 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:44.057 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.182564+0000", 2024-09-21T22:07:44.057 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0275000000000001e-05 2024-09-21T22:07:44.057 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.057 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.057 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:44.057 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.182609+0000", 2024-09-21T22:07:44.057 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5198000000000003e-05 2024-09-21T22:07:44.058 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:44.058 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:44.058 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:44.058 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.058 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.058 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5008.0:172 2.0 2:08b4f96a:::rbd_data.126483d75f2d.000000000000000f:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e72)", 2024-09-21T22:07:44.058 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:39.193798+0000", 2024-09-21T22:07:44.058 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.7414582279999999, 2024-09-21T22:07:44.058 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.050829119999999998, 2024-09-21T22:07:44.058 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:44.058 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:44.058 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:44.059 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:44.059 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5008", 2024-09-21T22:07:44.059 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/1400114666", 2024-09-21T22:07:44.059 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 172 2024-09-21T22:07:44.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.059 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:44.059 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.059 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:44.059 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.193798+0000", 2024-09-21T22:07:44.059 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:44.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.059 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.059 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:44.059 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.193798+0000", 2024-09-21T22:07:44.060 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:44.060 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.060 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.060 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:44.060 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.193800+0000", 2024-09-21T22:07:44.060 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4770000000000002e-06 2024-09-21T22:07:44.060 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.060 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.060 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:44.060 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.194508+0000", 2024-09-21T22:07:44.061 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00070776900000000002 2024-09-21T22:07:44.061 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.061 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.061 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:44.061 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.194540+0000", 2024-09-21T22:07:44.061 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2647000000000001e-05 2024-09-21T22:07:44.061 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.061 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.061 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:44.061 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.194556+0000", 2024-09-21T22:07:44.061 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5492999999999999e-05 2024-09-21T22:07:44.061 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.062 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.062 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:44.062 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.194747+0000", 2024-09-21T22:07:44.062 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019105899999999999 2024-09-21T22:07:44.062 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.062 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.062 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:44.062 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.194886+0000", 2024-09-21T22:07:44.062 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013939599999999999 2024-09-21T22:07:44.062 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.063 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.063 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-09-21T22:07:44.063 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.195089+0000", 2024-09-21T22:07:44.063 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020208499999999999 2024-09-21T22:07:44.063 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.063 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.063 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:44.063 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.243304+0000", 2024-09-21T22:07:44.063 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.048215757999999997 2024-09-21T22:07:44.063 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.063 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.064 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:44.064 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.244466+0000", 2024-09-21T22:07:44.064 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011612969999999999 2024-09-21T22:07:44.064 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.064 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.064 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:44.064 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.244561+0000", 2024-09-21T22:07:44.064 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4995999999999996e-05 2024-09-21T22:07:44.064 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.064 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.064 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:44.065 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.244627+0000", 2024-09-21T22:07:44.065 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6142999999999993e-05 2024-09-21T22:07:44.065 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:44.065 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:44.065 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:44.065 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.065 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5008.0:174 2.0 2:08b4f96a:::rbd_data.126483d75f2d.000000000000000f:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e72)", 2024-09-21T22:07:44.065 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:39.256840+0000", 2024-09-21T22:07:44.065 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.6784153430000002, 2024-09-21T22:07:44.065 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.048802131999999998, 2024-09-21T22:07:44.066 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:44.066 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:44.066 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:44.066 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:44.066 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5008", 2024-09-21T22:07:44.066 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/1400114666", 2024-09-21T22:07:44.066 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 174 2024-09-21T22:07:44.066 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.066 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:44.066 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.066 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:44.066 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.256840+0000", 2024-09-21T22:07:44.067 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:44.067 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.067 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.067 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:44.067 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.256840+0000", 2024-09-21T22:07:44.067 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:44.067 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.067 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.067 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:44.067 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.256842+0000", 2024-09-21T22:07:44.067 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.55e-06 2024-09-21T22:07:44.068 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.068 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.068 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:44.068 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.257667+0000", 2024-09-21T22:07:44.068 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00082503100000000005 2024-09-21T22:07:44.068 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.068 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.068 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:44.068 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.257696+0000", 2024-09-21T22:07:44.068 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8629999999999999e-05 2024-09-21T22:07:44.068 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.069 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.069 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:44.069 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.257710+0000", 2024-09-21T22:07:44.069 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4323e-05 2024-09-21T22:07:44.069 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.069 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.069 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:44.069 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.257847+0000", 2024-09-21T22:07:44.069 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000137465 2024-09-21T22:07:44.069 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.069 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.069 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:44.070 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.257944+0000", 2024-09-21T22:07:44.070 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6866000000000004e-05 2024-09-21T22:07:44.070 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.070 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.070 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-09-21T22:07:44.070 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.258081+0000", 2024-09-21T22:07:44.070 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000136441 2024-09-21T22:07:44.070 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.070 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.070 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:44.071 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.299971+0000", 2024-09-21T22:07:44.071 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.041890278000000003 2024-09-21T22:07:44.071 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.071 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.071 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:44.071 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.305494+0000", 2024-09-21T22:07:44.071 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0055228320000000001 2024-09-21T22:07:44.071 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.071 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.071 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:44.071 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.305581+0000", 2024-09-21T22:07:44.071 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7157000000000006e-05 2024-09-21T22:07:44.072 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.072 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.072 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:44.072 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.305643+0000", 2024-09-21T22:07:44.072 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1558999999999998e-05 2024-09-21T22:07:44.072 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:44.072 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:44.072 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:44.072 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.072 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.072 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5008.0:176 2.0 2:08b4f96a:::rbd_data.126483d75f2d.000000000000000f:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e72)", 2024-09-21T22:07:44.072 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:39.315531+0000", 2024-09-21T22:07:44.073 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.6197245789999997, 2024-09-21T22:07:44.073 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047857522999999999, 2024-09-21T22:07:44.073 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:44.073 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:44.073 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:44.073 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:44.073 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5008", 2024-09-21T22:07:44.073 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/1400114666", 2024-09-21T22:07:44.073 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 176 2024-09-21T22:07:44.073 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.073 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:44.074 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.074 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:44.074 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.315531+0000", 2024-09-21T22:07:44.074 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:44.074 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.074 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.074 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:44.074 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.315531+0000", 2024-09-21T22:07:44.074 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:44.074 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.075 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.075 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:44.075 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.315533+0000", 2024-09-21T22:07:44.075 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2000000000000001e-06 2024-09-21T22:07:44.075 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.075 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.075 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:44.075 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.316293+0000", 2024-09-21T22:07:44.075 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00075914600000000004 2024-09-21T22:07:44.075 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.075 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.075 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:44.076 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.316325+0000", 2024-09-21T22:07:44.076 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2481999999999998e-05 2024-09-21T22:07:44.076 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.076 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.076 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:44.076 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.316340+0000", 2024-09-21T22:07:44.076 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5057e-05 2024-09-21T22:07:44.076 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.076 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.076 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:44.076 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.316486+0000", 2024-09-21T22:07:44.077 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014624000000000001 2024-09-21T22:07:44.077 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.077 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.077 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:44.077 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.316635+0000", 2024-09-21T22:07:44.077 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000148529 2024-09-21T22:07:44.077 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.077 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.077 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-09-21T22:07:44.077 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.316785+0000", 2024-09-21T22:07:44.077 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015023900000000001 2024-09-21T22:07:44.077 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.077 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.078 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:44.078 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.360089+0000", 2024-09-21T22:07:44.078 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.043303361999999998 2024-09-21T22:07:44.078 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.078 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.078 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:44.078 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.363289+0000", 2024-09-21T22:07:44.078 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0032002200000000001 2024-09-21T22:07:44.078 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.078 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.078 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:44.078 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.363347+0000", 2024-09-21T22:07:44.078 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8662000000000002e-05 2024-09-21T22:07:44.078 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.079 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.079 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:44.079 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.363389+0000", 2024-09-21T22:07:44.079 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1386e-05 2024-09-21T22:07:44.079 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:44.079 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:44.079 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:44.079 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:44.079 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:44.079 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:07:44.081 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T22:07:44.081 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:44.080+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:44.182 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:44.178+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:44.255 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:07:44.255 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:07:44.255 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-09-21T22:07:44.255 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:07:44.269 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T22:07:44.282 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:44.278+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:44.381 INFO:teuthology.orchestra.run.smithi039.stderr:noscrub is unset 2024-09-21T22:07:44.383 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:44.378+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:44.397 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:07:44.397 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:07:44.397 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-21T22:07:44.397 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-21T22:07:44.397 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:07:44.413 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T22:07:44.422 INFO:tasks.workunit.client.0.smithi157.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-09-21T22:07:44.440 INFO:tasks.workunit.client.0.smithi157.stderr:++ _sudo rbd device --device-type nbd map rbd/ns/testrbdnbd21553@snap 2024-09-21T22:07:44.440 INFO:tasks.workunit.client.0.smithi157.stderr:++ local cmd 2024-09-21T22:07:44.441 INFO:tasks.workunit.client.0.smithi157.stderr:+++ id -u 2024-09-21T22:07:44.443 INFO:tasks.workunit.client.0.smithi157.stderr:++ '[' 1000 -eq 0 ']' 2024-09-21T22:07:44.443 INFO:tasks.workunit.client.0.smithi157.stderr:+++ which rbd 2024-09-21T22:07:44.445 INFO:tasks.workunit.client.0.smithi157.stderr:++ cmd=/usr/bin/rbd 2024-09-21T22:07:44.445 INFO:tasks.workunit.client.0.smithi157.stderr:++ shift 2024-09-21T22:07:44.445 INFO:tasks.workunit.client.0.smithi157.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map rbd/ns/testrbdnbd21553@snap 2024-09-21T22:07:44.468 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T22:07:44.512 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:44.510+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:44.544 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:07:44.544 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-21T22:07:44.545 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-21T22:07:44.545 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-21T22:07:44.545 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.545 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4887.0:1 3.3 3:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-21T22:07:44.545 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:17.151514+0000", 2024-09-21T22:07:44.545 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 27.390873831, 2024-09-21T22:07:44.545 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.01157512, 2024-09-21T22:07:44.545 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:44.545 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:44.545 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:44.545 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:44.545 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4887", 2024-09-21T22:07:44.545 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/3431621434", 2024-09-21T22:07:44.545 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1 2024-09-21T22:07:44.546 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.546 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:44.546 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.546 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:44.546 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.151514+0000", 2024-09-21T22:07:44.546 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:44.546 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.546 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.546 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:44.546 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.151514+0000", 2024-09-21T22:07:44.546 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:44.546 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.546 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.547 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:44.547 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.151515+0000", 2024-09-21T22:07:44.547 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.469e-06 2024-09-21T22:07:44.547 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.547 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.547 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:44.547 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.151522+0000", 2024-09-21T22:07:44.547 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1330000000000001e-06 2024-09-21T22:07:44.547 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.547 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.547 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:44.547 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.151536+0000", 2024-09-21T22:07:44.547 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3563e-05 2024-09-21T22:07:44.547 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.548 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.548 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:44.548 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.151574+0000", 2024-09-21T22:07:44.548 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8467999999999998e-05 2024-09-21T22:07:44.548 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.548 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.548 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:44.548 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.151665+0000", 2024-09-21T22:07:44.548 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0668000000000007e-05 2024-09-21T22:07:44.548 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.548 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.548 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-21T22:07:44.548 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.151676+0000", 2024-09-21T22:07:44.548 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1443999999999999e-05 2024-09-21T22:07:44.549 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.549 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.549 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:44.549 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.161694+0000", 2024-09-21T22:07:44.549 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010017402999999999 2024-09-21T22:07:44.549 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.549 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.549 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:44.549 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.161805+0000", 2024-09-21T22:07:44.549 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011122500000000001 2024-09-21T22:07:44.549 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.549 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.549 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:44.549 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.161937+0000", 2024-09-21T22:07:44.550 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013177299999999999 2024-09-21T22:07:44.550 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.550 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.550 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:44.550 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.162555+0000", 2024-09-21T22:07:44.550 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00061786199999999995 2024-09-21T22:07:44.550 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.550 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.550 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:44.550 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.163021+0000", 2024-09-21T22:07:44.550 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000465749 2024-09-21T22:07:44.550 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.550 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.551 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:44.551 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.163059+0000", 2024-09-21T22:07:44.551 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8238000000000002e-05 2024-09-21T22:07:44.551 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.551 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.551 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:44.551 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.163089+0000", 2024-09-21T22:07:44.551 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0125e-05 2024-09-21T22:07:44.551 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:44.551 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:44.551 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:44.551 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.551 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.551 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4840.0:675 2.7 e63/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 63'56, mlcod=63'54)", 2024-09-21T22:07:44.552 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.618817+0000", 2024-09-21T22:07:44.552 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 19.923571042999999, 2024-09-21T22:07:44.552 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0060329809999999998, 2024-09-21T22:07:44.552 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:44.552 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:44.552 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:44.552 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:44.552 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.552 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:44.552 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.618817+0000", 2024-09-21T22:07:44.552 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:44.552 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.552 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.552 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:44.553 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.618817+0000", 2024-09-21T22:07:44.553 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:44.553 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.553 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.553 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:44.553 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.618819+0000", 2024-09-21T22:07:44.553 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3070000000000001e-06 2024-09-21T22:07:44.553 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.553 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.553 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:44.553 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.618934+0000", 2024-09-21T22:07:44.553 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000114669 2024-09-21T22:07:44.553 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.553 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.554 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:44.554 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.618969+0000", 2024-09-21T22:07:44.554 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5985999999999997e-05 2024-09-21T22:07:44.554 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.554 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.554 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:44.554 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.618985+0000", 2024-09-21T22:07:44.554 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5840000000000001e-05 2024-09-21T22:07:44.554 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.554 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.554 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:44.554 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.619101+0000", 2024-09-21T22:07:44.554 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000115591 2024-09-21T22:07:44.554 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.555 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.555 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:44.555 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.619136+0000", 2024-09-21T22:07:44.555 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5287000000000003e-05 2024-09-21T22:07:44.555 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.555 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.555 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:44.555 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.624766+0000", 2024-09-21T22:07:44.555 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0056301500000000004 2024-09-21T22:07:44.555 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.555 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.555 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:44.555 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.624850+0000", 2024-09-21T22:07:44.555 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3151000000000004e-05 2024-09-21T22:07:44.556 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:44.556 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:44.556 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:44.556 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.556 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.556 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4840.0:676 2.7 e63/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 63'57, mlcod=63'55)", 2024-09-21T22:07:44.556 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.627276+0000", 2024-09-21T22:07:44.556 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 19.915111283000002, 2024-09-21T22:07:44.556 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.037500787000000001, 2024-09-21T22:07:44.556 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:44.556 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:44.556 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:44.556 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:44.557 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.557 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:44.557 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.627276+0000", 2024-09-21T22:07:44.557 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:44.557 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.557 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.557 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:44.557 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.627276+0000", 2024-09-21T22:07:44.557 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:44.557 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.557 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.557 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:44.557 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.627277+0000", 2024-09-21T22:07:44.557 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.068e-06 2024-09-21T22:07:44.557 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.558 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.558 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:44.558 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.628148+0000", 2024-09-21T22:07:44.558 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00087069999999999997 2024-09-21T22:07:44.558 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.558 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.558 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:44.558 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.628170+0000", 2024-09-21T22:07:44.558 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2147999999999999e-05 2024-09-21T22:07:44.558 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.558 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.558 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:44.558 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.628179+0000", 2024-09-21T22:07:44.558 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2450000000000004e-06 2024-09-21T22:07:44.559 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.559 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.559 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:44.559 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.628279+0000", 2024-09-21T22:07:44.559 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9298999999999999e-05 2024-09-21T22:07:44.559 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.559 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.559 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:44.559 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.628305+0000", 2024-09-21T22:07:44.559 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6316999999999999e-05 2024-09-21T22:07:44.559 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.559 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.559 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:44.559 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.664714+0000", 2024-09-21T22:07:44.560 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.036408923000000003 2024-09-21T22:07:44.560 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.560 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.560 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:44.560 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.664777+0000", 2024-09-21T22:07:44.560 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3087000000000001e-05 2024-09-21T22:07:44.560 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:44.560 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:44.560 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:44.560 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.560 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.560 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4840.0:677 2.7 e63/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 63'58, mlcod=63'56)", 2024-09-21T22:07:44.560 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.667210+0000", 2024-09-21T22:07:44.560 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 19.875178043999998, 2024-09-21T22:07:44.561 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010046522, 2024-09-21T22:07:44.561 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:44.561 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:44.561 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:44.561 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:44.561 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.561 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:44.561 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.667210+0000", 2024-09-21T22:07:44.561 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:44.561 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.561 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.561 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:44.561 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.667210+0000", 2024-09-21T22:07:44.561 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:44.561 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.562 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.562 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:44.562 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.667211+0000", 2024-09-21T22:07:44.562 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3379999999999999e-06 2024-09-21T22:07:44.562 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.562 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.562 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:44.562 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.667272+0000", 2024-09-21T22:07:44.562 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1257e-05 2024-09-21T22:07:44.562 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.562 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.562 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:44.562 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.667295+0000", 2024-09-21T22:07:44.562 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2637999999999999e-05 2024-09-21T22:07:44.563 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.563 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.563 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:44.563 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.667305+0000", 2024-09-21T22:07:44.563 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0044000000000001e-05 2024-09-21T22:07:44.563 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.563 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.563 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:44.563 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.667425+0000", 2024-09-21T22:07:44.563 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000120605 2024-09-21T22:07:44.563 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.563 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.563 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:44.563 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.667476+0000", 2024-09-21T22:07:44.564 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0843000000000001e-05 2024-09-21T22:07:44.564 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.564 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.564 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:44.564 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.677180+0000", 2024-09-21T22:07:44.564 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0097035860000000002 2024-09-21T22:07:44.564 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.564 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.564 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:44.564 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.677256+0000", 2024-09-21T22:07:44.564 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6211000000000006e-05 2024-09-21T22:07:44.564 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:44.564 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:44.564 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:44.564 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.565 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.565 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4840.0:678 2.7 e63/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 63'59, mlcod=63'57)", 2024-09-21T22:07:44.565 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.679689+0000", 2024-09-21T22:07:44.565 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 19.862698782999999, 2024-09-21T22:07:44.565 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053308406000000003, 2024-09-21T22:07:44.565 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:44.565 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:44.565 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:44.565 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:44.565 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.565 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:44.565 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.679689+0000", 2024-09-21T22:07:44.565 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:44.565 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.566 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.566 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:44.566 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.679689+0000", 2024-09-21T22:07:44.566 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:44.566 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.566 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.566 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:44.566 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.679691+0000", 2024-09-21T22:07:44.566 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9400000000000001e-06 2024-09-21T22:07:44.566 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.566 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.566 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:44.566 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.680245+0000", 2024-09-21T22:07:44.566 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00055382999999999995 2024-09-21T22:07:44.567 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.567 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.567 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:44.567 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.680276+0000", 2024-09-21T22:07:44.567 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1123e-05 2024-09-21T22:07:44.567 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.567 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.567 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:44.567 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.680290+0000", 2024-09-21T22:07:44.567 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4056999999999999e-05 2024-09-21T22:07:44.567 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.567 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.567 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:44.567 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.680418+0000", 2024-09-21T22:07:44.568 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000127792 2024-09-21T22:07:44.568 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.568 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.568 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:44.568 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.680465+0000", 2024-09-21T22:07:44.568 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7360000000000001e-05 2024-09-21T22:07:44.568 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.568 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.568 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:44.568 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.732923+0000", 2024-09-21T22:07:44.568 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.052458312999999999 2024-09-21T22:07:44.568 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.568 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.568 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:44.568 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.732997+0000", 2024-09-21T22:07:44.569 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3991000000000004e-05 2024-09-21T22:07:44.569 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:44.569 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:44.569 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:44.569 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.569 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.569 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4840.0:679 2.7 e63/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 63'60, mlcod=63'58)", 2024-09-21T22:07:44.569 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.735615+0000", 2024-09-21T22:07:44.569 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 19.806772630000001, 2024-09-21T22:07:44.569 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0077878399999999999, 2024-09-21T22:07:44.569 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:44.569 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:44.569 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:44.569 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:44.570 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.570 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:44.570 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.735615+0000", 2024-09-21T22:07:44.570 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:44.570 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.570 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.570 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:44.570 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.735615+0000", 2024-09-21T22:07:44.570 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:44.570 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.570 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.570 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:44.570 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.735617+0000", 2024-09-21T22:07:44.570 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7999999999999999e-06 2024-09-21T22:07:44.571 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.571 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.571 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:44.571 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.735706+0000", 2024-09-21T22:07:44.571 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8839999999999996e-05 2024-09-21T22:07:44.571 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.571 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.571 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:44.571 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.735736+0000", 2024-09-21T22:07:44.571 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9955999999999999e-05 2024-09-21T22:07:44.571 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.571 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.571 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:44.571 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.735755+0000", 2024-09-21T22:07:44.571 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9283000000000002e-05 2024-09-21T22:07:44.572 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.572 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.572 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:44.572 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.735857+0000", 2024-09-21T22:07:44.572 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000101681 2024-09-21T22:07:44.572 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.572 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.572 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:44.572 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.735894+0000", 2024-09-21T22:07:44.572 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7487999999999998e-05 2024-09-21T22:07:44.572 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.572 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.572 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:44.572 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.743321+0000", 2024-09-21T22:07:44.572 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0074270050000000004 2024-09-21T22:07:44.573 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.573 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.573 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:44.573 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.743403+0000", 2024-09-21T22:07:44.573 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1786999999999995e-05 2024-09-21T22:07:44.573 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:44.573 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:44.573 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:44.573 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.573 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.573 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4840.0:680 2.7 e63/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 63'61, mlcod=63'59)", 2024-09-21T22:07:44.573 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.745996+0000", 2024-09-21T22:07:44.573 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 19.796391270000001, 2024-09-21T22:07:44.573 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.061804132999999997, 2024-09-21T22:07:44.574 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:44.574 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:44.574 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:44.574 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:44.574 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.574 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:44.574 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.745996+0000", 2024-09-21T22:07:44.574 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:44.574 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.574 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.574 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:44.574 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.745996+0000", 2024-09-21T22:07:44.574 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:44.574 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.575 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.575 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:44.575 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.745998+0000", 2024-09-21T22:07:44.575 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.717e-06 2024-09-21T22:07:44.575 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.575 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.575 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:44.575 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.746523+0000", 2024-09-21T22:07:44.575 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00052525399999999996 2024-09-21T22:07:44.575 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.575 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.575 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:44.575 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.746554+0000", 2024-09-21T22:07:44.575 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0586000000000002e-05 2024-09-21T22:07:44.575 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.576 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.576 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:44.576 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.746577+0000", 2024-09-21T22:07:44.576 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3079e-05 2024-09-21T22:07:44.576 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.576 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.576 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:44.576 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.746689+0000", 2024-09-21T22:07:44.576 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011177 2024-09-21T22:07:44.576 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.576 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.576 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:44.576 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.746722+0000", 2024-09-21T22:07:44.576 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2870000000000002e-05 2024-09-21T22:07:44.576 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.577 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.577 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:44.577 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.807740+0000", 2024-09-21T22:07:44.577 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.061018612 2024-09-21T22:07:44.577 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.577 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.577 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:44.577 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.807800+0000", 2024-09-21T22:07:44.577 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0244999999999997e-05 2024-09-21T22:07:44.577 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:44.577 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:44.577 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:44.577 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.577 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.578 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4840.0:681 2.7 e63/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 63'62, mlcod=63'60)", 2024-09-21T22:07:44.578 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.810646+0000", 2024-09-21T22:07:44.578 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 19.731741845999998, 2024-09-21T22:07:44.578 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010715685000000001, 2024-09-21T22:07:44.578 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:44.578 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:44.578 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:44.578 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:44.578 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.578 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:44.578 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.810646+0000", 2024-09-21T22:07:44.578 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:44.578 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.578 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.579 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:44.579 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.810646+0000", 2024-09-21T22:07:44.579 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:44.579 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.579 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.579 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:44.579 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.810649+0000", 2024-09-21T22:07:44.579 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8569999999999999e-06 2024-09-21T22:07:44.579 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.579 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.579 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:44.579 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.810780+0000", 2024-09-21T22:07:44.579 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013114600000000001 2024-09-21T22:07:44.579 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.579 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.580 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:44.580 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.810819+0000", 2024-09-21T22:07:44.580 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9363000000000002e-05 2024-09-21T22:07:44.580 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.580 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.580 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:44.580 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.810839+0000", 2024-09-21T22:07:44.580 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9865999999999999e-05 2024-09-21T22:07:44.580 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.580 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.580 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:44.580 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.810992+0000", 2024-09-21T22:07:44.580 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015333299999999999 2024-09-21T22:07:44.580 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.581 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.581 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:44.581 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.811037+0000", 2024-09-21T22:07:44.581 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.464e-05 2024-09-21T22:07:44.581 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.581 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.581 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:44.581 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.821280+0000", 2024-09-21T22:07:44.581 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010243254 2024-09-21T22:07:44.581 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.581 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.581 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:44.581 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.821361+0000", 2024-09-21T22:07:44.581 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1225999999999998e-05 2024-09-21T22:07:44.582 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:44.582 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:44.582 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:44.582 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.582 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.582 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4840.0:682 2.7 e63/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 63'63, mlcod=63'61)", 2024-09-21T22:07:44.582 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.824126+0000", 2024-09-21T22:07:44.582 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 19.718261700999999, 2024-09-21T22:07:44.582 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.038159327, 2024-09-21T22:07:44.582 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:44.582 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:44.582 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:44.582 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:44.582 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.582 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:44.583 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.824126+0000", 2024-09-21T22:07:44.583 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:44.583 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.583 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.583 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:44.583 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.824126+0000", 2024-09-21T22:07:44.583 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:44.583 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.583 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.583 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:44.583 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.824128+0000", 2024-09-21T22:07:44.584 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9659999999999999e-06 2024-09-21T22:07:44.584 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.584 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.584 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:44.584 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.824842+0000", 2024-09-21T22:07:44.584 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00071423999999999999 2024-09-21T22:07:44.584 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.584 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.584 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:44.584 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.824883+0000", 2024-09-21T22:07:44.584 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1131999999999999e-05 2024-09-21T22:07:44.584 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.584 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.584 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:44.585 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.824931+0000", 2024-09-21T22:07:44.585 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7327000000000003e-05 2024-09-21T22:07:44.585 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.585 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.585 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:44.585 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.825058+0000", 2024-09-21T22:07:44.585 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012790499999999999 2024-09-21T22:07:44.585 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.585 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.585 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:44.585 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.825087+0000", 2024-09-21T22:07:44.585 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8192e-05 2024-09-21T22:07:44.585 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.585 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.585 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:44.586 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.862225+0000", 2024-09-21T22:07:44.586 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.037138309000000001 2024-09-21T22:07:44.586 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.586 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.586 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:44.586 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.862285+0000", 2024-09-21T22:07:44.586 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0256e-05 2024-09-21T22:07:44.586 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:44.586 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:44.586 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:44.586 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.586 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.586 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5008.0:115 2.7 e71/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 71'147, mlcod=71'145)", 2024-09-21T22:07:44.586 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:37.860188+0000", 2024-09-21T22:07:44.587 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 6.6821996669999999, 2024-09-21T22:07:44.587 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0054120000000000001, 2024-09-21T22:07:44.587 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:44.587 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:44.587 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:44.587 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:44.587 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.587 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:44.587 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.860188+0000", 2024-09-21T22:07:44.587 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:44.587 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.587 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.587 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:44.587 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.860188+0000", 2024-09-21T22:07:44.588 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:44.588 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.588 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.588 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:44.588 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.860189+0000", 2024-09-21T22:07:44.588 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0800000000000004e-07 2024-09-21T22:07:44.588 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.588 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.588 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:44.588 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.860243+0000", 2024-09-21T22:07:44.588 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4666000000000001e-05 2024-09-21T22:07:44.588 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.588 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.588 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:44.589 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.860255+0000", 2024-09-21T22:07:44.589 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.171e-05 2024-09-21T22:07:44.589 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.589 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.589 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:44.589 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.860260+0000", 2024-09-21T22:07:44.589 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1159999999999998e-06 2024-09-21T22:07:44.589 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.589 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.589 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:44.589 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.860325+0000", 2024-09-21T22:07:44.589 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4796999999999996e-05 2024-09-21T22:07:44.589 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.589 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.589 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:44.590 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.860344+0000", 2024-09-21T22:07:44.590 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9403e-05 2024-09-21T22:07:44.590 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.590 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.590 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:44.590 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.865563+0000", 2024-09-21T22:07:44.590 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0052183280000000004 2024-09-21T22:07:44.590 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.590 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.590 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:44.590 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.865600+0000", 2024-09-21T22:07:44.590 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7172000000000003e-05 2024-09-21T22:07:44.590 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:44.590 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:44.591 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:44.591 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.591 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.591 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5008.0:116 2.7 e71/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 71'148, mlcod=71'146)", 2024-09-21T22:07:44.591 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:37.867612+0000", 2024-09-21T22:07:44.591 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 6.6747752059999996, 2024-09-21T22:07:44.591 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.045123155999999998, 2024-09-21T22:07:44.591 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:44.591 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:44.591 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:44.591 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:44.591 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.591 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:44.591 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.867612+0000", 2024-09-21T22:07:44.591 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:44.592 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.592 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.592 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:44.592 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.867612+0000", 2024-09-21T22:07:44.592 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:44.592 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.592 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.592 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:44.592 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.867613+0000", 2024-09-21T22:07:44.592 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1200000000000002e-07 2024-09-21T22:07:44.592 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.592 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.592 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:44.592 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.868135+0000", 2024-09-21T22:07:44.593 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00052132700000000003 2024-09-21T22:07:44.593 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.593 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.593 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:44.593 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.868150+0000", 2024-09-21T22:07:44.593 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5271e-05 2024-09-21T22:07:44.593 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.593 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.593 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:44.593 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.868156+0000", 2024-09-21T22:07:44.593 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4969999999999998e-06 2024-09-21T22:07:44.593 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.593 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.594 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:44.594 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.868213+0000", 2024-09-21T22:07:44.594 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7114999999999998e-05 2024-09-21T22:07:44.594 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.594 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.594 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:44.594 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.868238+0000", 2024-09-21T22:07:44.594 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4987999999999999e-05 2024-09-21T22:07:44.594 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.594 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.594 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:44.594 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.912677+0000", 2024-09-21T22:07:44.594 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.044439019000000003 2024-09-21T22:07:44.594 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.595 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.595 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:44.595 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.912736+0000", 2024-09-21T22:07:44.595 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8127000000000001e-05 2024-09-21T22:07:44.595 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:44.595 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:44.595 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:44.595 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.595 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.595 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5008.0:117 2.7 e71/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 71'149, mlcod=71'147)", 2024-09-21T22:07:44.595 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:37.914937+0000", 2024-09-21T22:07:44.595 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 6.6274501939999997, 2024-09-21T22:07:44.595 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0076223819999999996, 2024-09-21T22:07:44.596 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:44.596 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:44.596 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:44.596 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:44.596 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.596 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:44.596 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.914937+0000", 2024-09-21T22:07:44.596 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:44.596 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.596 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.596 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:44.596 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.914937+0000", 2024-09-21T22:07:44.596 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:44.596 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.596 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.597 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:44.597 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.914939+0000", 2024-09-21T22:07:44.597 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.826e-06 2024-09-21T22:07:44.597 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.597 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.597 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:44.597 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.915021+0000", 2024-09-21T22:07:44.597 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2244999999999996e-05 2024-09-21T22:07:44.597 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.597 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.597 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:44.597 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.915046+0000", 2024-09-21T22:07:44.597 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4794e-05 2024-09-21T22:07:44.597 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.597 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.598 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:44.598 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.915063+0000", 2024-09-21T22:07:44.598 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6773999999999999e-05 2024-09-21T22:07:44.598 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.598 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.598 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:44.598 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.915143+0000", 2024-09-21T22:07:44.598 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9516000000000002e-05 2024-09-21T22:07:44.598 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.598 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.598 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:44.598 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.915185+0000", 2024-09-21T22:07:44.598 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2324000000000003e-05 2024-09-21T22:07:44.598 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.599 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.599 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:44.599 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.922508+0000", 2024-09-21T22:07:44.599 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0073226419999999999 2024-09-21T22:07:44.599 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.599 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.599 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:44.599 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.922560+0000", 2024-09-21T22:07:44.599 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2261e-05 2024-09-21T22:07:44.599 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:44.599 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:44.599 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:44.599 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.599 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.599 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5008.0:118 2.7 e71/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 71'150, mlcod=71'148)", 2024-09-21T22:07:44.600 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:37.924700+0000", 2024-09-21T22:07:44.600 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 6.6176880149999997, 2024-09-21T22:07:44.600 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.034650561000000003, 2024-09-21T22:07:44.600 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:44.600 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:44.600 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:44.600 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:44.600 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.600 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:44.600 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.924700+0000", 2024-09-21T22:07:44.600 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:44.600 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.600 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.600 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:44.601 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.924700+0000", 2024-09-21T22:07:44.601 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:44.601 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.601 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.601 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:44.601 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.924702+0000", 2024-09-21T22:07:44.601 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9460000000000001e-06 2024-09-21T22:07:44.601 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.601 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.601 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:44.601 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.925132+0000", 2024-09-21T22:07:44.601 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00043021200000000002 2024-09-21T22:07:44.601 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.601 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.602 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:44.602 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.925156+0000", 2024-09-21T22:07:44.602 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4451999999999999e-05 2024-09-21T22:07:44.602 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.602 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.602 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:44.602 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.925168+0000", 2024-09-21T22:07:44.602 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2249000000000001e-05 2024-09-21T22:07:44.602 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.602 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.602 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:44.602 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.925241+0000", 2024-09-21T22:07:44.602 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2502000000000005e-05 2024-09-21T22:07:44.602 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.603 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.603 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:44.603 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.925267+0000", 2024-09-21T22:07:44.603 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6471000000000001e-05 2024-09-21T22:07:44.603 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.603 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.603 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:44.603 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.959217+0000", 2024-09-21T22:07:44.603 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.033949569999999998 2024-09-21T22:07:44.603 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.603 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.603 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:44.603 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.959350+0000", 2024-09-21T22:07:44.603 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013315899999999999 2024-09-21T22:07:44.604 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:44.604 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:44.604 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:44.604 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.604 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.604 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5008.0:119 2.7 e71/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 71'151, mlcod=71'149)", 2024-09-21T22:07:44.604 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:37.961607+0000", 2024-09-21T22:07:44.604 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 6.580780377, 2024-09-21T22:07:44.604 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0058888020000000003, 2024-09-21T22:07:44.604 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:44.604 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:44.604 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:44.604 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:44.604 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.605 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:44.605 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.961607+0000", 2024-09-21T22:07:44.605 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:44.605 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.605 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.605 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:44.605 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.961607+0000", 2024-09-21T22:07:44.605 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:44.605 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.605 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.605 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:44.605 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.961608+0000", 2024-09-21T22:07:44.605 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.017e-06 2024-09-21T22:07:44.605 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.606 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.606 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:44.606 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.961669+0000", 2024-09-21T22:07:44.606 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0905e-05 2024-09-21T22:07:44.606 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.606 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.606 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:44.606 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.961686+0000", 2024-09-21T22:07:44.606 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6665999999999999e-05 2024-09-21T22:07:44.606 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.606 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.606 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:44.606 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.961696+0000", 2024-09-21T22:07:44.606 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.063e-05 2024-09-21T22:07:44.607 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.607 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.607 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:44.607 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.961771+0000", 2024-09-21T22:07:44.607 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4443000000000005e-05 2024-09-21T22:07:44.607 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.607 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.607 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:44.607 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.961794+0000", 2024-09-21T22:07:44.607 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3608000000000002e-05 2024-09-21T22:07:44.607 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.607 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.607 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:44.607 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.967374+0000", 2024-09-21T22:07:44.607 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0055799969999999997 2024-09-21T22:07:44.608 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.608 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.608 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:44.608 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.967496+0000", 2024-09-21T22:07:44.608 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000121536 2024-09-21T22:07:44.608 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:44.608 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:44.608 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:44.608 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.608 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.608 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5008.0:120 2.7 e71/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 71'152, mlcod=71'150)", 2024-09-21T22:07:44.608 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:37.970837+0000", 2024-09-21T22:07:44.608 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 6.5715507640000004, 2024-09-21T22:07:44.608 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.03044943, 2024-09-21T22:07:44.608 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:44.609 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:44.609 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:44.609 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:44.609 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.609 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:44.609 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.970837+0000", 2024-09-21T22:07:44.609 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:44.609 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.609 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.609 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:44.609 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.970837+0000", 2024-09-21T22:07:44.609 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:44.609 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.609 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.610 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:44.610 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.970838+0000", 2024-09-21T22:07:44.610 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4649999999999999e-06 2024-09-21T22:07:44.610 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.610 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.610 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:44.610 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.971265+0000", 2024-09-21T22:07:44.610 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00042665800000000001 2024-09-21T22:07:44.610 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.610 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.610 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:44.610 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.971285+0000", 2024-09-21T22:07:44.610 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9639999999999998e-05 2024-09-21T22:07:44.610 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.611 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.611 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:44.611 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.971294+0000", 2024-09-21T22:07:44.611 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7210000000000006e-06 2024-09-21T22:07:44.611 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.611 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.611 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:44.611 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.971354+0000", 2024-09-21T22:07:44.611 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9613e-05 2024-09-21T22:07:44.611 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.611 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.611 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:44.611 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.971381+0000", 2024-09-21T22:07:44.611 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7538999999999998e-05 2024-09-21T22:07:44.611 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.612 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.612 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:44.612 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.001199+0000", 2024-09-21T22:07:44.612 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029817537000000002 2024-09-21T22:07:44.612 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.612 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.612 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:44.612 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.001286+0000", 2024-09-21T22:07:44.612 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7256999999999995e-05 2024-09-21T22:07:44.612 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:44.613 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:44.613 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:44.613 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.613 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.613 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5008.0:121 2.7 e71/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 71'153, mlcod=71'151)", 2024-09-21T22:07:44.613 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:38.007198+0000", 2024-09-21T22:07:44.613 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 6.5351898439999996, 2024-09-21T22:07:44.613 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0047526499999999998, 2024-09-21T22:07:44.613 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:44.613 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:44.613 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:44.613 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:44.613 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.613 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:44.614 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.007198+0000", 2024-09-21T22:07:44.614 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:44.614 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.614 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.614 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:44.614 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.007198+0000", 2024-09-21T22:07:44.614 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:44.614 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.614 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.614 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:44.614 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.007199+0000", 2024-09-21T22:07:44.614 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4200000000000005e-07 2024-09-21T22:07:44.614 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.614 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.615 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:44.615 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.007255+0000", 2024-09-21T22:07:44.615 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6406000000000002e-05 2024-09-21T22:07:44.615 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.615 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.615 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:44.615 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.007268+0000", 2024-09-21T22:07:44.615 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2824e-05 2024-09-21T22:07:44.615 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.615 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.615 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:44.615 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.007276+0000", 2024-09-21T22:07:44.615 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1640000000000001e-06 2024-09-21T22:07:44.615 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.616 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.616 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:44.616 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.007335+0000", 2024-09-21T22:07:44.616 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8610999999999998e-05 2024-09-21T22:07:44.616 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.616 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.616 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:44.616 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.007355+0000", 2024-09-21T22:07:44.616 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9987999999999999e-05 2024-09-21T22:07:44.616 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.616 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.616 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:44.616 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.011869+0000", 2024-09-21T22:07:44.616 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.004514784 2024-09-21T22:07:44.616 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.617 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.617 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:44.617 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.011950+0000", 2024-09-21T22:07:44.617 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1031000000000004e-05 2024-09-21T22:07:44.617 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:44.617 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:44.617 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:44.617 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.617 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.617 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5008.0:122 2.7 e71/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 71'154, mlcod=71'152)", 2024-09-21T22:07:44.617 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:38.014709+0000", 2024-09-21T22:07:44.617 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 6.5276788889999997, 2024-09-21T22:07:44.617 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029567408999999999, 2024-09-21T22:07:44.617 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:44.617 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:44.618 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:44.618 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:44.618 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.618 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:44.618 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.014709+0000", 2024-09-21T22:07:44.618 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:44.618 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.618 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.618 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:44.618 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.014709+0000", 2024-09-21T22:07:44.618 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:44.618 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.618 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.618 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:44.619 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.014709+0000", 2024-09-21T22:07:44.619 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9599999999999999e-07 2024-09-21T22:07:44.619 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.619 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.619 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:44.619 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.015201+0000", 2024-09-21T22:07:44.619 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00049175599999999999 2024-09-21T22:07:44.619 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.619 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.619 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:44.619 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.015215+0000", 2024-09-21T22:07:44.619 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3692999999999999e-05 2024-09-21T22:07:44.619 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.619 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.620 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:44.620 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.015225+0000", 2024-09-21T22:07:44.620 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.815e-06 2024-09-21T22:07:44.620 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.620 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.620 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:44.620 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.015287+0000", 2024-09-21T22:07:44.620 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1858000000000003e-05 2024-09-21T22:07:44.620 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.620 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.620 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:44.620 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.015315+0000", 2024-09-21T22:07:44.620 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8490999999999999e-05 2024-09-21T22:07:44.620 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.620 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.621 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:44.621 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.044182+0000", 2024-09-21T22:07:44.621 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.028866936999999999 2024-09-21T22:07:44.621 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.621 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.621 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:44.621 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.044276+0000", 2024-09-21T22:07:44.621 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4163e-05 2024-09-21T22:07:44.621 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:44.621 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:44.621 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:44.621 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.621 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.621 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5008.0:132 2.7 e72/16 2:f45e3993:::rbd_object_map.126483d75f2d:head v 72'156, mlcod=71'154)", 2024-09-21T22:07:44.621 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:38.335972+0000", 2024-09-21T22:07:44.622 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 6.2064154819999997, 2024-09-21T22:07:44.622 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003515944, 2024-09-21T22:07:44.622 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:44.622 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:44.622 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:44.622 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:44.622 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.622 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:44.622 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.335972+0000", 2024-09-21T22:07:44.622 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:44.622 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.622 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.622 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:44.622 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.335972+0000", 2024-09-21T22:07:44.623 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:44.623 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.623 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.623 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:44.623 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.335973+0000", 2024-09-21T22:07:44.623 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5199999999999997e-07 2024-09-21T22:07:44.623 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.623 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.623 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:44.623 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.335981+0000", 2024-09-21T22:07:44.623 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3720000000000005e-06 2024-09-21T22:07:44.623 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.623 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.623 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:44.623 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.336000+0000", 2024-09-21T22:07:44.624 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8663000000000001e-05 2024-09-21T22:07:44.624 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.624 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.624 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:44.624 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.336015+0000", 2024-09-21T22:07:44.624 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5372000000000001e-05 2024-09-21T22:07:44.624 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.624 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.624 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:44.624 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.336111+0000", 2024-09-21T22:07:44.624 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5968e-05 2024-09-21T22:07:44.624 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.624 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.624 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:44.624 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.336146+0000", 2024-09-21T22:07:44.625 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4882000000000002e-05 2024-09-21T22:07:44.625 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.625 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.625 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:44.625 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.339438+0000", 2024-09-21T22:07:44.625 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0032919709999999999 2024-09-21T22:07:44.625 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.625 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.625 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:44.625 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.339488+0000", 2024-09-21T22:07:44.625 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0164000000000003e-05 2024-09-21T22:07:44.625 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:44.625 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:44.625 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:44.625 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.626 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.626 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5008.0:150 2.7 e72/16 2:f45e3993:::rbd_object_map.126483d75f2d:head v 72'158, mlcod=72'156)", 2024-09-21T22:07:44.626 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:38.744508+0000", 2024-09-21T22:07:44.626 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 5.7978793340000001, 2024-09-21T22:07:44.626 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0022832690000000001, 2024-09-21T22:07:44.626 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:44.626 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:44.626 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:44.626 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:44.626 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.626 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:44.626 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.744508+0000", 2024-09-21T22:07:44.626 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:44.626 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.627 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.627 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:44.627 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.744508+0000", 2024-09-21T22:07:44.627 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:44.627 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.627 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.627 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:44.627 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.744511+0000", 2024-09-21T22:07:44.627 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4870000000000001e-06 2024-09-21T22:07:44.627 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.627 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.627 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:44.627 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.744530+0000", 2024-09-21T22:07:44.627 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8966000000000001e-05 2024-09-21T22:07:44.627 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.628 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.628 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:44.628 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.744566+0000", 2024-09-21T22:07:44.628 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6627e-05 2024-09-21T22:07:44.628 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.628 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.628 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:44.628 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.744588+0000", 2024-09-21T22:07:44.628 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1716e-05 2024-09-21T22:07:44.628 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.628 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.628 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:44.628 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.744750+0000", 2024-09-21T22:07:44.628 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016171600000000001 2024-09-21T22:07:44.629 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.629 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.629 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:44.629 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.744813+0000", 2024-09-21T22:07:44.629 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3689999999999995e-05 2024-09-21T22:07:44.629 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.629 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.629 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:44.629 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.746701+0000", 2024-09-21T22:07:44.629 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0018874479999999999 2024-09-21T22:07:44.629 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.629 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.629 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:44.629 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.746792+0000", 2024-09-21T22:07:44.630 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0618999999999994e-05 2024-09-21T22:07:44.630 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:44.630 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:44.630 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:44.630 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.630 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.630 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5008.0:159 2.7 e72/16 2:f45e3993:::rbd_object_map.126483d75f2d:head v 72'159, mlcod=72'157)", 2024-09-21T22:07:44.630 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:38.914368+0000", 2024-09-21T22:07:44.630 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 5.6280199550000001, 2024-09-21T22:07:44.630 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021177169999999999, 2024-09-21T22:07:44.630 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:44.630 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:44.630 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:44.630 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:44.631 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.631 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:44.631 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.914368+0000", 2024-09-21T22:07:44.631 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:44.631 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.631 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.631 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:44.631 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.914368+0000", 2024-09-21T22:07:44.631 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:44.631 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.631 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.631 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:44.631 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.914370+0000", 2024-09-21T22:07:44.631 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9400000000000001e-06 2024-09-21T22:07:44.631 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.632 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.632 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:44.632 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.914382+0000", 2024-09-21T22:07:44.632 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2427e-05 2024-09-21T22:07:44.632 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.632 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.632 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:44.632 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.914412+0000", 2024-09-21T22:07:44.632 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9583000000000001e-05 2024-09-21T22:07:44.632 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.632 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.632 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:44.632 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.914427+0000", 2024-09-21T22:07:44.632 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5420000000000001e-05 2024-09-21T22:07:44.633 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.633 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.633 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:44.633 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.914569+0000", 2024-09-21T22:07:44.633 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014215899999999999 2024-09-21T22:07:44.633 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.633 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.633 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:44.633 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.914635+0000", 2024-09-21T22:07:44.633 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5566000000000002e-05 2024-09-21T22:07:44.633 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.633 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.633 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:44.633 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.916409+0000", 2024-09-21T22:07:44.633 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017742229999999999 2024-09-21T22:07:44.634 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:44.634 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:44.634 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:44.634 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.916485+0000", 2024-09-21T22:07:44.634 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6398999999999998e-05 2024-09-21T22:07:44.634 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:44.634 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:44.634 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:44.634 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:44.634 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:44.634 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:07:44.635 INFO:teuthology.orchestra.run.smithi039.stdout:osd.0: {} 2024-09-21T22:07:44.636 INFO:teuthology.orchestra.run.smithi039.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T22:07:44.636 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:44.630+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:44.637 INFO:tasks.workunit.client.0.smithi157.stderr:+ DEV=/dev/nbd0 2024-09-21T22:07:44.637 INFO:tasks.workunit.client.0.smithi157.stderr:+ get_pid rbd ns 2024-09-21T22:07:44.637 INFO:tasks.workunit.client.0.smithi157.stderr:+ local pool=rbd 2024-09-21T22:07:44.637 INFO:tasks.workunit.client.0.smithi157.stderr:+ local ns=ns 2024-09-21T22:07:44.637 INFO:tasks.workunit.client.0.smithi157.stderr:++ rbd device --device-type nbd --format xml list 2024-09-21T22:07:44.637 INFO:tasks.workunit.client.0.smithi157.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\''ns'\''][image='\''testrbdnbd21553'\''][device='\''/dev/nbd0'\'']/id' 2024-09-21T22:07:44.637 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T22:07:44.644 INFO:teuthology.orchestra.run.smithi039.stdout:osd.1: {} 2024-09-21T22:07:44.645 INFO:teuthology.orchestra.run.smithi039.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T22:07:44.652 INFO:tasks.workunit.client.0.smithi157.stderr:+ PID=22980 2024-09-21T22:07:44.653 INFO:tasks.workunit.client.0.smithi157.stderr:+ test -n 22980 2024-09-21T22:07:44.653 INFO:tasks.workunit.client.0.smithi157.stderr:+ ps -p 22980 -C rbd-nbd 2024-09-21T22:07:44.660 INFO:tasks.workunit.client.0.smithi157.stdout: PID TTY TIME CMD 2024-09-21T22:07:44.660 INFO:tasks.workunit.client.0.smithi157.stdout: 22980 ? 00:00:00 rbd-nbd 2024-09-21T22:07:44.660 INFO:tasks.workunit.client.0.smithi157.stderr:+ unmap_device rbd/ns/testrbdnbd21553@snap 22980 2024-09-21T22:07:44.660 INFO:tasks.workunit.client.0.smithi157.stderr:+ local args=rbd/ns/testrbdnbd21553@snap 2024-09-21T22:07:44.661 INFO:tasks.workunit.client.0.smithi157.stderr:+ local pid=22980 2024-09-21T22:07:44.661 INFO:tasks.workunit.client.0.smithi157.stderr:+ _sudo rbd device --device-type nbd unmap rbd/ns/testrbdnbd21553@snap 2024-09-21T22:07:44.661 INFO:tasks.workunit.client.0.smithi157.stderr:+ local cmd 2024-09-21T22:07:44.661 INFO:tasks.workunit.client.0.smithi157.stderr:++ id -u 2024-09-21T22:07:44.661 INFO:teuthology.orchestra.run.smithi039.stdout:osd.2: {} 2024-09-21T22:07:44.661 INFO:teuthology.orchestra.run.smithi039.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T22:07:44.662 INFO:tasks.workunit.client.0.smithi157.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T22:07:44.662 INFO:tasks.workunit.client.0.smithi157.stderr:++ which rbd 2024-09-21T22:07:44.663 INFO:tasks.workunit.client.0.smithi157.stderr:+ cmd=/usr/bin/rbd 2024-09-21T22:07:44.663 INFO:tasks.workunit.client.0.smithi157.stderr:+ shift 2024-09-21T22:07:44.663 INFO:tasks.workunit.client.0.smithi157.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap rbd/ns/testrbdnbd21553@snap 2024-09-21T22:07:44.680 INFO:teuthology.orchestra.run.smithi039.stdout:osd.3: {} 2024-09-21T22:07:44.680 INFO:teuthology.orchestra.run.smithi039.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T22:07:44.698 INFO:teuthology.orchestra.run.smithi039.stdout:osd.4: {} 2024-09-21T22:07:44.699 INFO:teuthology.orchestra.run.smithi039.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T22:07:44.718 INFO:teuthology.orchestra.run.smithi039.stdout:osd.5: {} 2024-09-21T22:07:44.718 INFO:teuthology.orchestra.run.smithi039.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T22:07:44.736 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:07:44.732+0000 7f2075a06640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31720) UID: 0 2024-09-21T22:07:44.742 INFO:tasks.workunit.client.0.smithi157.stderr:+ rbd device --device-type nbd list 2024-09-21T22:07:44.742 INFO:tasks.workunit.client.0.smithi157.stderr:+ expect_false grep '^22980\b' 2024-09-21T22:07:44.742 INFO:tasks.workunit.client.0.smithi157.stderr:+ grep '^22980\b' 2024-09-21T22:07:44.770 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:44.770 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:44.786 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T22:07:44.795 INFO:tasks.workunit.client.0.smithi157.stderr:+ return 0 2024-09-21T22:07:44.796 INFO:tasks.workunit.client.0.smithi157.stderr:+ ps -C rbd-nbd 2024-09-21T22:07:44.796 INFO:tasks.workunit.client.0.smithi157.stderr:+ expect_false grep '^ *22980\b' 2024-09-21T22:07:44.796 INFO:tasks.workunit.client.0.smithi157.stderr:+ grep '^ *22980\b' 2024-09-21T22:07:44.802 INFO:tasks.workunit.client.0.smithi157.stderr:+ return 0 2024-09-21T22:07:44.803 INFO:tasks.workunit.client.0.smithi157.stderr:+ sleep 0.5 2024-09-21T22:07:44.874 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:44.872+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:44.937 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:07:44.932+0000 7f2075a06640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31720) UID: 0 2024-09-21T22:07:44.959 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:44.960 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:44.974 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T22:07:45.062 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:45.060+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:45.138 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:45.138 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:45.139 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:45.134+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:45.152 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T22:07:45.239 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:45.238+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:45.288 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:45.289 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:45.303 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T22:07:45.304 INFO:tasks.workunit.client.0.smithi157.stderr:+ DEV= 2024-09-21T22:07:45.305 INFO:tasks.workunit.client.0.smithi157.stderr:++ rbd snap ls rbd/ns/testrbdnbd21553 2024-09-21T22:07:45.305 INFO:tasks.workunit.client.0.smithi157.stderr:++ awk '$2 == "snap" {print $1}' 2024-09-21T22:07:45.341 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:45.336+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:45.365 INFO:tasks.workunit.client.0.smithi157.stderr:+ SNAPID=4 2024-09-21T22:07:45.365 INFO:tasks.workunit.client.0.smithi157.stderr:++ _sudo rbd device --device-type nbd map --snap-id 4 rbd/ns/testrbdnbd21553 2024-09-21T22:07:45.366 INFO:tasks.workunit.client.0.smithi157.stderr:++ local cmd 2024-09-21T22:07:45.366 INFO:tasks.workunit.client.0.smithi157.stderr:+++ id -u 2024-09-21T22:07:45.366 INFO:tasks.workunit.client.0.smithi157.stderr:++ '[' 1000 -eq 0 ']' 2024-09-21T22:07:45.367 INFO:tasks.workunit.client.0.smithi157.stderr:+++ which rbd 2024-09-21T22:07:45.368 INFO:tasks.workunit.client.0.smithi157.stderr:++ cmd=/usr/bin/rbd 2024-09-21T22:07:45.368 INFO:tasks.workunit.client.0.smithi157.stderr:++ shift 2024-09-21T22:07:45.368 INFO:tasks.workunit.client.0.smithi157.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map --snap-id 4 rbd/ns/testrbdnbd21553 2024-09-21T22:07:45.441 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:45.436+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:45.480 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:45.480 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:45.495 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T22:07:45.501 INFO:tasks.workunit.client.0.smithi157.stderr:+ DEV=/dev/nbd0 2024-09-21T22:07:45.502 INFO:tasks.workunit.client.0.smithi157.stderr:+ get_pid rbd ns 2024-09-21T22:07:45.502 INFO:tasks.workunit.client.0.smithi157.stderr:+ local pool=rbd 2024-09-21T22:07:45.502 INFO:tasks.workunit.client.0.smithi157.stderr:+ local ns=ns 2024-09-21T22:07:45.502 INFO:tasks.workunit.client.0.smithi157.stderr:++ rbd device --device-type nbd --format xml list 2024-09-21T22:07:45.502 INFO:tasks.workunit.client.0.smithi157.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\''ns'\''][image='\''testrbdnbd21553'\''][device='\''/dev/nbd0'\'']/id' 2024-09-21T22:07:45.542 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:45.538+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:45.568 INFO:tasks.workunit.client.0.smithi157.stderr:+ PID=23064 2024-09-21T22:07:45.568 INFO:tasks.workunit.client.0.smithi157.stderr:+ test -n 23064 2024-09-21T22:07:45.568 INFO:tasks.workunit.client.0.smithi157.stderr:+ ps -p 23064 -C rbd-nbd 2024-09-21T22:07:45.573 INFO:tasks.workunit.client.0.smithi157.stdout: PID TTY TIME CMD 2024-09-21T22:07:45.574 INFO:tasks.workunit.client.0.smithi157.stdout: 23064 ? 00:00:00 rbd-nbd 2024-09-21T22:07:45.574 INFO:tasks.workunit.client.0.smithi157.stderr:+ unmap_device '--snap-id 4 rbd/ns/testrbdnbd21553' 23064 2024-09-21T22:07:45.575 INFO:tasks.workunit.client.0.smithi157.stderr:+ local 'args=--snap-id 4 rbd/ns/testrbdnbd21553' 2024-09-21T22:07:45.575 INFO:tasks.workunit.client.0.smithi157.stderr:+ local pid=23064 2024-09-21T22:07:45.575 INFO:tasks.workunit.client.0.smithi157.stderr:+ _sudo rbd device --device-type nbd unmap --snap-id 4 rbd/ns/testrbdnbd21553 2024-09-21T22:07:45.575 INFO:tasks.workunit.client.0.smithi157.stderr:+ local cmd 2024-09-21T22:07:45.575 INFO:tasks.workunit.client.0.smithi157.stderr:++ id -u 2024-09-21T22:07:45.575 INFO:tasks.workunit.client.0.smithi157.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T22:07:45.575 INFO:tasks.workunit.client.0.smithi157.stderr:++ which rbd 2024-09-21T22:07:45.577 INFO:tasks.workunit.client.0.smithi157.stderr:+ cmd=/usr/bin/rbd 2024-09-21T22:07:45.577 INFO:tasks.workunit.client.0.smithi157.stderr:+ shift 2024-09-21T22:07:45.577 INFO:tasks.workunit.client.0.smithi157.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap --snap-id 4 rbd/ns/testrbdnbd21553 2024-09-21T22:07:45.616 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:45.616 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:45.632 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T22:07:45.643 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:45.640+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:45.663 INFO:tasks.workunit.client.0.smithi157.stderr:+ rbd device --device-type nbd list 2024-09-21T22:07:45.663 INFO:tasks.workunit.client.0.smithi157.stderr:+ expect_false grep '^23064\b' 2024-09-21T22:07:45.663 INFO:tasks.workunit.client.0.smithi157.stderr:+ grep '^23064\b' 2024-09-21T22:07:45.714 INFO:tasks.workunit.client.0.smithi157.stderr:+ return 0 2024-09-21T22:07:45.715 INFO:tasks.workunit.client.0.smithi157.stderr:+ ps -C rbd-nbd 2024-09-21T22:07:45.715 INFO:tasks.workunit.client.0.smithi157.stderr:+ expect_false grep '^ *23064\b' 2024-09-21T22:07:45.715 INFO:tasks.workunit.client.0.smithi157.stderr:+ grep '^ *23064\b' 2024-09-21T22:07:45.721 INFO:tasks.workunit.client.0.smithi157.stderr:+ return 0 2024-09-21T22:07:45.721 INFO:tasks.workunit.client.0.smithi157.stderr:+ sleep 0.5 2024-09-21T22:07:45.744 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:07:45.744+0000 7f2075a06640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31720) UID: 0 2024-09-21T22:07:45.800 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:45.800 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:45.814 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T22:07:45.860 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:45.858+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:45.941 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:45.941 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:45.945 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:45.942+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:45.955 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T22:07:46.048 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:46.046+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:46.132 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:46.133 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:46.147 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T22:07:46.148 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:46.144+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:46.222 INFO:tasks.workunit.client.0.smithi157.stderr:+ DEV= 2024-09-21T22:07:46.223 INFO:tasks.workunit.client.0.smithi157.stderr:++ _sudo rbd device --device-type nbd map --pool rbd --namespace ns --image testrbdnbd21553 2024-09-21T22:07:46.223 INFO:tasks.workunit.client.0.smithi157.stderr:++ local cmd 2024-09-21T22:07:46.223 INFO:tasks.workunit.client.0.smithi157.stderr:+++ id -u 2024-09-21T22:07:46.225 INFO:tasks.workunit.client.0.smithi157.stderr:++ '[' 1000 -eq 0 ']' 2024-09-21T22:07:46.226 INFO:tasks.workunit.client.0.smithi157.stderr:+++ which rbd 2024-09-21T22:07:46.227 INFO:tasks.workunit.client.0.smithi157.stderr:++ cmd=/usr/bin/rbd 2024-09-21T22:07:46.227 INFO:tasks.workunit.client.0.smithi157.stderr:++ shift 2024-09-21T22:07:46.227 INFO:tasks.workunit.client.0.smithi157.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map --pool rbd --namespace ns --image testrbdnbd21553 2024-09-21T22:07:46.248 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:46.246+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:46.297 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 5, 0] out_osds: [2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-09-21T22:07:46.298 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T22:07:46.298 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-09-21T22:07:46.298 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-09-21T22:07:46.315 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:46.315 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:46.332 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T22:07:46.349 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:46.344+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:46.390 INFO:tasks.workunit.client.0.smithi157.stderr:+ DEV=/dev/nbd0 2024-09-21T22:07:46.390 INFO:tasks.workunit.client.0.smithi157.stderr:+ get_pid rbd ns 2024-09-21T22:07:46.390 INFO:tasks.workunit.client.0.smithi157.stderr:+ local pool=rbd 2024-09-21T22:07:46.390 INFO:tasks.workunit.client.0.smithi157.stderr:+ local ns=ns 2024-09-21T22:07:46.391 INFO:tasks.workunit.client.0.smithi157.stderr:++ rbd device --device-type nbd --format xml list 2024-09-21T22:07:46.391 INFO:tasks.workunit.client.0.smithi157.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\''ns'\''][image='\''testrbdnbd21553'\''][device='\''/dev/nbd0'\'']/id' 2024-09-21T22:07:46.413 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-21T22:07:46.450 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:46.448+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:46.456 INFO:tasks.workunit.client.0.smithi157.stderr:+ PID=23126 2024-09-21T22:07:46.456 INFO:tasks.workunit.client.0.smithi157.stderr:+ test -n 23126 2024-09-21T22:07:46.456 INFO:tasks.workunit.client.0.smithi157.stderr:+ ps -p 23126 -C rbd-nbd 2024-09-21T22:07:46.462 INFO:tasks.workunit.client.0.smithi157.stdout: PID TTY TIME CMD 2024-09-21T22:07:46.462 INFO:tasks.workunit.client.0.smithi157.stdout: 23126 ? 00:00:00 rbd-nbd 2024-09-21T22:07:46.463 INFO:tasks.workunit.client.0.smithi157.stderr:+ unmap_device '--pool rbd --namespace ns --image testrbdnbd21553' 23126 2024-09-21T22:07:46.463 INFO:tasks.workunit.client.0.smithi157.stderr:+ local 'args=--pool rbd --namespace ns --image testrbdnbd21553' 2024-09-21T22:07:46.463 INFO:tasks.workunit.client.0.smithi157.stderr:+ local pid=23126 2024-09-21T22:07:46.463 INFO:tasks.workunit.client.0.smithi157.stderr:+ _sudo rbd device --device-type nbd unmap --pool rbd --namespace ns --image testrbdnbd21553 2024-09-21T22:07:46.463 INFO:tasks.workunit.client.0.smithi157.stderr:+ local cmd 2024-09-21T22:07:46.463 INFO:tasks.workunit.client.0.smithi157.stderr:++ id -u 2024-09-21T22:07:46.463 INFO:tasks.workunit.client.0.smithi157.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T22:07:46.463 INFO:tasks.workunit.client.0.smithi157.stderr:++ which rbd 2024-09-21T22:07:46.465 INFO:tasks.workunit.client.0.smithi157.stderr:+ cmd=/usr/bin/rbd 2024-09-21T22:07:46.465 INFO:tasks.workunit.client.0.smithi157.stderr:+ shift 2024-09-21T22:07:46.465 INFO:tasks.workunit.client.0.smithi157.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap --pool rbd --namespace ns --image testrbdnbd21553 2024-09-21T22:07:46.488 INFO:teuthology.orchestra.run.smithi039.stderr:no valid command found; 10 closest matches: 2024-09-21T22:07:46.488 INFO:teuthology.orchestra.run.smithi039.stderr:0 2024-09-21T22:07:46.488 INFO:teuthology.orchestra.run.smithi039.stderr:1 2024-09-21T22:07:46.489 INFO:teuthology.orchestra.run.smithi039.stderr:2 2024-09-21T22:07:46.489 INFO:teuthology.orchestra.run.smithi039.stderr:abort 2024-09-21T22:07:46.489 INFO:teuthology.orchestra.run.smithi039.stderr:assert 2024-09-21T22:07:46.489 INFO:teuthology.orchestra.run.smithi039.stderr:bench [] [] [] [] 2024-09-21T22:07:46.489 INFO:teuthology.orchestra.run.smithi039.stderr:bluefs debug_inject_read_zeros 2024-09-21T22:07:46.489 INFO:teuthology.orchestra.run.smithi039.stderr:bluefs files list 2024-09-21T22:07:46.489 INFO:teuthology.orchestra.run.smithi039.stderr:bluefs stats 2024-09-21T22:07:46.489 INFO:teuthology.orchestra.run.smithi039.stderr:bluestore allocator dump block 2024-09-21T22:07:46.489 INFO:teuthology.orchestra.run.smithi039.stderr:Error EINVAL: invalid command 2024-09-21T22:07:46.493 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-21T22:07:46.493 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-21T22:07:46.494 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-09-21T22:07:46.518 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:46.518 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:46.534 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T22:07:46.549 INFO:tasks.workunit.client.0.smithi157.stderr:+ rbd device --device-type nbd list 2024-09-21T22:07:46.549 INFO:tasks.workunit.client.0.smithi157.stderr:+ expect_false grep '^23126\b' 2024-09-21T22:07:46.549 INFO:tasks.workunit.client.0.smithi157.stderr:+ grep '^23126\b' 2024-09-21T22:07:46.576 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:46.574+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:46.603 INFO:tasks.workunit.client.0.smithi157.stderr:+ return 0 2024-09-21T22:07:46.604 INFO:tasks.workunit.client.0.smithi157.stderr:+ ps -C rbd-nbd 2024-09-21T22:07:46.604 INFO:tasks.workunit.client.0.smithi157.stderr:+ expect_false grep '^ *23126\b' 2024-09-21T22:07:46.604 INFO:tasks.workunit.client.0.smithi157.stderr:+ grep '^ *23126\b' 2024-09-21T22:07:46.610 INFO:tasks.workunit.client.0.smithi157.stderr:+ return 0 2024-09-21T22:07:46.610 INFO:tasks.workunit.client.0.smithi157.stderr:+ sleep 0.5 2024-09-21T22:07:46.651 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:07:46.648+0000 7f2075a06640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31720) UID: 0 2024-09-21T22:07:46.658 INFO:teuthology.orchestra.run.smithi039.stderr:no valid command found; 10 closest matches: 2024-09-21T22:07:46.658 INFO:teuthology.orchestra.run.smithi039.stderr:0 2024-09-21T22:07:46.658 INFO:teuthology.orchestra.run.smithi039.stderr:1 2024-09-21T22:07:46.658 INFO:teuthology.orchestra.run.smithi039.stderr:2 2024-09-21T22:07:46.658 INFO:teuthology.orchestra.run.smithi039.stderr:abort 2024-09-21T22:07:46.658 INFO:teuthology.orchestra.run.smithi039.stderr:assert 2024-09-21T22:07:46.658 INFO:teuthology.orchestra.run.smithi039.stderr:bench [] [] [] [] 2024-09-21T22:07:46.658 INFO:teuthology.orchestra.run.smithi039.stderr:bluefs debug_inject_read_zeros 2024-09-21T22:07:46.659 INFO:teuthology.orchestra.run.smithi039.stderr:bluefs files list 2024-09-21T22:07:46.659 INFO:teuthology.orchestra.run.smithi039.stderr:bluefs stats 2024-09-21T22:07:46.659 INFO:teuthology.orchestra.run.smithi039.stderr:bluestore allocator dump block 2024-09-21T22:07:46.659 INFO:teuthology.orchestra.run.smithi039.stderr:Error EINVAL: invalid command 2024-09-21T22:07:46.661 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-21T22:07:46.661 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-21T22:07:46.662 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 'trim stale osdmaps' 2024-09-21T22:07:46.695 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:46.695 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:46.710 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T22:07:46.736 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T22:07:46.752 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:46.750+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:46.798 INFO:teuthology.orchestra.run.smithi039.stderr:nodeep-scrub is unset 2024-09-21T22:07:46.817 INFO:teuthology.orchestra.run.smithi039.stderr:no valid command found; 10 closest matches: 2024-09-21T22:07:46.817 INFO:teuthology.orchestra.run.smithi039.stderr:0 2024-09-21T22:07:46.817 INFO:teuthology.orchestra.run.smithi039.stderr:1 2024-09-21T22:07:46.818 INFO:teuthology.orchestra.run.smithi039.stderr:2 2024-09-21T22:07:46.818 INFO:teuthology.orchestra.run.smithi039.stderr:abort 2024-09-21T22:07:46.818 INFO:teuthology.orchestra.run.smithi039.stderr:assert 2024-09-21T22:07:46.818 INFO:teuthology.orchestra.run.smithi039.stderr:bench [] [] [] [] 2024-09-21T22:07:46.818 INFO:teuthology.orchestra.run.smithi039.stderr:bluefs debug_inject_read_zeros 2024-09-21T22:07:46.818 INFO:teuthology.orchestra.run.smithi039.stderr:bluefs files list 2024-09-21T22:07:46.818 INFO:teuthology.orchestra.run.smithi039.stderr:bluefs stats 2024-09-21T22:07:46.818 INFO:teuthology.orchestra.run.smithi039.stderr:bluestore allocator dump block 2024-09-21T22:07:46.818 INFO:teuthology.orchestra.run.smithi039.stderr:Error EINVAL: invalid command 2024-09-21T22:07:46.819 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-21T22:07:46.820 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-21T22:07:46.820 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 'trim stale osdmaps' 2024-09-21T22:07:46.850 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:46.851 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:46.864 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:46.862+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:46.865 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T22:07:46.893 INFO:teuthology.orchestra.run.smithi039.stdout:osd.0: {} 2024-09-21T22:07:46.894 INFO:teuthology.orchestra.run.smithi039.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T22:07:46.912 INFO:teuthology.orchestra.run.smithi039.stdout:osd.1: {} 2024-09-21T22:07:46.912 INFO:teuthology.orchestra.run.smithi039.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T22:07:46.928 INFO:teuthology.orchestra.run.smithi039.stdout:osd.2: {} 2024-09-21T22:07:46.929 INFO:teuthology.orchestra.run.smithi039.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T22:07:46.949 INFO:teuthology.orchestra.run.smithi039.stdout:osd.3: {} 2024-09-21T22:07:46.949 INFO:teuthology.orchestra.run.smithi039.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T22:07:46.953 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:46.950+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:46.969 INFO:teuthology.orchestra.run.smithi039.stdout:osd.4: {} 2024-09-21T22:07:46.969 INFO:teuthology.orchestra.run.smithi039.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T22:07:46.972 INFO:teuthology.orchestra.run.smithi039.stderr:no valid command found; 10 closest matches: 2024-09-21T22:07:46.972 INFO:teuthology.orchestra.run.smithi039.stderr:0 2024-09-21T22:07:46.973 INFO:teuthology.orchestra.run.smithi039.stderr:1 2024-09-21T22:07:46.973 INFO:teuthology.orchestra.run.smithi039.stderr:2 2024-09-21T22:07:46.973 INFO:teuthology.orchestra.run.smithi039.stderr:abort 2024-09-21T22:07:46.973 INFO:teuthology.orchestra.run.smithi039.stderr:assert 2024-09-21T22:07:46.973 INFO:teuthology.orchestra.run.smithi039.stderr:bench [] [] [] [] 2024-09-21T22:07:46.973 INFO:teuthology.orchestra.run.smithi039.stderr:bluefs debug_inject_read_zeros 2024-09-21T22:07:46.973 INFO:teuthology.orchestra.run.smithi039.stderr:bluefs files list 2024-09-21T22:07:46.973 INFO:teuthology.orchestra.run.smithi039.stderr:bluefs stats 2024-09-21T22:07:46.973 INFO:teuthology.orchestra.run.smithi039.stderr:bluestore allocator dump block 2024-09-21T22:07:46.973 INFO:teuthology.orchestra.run.smithi039.stderr:Error EINVAL: invalid command 2024-09-21T22:07:46.975 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-21T22:07:46.976 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-21T22:07:46.976 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-09-21T22:07:46.990 INFO:teuthology.orchestra.run.smithi039.stdout:osd.5: {} 2024-09-21T22:07:46.990 INFO:teuthology.orchestra.run.smithi039.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T22:07:46.991 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:07:46.991 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:07:46.991 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-21T22:07:46.991 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-21T22:07:46.991 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:07:47.007 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T22:07:47.054 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:47.052+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:47.112 INFO:tasks.workunit.client.0.smithi157.stderr:++ _sudo rbd device --device-type nbd map --pool rbd --namespace ns --image testrbdnbd21553 --snap snap 2024-09-21T22:07:47.112 INFO:tasks.workunit.client.0.smithi157.stderr:++ local cmd 2024-09-21T22:07:47.113 INFO:tasks.workunit.client.0.smithi157.stderr:+++ id -u 2024-09-21T22:07:47.114 INFO:tasks.workunit.client.0.smithi157.stderr:++ '[' 1000 -eq 0 ']' 2024-09-21T22:07:47.115 INFO:tasks.workunit.client.0.smithi157.stderr:+++ which rbd 2024-09-21T22:07:47.117 INFO:tasks.workunit.client.0.smithi157.stderr:++ cmd=/usr/bin/rbd 2024-09-21T22:07:47.117 INFO:tasks.workunit.client.0.smithi157.stderr:++ shift 2024-09-21T22:07:47.117 INFO:tasks.workunit.client.0.smithi157.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map --pool rbd --namespace ns --image testrbdnbd21553 --snap snap 2024-09-21T22:07:47.138 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:07:47.138 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-21T22:07:47.138 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-21T22:07:47.138 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-21T22:07:47.139 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.139 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:07:47.139 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.036355+0000", 2024-09-21T22:07:47.139 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 105.099378446, 2024-09-21T22:07:47.139 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.062441934999999997, 2024-09-21T22:07:47.139 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:47.139 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:47.139 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:47.139 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:47.139 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:07:47.139 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:07:47.139 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1 2024-09-21T22:07:47.139 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.139 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:47.140 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.140 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:47.140 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036355+0000", 2024-09-21T22:07:47.140 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:47.140 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.140 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.140 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:47.140 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036355+0000", 2024-09-21T22:07:47.140 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:47.140 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.140 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.140 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:47.140 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036357+0000", 2024-09-21T22:07:47.140 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.446e-06 2024-09-21T22:07:47.141 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.141 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.141 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:47.141 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036373+0000", 2024-09-21T22:07:47.141 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5447e-05 2024-09-21T22:07:47.141 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.141 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.141 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:47.141 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036399+0000", 2024-09-21T22:07:47.141 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6423000000000001e-05 2024-09-21T22:07:47.141 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.141 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.141 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:47.141 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036420+0000", 2024-09-21T22:07:47.141 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1129999999999999e-05 2024-09-21T22:07:47.142 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.142 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.142 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:47.142 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036557+0000", 2024-09-21T22:07:47.142 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013686899999999999 2024-09-21T22:07:47.142 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.142 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.142 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-21T22:07:47.142 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036585+0000", 2024-09-21T22:07:47.142 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8362999999999999e-05 2024-09-21T22:07:47.142 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.142 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.142 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:47.142 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.095068+0000", 2024-09-21T22:07:47.143 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058482972000000001 2024-09-21T22:07:47.143 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.143 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.143 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:47.143 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.095516+0000", 2024-09-21T22:07:47.143 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00044802999999999998 2024-09-21T22:07:47.143 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.143 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.143 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:47.143 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.095869+0000", 2024-09-21T22:07:47.143 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000352104 2024-09-21T22:07:47.143 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.143 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.143 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:47.144 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.097055+0000", 2024-09-21T22:07:47.144 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011865599999999999 2024-09-21T22:07:47.144 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.144 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.144 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:47.144 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.098637+0000", 2024-09-21T22:07:47.144 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001581458 2024-09-21T22:07:47.144 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.144 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.144 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:47.144 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.098728+0000", 2024-09-21T22:07:47.144 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1043000000000002e-05 2024-09-21T22:07:47.144 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.144 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.145 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:47.145 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.098797+0000", 2024-09-21T22:07:47.145 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9090000000000004e-05 2024-09-21T22:07:47.145 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:47.145 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:47.145 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:47.145 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.145 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.145 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4840.0:678 2.7 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:07:47.145 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.678289+0000", 2024-09-21T22:07:47.145 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 22.457444390999999, 2024-09-21T22:07:47.145 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.055089659999999999, 2024-09-21T22:07:47.145 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:47.145 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:47.146 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:47.146 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:47.146 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4840", 2024-09-21T22:07:47.146 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:47.146 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 678 2024-09-21T22:07:47.146 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.146 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:47.146 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.146 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:47.146 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.678289+0000", 2024-09-21T22:07:47.146 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:47.146 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.146 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.146 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:47.147 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.678289+0000", 2024-09-21T22:07:47.147 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:47.147 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.147 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.147 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:47.147 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.678291+0000", 2024-09-21T22:07:47.147 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7400000000000001e-06 2024-09-21T22:07:47.147 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.147 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.147 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:47.147 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.679129+0000", 2024-09-21T22:07:47.147 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00083801700000000004 2024-09-21T22:07:47.147 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.147 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.147 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:47.148 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.679161+0000", 2024-09-21T22:07:47.148 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2398999999999999e-05 2024-09-21T22:07:47.148 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.148 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.148 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:47.148 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.679176+0000", 2024-09-21T22:07:47.148 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5396999999999998e-05 2024-09-21T22:07:47.148 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.148 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.148 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:47.148 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.679256+0000", 2024-09-21T22:07:47.148 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9271999999999996e-05 2024-09-21T22:07:47.148 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.148 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.149 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:47.149 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.679333+0000", 2024-09-21T22:07:47.149 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7682999999999995e-05 2024-09-21T22:07:47.149 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.149 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.149 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-09-21T22:07:47.149 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.679457+0000", 2024-09-21T22:07:47.149 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012327900000000001 2024-09-21T22:07:47.149 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.149 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.149 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:47.149 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.713209+0000", 2024-09-21T22:07:47.149 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.033752058000000001 2024-09-21T22:07:47.149 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.149 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.149 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:47.150 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.733243+0000", 2024-09-21T22:07:47.150 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020034337999999999 2024-09-21T22:07:47.150 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.150 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.150 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:47.150 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.733320+0000", 2024-09-21T22:07:47.150 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.7354999999999994e-05 2024-09-21T22:07:47.150 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.150 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.150 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:47.150 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.733378+0000", 2024-09-21T22:07:47.150 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8121999999999997e-05 2024-09-21T22:07:47.150 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:47.150 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:47.150 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:47.151 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.151 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.151 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4840.0:680 2.7 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:07:47.151 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.744506+0000", 2024-09-21T22:07:47.151 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 22.391227592, 2024-09-21T22:07:47.151 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.063865055000000004, 2024-09-21T22:07:47.151 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:47.151 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:47.151 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:47.151 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:47.151 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4840", 2024-09-21T22:07:47.151 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:47.151 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 680 2024-09-21T22:07:47.152 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.152 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:47.152 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.152 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:47.152 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.744506+0000", 2024-09-21T22:07:47.152 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:47.152 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.152 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.152 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:47.152 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.744506+0000", 2024-09-21T22:07:47.152 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:47.152 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.152 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.152 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:47.152 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.744508+0000", 2024-09-21T22:07:47.153 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.103e-06 2024-09-21T22:07:47.153 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.153 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.153 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:47.153 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.745371+0000", 2024-09-21T22:07:47.153 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00086322199999999997 2024-09-21T22:07:47.153 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.153 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.153 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:47.153 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.745401+0000", 2024-09-21T22:07:47.153 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9955999999999999e-05 2024-09-21T22:07:47.153 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.153 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.153 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:47.154 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.745416+0000", 2024-09-21T22:07:47.154 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4873e-05 2024-09-21T22:07:47.154 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.154 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.154 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:47.154 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.745509+0000", 2024-09-21T22:07:47.154 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3171000000000006e-05 2024-09-21T22:07:47.154 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.154 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.154 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:47.154 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.745632+0000", 2024-09-21T22:07:47.154 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012272699999999999 2024-09-21T22:07:47.154 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.154 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.154 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-09-21T22:07:47.155 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.745789+0000", 2024-09-21T22:07:47.155 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015781399999999999 2024-09-21T22:07:47.155 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.155 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.155 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:47.155 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.785494+0000", 2024-09-21T22:07:47.155 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.039704401 2024-09-21T22:07:47.155 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.155 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.155 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:47.155 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.808090+0000", 2024-09-21T22:07:47.155 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022596207 2024-09-21T22:07:47.155 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.155 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.155 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:47.156 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.808246+0000", 2024-09-21T22:07:47.156 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015567200000000001 2024-09-21T22:07:47.156 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.156 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.156 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:47.156 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.808371+0000", 2024-09-21T22:07:47.156 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012490900000000001 2024-09-21T22:07:47.156 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:47.156 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:47.156 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:47.156 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.156 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.156 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4840.0:682 2.7 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:07:47.156 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.822415+0000", 2024-09-21T22:07:47.157 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 22.313318029000001, 2024-09-21T22:07:47.157 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.065044633000000004, 2024-09-21T22:07:47.157 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:47.157 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:47.157 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:47.157 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:47.157 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4840", 2024-09-21T22:07:47.157 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:47.157 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 682 2024-09-21T22:07:47.157 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.157 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:47.157 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.157 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:47.157 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.822415+0000", 2024-09-21T22:07:47.157 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:47.157 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.158 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.158 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:47.158 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.822415+0000", 2024-09-21T22:07:47.158 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:47.158 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.158 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.158 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:47.158 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.822417+0000", 2024-09-21T22:07:47.158 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8029999999999999e-06 2024-09-21T22:07:47.158 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.158 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.158 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:47.158 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.823308+0000", 2024-09-21T22:07:47.158 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00089146500000000003 2024-09-21T22:07:47.159 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.159 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.159 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:47.159 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.823339+0000", 2024-09-21T22:07:47.159 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0493000000000001e-05 2024-09-21T22:07:47.159 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.159 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.159 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:47.159 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.823354+0000", 2024-09-21T22:07:47.159 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5472999999999999e-05 2024-09-21T22:07:47.159 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.159 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.159 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:47.159 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.823476+0000", 2024-09-21T22:07:47.159 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000121209 2024-09-21T22:07:47.160 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.160 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.160 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:47.160 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.823650+0000", 2024-09-21T22:07:47.160 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017394599999999999 2024-09-21T22:07:47.160 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.160 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.160 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-09-21T22:07:47.160 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.823873+0000", 2024-09-21T22:07:47.160 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022314899999999999 2024-09-21T22:07:47.160 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.160 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.160 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:47.160 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.884396+0000", 2024-09-21T22:07:47.160 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.060523674999999999 2024-09-21T22:07:47.161 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.161 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.161 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:47.161 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.887238+0000", 2024-09-21T22:07:47.161 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028420450000000001 2024-09-21T22:07:47.161 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.161 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.161 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:47.161 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.887369+0000", 2024-09-21T22:07:47.161 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013010699999999999 2024-09-21T22:07:47.161 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.161 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.161 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:47.161 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.887460+0000", 2024-09-21T22:07:47.161 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1267999999999994e-05 2024-09-21T22:07:47.162 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:47.162 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:47.162 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:47.162 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.162 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.162 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4840.0:692 2.5 2:b1d55633:::rbd_data.126483d75f2d.000000000000000b:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:07:47.162 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:25.091048+0000", 2024-09-21T22:07:47.162 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 22.044685495, 2024-09-21T22:07:47.162 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.060917092999999999, 2024-09-21T22:07:47.162 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:47.162 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:47.162 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:47.162 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:47.162 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4840", 2024-09-21T22:07:47.162 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:47.163 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 692 2024-09-21T22:07:47.163 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.163 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:47.163 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.163 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:47.163 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.091048+0000", 2024-09-21T22:07:47.163 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:47.163 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.163 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.163 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:47.163 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.091048+0000", 2024-09-21T22:07:47.163 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:47.163 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.163 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.164 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:47.164 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.091050+0000", 2024-09-21T22:07:47.164 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.03e-06 2024-09-21T22:07:47.164 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.164 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.164 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:47.164 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.091989+0000", 2024-09-21T22:07:47.164 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00093932100000000001 2024-09-21T22:07:47.164 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.164 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.164 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:47.164 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.092023+0000", 2024-09-21T22:07:47.164 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4310000000000002e-05 2024-09-21T22:07:47.164 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.164 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.165 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:47.165 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.092040+0000", 2024-09-21T22:07:47.165 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6426000000000002e-05 2024-09-21T22:07:47.165 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.165 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.165 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:47.165 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.092175+0000", 2024-09-21T22:07:47.165 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013500299999999999 2024-09-21T22:07:47.165 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.165 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.165 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:47.165 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.092307+0000", 2024-09-21T22:07:47.165 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000132562 2024-09-21T22:07:47.165 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.166 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.166 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:47.166 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.092499+0000", 2024-09-21T22:07:47.166 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000191972 2024-09-21T22:07:47.166 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.166 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.166 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:47.166 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.151065+0000", 2024-09-21T22:07:47.166 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058565514999999999 2024-09-21T22:07:47.166 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.166 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.166 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:47.166 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.151844+0000", 2024-09-21T22:07:47.166 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00077930800000000004 2024-09-21T22:07:47.167 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.167 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.167 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:47.167 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.151916+0000", 2024-09-21T22:07:47.167 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1921999999999993e-05 2024-09-21T22:07:47.167 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.167 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.167 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:47.167 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.151965+0000", 2024-09-21T22:07:47.167 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8724000000000003e-05 2024-09-21T22:07:47.167 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:47.168 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:47.168 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:47.168 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.168 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.168 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4840.0:694 2.5 2:b1d55633:::rbd_data.126483d75f2d.000000000000000b:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:07:47.168 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:25.162829+0000", 2024-09-21T22:07:47.168 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 21.972904161999999, 2024-09-21T22:07:47.168 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.045653405000000001, 2024-09-21T22:07:47.168 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:47.168 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:47.168 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:47.168 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:47.168 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4840", 2024-09-21T22:07:47.169 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:47.169 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 694 2024-09-21T22:07:47.169 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.169 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:47.169 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.169 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:47.169 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.162829+0000", 2024-09-21T22:07:47.169 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:47.169 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.169 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.169 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:47.169 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.162829+0000", 2024-09-21T22:07:47.169 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:47.169 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.169 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.170 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:47.170 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.162831+0000", 2024-09-21T22:07:47.170 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7799999999999999e-06 2024-09-21T22:07:47.170 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.170 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.170 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:47.170 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.163904+0000", 2024-09-21T22:07:47.170 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001073673 2024-09-21T22:07:47.170 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.170 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.170 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:47.170 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.163937+0000", 2024-09-21T22:07:47.170 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2100000000000001e-05 2024-09-21T22:07:47.171 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.171 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.171 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:47.171 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.163952+0000", 2024-09-21T22:07:47.171 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5517000000000001e-05 2024-09-21T22:07:47.171 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.171 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.171 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:47.171 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.164060+0000", 2024-09-21T22:07:47.171 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000107909 2024-09-21T22:07:47.171 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.171 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.171 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:47.171 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.164162+0000", 2024-09-21T22:07:47.172 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000101619 2024-09-21T22:07:47.172 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.172 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.172 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:47.172 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.164318+0000", 2024-09-21T22:07:47.172 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015683599999999999 2024-09-21T22:07:47.172 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.172 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.172 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:47.172 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.207369+0000", 2024-09-21T22:07:47.172 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.043050401000000002 2024-09-21T22:07:47.172 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.172 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.172 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:47.172 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.208256+0000", 2024-09-21T22:07:47.173 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00088747800000000003 2024-09-21T22:07:47.173 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.173 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.173 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:47.173 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.208350+0000", 2024-09-21T22:07:47.173 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4129000000000006e-05 2024-09-21T22:07:47.173 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.173 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.173 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:47.173 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.208482+0000", 2024-09-21T22:07:47.173 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000131963 2024-09-21T22:07:47.173 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:47.173 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:47.173 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:47.173 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.174 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.174 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4840.0:696 2.5 2:b1d55633:::rbd_data.126483d75f2d.000000000000000b:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:07:47.174 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:25.218641+0000", 2024-09-21T22:07:47.174 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 21.917092227000001, 2024-09-21T22:07:47.174 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058363669999999999, 2024-09-21T22:07:47.174 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:47.174 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:47.174 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:47.174 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:47.174 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4840", 2024-09-21T22:07:47.174 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:47.174 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 696 2024-09-21T22:07:47.174 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.174 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:47.175 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.175 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:47.175 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.218641+0000", 2024-09-21T22:07:47.175 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:47.175 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.175 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.175 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:47.175 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.218641+0000", 2024-09-21T22:07:47.175 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:47.175 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.175 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.175 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:47.175 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.218642+0000", 2024-09-21T22:07:47.175 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.266e-06 2024-09-21T22:07:47.175 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.176 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.176 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:47.176 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.219432+0000", 2024-09-21T22:07:47.176 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00078986300000000002 2024-09-21T22:07:47.176 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.176 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.176 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:47.176 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.219450+0000", 2024-09-21T22:07:47.176 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7526000000000001e-05 2024-09-21T22:07:47.176 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.176 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.176 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:47.176 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.219459+0000", 2024-09-21T22:07:47.176 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3000000000000007e-06 2024-09-21T22:07:47.176 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.177 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.177 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:47.177 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.219655+0000", 2024-09-21T22:07:47.177 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019599099999999999 2024-09-21T22:07:47.177 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.177 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.177 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:47.177 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.219917+0000", 2024-09-21T22:07:47.177 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026198200000000001 2024-09-21T22:07:47.177 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.177 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.177 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:47.177 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.220198+0000", 2024-09-21T22:07:47.177 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00028154500000000002 2024-09-21T22:07:47.177 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.177 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.178 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:47.178 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.268856+0000", 2024-09-21T22:07:47.178 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.048657687999999998 2024-09-21T22:07:47.178 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.178 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.178 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:47.178 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.276903+0000", 2024-09-21T22:07:47.178 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0080472249999999999 2024-09-21T22:07:47.178 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.178 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.178 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:47.178 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.276958+0000", 2024-09-21T22:07:47.178 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4184999999999997e-05 2024-09-21T22:07:47.178 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.179 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.179 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:47.179 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.277005+0000", 2024-09-21T22:07:47.179 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7098999999999997e-05 2024-09-21T22:07:47.179 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:47.179 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:47.179 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:47.179 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.179 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.179 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4840.0:698 2.5 2:b1d55633:::rbd_data.126483d75f2d.000000000000000b:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:07:47.179 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:25.289380+0000", 2024-09-21T22:07:47.179 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 21.84635368, 2024-09-21T22:07:47.179 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.055691072000000001, 2024-09-21T22:07:47.179 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:47.179 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:47.180 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:47.180 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:47.180 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4840", 2024-09-21T22:07:47.180 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:47.180 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 698 2024-09-21T22:07:47.180 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.180 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:47.180 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.180 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:47.180 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.289380+0000", 2024-09-21T22:07:47.180 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:47.180 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.180 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.180 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:47.180 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.289380+0000", 2024-09-21T22:07:47.181 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:47.181 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.181 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.181 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:47.181 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.289381+0000", 2024-09-21T22:07:47.181 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3540000000000001e-06 2024-09-21T22:07:47.181 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.181 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.181 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:47.181 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.290221+0000", 2024-09-21T22:07:47.181 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00083999900000000002 2024-09-21T22:07:47.181 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.181 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.181 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:47.181 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.290238+0000", 2024-09-21T22:07:47.182 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7130999999999999e-05 2024-09-21T22:07:47.182 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.182 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.182 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:47.182 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.290247+0000", 2024-09-21T22:07:47.182 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6500000000000002e-06 2024-09-21T22:07:47.182 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.182 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.182 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:47.182 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.290320+0000", 2024-09-21T22:07:47.182 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3530999999999998e-05 2024-09-21T22:07:47.182 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.182 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.182 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:47.183 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.290433+0000", 2024-09-21T22:07:47.183 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000112744 2024-09-21T22:07:47.183 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.183 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.183 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:47.183 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.290605+0000", 2024-09-21T22:07:47.183 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017253700000000001 2024-09-21T22:07:47.183 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.183 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.183 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:47.183 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.320545+0000", 2024-09-21T22:07:47.183 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029939582999999999 2024-09-21T22:07:47.183 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.183 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.183 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:47.184 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.344991+0000", 2024-09-21T22:07:47.184 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024445956000000001 2024-09-21T22:07:47.184 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.184 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.184 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:47.184 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.345034+0000", 2024-09-21T22:07:47.184 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2806000000000003e-05 2024-09-21T22:07:47.184 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.184 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.184 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:47.184 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.345071+0000", 2024-09-21T22:07:47.184 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6780999999999999e-05 2024-09-21T22:07:47.184 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:47.184 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:47.184 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:47.185 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.185 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.185 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4840.0:725 2.0 2:08b4f96a:::rbd_data.126483d75f2d.000000000000000f:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:07:47.185 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:25.936440+0000", 2024-09-21T22:07:47.185 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 21.199293182000002, 2024-09-21T22:07:47.185 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.049962899999999998, 2024-09-21T22:07:47.185 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:47.185 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:47.185 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:47.185 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:47.185 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4840", 2024-09-21T22:07:47.185 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:47.185 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 725 2024-09-21T22:07:47.185 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.186 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:47.186 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.186 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:47.186 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.936440+0000", 2024-09-21T22:07:47.186 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:47.186 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.186 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.186 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:47.186 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.936440+0000", 2024-09-21T22:07:47.186 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:47.186 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.186 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.186 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:47.186 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.936442+0000", 2024-09-21T22:07:47.187 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0360000000000001e-06 2024-09-21T22:07:47.187 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.187 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.187 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:47.187 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.937560+0000", 2024-09-21T22:07:47.187 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011176300000000001 2024-09-21T22:07:47.187 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.187 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.187 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:47.187 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.937592+0000", 2024-09-21T22:07:47.187 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2743000000000001e-05 2024-09-21T22:07:47.187 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.187 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.187 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:47.187 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.937608+0000", 2024-09-21T22:07:47.188 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5477000000000001e-05 2024-09-21T22:07:47.188 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.188 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.188 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:47.188 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.937695+0000", 2024-09-21T22:07:47.188 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7220999999999997e-05 2024-09-21T22:07:47.188 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.188 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.188 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:47.188 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.937792+0000", 2024-09-21T22:07:47.188 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7261000000000003e-05 2024-09-21T22:07:47.188 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.188 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.188 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-09-21T22:07:47.189 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.937934+0000", 2024-09-21T22:07:47.189 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000141376 2024-09-21T22:07:47.189 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.189 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.189 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:47.189 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.975222+0000", 2024-09-21T22:07:47.189 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.037287995999999997 2024-09-21T22:07:47.189 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.189 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.189 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:47.189 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.986285+0000", 2024-09-21T22:07:47.189 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011063329 2024-09-21T22:07:47.189 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.189 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.189 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:47.190 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.986348+0000", 2024-09-21T22:07:47.190 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3330000000000005e-05 2024-09-21T22:07:47.190 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.190 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.190 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:47.190 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.986403+0000", 2024-09-21T22:07:47.190 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4500999999999999e-05 2024-09-21T22:07:47.190 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:47.190 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:47.190 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:47.190 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.190 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.190 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4840.0:727 2.0 2:08b4f96a:::rbd_data.126483d75f2d.000000000000000f:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:07:47.190 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:25.998176+0000", 2024-09-21T22:07:47.191 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 21.137556985, 2024-09-21T22:07:47.191 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.054236470000000002, 2024-09-21T22:07:47.191 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:47.191 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:47.191 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:47.191 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:47.191 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4840", 2024-09-21T22:07:47.191 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:47.191 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 727 2024-09-21T22:07:47.191 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.191 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:47.191 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.191 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:47.191 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.998176+0000", 2024-09-21T22:07:47.191 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:47.192 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.192 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.192 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:47.192 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.998176+0000", 2024-09-21T22:07:47.192 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:47.192 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.192 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.192 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:47.192 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.998179+0000", 2024-09-21T22:07:47.192 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3870000000000002e-06 2024-09-21T22:07:47.192 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.192 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.192 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:47.192 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.999060+0000", 2024-09-21T22:07:47.193 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00088148800000000004 2024-09-21T22:07:47.193 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.193 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.193 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:47.193 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.999094+0000", 2024-09-21T22:07:47.193 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3779000000000003e-05 2024-09-21T22:07:47.193 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.193 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.193 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:47.193 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.999111+0000", 2024-09-21T22:07:47.193 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6990000000000002e-05 2024-09-21T22:07:47.193 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.193 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.193 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:47.193 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.999190+0000", 2024-09-21T22:07:47.194 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9354e-05 2024-09-21T22:07:47.194 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.194 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.194 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:47.194 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.999251+0000", 2024-09-21T22:07:47.194 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0733000000000003e-05 2024-09-21T22:07:47.194 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.194 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.194 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-09-21T22:07:47.194 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.999342+0000", 2024-09-21T22:07:47.194 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1376000000000001e-05 2024-09-21T22:07:47.194 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.194 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.194 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:47.194 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.032392+0000", 2024-09-21T22:07:47.195 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.033049626999999998 2024-09-21T22:07:47.195 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.195 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.195 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:47.195 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.052249+0000", 2024-09-21T22:07:47.195 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019856944000000001 2024-09-21T22:07:47.195 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.195 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.195 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:47.195 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.052342+0000", 2024-09-21T22:07:47.195 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2650000000000002e-05 2024-09-21T22:07:47.195 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.195 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.195 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:47.196 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.052413+0000", 2024-09-21T22:07:47.196 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1142000000000004e-05 2024-09-21T22:07:47.196 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:47.196 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:47.196 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:47.196 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.196 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.196 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4840.0:729 2.0 2:08b4f96a:::rbd_data.126483d75f2d.000000000000000f:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:07:47.196 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:26.062012+0000", 2024-09-21T22:07:47.196 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 21.073720993999999, 2024-09-21T22:07:47.196 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.060656823999999998, 2024-09-21T22:07:47.196 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:47.196 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:47.196 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:47.196 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:47.197 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4840", 2024-09-21T22:07:47.197 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:47.197 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 729 2024-09-21T22:07:47.197 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.197 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:47.197 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.197 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:47.197 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.062012+0000", 2024-09-21T22:07:47.197 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:47.197 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.198 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.198 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:47.198 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.062012+0000", 2024-09-21T22:07:47.198 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:47.198 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.198 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.198 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:47.198 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.062013+0000", 2024-09-21T22:07:47.198 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.243e-06 2024-09-21T22:07:47.198 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.198 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.198 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:47.198 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.062827+0000", 2024-09-21T22:07:47.198 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00081374799999999999 2024-09-21T22:07:47.198 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.199 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.199 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:47.199 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.062845+0000", 2024-09-21T22:07:47.199 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.808e-05 2024-09-21T22:07:47.199 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.199 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.199 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:47.199 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.062854+0000", 2024-09-21T22:07:47.199 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1039999999999996e-06 2024-09-21T22:07:47.199 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.199 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.199 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:47.199 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.062970+0000", 2024-09-21T22:07:47.199 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011601 2024-09-21T22:07:47.200 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.200 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.200 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:47.200 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.063230+0000", 2024-09-21T22:07:47.200 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025918799999999999 2024-09-21T22:07:47.200 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.200 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.200 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-09-21T22:07:47.200 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.063641+0000", 2024-09-21T22:07:47.200 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00041152799999999998 2024-09-21T22:07:47.200 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.200 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.200 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:47.200 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.122015+0000", 2024-09-21T22:07:47.200 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058374246999999997 2024-09-21T22:07:47.201 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.201 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.201 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:47.201 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.122451+0000", 2024-09-21T22:07:47.201 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000435771 2024-09-21T22:07:47.201 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.201 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.201 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:47.201 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.122576+0000", 2024-09-21T22:07:47.201 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012469099999999999 2024-09-21T22:07:47.201 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.201 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.201 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:47.201 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.122669+0000", 2024-09-21T22:07:47.201 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3214000000000006e-05 2024-09-21T22:07:47.202 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:47.202 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:47.202 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:47.202 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.202 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.202 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4840.0:731 2.0 2:08b4f96a:::rbd_data.126483d75f2d.000000000000000f:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:07:47.202 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:26.135603+0000", 2024-09-21T22:07:47.202 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 21.000130494, 2024-09-21T22:07:47.202 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.056838474999999999, 2024-09-21T22:07:47.202 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:47.202 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:47.202 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:47.202 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:47.202 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4840", 2024-09-21T22:07:47.203 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:47.203 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 731 2024-09-21T22:07:47.203 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.203 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:47.203 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.203 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:47.203 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.135603+0000", 2024-09-21T22:07:47.203 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:47.203 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.203 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.203 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:47.203 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.135603+0000", 2024-09-21T22:07:47.203 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:47.203 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.203 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.204 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:47.204 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.135605+0000", 2024-09-21T22:07:47.204 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9700000000000002e-06 2024-09-21T22:07:47.204 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.204 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.204 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:47.204 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.136554+0000", 2024-09-21T22:07:47.204 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00094886700000000005 2024-09-21T22:07:47.204 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.204 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.204 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:47.204 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.136582+0000", 2024-09-21T22:07:47.204 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8393000000000001e-05 2024-09-21T22:07:47.204 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.205 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.205 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:47.205 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.136596+0000", 2024-09-21T22:07:47.205 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.43e-05 2024-09-21T22:07:47.205 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.205 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.205 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:47.205 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.136691+0000", 2024-09-21T22:07:47.205 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4843e-05 2024-09-21T22:07:47.205 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.205 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.205 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:47.205 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.136873+0000", 2024-09-21T22:07:47.205 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018237099999999999 2024-09-21T22:07:47.206 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.206 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.206 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-09-21T22:07:47.206 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.137140+0000", 2024-09-21T22:07:47.206 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026642700000000002 2024-09-21T22:07:47.206 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.206 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.206 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:47.206 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.180580+0000", 2024-09-21T22:07:47.206 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.043440078 2024-09-21T22:07:47.206 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.206 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.206 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:47.206 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.192175+0000", 2024-09-21T22:07:47.207 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011594959 2024-09-21T22:07:47.207 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.207 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.207 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:47.207 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.192353+0000", 2024-09-21T22:07:47.207 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017850200000000001 2024-09-21T22:07:47.207 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.207 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.207 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:47.207 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.192441+0000", 2024-09-21T22:07:47.207 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7764999999999997e-05 2024-09-21T22:07:47.207 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:47.207 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:47.207 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:47.207 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.208 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.208 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5008.0:116 2.7 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-21T22:07:47.208 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:37.866431+0000", 2024-09-21T22:07:47.208 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.2693022220000003, 2024-09-21T22:07:47.208 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.046674751, 2024-09-21T22:07:47.208 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:47.208 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:47.208 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:47.208 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:47.208 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5008", 2024-09-21T22:07:47.208 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/1400114666", 2024-09-21T22:07:47.208 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 116 2024-09-21T22:07:47.208 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.208 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:47.209 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.209 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:47.209 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.866431+0000", 2024-09-21T22:07:47.209 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:47.209 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.209 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.209 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:47.209 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.866431+0000", 2024-09-21T22:07:47.209 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:47.209 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.209 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.209 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:47.209 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.866433+0000", 2024-09-21T22:07:47.209 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.592e-06 2024-09-21T22:07:47.210 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.210 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.210 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:47.210 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.867223+0000", 2024-09-21T22:07:47.210 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00079020900000000005 2024-09-21T22:07:47.210 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.210 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.210 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:47.210 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.867243+0000", 2024-09-21T22:07:47.210 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0333e-05 2024-09-21T22:07:47.210 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.210 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.210 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:47.210 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.867254+0000", 2024-09-21T22:07:47.210 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0558e-05 2024-09-21T22:07:47.211 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.211 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.211 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:47.211 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.867324+0000", 2024-09-21T22:07:47.211 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0290999999999995e-05 2024-09-21T22:07:47.211 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.211 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.211 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:47.211 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.867398+0000", 2024-09-21T22:07:47.211 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4509000000000001e-05 2024-09-21T22:07:47.211 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.211 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.211 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-09-21T22:07:47.211 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.867514+0000", 2024-09-21T22:07:47.211 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011581799999999999 2024-09-21T22:07:47.212 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.212 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.212 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:47.212 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.899084+0000", 2024-09-21T22:07:47.212 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.031569901999999997 2024-09-21T22:07:47.212 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.212 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.212 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:47.212 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.913011+0000", 2024-09-21T22:07:47.212 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.013926945 2024-09-21T22:07:47.212 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.212 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.212 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:47.212 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.913063+0000", 2024-09-21T22:07:47.213 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2352999999999998e-05 2024-09-21T22:07:47.213 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.213 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.213 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:47.213 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.913106+0000", 2024-09-21T22:07:47.213 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2240999999999998e-05 2024-09-21T22:07:47.213 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:47.213 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:47.213 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:47.213 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.213 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.213 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5008.0:143 2.5 2:a70ae99f:::rbd_data.126483d75f2d.000000000000000c:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e72)", 2024-09-21T22:07:47.213 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:38.527661+0000", 2024-09-21T22:07:47.213 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.6080724120000003, 2024-09-21T22:07:47.213 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.042001680999999999, 2024-09-21T22:07:47.214 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:47.214 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:47.214 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:47.214 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:47.214 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5008", 2024-09-21T22:07:47.214 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/1400114666", 2024-09-21T22:07:47.214 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 143 2024-09-21T22:07:47.214 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.214 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:47.214 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.214 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:47.214 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.527661+0000", 2024-09-21T22:07:47.214 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:47.214 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.215 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.215 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:47.215 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.527661+0000", 2024-09-21T22:07:47.215 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:47.215 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.215 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.215 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:47.215 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.527662+0000", 2024-09-21T22:07:47.215 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1540000000000001e-06 2024-09-21T22:07:47.215 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.215 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.215 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:47.215 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.528503+0000", 2024-09-21T22:07:47.215 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00084101800000000002 2024-09-21T22:07:47.215 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.216 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.216 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:47.216 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.528523+0000", 2024-09-21T22:07:47.216 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0322e-05 2024-09-21T22:07:47.216 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.216 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.216 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:47.216 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.528534+0000", 2024-09-21T22:07:47.216 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0564000000000001e-05 2024-09-21T22:07:47.216 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.216 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.216 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:47.216 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.528636+0000", 2024-09-21T22:07:47.216 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010234300000000001 2024-09-21T22:07:47.216 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.217 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.217 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:47.217 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.528736+0000", 2024-09-21T22:07:47.217 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9910999999999999e-05 2024-09-21T22:07:47.217 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.217 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.217 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:47.217 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.528896+0000", 2024-09-21T22:07:47.217 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016024 2024-09-21T22:07:47.217 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.217 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.217 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:47.217 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.566383+0000", 2024-09-21T22:07:47.217 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.037487174999999998 2024-09-21T22:07:47.217 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.218 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.218 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:47.218 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.569570+0000", 2024-09-21T22:07:47.218 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0031864269999999999 2024-09-21T22:07:47.218 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.218 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.218 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:47.218 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.569620+0000", 2024-09-21T22:07:47.218 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0321000000000002e-05 2024-09-21T22:07:47.218 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.218 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.218 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:47.218 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.569662+0000", 2024-09-21T22:07:47.219 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2206000000000002e-05 2024-09-21T22:07:47.219 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:47.219 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:47.219 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:47.219 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.219 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.219 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5008.0:147 2.5 2:a70ae99f:::rbd_data.126483d75f2d.000000000000000c:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e72)", 2024-09-21T22:07:47.219 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:38.627134+0000", 2024-09-21T22:07:47.219 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.5085993250000005, 2024-09-21T22:07:47.219 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.052909400000000002, 2024-09-21T22:07:47.219 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:47.219 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:47.219 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:47.219 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:47.219 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5008", 2024-09-21T22:07:47.220 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/1400114666", 2024-09-21T22:07:47.220 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 147 2024-09-21T22:07:47.220 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.220 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:47.220 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.220 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:47.220 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.627134+0000", 2024-09-21T22:07:47.220 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:47.220 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.220 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.220 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:47.220 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.627134+0000", 2024-09-21T22:07:47.220 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:47.220 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.221 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.221 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:47.221 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.627136+0000", 2024-09-21T22:07:47.221 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2129999999999998e-06 2024-09-21T22:07:47.221 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.221 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.221 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:47.221 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.628233+0000", 2024-09-21T22:07:47.221 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001097223 2024-09-21T22:07:47.221 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.221 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.221 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:47.221 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.628266+0000", 2024-09-21T22:07:47.221 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2382999999999998e-05 2024-09-21T22:07:47.222 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.222 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.222 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:47.222 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.628281+0000", 2024-09-21T22:07:47.222 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5417000000000002e-05 2024-09-21T22:07:47.222 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.222 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.222 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:47.222 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.628412+0000", 2024-09-21T22:07:47.222 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013106700000000001 2024-09-21T22:07:47.222 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.222 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.222 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:47.223 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.628529+0000", 2024-09-21T22:07:47.223 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000117269 2024-09-21T22:07:47.223 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.223 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.223 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:47.223 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.628704+0000", 2024-09-21T22:07:47.223 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017447599999999999 2024-09-21T22:07:47.223 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.223 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.223 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:47.223 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.676772+0000", 2024-09-21T22:07:47.223 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.048068196000000001 2024-09-21T22:07:47.223 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.223 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.223 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:47.224 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.679874+0000", 2024-09-21T22:07:47.224 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0031019770000000001 2024-09-21T22:07:47.224 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.224 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.224 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:47.224 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.679974+0000", 2024-09-21T22:07:47.224 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9579000000000003e-05 2024-09-21T22:07:47.224 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.224 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.224 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:47.224 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.680043+0000", 2024-09-21T22:07:47.224 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9599999999999998e-05 2024-09-21T22:07:47.224 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:47.224 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:47.224 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:47.225 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.225 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.225 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5008.0:149 2.5 2:a70ae99f:::rbd_data.126483d75f2d.000000000000000c:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e72)", 2024-09-21T22:07:47.225 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:38.690581+0000", 2024-09-21T22:07:47.225 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.4451523930000008, 2024-09-21T22:07:47.225 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.051360648000000002, 2024-09-21T22:07:47.225 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:47.225 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:47.225 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:47.225 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:47.225 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5008", 2024-09-21T22:07:47.225 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/1400114666", 2024-09-21T22:07:47.225 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 149 2024-09-21T22:07:47.225 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.226 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:47.226 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.226 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:47.226 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.690581+0000", 2024-09-21T22:07:47.226 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:47.226 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.226 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.226 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:47.226 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.690581+0000", 2024-09-21T22:07:47.226 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:47.226 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.226 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.226 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:47.226 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.690583+0000", 2024-09-21T22:07:47.227 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3e-06 2024-09-21T22:07:47.227 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.227 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.227 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:47.227 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.691373+0000", 2024-09-21T22:07:47.227 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00078976800000000005 2024-09-21T22:07:47.227 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.227 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.227 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:47.227 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.691406+0000", 2024-09-21T22:07:47.227 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2787000000000003e-05 2024-09-21T22:07:47.227 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.227 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.227 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:47.228 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.691422+0000", 2024-09-21T22:07:47.228 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6443000000000002e-05 2024-09-21T22:07:47.228 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.228 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.228 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:47.228 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.691597+0000", 2024-09-21T22:07:47.229 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017483599999999999 2024-09-21T22:07:47.229 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.230 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.230 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:47.230 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.691737+0000", 2024-09-21T22:07:47.230 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014013900000000001 2024-09-21T22:07:47.230 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.230 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.230 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:47.230 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.691914+0000", 2024-09-21T22:07:47.230 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017682 2024-09-21T22:07:47.230 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.230 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.230 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:47.230 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.732757+0000", 2024-09-21T22:07:47.230 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.040843438000000003 2024-09-21T22:07:47.231 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.231 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.231 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:47.231 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.741844+0000", 2024-09-21T22:07:47.231 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0090864139999999993 2024-09-21T22:07:47.231 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.231 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.231 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:47.231 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.741898+0000", 2024-09-21T22:07:47.231 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4336000000000003e-05 2024-09-21T22:07:47.231 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.231 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.231 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:47.231 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.741941+0000", 2024-09-21T22:07:47.231 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.3367e-05 2024-09-21T22:07:47.232 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:47.232 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:47.232 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:47.232 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.232 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.232 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5008.0:170 2.0 2:08b4f96a:::rbd_data.126483d75f2d.000000000000000f:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e72)", 2024-09-21T22:07:47.232 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:39.132043+0000", 2024-09-21T22:07:47.232 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.0036904680000003, 2024-09-21T22:07:47.232 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.050566569999999998, 2024-09-21T22:07:47.232 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:47.232 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:47.232 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:47.232 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:47.232 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5008", 2024-09-21T22:07:47.233 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/1400114666", 2024-09-21T22:07:47.233 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 170 2024-09-21T22:07:47.233 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.233 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:47.233 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.233 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:47.233 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.132043+0000", 2024-09-21T22:07:47.233 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:47.233 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.233 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.233 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:47.233 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.132043+0000", 2024-09-21T22:07:47.233 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:47.233 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.234 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.234 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:47.234 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.132065+0000", 2024-09-21T22:07:47.234 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1999999999999999e-05 2024-09-21T22:07:47.234 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.234 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.234 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:47.234 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.132972+0000", 2024-09-21T22:07:47.234 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00090684399999999997 2024-09-21T22:07:47.234 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.234 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.234 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:47.234 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.133002+0000", 2024-09-21T22:07:47.234 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0936000000000003e-05 2024-09-21T22:07:47.234 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.235 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.235 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:47.235 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.133019+0000", 2024-09-21T22:07:47.235 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6004000000000001e-05 2024-09-21T22:07:47.235 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.235 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.235 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:47.235 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.133170+0000", 2024-09-21T22:07:47.235 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015154099999999999 2024-09-21T22:07:47.235 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.235 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.235 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:47.235 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.133269+0000", 2024-09-21T22:07:47.235 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.8876999999999995e-05 2024-09-21T22:07:47.236 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.236 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.236 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-09-21T22:07:47.236 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.133413+0000", 2024-09-21T22:07:47.236 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000143931 2024-09-21T22:07:47.236 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.236 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.236 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:47.236 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.181524+0000", 2024-09-21T22:07:47.236 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.048111196000000002 2024-09-21T22:07:47.236 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.236 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.236 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:47.236 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.182494+0000", 2024-09-21T22:07:47.237 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00096976799999999998 2024-09-21T22:07:47.237 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.237 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.237 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:47.237 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.182564+0000", 2024-09-21T22:07:47.237 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0275000000000001e-05 2024-09-21T22:07:47.237 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.237 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.237 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:47.237 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.182609+0000", 2024-09-21T22:07:47.237 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5198000000000003e-05 2024-09-21T22:07:47.237 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:47.237 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:47.237 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:47.237 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.238 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.238 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5008.0:172 2.0 2:08b4f96a:::rbd_data.126483d75f2d.000000000000000f:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e72)", 2024-09-21T22:07:47.238 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:39.193798+0000", 2024-09-21T22:07:47.238 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 7.9419355859999996, 2024-09-21T22:07:47.238 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.050829119999999998, 2024-09-21T22:07:47.238 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:47.238 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:47.238 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:47.238 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:47.238 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5008", 2024-09-21T22:07:47.238 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/1400114666", 2024-09-21T22:07:47.238 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 172 2024-09-21T22:07:47.238 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.238 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:47.239 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.239 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:47.239 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.193798+0000", 2024-09-21T22:07:47.239 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:47.239 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.239 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.239 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:47.239 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.193798+0000", 2024-09-21T22:07:47.239 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:47.239 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.239 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.239 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:47.239 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.193800+0000", 2024-09-21T22:07:47.239 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4770000000000002e-06 2024-09-21T22:07:47.240 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.240 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.240 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:47.240 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.194508+0000", 2024-09-21T22:07:47.240 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00070776900000000002 2024-09-21T22:07:47.240 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.240 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.240 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:47.240 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.194540+0000", 2024-09-21T22:07:47.240 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2647000000000001e-05 2024-09-21T22:07:47.240 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.240 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.240 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:47.240 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.194556+0000", 2024-09-21T22:07:47.241 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5492999999999999e-05 2024-09-21T22:07:47.241 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.241 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.241 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:47.241 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.194747+0000", 2024-09-21T22:07:47.241 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019105899999999999 2024-09-21T22:07:47.241 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.241 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.241 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:47.241 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.194886+0000", 2024-09-21T22:07:47.241 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013939599999999999 2024-09-21T22:07:47.241 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.241 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.241 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-09-21T22:07:47.241 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.195089+0000", 2024-09-21T22:07:47.242 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020208499999999999 2024-09-21T22:07:47.242 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.242 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.242 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:47.242 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.243304+0000", 2024-09-21T22:07:47.242 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.048215757999999997 2024-09-21T22:07:47.242 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.242 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.242 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:47.242 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.244466+0000", 2024-09-21T22:07:47.242 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011612969999999999 2024-09-21T22:07:47.242 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.242 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.243 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:47.243 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.244561+0000", 2024-09-21T22:07:47.243 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4995999999999996e-05 2024-09-21T22:07:47.243 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.243 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.243 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:47.243 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.244627+0000", 2024-09-21T22:07:47.243 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6142999999999993e-05 2024-09-21T22:07:47.243 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:47.243 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:47.243 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:47.243 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.243 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.243 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5008.0:174 2.0 2:08b4f96a:::rbd_data.126483d75f2d.000000000000000f:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e72)", 2024-09-21T22:07:47.243 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:39.256840+0000", 2024-09-21T22:07:47.244 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 7.8788927009999998, 2024-09-21T22:07:47.244 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.048802131999999998, 2024-09-21T22:07:47.244 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:47.244 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:47.244 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:47.244 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:47.244 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5008", 2024-09-21T22:07:47.244 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/1400114666", 2024-09-21T22:07:47.244 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 174 2024-09-21T22:07:47.244 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.244 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:47.244 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.244 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:47.244 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.256840+0000", 2024-09-21T22:07:47.244 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:47.245 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.245 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.245 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:47.245 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.256840+0000", 2024-09-21T22:07:47.245 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:47.245 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.245 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.245 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:47.245 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.256842+0000", 2024-09-21T22:07:47.245 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.55e-06 2024-09-21T22:07:47.245 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.245 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.245 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:47.245 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.257667+0000", 2024-09-21T22:07:47.245 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00082503100000000005 2024-09-21T22:07:47.246 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.246 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.246 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:47.246 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.257696+0000", 2024-09-21T22:07:47.246 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8629999999999999e-05 2024-09-21T22:07:47.246 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.246 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.246 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:47.246 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.257710+0000", 2024-09-21T22:07:47.246 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4323e-05 2024-09-21T22:07:47.246 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.246 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.246 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:47.246 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.257847+0000", 2024-09-21T22:07:47.247 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000137465 2024-09-21T22:07:47.247 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.247 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.247 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:47.247 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.257944+0000", 2024-09-21T22:07:47.247 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.6866000000000004e-05 2024-09-21T22:07:47.247 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.247 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.247 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-09-21T22:07:47.247 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.258081+0000", 2024-09-21T22:07:47.247 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000136441 2024-09-21T22:07:47.247 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.247 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.247 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:47.247 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.299971+0000", 2024-09-21T22:07:47.248 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.041890278000000003 2024-09-21T22:07:47.248 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.248 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.248 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:47.248 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.305494+0000", 2024-09-21T22:07:47.248 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0055228320000000001 2024-09-21T22:07:47.248 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.248 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.248 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:47.248 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.305581+0000", 2024-09-21T22:07:47.248 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7157000000000006e-05 2024-09-21T22:07:47.248 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.248 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.248 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:47.248 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.305643+0000", 2024-09-21T22:07:47.249 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1558999999999998e-05 2024-09-21T22:07:47.249 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:47.249 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:47.249 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:47.249 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.249 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.249 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5008.0:176 2.0 2:08b4f96a:::rbd_data.126483d75f2d.000000000000000f:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e72)", 2024-09-21T22:07:47.249 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:39.315531+0000", 2024-09-21T22:07:47.249 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 7.8202019370000002, 2024-09-21T22:07:47.249 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047857522999999999, 2024-09-21T22:07:47.249 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:47.249 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:47.249 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:47.249 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:47.250 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5008", 2024-09-21T22:07:47.250 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/1400114666", 2024-09-21T22:07:47.250 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 176 2024-09-21T22:07:47.250 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.250 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:47.250 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.250 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:47.250 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.315531+0000", 2024-09-21T22:07:47.250 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:47.250 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.250 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.250 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:47.250 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.315531+0000", 2024-09-21T22:07:47.250 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:47.251 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.251 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.251 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:47.251 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.315533+0000", 2024-09-21T22:07:47.251 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2000000000000001e-06 2024-09-21T22:07:47.251 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.251 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.251 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:47.251 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.316293+0000", 2024-09-21T22:07:47.251 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00075914600000000004 2024-09-21T22:07:47.251 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.251 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.251 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:47.251 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.316325+0000", 2024-09-21T22:07:47.251 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2481999999999998e-05 2024-09-21T22:07:47.252 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.252 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.252 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:47.252 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.316340+0000", 2024-09-21T22:07:47.252 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5057e-05 2024-09-21T22:07:47.252 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.252 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.252 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:47.252 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.316486+0000", 2024-09-21T22:07:47.252 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014624000000000001 2024-09-21T22:07:47.252 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.252 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.252 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:47.252 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.316635+0000", 2024-09-21T22:07:47.252 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000148529 2024-09-21T22:07:47.253 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.253 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.253 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-09-21T22:07:47.253 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.316785+0000", 2024-09-21T22:07:47.253 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015023900000000001 2024-09-21T22:07:47.253 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.253 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.253 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:47.253 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.360089+0000", 2024-09-21T22:07:47.253 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.043303361999999998 2024-09-21T22:07:47.253 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.253 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.253 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:47.253 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.363289+0000", 2024-09-21T22:07:47.253 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0032002200000000001 2024-09-21T22:07:47.254 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.254 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.254 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:47.254 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.363347+0000", 2024-09-21T22:07:47.254 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8662000000000002e-05 2024-09-21T22:07:47.254 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.254 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.254 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:47.254 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:39.363389+0000", 2024-09-21T22:07:47.254 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1386e-05 2024-09-21T22:07:47.254 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:47.254 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:47.254 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:47.254 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:47.255 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:47.255 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:07:47.256 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-21T22:07:47.257 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T22:07:47.257 INFO:teuthology.orchestra.run.smithi039.stderr:no valid command found; 10 closest matches: 2024-09-21T22:07:47.257 INFO:teuthology.orchestra.run.smithi039.stderr:0 2024-09-21T22:07:47.257 INFO:teuthology.orchestra.run.smithi039.stderr:1 2024-09-21T22:07:47.257 INFO:teuthology.orchestra.run.smithi039.stderr:2 2024-09-21T22:07:47.257 INFO:teuthology.orchestra.run.smithi039.stderr:abort 2024-09-21T22:07:47.257 INFO:teuthology.orchestra.run.smithi039.stderr:assert 2024-09-21T22:07:47.258 INFO:teuthology.orchestra.run.smithi039.stderr:bench [] [] [] [] 2024-09-21T22:07:47.258 INFO:teuthology.orchestra.run.smithi039.stderr:bluefs debug_inject_read_zeros 2024-09-21T22:07:47.258 INFO:teuthology.orchestra.run.smithi039.stderr:bluefs files list 2024-09-21T22:07:47.258 INFO:teuthology.orchestra.run.smithi039.stderr:bluefs stats 2024-09-21T22:07:47.258 INFO:teuthology.orchestra.run.smithi039.stderr:bluestore allocator dump block 2024-09-21T22:07:47.258 INFO:teuthology.orchestra.run.smithi039.stderr:Error EINVAL: invalid command 2024-09-21T22:07:47.258 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-09-21T22:07:47.258 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:47.254+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:47.282 INFO:tasks.workunit.client.0.smithi157.stderr:+ DEV=/dev/nbd0 2024-09-21T22:07:47.282 INFO:tasks.workunit.client.0.smithi157.stderr:+ get_pid rbd ns 2024-09-21T22:07:47.282 INFO:tasks.workunit.client.0.smithi157.stderr:+ local pool=rbd 2024-09-21T22:07:47.282 INFO:tasks.workunit.client.0.smithi157.stderr:+ local ns=ns 2024-09-21T22:07:47.282 INFO:tasks.workunit.client.0.smithi157.stderr:++ rbd device --device-type nbd --format xml list 2024-09-21T22:07:47.283 INFO:tasks.workunit.client.0.smithi157.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\''ns'\''][image='\''testrbdnbd21553'\''][device='\''/dev/nbd0'\'']/id' 2024-09-21T22:07:47.348 INFO:tasks.workunit.client.0.smithi157.stderr:+ PID=23188 2024-09-21T22:07:47.348 INFO:tasks.workunit.client.0.smithi157.stderr:+ test -n 23188 2024-09-21T22:07:47.348 INFO:tasks.workunit.client.0.smithi157.stderr:+ ps -p 23188 -C rbd-nbd 2024-09-21T22:07:47.354 INFO:tasks.workunit.client.0.smithi157.stdout: PID TTY TIME CMD 2024-09-21T22:07:47.354 INFO:tasks.workunit.client.0.smithi157.stdout: 23188 ? 00:00:00 rbd-nbd 2024-09-21T22:07:47.355 INFO:tasks.workunit.client.0.smithi157.stderr:+ unmap_device '--pool rbd --namespace ns --image testrbdnbd21553 --snap snap' 23188 2024-09-21T22:07:47.355 INFO:tasks.workunit.client.0.smithi157.stderr:+ local 'args=--pool rbd --namespace ns --image testrbdnbd21553 --snap snap' 2024-09-21T22:07:47.355 INFO:tasks.workunit.client.0.smithi157.stderr:+ local pid=23188 2024-09-21T22:07:47.355 INFO:tasks.workunit.client.0.smithi157.stderr:+ _sudo rbd device --device-type nbd unmap --pool rbd --namespace ns --image testrbdnbd21553 --snap snap 2024-09-21T22:07:47.355 INFO:tasks.workunit.client.0.smithi157.stderr:+ local cmd 2024-09-21T22:07:47.355 INFO:tasks.workunit.client.0.smithi157.stderr:++ id -u 2024-09-21T22:07:47.356 INFO:tasks.workunit.client.0.smithi157.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T22:07:47.356 INFO:tasks.workunit.client.0.smithi157.stderr:++ which rbd 2024-09-21T22:07:47.356 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:07:47.352+0000 7f2075a06640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31720) UID: 0 2024-09-21T22:07:47.357 INFO:tasks.workunit.client.0.smithi157.stderr:+ cmd=/usr/bin/rbd 2024-09-21T22:07:47.357 INFO:tasks.workunit.client.0.smithi157.stderr:+ shift 2024-09-21T22:07:47.357 INFO:tasks.workunit.client.0.smithi157.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap --pool rbd --namespace ns --image testrbdnbd21553 --snap snap 2024-09-21T22:07:47.430 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:07:47.430 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:07:47.430 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-09-21T22:07:47.430 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:07:47.444 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T22:07:47.445 INFO:tasks.workunit.client.0.smithi157.stderr:+ rbd device --device-type nbd list 2024-09-21T22:07:47.446 INFO:tasks.workunit.client.0.smithi157.stderr:+ expect_false grep '^23188\b' 2024-09-21T22:07:47.446 INFO:tasks.workunit.client.0.smithi157.stderr:+ grep '^23188\b' 2024-09-21T22:07:47.457 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:47.454+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:47.499 INFO:tasks.workunit.client.0.smithi157.stderr:+ return 0 2024-09-21T22:07:47.500 INFO:tasks.workunit.client.0.smithi157.stderr:+ ps -C rbd-nbd 2024-09-21T22:07:47.500 INFO:tasks.workunit.client.0.smithi157.stderr:+ expect_false grep '^ *23188\b' 2024-09-21T22:07:47.500 INFO:tasks.workunit.client.0.smithi157.stderr:+ grep '^ *23188\b' 2024-09-21T22:07:47.506 INFO:tasks.workunit.client.0.smithi157.stderr:+ return 0 2024-09-21T22:07:47.506 INFO:tasks.workunit.client.0.smithi157.stderr:+ sleep 0.5 2024-09-21T22:07:47.558 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:47.554+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:47.613 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:07:47.613 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:07:47.613 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-21T22:07:47.614 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-21T22:07:47.614 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:07:47.628 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T22:07:47.674 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:47.672+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:47.758 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:07:47.759 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-21T22:07:47.759 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-21T22:07:47.759 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-21T22:07:47.759 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.759 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4887.0:1 3.3 3:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e59)", 2024-09-21T22:07:47.759 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:17.151514+0000", 2024-09-21T22:07:47.759 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 30.605032076000001, 2024-09-21T22:07:47.759 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.01157512, 2024-09-21T22:07:47.759 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:47.759 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:47.759 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:47.759 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:47.759 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4887", 2024-09-21T22:07:47.760 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/3431621434", 2024-09-21T22:07:47.760 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1 2024-09-21T22:07:47.760 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.760 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:47.760 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.760 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:47.760 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.151514+0000", 2024-09-21T22:07:47.760 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:47.760 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.760 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.761 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:47.761 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.151514+0000", 2024-09-21T22:07:47.761 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:47.761 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.761 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.761 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:47.761 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.151515+0000", 2024-09-21T22:07:47.761 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.469e-06 2024-09-21T22:07:47.761 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.761 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.761 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:47.761 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.151522+0000", 2024-09-21T22:07:47.761 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1330000000000001e-06 2024-09-21T22:07:47.761 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.762 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.762 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:47.762 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.151536+0000", 2024-09-21T22:07:47.762 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3563e-05 2024-09-21T22:07:47.762 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.762 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.762 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:47.762 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.151574+0000", 2024-09-21T22:07:47.762 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8467999999999998e-05 2024-09-21T22:07:47.762 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.762 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.762 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:47.762 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.151665+0000", 2024-09-21T22:07:47.762 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0668000000000007e-05 2024-09-21T22:07:47.763 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.763 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.763 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-21T22:07:47.763 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.151676+0000", 2024-09-21T22:07:47.763 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1443999999999999e-05 2024-09-21T22:07:47.763 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.763 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.763 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:47.763 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.161694+0000", 2024-09-21T22:07:47.763 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010017402999999999 2024-09-21T22:07:47.763 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.763 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.763 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:47.763 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.161805+0000", 2024-09-21T22:07:47.764 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011122500000000001 2024-09-21T22:07:47.764 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.764 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.764 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:47.764 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.161937+0000", 2024-09-21T22:07:47.764 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013177299999999999 2024-09-21T22:07:47.764 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.764 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.764 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:47.764 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.162555+0000", 2024-09-21T22:07:47.764 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00061786199999999995 2024-09-21T22:07:47.765 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.765 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.765 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:47.765 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.163021+0000", 2024-09-21T22:07:47.765 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000465749 2024-09-21T22:07:47.765 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.765 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.765 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:47.765 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.163059+0000", 2024-09-21T22:07:47.765 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8238000000000002e-05 2024-09-21T22:07:47.765 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.765 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.765 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:47.765 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:17.163089+0000", 2024-09-21T22:07:47.765 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0125e-05 2024-09-21T22:07:47.766 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:47.766 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:47.766 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:47.766 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.766 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.766 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4840.0:675 2.7 e63/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 63'56, mlcod=63'54)", 2024-09-21T22:07:47.766 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.618817+0000", 2024-09-21T22:07:47.766 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 23.137729287999999, 2024-09-21T22:07:47.766 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0060329809999999998, 2024-09-21T22:07:47.766 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:47.766 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:47.766 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:47.766 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:47.767 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.767 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:47.767 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.618817+0000", 2024-09-21T22:07:47.767 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:47.767 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.767 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.767 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:47.767 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.618817+0000", 2024-09-21T22:07:47.767 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:47.767 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.767 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.767 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:47.767 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.618819+0000", 2024-09-21T22:07:47.767 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3070000000000001e-06 2024-09-21T22:07:47.767 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.768 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.768 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:47.768 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.618934+0000", 2024-09-21T22:07:47.768 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000114669 2024-09-21T22:07:47.768 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.768 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.768 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:47.768 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.618969+0000", 2024-09-21T22:07:47.768 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5985999999999997e-05 2024-09-21T22:07:47.768 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.768 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.768 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:47.768 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.618985+0000", 2024-09-21T22:07:47.768 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5840000000000001e-05 2024-09-21T22:07:47.768 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.769 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.769 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:47.769 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.619101+0000", 2024-09-21T22:07:47.769 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000115591 2024-09-21T22:07:47.769 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.769 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.769 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:47.769 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.619136+0000", 2024-09-21T22:07:47.769 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5287000000000003e-05 2024-09-21T22:07:47.769 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.769 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.769 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:47.769 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.624766+0000", 2024-09-21T22:07:47.769 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0056301500000000004 2024-09-21T22:07:47.770 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.770 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.770 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:47.770 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.624850+0000", 2024-09-21T22:07:47.770 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3151000000000004e-05 2024-09-21T22:07:47.770 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:47.770 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:47.770 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:47.770 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.770 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.770 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4840.0:676 2.7 e63/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 63'57, mlcod=63'55)", 2024-09-21T22:07:47.770 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.627276+0000", 2024-09-21T22:07:47.770 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 23.129269527999998, 2024-09-21T22:07:47.770 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.037500787000000001, 2024-09-21T22:07:47.770 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:47.771 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:47.771 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:47.771 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:47.771 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.771 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:47.771 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.627276+0000", 2024-09-21T22:07:47.771 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:47.771 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.771 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.771 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:47.771 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.627276+0000", 2024-09-21T22:07:47.772 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:47.772 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.772 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.772 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:47.772 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.627277+0000", 2024-09-21T22:07:47.772 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.068e-06 2024-09-21T22:07:47.772 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.772 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.772 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:47.772 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.628148+0000", 2024-09-21T22:07:47.772 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00087069999999999997 2024-09-21T22:07:47.772 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.772 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.772 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:47.773 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.628170+0000", 2024-09-21T22:07:47.773 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2147999999999999e-05 2024-09-21T22:07:47.773 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.773 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.773 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:47.773 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.628179+0000", 2024-09-21T22:07:47.773 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2450000000000004e-06 2024-09-21T22:07:47.773 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.773 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.773 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:47.773 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.628279+0000", 2024-09-21T22:07:47.773 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9298999999999999e-05 2024-09-21T22:07:47.773 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.773 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.774 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:47.774 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.628305+0000", 2024-09-21T22:07:47.774 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6316999999999999e-05 2024-09-21T22:07:47.774 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.774 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.774 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:47.774 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.664714+0000", 2024-09-21T22:07:47.774 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.036408923000000003 2024-09-21T22:07:47.774 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.774 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.774 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:47.774 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.664777+0000", 2024-09-21T22:07:47.774 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3087000000000001e-05 2024-09-21T22:07:47.775 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:47.775 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:47.775 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:47.775 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.775 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.775 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4840.0:677 2.7 e63/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 63'58, mlcod=63'56)", 2024-09-21T22:07:47.775 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.667210+0000", 2024-09-21T22:07:47.775 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 23.089336288999998, 2024-09-21T22:07:47.775 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010046522, 2024-09-21T22:07:47.775 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:47.775 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:47.776 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:47.776 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:47.776 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.776 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:47.776 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.667210+0000", 2024-09-21T22:07:47.776 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:47.776 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.776 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.776 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:47.777 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.667210+0000", 2024-09-21T22:07:47.777 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:47.777 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.777 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.777 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:47.777 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.667211+0000", 2024-09-21T22:07:47.777 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3379999999999999e-06 2024-09-21T22:07:47.777 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.777 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.777 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:47.777 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.667272+0000", 2024-09-21T22:07:47.778 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1257e-05 2024-09-21T22:07:47.778 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.778 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.778 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:47.778 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.667295+0000", 2024-09-21T22:07:47.778 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2637999999999999e-05 2024-09-21T22:07:47.778 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.778 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.778 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:47.778 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.667305+0000", 2024-09-21T22:07:47.778 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0044000000000001e-05 2024-09-21T22:07:47.778 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.778 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.778 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:47.778 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.667425+0000", 2024-09-21T22:07:47.779 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000120605 2024-09-21T22:07:47.779 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.779 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.779 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:47.779 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.667476+0000", 2024-09-21T22:07:47.779 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0843000000000001e-05 2024-09-21T22:07:47.779 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.779 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.779 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:47.779 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.677180+0000", 2024-09-21T22:07:47.779 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0097035860000000002 2024-09-21T22:07:47.779 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.779 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.779 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:47.780 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.677256+0000", 2024-09-21T22:07:47.780 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6211000000000006e-05 2024-09-21T22:07:47.780 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:47.780 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:47.780 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:47.780 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.780 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.780 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4840.0:678 2.7 e63/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 63'59, mlcod=63'57)", 2024-09-21T22:07:47.780 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.679689+0000", 2024-09-21T22:07:47.780 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 23.076857027999999, 2024-09-21T22:07:47.780 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053308406000000003, 2024-09-21T22:07:47.780 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:47.781 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:47.781 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:47.781 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:47.781 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.781 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:47.781 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.679689+0000", 2024-09-21T22:07:47.781 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:47.781 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.781 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.781 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:47.781 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.679689+0000", 2024-09-21T22:07:47.781 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:47.781 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.781 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.782 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:47.782 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.679691+0000", 2024-09-21T22:07:47.782 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9400000000000001e-06 2024-09-21T22:07:47.782 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.782 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.782 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:47.782 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.680245+0000", 2024-09-21T22:07:47.782 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00055382999999999995 2024-09-21T22:07:47.782 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.782 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.782 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:47.782 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.680276+0000", 2024-09-21T22:07:47.782 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1123e-05 2024-09-21T22:07:47.782 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.783 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.783 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:47.783 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.680290+0000", 2024-09-21T22:07:47.783 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4056999999999999e-05 2024-09-21T22:07:47.783 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.783 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.783 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:47.783 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.680418+0000", 2024-09-21T22:07:47.783 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000127792 2024-09-21T22:07:47.783 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.783 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.783 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:47.783 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.680465+0000", 2024-09-21T22:07:47.783 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7360000000000001e-05 2024-09-21T22:07:47.784 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.784 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.784 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:47.784 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.732923+0000", 2024-09-21T22:07:47.784 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.052458312999999999 2024-09-21T22:07:47.784 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.784 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.784 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:47.784 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.732997+0000", 2024-09-21T22:07:47.784 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3991000000000004e-05 2024-09-21T22:07:47.784 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:47.784 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:47.784 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:47.784 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.784 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.785 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4840.0:679 2.7 e63/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 63'60, mlcod=63'58)", 2024-09-21T22:07:47.785 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.735615+0000", 2024-09-21T22:07:47.785 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 23.020930875000001, 2024-09-21T22:07:47.785 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0077878399999999999, 2024-09-21T22:07:47.785 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:47.785 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:47.785 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:47.785 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:47.785 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.785 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:47.785 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.735615+0000", 2024-09-21T22:07:47.785 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:47.785 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.785 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.786 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:47.786 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.735615+0000", 2024-09-21T22:07:47.786 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:47.786 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.786 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.786 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:47.786 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.735617+0000", 2024-09-21T22:07:47.786 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7999999999999999e-06 2024-09-21T22:07:47.786 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.786 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.786 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:47.786 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.735706+0000", 2024-09-21T22:07:47.786 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8839999999999996e-05 2024-09-21T22:07:47.786 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.787 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.787 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:47.787 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.735736+0000", 2024-09-21T22:07:47.787 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9955999999999999e-05 2024-09-21T22:07:47.787 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.787 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.787 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:47.787 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.735755+0000", 2024-09-21T22:07:47.787 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9283000000000002e-05 2024-09-21T22:07:47.787 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.787 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.787 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:47.787 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.735857+0000", 2024-09-21T22:07:47.787 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000101681 2024-09-21T22:07:47.787 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.788 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.788 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:47.788 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.735894+0000", 2024-09-21T22:07:47.788 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7487999999999998e-05 2024-09-21T22:07:47.788 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.788 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.788 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:47.788 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.743321+0000", 2024-09-21T22:07:47.788 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0074270050000000004 2024-09-21T22:07:47.788 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.788 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.788 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:47.788 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.743403+0000", 2024-09-21T22:07:47.788 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1786999999999995e-05 2024-09-21T22:07:47.789 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:47.789 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:47.789 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:47.789 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.789 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.789 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4840.0:680 2.7 e63/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 63'61, mlcod=63'59)", 2024-09-21T22:07:47.789 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.745996+0000", 2024-09-21T22:07:47.789 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 23.010549515000001, 2024-09-21T22:07:47.789 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.061804132999999997, 2024-09-21T22:07:47.789 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:47.789 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:47.789 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:47.789 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:47.789 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.790 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:47.790 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.745996+0000", 2024-09-21T22:07:47.790 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:47.790 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.790 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.790 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:47.790 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.745996+0000", 2024-09-21T22:07:47.790 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:47.790 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.790 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.791 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:47.791 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.745998+0000", 2024-09-21T22:07:47.791 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.717e-06 2024-09-21T22:07:47.791 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.791 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.791 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:47.791 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.746523+0000", 2024-09-21T22:07:47.791 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00052525399999999996 2024-09-21T22:07:47.791 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.791 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.791 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:47.791 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.746554+0000", 2024-09-21T22:07:47.791 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0586000000000002e-05 2024-09-21T22:07:47.791 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.791 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.792 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:47.792 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.746577+0000", 2024-09-21T22:07:47.792 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3079e-05 2024-09-21T22:07:47.792 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.792 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.792 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:47.792 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.746689+0000", 2024-09-21T22:07:47.792 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011177 2024-09-21T22:07:47.792 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.792 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.792 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:47.792 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.746722+0000", 2024-09-21T22:07:47.792 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2870000000000002e-05 2024-09-21T22:07:47.792 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.793 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.793 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:47.793 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.807740+0000", 2024-09-21T22:07:47.793 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.061018612 2024-09-21T22:07:47.793 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.793 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.793 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:47.793 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.807800+0000", 2024-09-21T22:07:47.794 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0244999999999997e-05 2024-09-21T22:07:47.794 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:47.794 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:47.794 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:47.794 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.794 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.794 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4840.0:681 2.7 e63/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 63'62, mlcod=63'60)", 2024-09-21T22:07:47.794 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.810646+0000", 2024-09-21T22:07:47.794 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 22.945900090999999, 2024-09-21T22:07:47.794 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010715685000000001, 2024-09-21T22:07:47.794 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:47.794 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:47.794 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:47.794 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:47.795 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.795 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:47.795 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.810646+0000", 2024-09-21T22:07:47.795 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:47.795 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.795 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.795 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:47.795 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.810646+0000", 2024-09-21T22:07:47.795 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:47.795 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.795 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.795 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:47.795 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.810649+0000", 2024-09-21T22:07:47.795 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8569999999999999e-06 2024-09-21T22:07:47.796 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.796 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.796 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:47.796 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.810780+0000", 2024-09-21T22:07:47.796 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013114600000000001 2024-09-21T22:07:47.796 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.796 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.796 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:47.796 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.810819+0000", 2024-09-21T22:07:47.796 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.9363000000000002e-05 2024-09-21T22:07:47.796 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.796 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.796 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:47.797 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.810839+0000", 2024-09-21T22:07:47.797 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9865999999999999e-05 2024-09-21T22:07:47.797 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.797 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.797 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:47.797 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.810992+0000", 2024-09-21T22:07:47.797 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015333299999999999 2024-09-21T22:07:47.797 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.797 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.797 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:47.797 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.811037+0000", 2024-09-21T22:07:47.797 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.464e-05 2024-09-21T22:07:47.797 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.797 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.798 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:47.798 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.821280+0000", 2024-09-21T22:07:47.798 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010243254 2024-09-21T22:07:47.798 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.798 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.798 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:47.798 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.821361+0000", 2024-09-21T22:07:47.798 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1225999999999998e-05 2024-09-21T22:07:47.798 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:47.798 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:47.798 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:47.798 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.798 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.798 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4840.0:682 2.7 e63/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 63'63, mlcod=63'61)", 2024-09-21T22:07:47.798 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.824126+0000", 2024-09-21T22:07:47.799 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 22.932419946, 2024-09-21T22:07:47.799 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.038159327, 2024-09-21T22:07:47.799 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:47.799 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:47.799 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:47.799 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:47.799 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.799 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:47.799 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.824126+0000", 2024-09-21T22:07:47.799 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:47.799 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.799 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.799 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:47.799 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.824126+0000", 2024-09-21T22:07:47.800 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:47.800 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.800 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.800 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:47.800 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.824128+0000", 2024-09-21T22:07:47.800 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9659999999999999e-06 2024-09-21T22:07:47.800 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.800 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.800 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:47.800 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.824842+0000", 2024-09-21T22:07:47.800 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00071423999999999999 2024-09-21T22:07:47.800 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.800 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.800 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:47.800 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.824883+0000", 2024-09-21T22:07:47.801 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1131999999999999e-05 2024-09-21T22:07:47.801 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.801 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.801 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:47.801 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.824931+0000", 2024-09-21T22:07:47.801 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7327000000000003e-05 2024-09-21T22:07:47.801 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.801 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.801 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:47.801 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.825058+0000", 2024-09-21T22:07:47.801 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012790499999999999 2024-09-21T22:07:47.802 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.802 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.802 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:47.802 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.825087+0000", 2024-09-21T22:07:47.802 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8192e-05 2024-09-21T22:07:47.802 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.802 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.802 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:47.802 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.862225+0000", 2024-09-21T22:07:47.802 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.037138309000000001 2024-09-21T22:07:47.802 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.802 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.803 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:47.803 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.862285+0000", 2024-09-21T22:07:47.803 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0256e-05 2024-09-21T22:07:47.803 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:47.803 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:47.803 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:47.803 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.803 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.803 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5008.0:115 2.7 e71/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 71'147, mlcod=71'145)", 2024-09-21T22:07:47.803 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:37.860188+0000", 2024-09-21T22:07:47.804 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.8963579119999991, 2024-09-21T22:07:47.804 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0054120000000000001, 2024-09-21T22:07:47.804 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:47.804 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:47.804 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:47.804 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:47.804 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.804 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:47.804 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.860188+0000", 2024-09-21T22:07:47.804 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:47.804 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.804 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.804 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:47.804 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.860188+0000", 2024-09-21T22:07:47.805 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:47.805 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.805 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.805 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:47.805 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.860189+0000", 2024-09-21T22:07:47.805 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.0800000000000004e-07 2024-09-21T22:07:47.805 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.805 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.805 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:47.805 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.860243+0000", 2024-09-21T22:07:47.805 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4666000000000001e-05 2024-09-21T22:07:47.805 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.805 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.805 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:47.806 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.860255+0000", 2024-09-21T22:07:47.806 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.171e-05 2024-09-21T22:07:47.806 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.806 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.806 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:47.806 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.860260+0000", 2024-09-21T22:07:47.806 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1159999999999998e-06 2024-09-21T22:07:47.806 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.806 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.806 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:47.806 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.860325+0000", 2024-09-21T22:07:47.806 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4796999999999996e-05 2024-09-21T22:07:47.806 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.806 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.807 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:47.807 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.860344+0000", 2024-09-21T22:07:47.807 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9403e-05 2024-09-21T22:07:47.807 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.807 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.807 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:47.807 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.865563+0000", 2024-09-21T22:07:47.807 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0052183280000000004 2024-09-21T22:07:47.807 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.807 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.807 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:47.807 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.865600+0000", 2024-09-21T22:07:47.807 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7172000000000003e-05 2024-09-21T22:07:47.807 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:47.808 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:47.808 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:47.808 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.808 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.808 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5008.0:116 2.7 e71/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 71'148, mlcod=71'146)", 2024-09-21T22:07:47.808 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:37.867612+0000", 2024-09-21T22:07:47.808 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.8889334509999998, 2024-09-21T22:07:47.808 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.045123155999999998, 2024-09-21T22:07:47.808 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:47.808 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:47.808 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:47.808 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:47.808 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.809 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:47.809 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.867612+0000", 2024-09-21T22:07:47.809 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:47.809 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.809 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.809 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:47.809 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.867612+0000", 2024-09-21T22:07:47.809 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:47.809 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.809 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.809 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:47.809 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.867613+0000", 2024-09-21T22:07:47.809 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1200000000000002e-07 2024-09-21T22:07:47.809 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.809 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.810 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:47.810 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.868135+0000", 2024-09-21T22:07:47.810 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00052132700000000003 2024-09-21T22:07:47.810 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.810 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.810 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:47.810 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.868150+0000", 2024-09-21T22:07:47.810 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5271e-05 2024-09-21T22:07:47.810 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.810 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.810 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:47.810 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.868156+0000", 2024-09-21T22:07:47.810 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4969999999999998e-06 2024-09-21T22:07:47.811 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.811 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.811 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:47.811 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.868213+0000", 2024-09-21T22:07:47.811 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7114999999999998e-05 2024-09-21T22:07:47.811 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.811 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.811 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:47.811 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.868238+0000", 2024-09-21T22:07:47.811 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4987999999999999e-05 2024-09-21T22:07:47.811 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.811 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.811 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:47.811 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.912677+0000", 2024-09-21T22:07:47.811 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.044439019000000003 2024-09-21T22:07:47.812 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.812 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.812 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:47.812 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.912736+0000", 2024-09-21T22:07:47.812 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8127000000000001e-05 2024-09-21T22:07:47.812 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:47.812 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:47.812 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:47.812 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.812 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.812 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5008.0:117 2.7 e71/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 71'149, mlcod=71'147)", 2024-09-21T22:07:47.812 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:37.914937+0000", 2024-09-21T22:07:47.812 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.8416084389999998, 2024-09-21T22:07:47.812 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0076223819999999996, 2024-09-21T22:07:47.813 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:47.813 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:47.813 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:47.813 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:47.813 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.813 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:47.813 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.914937+0000", 2024-09-21T22:07:47.813 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:47.813 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.813 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.813 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:47.813 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.914937+0000", 2024-09-21T22:07:47.813 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:47.813 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.814 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.814 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:47.814 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.914939+0000", 2024-09-21T22:07:47.814 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.826e-06 2024-09-21T22:07:47.814 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.814 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.814 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:47.814 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.915021+0000", 2024-09-21T22:07:47.814 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.2244999999999996e-05 2024-09-21T22:07:47.814 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.814 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.814 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:47.814 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.915046+0000", 2024-09-21T22:07:47.814 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4794e-05 2024-09-21T22:07:47.815 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.815 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.815 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:47.815 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.915063+0000", 2024-09-21T22:07:47.815 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6773999999999999e-05 2024-09-21T22:07:47.815 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.815 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.815 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:47.815 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.915143+0000", 2024-09-21T22:07:47.815 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9516000000000002e-05 2024-09-21T22:07:47.815 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.815 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.815 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:47.815 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.915185+0000", 2024-09-21T22:07:47.815 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2324000000000003e-05 2024-09-21T22:07:47.816 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.816 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.816 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:47.816 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.922508+0000", 2024-09-21T22:07:47.816 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0073226419999999999 2024-09-21T22:07:47.816 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.816 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.816 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:47.816 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.922560+0000", 2024-09-21T22:07:47.816 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2261e-05 2024-09-21T22:07:47.816 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:47.816 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:47.816 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:47.816 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.816 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.817 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5008.0:118 2.7 e71/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 71'150, mlcod=71'148)", 2024-09-21T22:07:47.817 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:37.924700+0000", 2024-09-21T22:07:47.817 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.8318462600000007, 2024-09-21T22:07:47.817 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.034650561000000003, 2024-09-21T22:07:47.817 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:47.817 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:47.817 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:47.817 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:47.817 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.817 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:47.817 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.924700+0000", 2024-09-21T22:07:47.817 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:47.817 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.817 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.818 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:47.818 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.924700+0000", 2024-09-21T22:07:47.818 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:47.818 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.818 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.818 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:47.818 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.924702+0000", 2024-09-21T22:07:47.818 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9460000000000001e-06 2024-09-21T22:07:47.818 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.818 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.818 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:47.818 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.925132+0000", 2024-09-21T22:07:47.818 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00043021200000000002 2024-09-21T22:07:47.818 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.819 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.819 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:47.819 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.925156+0000", 2024-09-21T22:07:47.819 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4451999999999999e-05 2024-09-21T22:07:47.819 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.819 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.819 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:47.819 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.925168+0000", 2024-09-21T22:07:47.819 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2249000000000001e-05 2024-09-21T22:07:47.819 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.819 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.819 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:47.819 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.925241+0000", 2024-09-21T22:07:47.819 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2502000000000005e-05 2024-09-21T22:07:47.819 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.820 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.820 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:47.820 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.925267+0000", 2024-09-21T22:07:47.820 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6471000000000001e-05 2024-09-21T22:07:47.820 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.820 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.820 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:47.820 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.959217+0000", 2024-09-21T22:07:47.820 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.033949569999999998 2024-09-21T22:07:47.820 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.820 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.821 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:47.821 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.959350+0000", 2024-09-21T22:07:47.821 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013315899999999999 2024-09-21T22:07:47.821 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:47.821 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:47.821 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:47.821 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.821 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.821 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5008.0:119 2.7 e71/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 71'151, mlcod=71'149)", 2024-09-21T22:07:47.821 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:37.961607+0000", 2024-09-21T22:07:47.821 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.7949386220000001, 2024-09-21T22:07:47.821 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0058888020000000003, 2024-09-21T22:07:47.821 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:47.821 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:47.822 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:47.822 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:47.822 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.822 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:47.822 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.961607+0000", 2024-09-21T22:07:47.822 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:47.822 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.822 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.822 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:47.822 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.961607+0000", 2024-09-21T22:07:47.822 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:47.822 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.822 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.822 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:47.823 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.961608+0000", 2024-09-21T22:07:47.823 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.017e-06 2024-09-21T22:07:47.823 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.823 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.823 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:47.823 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.961669+0000", 2024-09-21T22:07:47.823 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0905e-05 2024-09-21T22:07:47.823 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.823 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.823 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:47.823 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.961686+0000", 2024-09-21T22:07:47.824 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6665999999999999e-05 2024-09-21T22:07:47.824 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.824 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.824 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:47.824 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.961696+0000", 2024-09-21T22:07:47.824 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.063e-05 2024-09-21T22:07:47.824 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.824 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.824 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:47.824 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.961771+0000", 2024-09-21T22:07:47.824 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4443000000000005e-05 2024-09-21T22:07:47.824 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.824 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.824 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:47.825 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.961794+0000", 2024-09-21T22:07:47.825 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3608000000000002e-05 2024-09-21T22:07:47.825 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.825 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.825 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:47.825 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.967374+0000", 2024-09-21T22:07:47.825 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0055799969999999997 2024-09-21T22:07:47.825 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.825 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.825 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:47.826 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.967496+0000", 2024-09-21T22:07:47.826 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000121536 2024-09-21T22:07:47.826 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:47.826 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:47.826 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:47.826 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.826 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.826 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5008.0:120 2.7 e71/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 71'152, mlcod=71'150)", 2024-09-21T22:07:47.826 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:37.970837+0000", 2024-09-21T22:07:47.826 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.7857090089999996, 2024-09-21T22:07:47.826 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.03044943, 2024-09-21T22:07:47.827 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:47.827 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:47.827 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:47.827 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:47.827 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.827 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:47.827 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.970837+0000", 2024-09-21T22:07:47.827 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:47.827 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.828 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.828 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:47.828 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.970837+0000", 2024-09-21T22:07:47.828 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:47.828 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.828 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.828 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:47.828 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.970838+0000", 2024-09-21T22:07:47.828 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4649999999999999e-06 2024-09-21T22:07:47.828 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.828 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.828 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:47.828 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.971265+0000", 2024-09-21T22:07:47.829 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00042665800000000001 2024-09-21T22:07:47.829 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.829 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.829 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:47.829 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.971285+0000", 2024-09-21T22:07:47.829 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9639999999999998e-05 2024-09-21T22:07:47.829 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.829 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.829 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:47.829 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.971294+0000", 2024-09-21T22:07:47.829 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7210000000000006e-06 2024-09-21T22:07:47.830 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.830 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.830 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:47.830 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.971354+0000", 2024-09-21T22:07:47.830 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9613e-05 2024-09-21T22:07:47.830 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.830 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.830 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:47.830 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.971381+0000", 2024-09-21T22:07:47.830 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7538999999999998e-05 2024-09-21T22:07:47.830 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.831 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.831 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:47.831 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.001199+0000", 2024-09-21T22:07:47.831 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029817537000000002 2024-09-21T22:07:47.831 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.831 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.831 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:47.831 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.001286+0000", 2024-09-21T22:07:47.831 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7256999999999995e-05 2024-09-21T22:07:47.831 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:47.831 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:47.831 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:47.831 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.831 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.832 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5008.0:121 2.7 e71/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 71'153, mlcod=71'151)", 2024-09-21T22:07:47.832 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:38.007198+0000", 2024-09-21T22:07:47.832 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.7493480889999997, 2024-09-21T22:07:47.832 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0047526499999999998, 2024-09-21T22:07:47.832 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:47.832 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:47.832 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:47.832 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:47.832 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.833 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:47.833 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.007198+0000", 2024-09-21T22:07:47.833 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:47.833 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.833 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.833 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:47.833 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.007198+0000", 2024-09-21T22:07:47.833 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:47.833 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.833 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.833 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:47.833 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.007199+0000", 2024-09-21T22:07:47.833 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4200000000000005e-07 2024-09-21T22:07:47.834 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.834 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.834 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:47.834 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.007255+0000", 2024-09-21T22:07:47.834 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6406000000000002e-05 2024-09-21T22:07:47.834 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.834 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.834 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:47.834 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.007268+0000", 2024-09-21T22:07:47.834 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2824e-05 2024-09-21T22:07:47.834 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.834 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.834 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:47.835 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.007276+0000", 2024-09-21T22:07:47.835 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1640000000000001e-06 2024-09-21T22:07:47.835 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.835 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.835 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:47.835 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.007335+0000", 2024-09-21T22:07:47.835 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8610999999999998e-05 2024-09-21T22:07:47.835 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.835 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.835 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:47.835 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.007355+0000", 2024-09-21T22:07:47.835 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9987999999999999e-05 2024-09-21T22:07:47.835 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.835 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.835 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:47.836 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.011869+0000", 2024-09-21T22:07:47.836 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.004514784 2024-09-21T22:07:47.836 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.836 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.836 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:47.836 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.011950+0000", 2024-09-21T22:07:47.836 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1031000000000004e-05 2024-09-21T22:07:47.836 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:47.836 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:47.836 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:47.836 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.836 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.836 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5008.0:122 2.7 e71/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 71'154, mlcod=71'152)", 2024-09-21T22:07:47.836 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:38.014709+0000", 2024-09-21T22:07:47.837 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.7418371340000007, 2024-09-21T22:07:47.837 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029567408999999999, 2024-09-21T22:07:47.837 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:47.837 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:47.837 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:47.837 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:47.837 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.837 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:47.837 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.014709+0000", 2024-09-21T22:07:47.838 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:47.838 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.838 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.838 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:47.838 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.014709+0000", 2024-09-21T22:07:47.838 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:47.838 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.838 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.838 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:47.838 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.014709+0000", 2024-09-21T22:07:47.839 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9599999999999999e-07 2024-09-21T22:07:47.839 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.839 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.839 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:47.839 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.015201+0000", 2024-09-21T22:07:47.839 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00049175599999999999 2024-09-21T22:07:47.839 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.839 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.839 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:47.839 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.015215+0000", 2024-09-21T22:07:47.839 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3692999999999999e-05 2024-09-21T22:07:47.839 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.839 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.839 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:47.839 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.015225+0000", 2024-09-21T22:07:47.840 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.815e-06 2024-09-21T22:07:47.840 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.840 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.840 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:47.840 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.015287+0000", 2024-09-21T22:07:47.840 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1858000000000003e-05 2024-09-21T22:07:47.840 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.840 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.840 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:47.840 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.015315+0000", 2024-09-21T22:07:47.840 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8490999999999999e-05 2024-09-21T22:07:47.840 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.840 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.840 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:47.841 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.044182+0000", 2024-09-21T22:07:47.841 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.028866936999999999 2024-09-21T22:07:47.841 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.841 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.841 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:47.841 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.044276+0000", 2024-09-21T22:07:47.841 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4163e-05 2024-09-21T22:07:47.841 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:47.841 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:47.841 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:47.841 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.841 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.841 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5008.0:132 2.7 e72/16 2:f45e3993:::rbd_object_map.126483d75f2d:head v 72'156, mlcod=71'154)", 2024-09-21T22:07:47.842 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:38.335972+0000", 2024-09-21T22:07:47.842 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.4205737270000007, 2024-09-21T22:07:47.842 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.003515944, 2024-09-21T22:07:47.842 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:47.842 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:47.842 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:47.842 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:47.842 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.842 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:47.842 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.335972+0000", 2024-09-21T22:07:47.843 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:47.843 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.843 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.843 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:47.843 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.335972+0000", 2024-09-21T22:07:47.843 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:47.843 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.843 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.843 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:47.843 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.335973+0000", 2024-09-21T22:07:47.843 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5199999999999997e-07 2024-09-21T22:07:47.843 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.844 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.844 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:47.844 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.335981+0000", 2024-09-21T22:07:47.844 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.3720000000000005e-06 2024-09-21T22:07:47.844 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.844 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.844 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:47.844 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.336000+0000", 2024-09-21T22:07:47.844 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8663000000000001e-05 2024-09-21T22:07:47.844 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.844 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.845 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:47.845 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.336015+0000", 2024-09-21T22:07:47.845 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5372000000000001e-05 2024-09-21T22:07:47.845 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.845 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.845 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:47.845 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.336111+0000", 2024-09-21T22:07:47.845 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5968e-05 2024-09-21T22:07:47.845 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.845 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.845 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:47.845 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.336146+0000", 2024-09-21T22:07:47.845 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4882000000000002e-05 2024-09-21T22:07:47.845 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.846 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.846 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:47.846 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.339438+0000", 2024-09-21T22:07:47.846 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0032919709999999999 2024-09-21T22:07:47.846 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.846 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.846 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:47.846 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.339488+0000", 2024-09-21T22:07:47.846 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0164000000000003e-05 2024-09-21T22:07:47.846 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:47.846 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:47.846 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:47.846 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.846 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.847 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5008.0:150 2.7 e72/16 2:f45e3993:::rbd_object_map.126483d75f2d:head v 72'158, mlcod=72'156)", 2024-09-21T22:07:47.847 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:38.744508+0000", 2024-09-21T22:07:47.847 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.0120375789999994, 2024-09-21T22:07:47.847 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0022832690000000001, 2024-09-21T22:07:47.847 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:47.847 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:47.847 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:47.847 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:47.847 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.847 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:47.847 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.744508+0000", 2024-09-21T22:07:47.847 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:47.847 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.847 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.847 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:47.848 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.744508+0000", 2024-09-21T22:07:47.848 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:47.848 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.848 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.848 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:47.848 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.744511+0000", 2024-09-21T22:07:47.848 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4870000000000001e-06 2024-09-21T22:07:47.848 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.848 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.848 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:47.848 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.744530+0000", 2024-09-21T22:07:47.848 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8966000000000001e-05 2024-09-21T22:07:47.848 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.848 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.849 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:47.849 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.744566+0000", 2024-09-21T22:07:47.849 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6627e-05 2024-09-21T22:07:47.849 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.849 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.849 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:47.849 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.744588+0000", 2024-09-21T22:07:47.849 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1716e-05 2024-09-21T22:07:47.849 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.849 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.849 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:47.849 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.744750+0000", 2024-09-21T22:07:47.849 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016171600000000001 2024-09-21T22:07:47.849 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.850 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.850 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:47.850 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.744813+0000", 2024-09-21T22:07:47.850 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3689999999999995e-05 2024-09-21T22:07:47.850 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.850 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.850 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:47.850 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.746701+0000", 2024-09-21T22:07:47.850 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0018874479999999999 2024-09-21T22:07:47.850 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.850 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.850 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:47.851 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.746792+0000", 2024-09-21T22:07:47.851 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0618999999999994e-05 2024-09-21T22:07:47.851 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:47.851 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:47.851 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:47.851 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.851 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.851 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5008.0:159 2.7 e72/16 2:f45e3993:::rbd_object_map.126483d75f2d:head v 72'159, mlcod=72'157)", 2024-09-21T22:07:47.851 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:38.914368+0000", 2024-09-21T22:07:47.851 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.8421781999999993, 2024-09-21T22:07:47.851 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0021177169999999999, 2024-09-21T22:07:47.851 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:47.851 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:47.851 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:47.851 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:47.852 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.852 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:47.852 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.914368+0000", 2024-09-21T22:07:47.852 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:47.852 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.852 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.852 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:47.852 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.914368+0000", 2024-09-21T22:07:47.852 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:47.852 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.852 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.853 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:47.853 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.914370+0000", 2024-09-21T22:07:47.853 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9400000000000001e-06 2024-09-21T22:07:47.853 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.853 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.853 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:47.853 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.914382+0000", 2024-09-21T22:07:47.853 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2427e-05 2024-09-21T22:07:47.853 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.853 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.854 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:47.854 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.914412+0000", 2024-09-21T22:07:47.854 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9583000000000001e-05 2024-09-21T22:07:47.854 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.854 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.854 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:47.854 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.914427+0000", 2024-09-21T22:07:47.854 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5420000000000001e-05 2024-09-21T22:07:47.854 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.854 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.854 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:47.854 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.914569+0000", 2024-09-21T22:07:47.854 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014215899999999999 2024-09-21T22:07:47.855 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.855 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.855 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:47.855 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.914635+0000", 2024-09-21T22:07:47.855 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5566000000000002e-05 2024-09-21T22:07:47.855 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.855 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.855 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:47.855 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.916409+0000", 2024-09-21T22:07:47.855 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0017742229999999999 2024-09-21T22:07:47.855 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:47.856 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:47.856 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:47.856 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.916485+0000", 2024-09-21T22:07:47.856 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6398999999999998e-05 2024-09-21T22:07:47.856 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:47.856 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:47.856 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:47.856 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:47.856 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:47.856 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:07:47.858 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T22:07:47.858 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:47.852+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:47.959 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:47.954+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:48.007 INFO:tasks.workunit.client.0.smithi157.stderr:+ DEV= 2024-09-21T22:07:48.008 INFO:tasks.workunit.client.0.smithi157.stderr:++ _sudo rbd device --device-type nbd map rbd/testrbdnbd21553 2024-09-21T22:07:48.008 INFO:tasks.workunit.client.0.smithi157.stderr:++ local cmd 2024-09-21T22:07:48.008 INFO:tasks.workunit.client.0.smithi157.stderr:+++ id -u 2024-09-21T22:07:48.010 INFO:tasks.workunit.client.0.smithi157.stderr:++ '[' 1000 -eq 0 ']' 2024-09-21T22:07:48.010 INFO:tasks.workunit.client.0.smithi157.stderr:+++ which rbd 2024-09-21T22:07:48.012 INFO:tasks.workunit.client.0.smithi157.stderr:++ cmd=/usr/bin/rbd 2024-09-21T22:07:48.013 INFO:tasks.workunit.client.0.smithi157.stderr:++ shift 2024-09-21T22:07:48.013 INFO:tasks.workunit.client.0.smithi157.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map rbd/testrbdnbd21553 2024-09-21T22:07:48.033 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:07:48.033 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:07:48.034 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-09-21T22:07:48.034 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:07:48.048 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T22:07:48.098 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:07:48.096+0000 7f2075a06640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31720) UID: 0 2024-09-21T22:07:48.144 INFO:tasks.workunit.client.0.smithi157.stderr:+ DEV=/dev/nbd0 2024-09-21T22:07:48.144 INFO:tasks.workunit.client.0.smithi157.stderr:+ get_pid rbd 2024-09-21T22:07:48.144 INFO:tasks.workunit.client.0.smithi157.stderr:+ local pool=rbd 2024-09-21T22:07:48.144 INFO:tasks.workunit.client.0.smithi157.stderr:+ local ns= 2024-09-21T22:07:48.145 INFO:tasks.workunit.client.0.smithi157.stderr:++ rbd device --device-type nbd --format xml list 2024-09-21T22:07:48.145 INFO:tasks.workunit.client.0.smithi157.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd21553'\''][device='\''/dev/nbd0'\'']/id' 2024-09-21T22:07:48.159 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:48.154+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:48.176 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:07:48.177 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:07:48.177 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-21T22:07:48.177 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-21T22:07:48.177 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:07:48.191 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T22:07:48.212 INFO:tasks.workunit.client.0.smithi157.stderr:+ PID=23250 2024-09-21T22:07:48.212 INFO:tasks.workunit.client.0.smithi157.stderr:+ test -n 23250 2024-09-21T22:07:48.212 INFO:tasks.workunit.client.0.smithi157.stderr:+ ps -p 23250 -C rbd-nbd 2024-09-21T22:07:48.221 INFO:tasks.workunit.client.0.smithi157.stdout: PID TTY TIME CMD 2024-09-21T22:07:48.221 INFO:tasks.workunit.client.0.smithi157.stdout: 23250 ? 00:00:00 rbd-nbd 2024-09-21T22:07:48.222 INFO:tasks.workunit.client.0.smithi157.stderr:+ pid=23250 2024-09-21T22:07:48.222 INFO:tasks.workunit.client.0.smithi157.stderr:++ _sudo rbd device --device-type nbd map rbd/ns/testrbdnbd21553 2024-09-21T22:07:48.222 INFO:tasks.workunit.client.0.smithi157.stderr:++ local cmd 2024-09-21T22:07:48.223 INFO:tasks.workunit.client.0.smithi157.stderr:+++ id -u 2024-09-21T22:07:48.224 INFO:tasks.workunit.client.0.smithi157.stderr:++ '[' 1000 -eq 0 ']' 2024-09-21T22:07:48.225 INFO:tasks.workunit.client.0.smithi157.stderr:+++ which rbd 2024-09-21T22:07:48.226 INFO:tasks.workunit.client.0.smithi157.stderr:++ cmd=/usr/bin/rbd 2024-09-21T22:07:48.226 INFO:tasks.workunit.client.0.smithi157.stderr:++ shift 2024-09-21T22:07:48.226 INFO:tasks.workunit.client.0.smithi157.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map rbd/ns/testrbdnbd21553 2024-09-21T22:07:48.278 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:07:48.276+0000 7f2075a06640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31720) UID: 0 2024-09-21T22:07:48.361 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:48.356+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:48.368 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:07:48.369 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-21T22:07:48.369 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-21T22:07:48.369 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-21T22:07:48.369 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.369 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5031.0:3 2.2 2:53d5e6d0:::rbd_header.126483d75f2d:head [call rbd.get_size in=11b,call,call rbd.get_features in=15b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e65)", 2024-09-21T22:07:48.369 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:27.373436+0000", 2024-09-21T22:07:48.369 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 20.992997526, 2024-09-21T22:07:48.369 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.86524013700000002, 2024-09-21T22:07:48.369 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:48.369 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:48.369 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:07:48.369 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:48.369 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5031", 2024-09-21T22:07:48.370 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/3232705659", 2024-09-21T22:07:48.370 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3 2024-09-21T22:07:48.370 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.370 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:48.370 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.370 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:48.370 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:27.373436+0000", 2024-09-21T22:07:48.370 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:48.370 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.370 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.370 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:48.370 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:27.373436+0000", 2024-09-21T22:07:48.370 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:48.370 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.371 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.371 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:48.371 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:27.373439+0000", 2024-09-21T22:07:48.371 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5969999999999999e-06 2024-09-21T22:07:48.371 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.371 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.371 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:48.371 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:27.373450+0000", 2024-09-21T22:07:48.371 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1326e-05 2024-09-21T22:07:48.371 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.371 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.371 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:48.371 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:27.373471+0000", 2024-09-21T22:07:48.371 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0169999999999998e-05 2024-09-21T22:07:48.372 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.372 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.372 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:48.372 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:27.373486+0000", 2024-09-21T22:07:48.372 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5483000000000001e-05 2024-09-21T22:07:48.372 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.372 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.372 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:48.372 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:27.373602+0000", 2024-09-21T22:07:48.372 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000115603 2024-09-21T22:07:48.372 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.372 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.372 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-21T22:07:48.372 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:27.373617+0000", 2024-09-21T22:07:48.372 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4956999999999999e-05 2024-09-21T22:07:48.373 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.373 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.373 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:48.373 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.234578+0000", 2024-09-21T22:07:48.373 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.86096097900000002 2024-09-21T22:07:48.373 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.373 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.373 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for missing object", 2024-09-21T22:07:48.373 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.234771+0000", 2024-09-21T22:07:48.373 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019394700000000001 2024-09-21T22:07:48.373 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.373 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.373 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:48.373 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.237470+0000", 2024-09-21T22:07:48.374 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026981319999999998 2024-09-21T22:07:48.374 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.374 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.374 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:48.374 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.237791+0000", 2024-09-21T22:07:48.374 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00032138499999999999 2024-09-21T22:07:48.374 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.374 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.374 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:48.374 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.238677+0000", 2024-09-21T22:07:48.374 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00088555800000000005 2024-09-21T22:07:48.374 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:48.374 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:48.374 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:48.375 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.375 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.375 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.2 66/65 [PushOp(2:4481fad7:ns::rbd_data.127f79bc213d.0000000000000006:head, version: 57'9, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:4481fad7:ns::rbd_data.127f79bc213d.0000000000000006:head@57'9, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T22:07:48.375 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:28.247056+0000", 2024-09-21T22:07:48.375 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 20.119377863, 2024-09-21T22:07:48.375 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.045025012000000003, 2024-09-21T22:07:48.375 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:48.375 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:48.375 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:07:48.375 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:48.375 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.375 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:48.375 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.247056+0000", 2024-09-21T22:07:48.375 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:48.376 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.376 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.376 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:48.376 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.247056+0000", 2024-09-21T22:07:48.376 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:48.376 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.376 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.376 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:48.376 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.247057+0000", 2024-09-21T22:07:48.376 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2500000000000001e-06 2024-09-21T22:07:48.376 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.376 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.376 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:48.376 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.249384+0000", 2024-09-21T22:07:48.376 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0023265619999999999 2024-09-21T22:07:48.377 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.377 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.377 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:48.377 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.249429+0000", 2024-09-21T22:07:48.377 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5392999999999997e-05 2024-09-21T22:07:48.377 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.377 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.377 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:48.377 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.249476+0000", 2024-09-21T22:07:48.377 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6570000000000003e-05 2024-09-21T22:07:48.377 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.377 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.377 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:48.377 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.249611+0000", 2024-09-21T22:07:48.377 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000135143 2024-09-21T22:07:48.378 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.378 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.378 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:48.378 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.249626+0000", 2024-09-21T22:07:48.378 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.469e-05 2024-09-21T22:07:48.378 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.378 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.378 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:48.378 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.292081+0000", 2024-09-21T22:07:48.378 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.042455404000000002 2024-09-21T22:07:48.378 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:48.378 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:48.378 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:48.378 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.379 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.379 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5031.0:9 2.2 2:53d5e6d0:::rbd_header.126483d75f2d:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-21T22:07:48.379 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:28.250163+0000", 2024-09-21T22:07:48.379 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 20.116270469, 2024-09-21T22:07:48.379 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.042986430999999999, 2024-09-21T22:07:48.379 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:48.379 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:48.379 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:07:48.379 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:48.379 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5031", 2024-09-21T22:07:48.379 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/3232705659", 2024-09-21T22:07:48.379 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 9 2024-09-21T22:07:48.379 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.379 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:48.379 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.380 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:48.380 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.250163+0000", 2024-09-21T22:07:48.380 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:48.380 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.380 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.380 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:48.380 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.250163+0000", 2024-09-21T22:07:48.380 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:48.380 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.380 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.380 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:48.380 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.250166+0000", 2024-09-21T22:07:48.380 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7039999999999999e-06 2024-09-21T22:07:48.380 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.381 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.381 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:48.381 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.250175+0000", 2024-09-21T22:07:48.381 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7560000000000005e-06 2024-09-21T22:07:48.381 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.381 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.381 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:48.381 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.250199+0000", 2024-09-21T22:07:48.381 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4049999999999998e-05 2024-09-21T22:07:48.381 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.381 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.381 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:48.381 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.250218+0000", 2024-09-21T22:07:48.381 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9236e-05 2024-09-21T22:07:48.381 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.382 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.382 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:48.382 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.292193+0000", 2024-09-21T22:07:48.382 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.041974694 2024-09-21T22:07:48.382 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.382 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.382 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:48.382 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.292492+0000", 2024-09-21T22:07:48.382 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00029900100000000002 2024-09-21T22:07:48.382 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.382 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.382 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:48.382 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.293150+0000", 2024-09-21T22:07:48.382 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00065799000000000001 2024-09-21T22:07:48.383 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:48.383 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:48.383 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:48.383 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.383 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.383 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5031.0:12 2.2 2:53d5e6d0:::rbd_header.126483d75f2d:head [watch watch cookie 140495695137040] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-09-21T22:07:48.383 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:28.295580+0000", 2024-09-21T22:07:48.383 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 20.070853582000002, 2024-09-21T22:07:48.383 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.033090870000000001, 2024-09-21T22:07:48.383 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:48.383 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:48.383 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:48.384 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:48.384 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5031", 2024-09-21T22:07:48.384 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/3232705659", 2024-09-21T22:07:48.385 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 12 2024-09-21T22:07:48.385 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.385 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:48.385 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.385 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:48.385 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.295580+0000", 2024-09-21T22:07:48.385 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:48.385 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.385 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.385 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:48.385 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.295580+0000", 2024-09-21T22:07:48.385 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:48.385 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.385 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.385 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:48.386 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.295581+0000", 2024-09-21T22:07:48.386 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6799999999999999e-07 2024-09-21T22:07:48.386 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.386 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.386 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:48.386 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.295586+0000", 2024-09-21T22:07:48.386 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8659999999999997e-06 2024-09-21T22:07:48.386 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.386 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.386 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:48.386 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.295599+0000", 2024-09-21T22:07:48.386 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2897999999999999e-05 2024-09-21T22:07:48.386 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.386 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.387 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:48.387 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.295614+0000", 2024-09-21T22:07:48.387 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5406000000000001e-05 2024-09-21T22:07:48.387 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.387 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.387 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:48.387 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.295665+0000", 2024-09-21T22:07:48.387 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0501000000000003e-05 2024-09-21T22:07:48.387 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.387 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.387 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:48.387 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.295727+0000", 2024-09-21T22:07:48.387 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2073e-05 2024-09-21T22:07:48.387 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.387 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.388 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-09-21T22:07:48.388 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.295849+0000", 2024-09-21T22:07:48.388 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012190500000000001 2024-09-21T22:07:48.388 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.388 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.388 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:48.388 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.296429+0000", 2024-09-21T22:07:48.388 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00058019699999999996 2024-09-21T22:07:48.388 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.388 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.388 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:48.388 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.328308+0000", 2024-09-21T22:07:48.388 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.031878849000000001 2024-09-21T22:07:48.388 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.388 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.389 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:48.389 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.328421+0000", 2024-09-21T22:07:48.389 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000113322 2024-09-21T22:07:48.389 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.389 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.389 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:48.389 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.328671+0000", 2024-09-21T22:07:48.389 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024998499999999999 2024-09-21T22:07:48.389 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:48.389 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:48.389 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:48.389 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.389 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.389 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.2 66/65 [PushOp(2:4481fad7:ns::rbd_data.127f79bc213d.0000000000000006:head, version: 57'9, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:4481fad7:ns::rbd_data.127f79bc213d.0000000000000006:head@57'9, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:07:48.390 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:28.296301+0000", 2024-09-21T22:07:48.390 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 20.070132660999999, 2024-09-21T22:07:48.390 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.031905291000000002, 2024-09-21T22:07:48.390 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:48.390 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:48.390 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:07:48.390 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:48.390 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.390 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:48.390 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.296301+0000", 2024-09-21T22:07:48.390 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:48.390 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.390 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.390 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:48.390 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.296301+0000", 2024-09-21T22:07:48.391 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:48.391 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.391 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.391 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:48.391 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.296315+0000", 2024-09-21T22:07:48.391 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3867e-05 2024-09-21T22:07:48.391 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.391 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.391 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:48.391 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.297432+0000", 2024-09-21T22:07:48.391 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011164759999999999 2024-09-21T22:07:48.391 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.391 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.391 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:48.391 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.297454+0000", 2024-09-21T22:07:48.392 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2813e-05 2024-09-21T22:07:48.392 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.392 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.392 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:48.392 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.297479+0000", 2024-09-21T22:07:48.392 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4817e-05 2024-09-21T22:07:48.392 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.392 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.392 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:48.392 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.297582+0000", 2024-09-21T22:07:48.392 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010261500000000001 2024-09-21T22:07:48.392 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.392 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.392 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:48.392 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.297600+0000", 2024-09-21T22:07:48.393 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8323000000000001e-05 2024-09-21T22:07:48.393 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.393 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.393 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:48.393 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.328207+0000", 2024-09-21T22:07:48.393 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.030606379999999999 2024-09-21T22:07:48.393 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:48.393 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:48.393 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:48.393 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.393 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.393 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.5031.0:12 2.2 e66/65 ondisk, result = 0)", 2024-09-21T22:07:48.393 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:28.297512+0000", 2024-09-21T22:07:48.393 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 20.068922088000001, 2024-09-21T22:07:48.394 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.031170693999999999, 2024-09-21T22:07:48.394 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:48.394 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:48.394 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:07:48.394 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:48.394 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.394 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:48.394 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.297512+0000", 2024-09-21T22:07:48.394 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:48.394 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.394 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.395 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:48.395 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.297512+0000", 2024-09-21T22:07:48.395 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:48.395 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.395 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.395 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:48.395 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.297512+0000", 2024-09-21T22:07:48.395 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7700000000000005e-07 2024-09-21T22:07:48.395 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.395 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.395 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:48.395 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.297517+0000", 2024-09-21T22:07:48.395 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1039999999999998e-06 2024-09-21T22:07:48.395 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.396 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.396 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:48.396 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.297528+0000", 2024-09-21T22:07:48.396 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0633e-05 2024-09-21T22:07:48.396 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.396 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.396 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:48.396 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.297537+0000", 2024-09-21T22:07:48.396 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7199999999999995e-06 2024-09-21T22:07:48.396 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.396 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.396 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:48.396 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.328273+0000", 2024-09-21T22:07:48.396 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.030736574999999999 2024-09-21T22:07:48.396 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.397 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.397 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:48.397 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.328293+0000", 2024-09-21T22:07:48.397 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9304e-05 2024-09-21T22:07:48.397 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.397 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.397 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:48.397 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.328683+0000", 2024-09-21T22:07:48.397 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00038988100000000002 2024-09-21T22:07:48.397 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:48.397 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:48.397 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:48.397 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.397 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.397 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.2 66/65 [PushOp(2:4481fad7:ns::rbd_data.127f79bc213d.0000000000000006:head, version: 57'9, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:4481fad7:ns::rbd_data.127f79bc213d.0000000000000006:head@57'9, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:07:48.398 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:28.331166+0000", 2024-09-21T22:07:48.398 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 20.035267719, 2024-09-21T22:07:48.398 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.039995378999999998, 2024-09-21T22:07:48.398 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:48.398 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:48.398 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:07:48.398 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:48.398 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.398 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:48.398 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.331166+0000", 2024-09-21T22:07:48.398 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:48.398 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.398 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.398 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:48.399 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.331166+0000", 2024-09-21T22:07:48.399 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:48.399 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.399 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.399 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:48.399 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.331167+0000", 2024-09-21T22:07:48.399 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9199999999999999e-07 2024-09-21T22:07:48.399 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.399 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.399 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:48.399 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.332127+0000", 2024-09-21T22:07:48.399 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00095969800000000004 2024-09-21T22:07:48.399 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.399 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.400 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:48.400 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.332149+0000", 2024-09-21T22:07:48.400 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1974999999999999e-05 2024-09-21T22:07:48.400 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.400 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.400 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:48.400 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.332161+0000", 2024-09-21T22:07:48.400 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2276000000000001e-05 2024-09-21T22:07:48.400 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.400 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.400 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:48.400 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.332290+0000", 2024-09-21T22:07:48.400 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000129214 2024-09-21T22:07:48.400 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.400 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.401 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:48.401 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.332312+0000", 2024-09-21T22:07:48.401 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1308999999999999e-05 2024-09-21T22:07:48.401 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.401 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.401 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:48.401 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.371162+0000", 2024-09-21T22:07:48.401 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.038850015000000002 2024-09-21T22:07:48.401 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:48.401 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:48.401 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:48.401 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.401 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.401 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4811.0:3 2.3 2:c4c92e5a:::rbd_trash:head [call rbd.trash_list in=13b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-21T22:07:48.401 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:28.475306+0000", 2024-09-21T22:07:48.402 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 19.891128174999999, 2024-09-21T22:07:48.402 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.77996670099999998, 2024-09-21T22:07:48.402 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:48.402 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:48.402 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:07:48.402 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:48.402 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4811", 2024-09-21T22:07:48.402 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/2028736390", 2024-09-21T22:07:48.402 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3 2024-09-21T22:07:48.402 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.402 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:48.402 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.402 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:48.402 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.475306+0000", 2024-09-21T22:07:48.402 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:48.403 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.403 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.403 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:48.403 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.475306+0000", 2024-09-21T22:07:48.403 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:48.403 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.403 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.403 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:48.403 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.475308+0000", 2024-09-21T22:07:48.403 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8959999999999999e-06 2024-09-21T22:07:48.403 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.403 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.403 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:48.403 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.475314+0000", 2024-09-21T22:07:48.403 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2280000000000003e-06 2024-09-21T22:07:48.404 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.404 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.404 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:48.404 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.475329+0000", 2024-09-21T22:07:48.404 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5054e-05 2024-09-21T22:07:48.404 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.404 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.404 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:48.404 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.475338+0000", 2024-09-21T22:07:48.404 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3770000000000002e-06 2024-09-21T22:07:48.404 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.404 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.404 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:48.404 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.475484+0000", 2024-09-21T22:07:48.404 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014612300000000001 2024-09-21T22:07:48.405 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.405 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.405 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-21T22:07:48.405 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.475506+0000", 2024-09-21T22:07:48.405 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1319999999999999e-05 2024-09-21T22:07:48.405 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.405 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.405 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:48.405 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.254433+0000", 2024-09-21T22:07:48.405 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.778927708 2024-09-21T22:07:48.405 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.405 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.405 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:48.405 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.254979+0000", 2024-09-21T22:07:48.405 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00054573100000000004 2024-09-21T22:07:48.406 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.406 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.406 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:48.406 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.255272+0000", 2024-09-21T22:07:48.406 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000293264 2024-09-21T22:07:48.406 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:48.406 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:48.406 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:48.406 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.406 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.406 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.3 67/66 [PushOp(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head, version: 63'11, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head@63'11, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T22:07:48.406 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:29.262595+0000", 2024-09-21T22:07:48.406 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 19.103838992, 2024-09-21T22:07:48.406 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.036027272999999999, 2024-09-21T22:07:48.406 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:48.407 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:48.407 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:07:48.407 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:48.407 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.407 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:48.407 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.262595+0000", 2024-09-21T22:07:48.407 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:48.407 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.407 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.407 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:48.407 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.262595+0000", 2024-09-21T22:07:48.407 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:48.407 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.407 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.408 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:48.408 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.262596+0000", 2024-09-21T22:07:48.408 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4300000000000001e-06 2024-09-21T22:07:48.408 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.408 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.408 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:48.408 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.264505+0000", 2024-09-21T22:07:48.408 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019085650000000001 2024-09-21T22:07:48.408 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.408 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.408 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:48.408 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.264557+0000", 2024-09-21T22:07:48.408 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2040000000000003e-05 2024-09-21T22:07:48.408 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.408 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.409 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:48.409 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.264589+0000", 2024-09-21T22:07:48.409 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1636e-05 2024-09-21T22:07:48.409 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.409 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.409 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:48.409 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.264778+0000", 2024-09-21T22:07:48.409 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018935600000000001 2024-09-21T22:07:48.409 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.409 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.409 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:48.409 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.264803+0000", 2024-09-21T22:07:48.409 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4692999999999999e-05 2024-09-21T22:07:48.409 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.409 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.410 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:48.410 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.298622+0000", 2024-09-21T22:07:48.410 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.033819553000000002 2024-09-21T22:07:48.410 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:48.410 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:48.410 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:48.410 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.410 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.410 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.3 67/66 [PushOp(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head, version: 63'11, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head@63'11, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:07:48.410 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:29.301490+0000", 2024-09-21T22:07:48.410 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 19.064944206, 2024-09-21T22:07:48.410 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.041178293999999997, 2024-09-21T22:07:48.410 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:48.410 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:48.411 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:07:48.411 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:48.411 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.411 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:48.411 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.301490+0000", 2024-09-21T22:07:48.411 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:48.411 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.411 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.411 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:48.411 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.301490+0000", 2024-09-21T22:07:48.411 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:48.411 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.411 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.411 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:48.411 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.301491+0000", 2024-09-21T22:07:48.411 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.513e-06 2024-09-21T22:07:48.412 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.412 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.412 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:48.412 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.302815+0000", 2024-09-21T22:07:48.412 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001323722 2024-09-21T22:07:48.412 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.412 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.412 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:48.412 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.302863+0000", 2024-09-21T22:07:48.412 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8449999999999999e-05 2024-09-21T22:07:48.412 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.412 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.412 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:48.412 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.302895+0000", 2024-09-21T22:07:48.412 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1072999999999999e-05 2024-09-21T22:07:48.413 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.413 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.413 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:48.413 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.303014+0000", 2024-09-21T22:07:48.413 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011907400000000001 2024-09-21T22:07:48.413 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.413 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.413 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:48.413 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.303029+0000", 2024-09-21T22:07:48.413 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5322e-05 2024-09-21T22:07:48.413 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.413 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.413 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:48.413 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.342668+0000", 2024-09-21T22:07:48.413 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.039639140000000003 2024-09-21T22:07:48.414 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:48.414 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:48.414 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:48.414 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.414 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.414 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.3 67/66 [PushOp(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head, version: 63'11, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head@63'11, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:07:48.414 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:29.345626+0000", 2024-09-21T22:07:48.414 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 19.020807472000001, 2024-09-21T22:07:48.414 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.034820216000000001, 2024-09-21T22:07:48.414 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:48.414 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:48.414 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:07:48.414 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:48.414 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.415 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:48.415 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.345626+0000", 2024-09-21T22:07:48.415 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:48.415 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.415 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.415 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:48.415 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.345626+0000", 2024-09-21T22:07:48.415 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:48.415 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.415 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.415 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:48.415 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.345628+0000", 2024-09-21T22:07:48.415 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3799999999999999e-06 2024-09-21T22:07:48.415 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.415 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.416 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:48.416 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.347215+0000", 2024-09-21T22:07:48.416 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015868760000000001 2024-09-21T22:07:48.416 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.416 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.416 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:48.416 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.347283+0000", 2024-09-21T22:07:48.416 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8267000000000002e-05 2024-09-21T22:07:48.416 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.416 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.416 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:48.416 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.347336+0000", 2024-09-21T22:07:48.416 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3455999999999998e-05 2024-09-21T22:07:48.416 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.416 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.417 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:48.417 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.347463+0000", 2024-09-21T22:07:48.417 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012614399999999999 2024-09-21T22:07:48.417 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.417 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.417 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:48.417 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.347473+0000", 2024-09-21T22:07:48.417 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0462000000000001e-05 2024-09-21T22:07:48.417 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.417 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.417 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:48.417 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.380447+0000", 2024-09-21T22:07:48.417 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.032973631000000003 2024-09-21T22:07:48.417 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:48.417 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:48.418 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:48.418 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.418 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.418 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.3 67/66 [PushOp(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head, version: 63'11, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head@63'11, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 4194304, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:07:48.418 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:29.383677+0000", 2024-09-21T22:07:48.418 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 18.982756595000001, 2024-09-21T22:07:48.418 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.031566611000000001, 2024-09-21T22:07:48.418 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:48.418 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:48.418 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:07:48.418 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:48.418 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.418 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:48.418 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.383677+0000", 2024-09-21T22:07:48.419 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:48.419 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.419 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.419 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:48.419 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.383677+0000", 2024-09-21T22:07:48.419 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:48.419 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.419 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.419 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:48.419 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.383679+0000", 2024-09-21T22:07:48.419 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4500000000000001e-06 2024-09-21T22:07:48.419 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.419 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.419 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:48.419 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.384982+0000", 2024-09-21T22:07:48.420 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001303069 2024-09-21T22:07:48.420 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.420 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.420 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:48.420 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.385028+0000", 2024-09-21T22:07:48.420 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5655999999999998e-05 2024-09-21T22:07:48.420 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.420 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.420 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:48.420 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.385056+0000", 2024-09-21T22:07:48.420 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8877000000000002e-05 2024-09-21T22:07:48.420 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.420 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.420 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:48.421 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.385161+0000", 2024-09-21T22:07:48.421 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000104325 2024-09-21T22:07:48.421 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.421 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.421 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:48.421 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.385173+0000", 2024-09-21T22:07:48.421 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2493e-05 2024-09-21T22:07:48.421 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.421 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.421 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:48.421 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.415244+0000", 2024-09-21T22:07:48.421 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.030070741000000002 2024-09-21T22:07:48.421 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:48.421 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:48.421 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:48.422 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.422 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.422 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.3 67/66 [PushOp(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head, version: 63'19, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head@63'19, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T22:07:48.422 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:29.420098+0000", 2024-09-21T22:07:48.422 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 18.946335637000001, 2024-09-21T22:07:48.422 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.067283846999999994, 2024-09-21T22:07:48.422 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:48.422 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:48.422 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:07:48.422 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:48.422 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.422 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:48.422 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.420098+0000", 2024-09-21T22:07:48.422 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:48.422 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.423 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.423 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:48.423 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.420098+0000", 2024-09-21T22:07:48.423 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:48.423 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.423 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.423 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:48.423 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.420100+0000", 2024-09-21T22:07:48.423 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.716e-06 2024-09-21T22:07:48.423 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.423 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.423 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:48.423 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.421949+0000", 2024-09-21T22:07:48.423 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0018485190000000001 2024-09-21T22:07:48.423 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.424 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.424 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:48.424 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.421996+0000", 2024-09-21T22:07:48.424 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7089000000000003e-05 2024-09-21T22:07:48.424 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.424 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.424 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:48.424 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.422042+0000", 2024-09-21T22:07:48.424 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6010000000000002e-05 2024-09-21T22:07:48.424 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.424 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.424 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:48.424 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.422231+0000", 2024-09-21T22:07:48.424 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018960900000000001 2024-09-21T22:07:48.425 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.425 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.425 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:48.425 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.422254+0000", 2024-09-21T22:07:48.425 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2864e-05 2024-09-21T22:07:48.425 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.425 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.425 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:48.425 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.487382+0000", 2024-09-21T22:07:48.425 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.065128039999999998 2024-09-21T22:07:48.425 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:48.425 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:48.425 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:48.425 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.425 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.425 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.3 67/66 [PushOp(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head, version: 63'19, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head@63'19, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:07:48.426 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:29.491783+0000", 2024-09-21T22:07:48.426 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 18.874651291999999, 2024-09-21T22:07:48.426 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.04476596, 2024-09-21T22:07:48.426 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:48.426 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:48.426 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:07:48.426 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:48.426 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.426 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:48.426 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.491783+0000", 2024-09-21T22:07:48.426 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:48.426 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.426 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.426 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:48.427 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.491783+0000", 2024-09-21T22:07:48.427 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:48.427 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.427 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.427 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:48.427 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.491784+0000", 2024-09-21T22:07:48.427 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.17e-06 2024-09-21T22:07:48.427 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.427 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.427 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:48.427 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.493126+0000", 2024-09-21T22:07:48.427 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0013424489999999999 2024-09-21T22:07:48.427 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.427 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.427 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:48.428 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.493170+0000", 2024-09-21T22:07:48.428 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4153000000000002e-05 2024-09-21T22:07:48.428 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.428 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.428 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:48.428 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.493218+0000", 2024-09-21T22:07:48.428 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7392999999999998e-05 2024-09-21T22:07:48.428 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.428 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.428 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:48.428 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.493392+0000", 2024-09-21T22:07:48.428 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017392699999999999 2024-09-21T22:07:48.428 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.428 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.429 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:48.429 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.493416+0000", 2024-09-21T22:07:48.429 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4117e-05 2024-09-21T22:07:48.429 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.429 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.429 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:48.429 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.536549+0000", 2024-09-21T22:07:48.429 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.043132750999999997 2024-09-21T22:07:48.429 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:48.429 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:48.429 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:48.429 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.429 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.429 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.3 67/66 [PushOp(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head, version: 63'19, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head@63'19, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:07:48.430 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:29.539831+0000", 2024-09-21T22:07:48.430 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 18.826602531999999, 2024-09-21T22:07:48.430 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.036233222000000002, 2024-09-21T22:07:48.430 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:48.430 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:48.430 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:07:48.430 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:48.430 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.430 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:48.430 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.539831+0000", 2024-09-21T22:07:48.430 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:48.430 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.430 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.430 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:48.430 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.539831+0000", 2024-09-21T22:07:48.431 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:48.431 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.431 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.431 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:48.431 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.539833+0000", 2024-09-21T22:07:48.431 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5200000000000001e-06 2024-09-21T22:07:48.431 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.431 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.431 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:48.431 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.541130+0000", 2024-09-21T22:07:48.431 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0012972719999999999 2024-09-21T22:07:48.431 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.431 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.431 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:48.431 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.541179+0000", 2024-09-21T22:07:48.432 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8507000000000002e-05 2024-09-21T22:07:48.432 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.432 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.432 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:48.432 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.541226+0000", 2024-09-21T22:07:48.432 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7663000000000001e-05 2024-09-21T22:07:48.432 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.432 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.432 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:48.432 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.541361+0000", 2024-09-21T22:07:48.432 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000135065 2024-09-21T22:07:48.432 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.432 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.432 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:48.433 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.541374+0000", 2024-09-21T22:07:48.433 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2295e-05 2024-09-21T22:07:48.433 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.433 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.433 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:48.433 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.576065+0000", 2024-09-21T22:07:48.433 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.034690899999999997 2024-09-21T22:07:48.433 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:48.433 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:48.433 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:48.433 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.433 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.433 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.3 67/66 [PushOp(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head, version: 63'19, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head@63'19, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 4194304, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:07:48.433 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:29.579455+0000", 2024-09-21T22:07:48.434 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 18.786979224, 2024-09-21T22:07:48.434 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.034295594999999998, 2024-09-21T22:07:48.434 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:48.434 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:48.434 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:07:48.434 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:48.434 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.434 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:48.434 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.579455+0000", 2024-09-21T22:07:48.434 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:48.434 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.434 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.434 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:48.434 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.579455+0000", 2024-09-21T22:07:48.434 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:48.435 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.435 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.435 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:48.435 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.579456+0000", 2024-09-21T22:07:48.435 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.404e-06 2024-09-21T22:07:48.435 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.435 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.435 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:48.435 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.580763+0000", 2024-09-21T22:07:48.435 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001307325 2024-09-21T22:07:48.435 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.435 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.435 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:48.435 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.580811+0000", 2024-09-21T22:07:48.435 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7843000000000003e-05 2024-09-21T22:07:48.436 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.436 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.436 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:48.436 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.580860+0000", 2024-09-21T22:07:48.436 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9036999999999998e-05 2024-09-21T22:07:48.436 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.436 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.436 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:48.436 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.580988+0000", 2024-09-21T22:07:48.436 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000127529 2024-09-21T22:07:48.436 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.436 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.436 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:48.436 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.580998+0000", 2024-09-21T22:07:48.436 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0617e-05 2024-09-21T22:07:48.437 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.437 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.437 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:48.437 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.613750+0000", 2024-09-21T22:07:48.437 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.032751839999999997 2024-09-21T22:07:48.437 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:48.437 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:48.437 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:48.437 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.437 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.437 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5008.0:125 2.3 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-21T22:07:48.437 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:38.066298+0000", 2024-09-21T22:07:48.437 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.300135559999999, 2024-09-21T22:07:48.437 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.062817136999999995, 2024-09-21T22:07:48.437 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:48.438 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:48.438 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:48.438 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:48.438 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5008", 2024-09-21T22:07:48.438 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/1400114666", 2024-09-21T22:07:48.438 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 125 2024-09-21T22:07:48.438 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.438 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:48.438 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.438 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:48.438 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.066298+0000", 2024-09-21T22:07:48.438 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:48.438 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.438 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.438 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:48.439 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.066298+0000", 2024-09-21T22:07:48.439 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:48.439 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.439 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.439 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:48.439 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.066300+0000", 2024-09-21T22:07:48.439 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.474e-06 2024-09-21T22:07:48.439 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.439 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.439 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:48.439 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.067443+0000", 2024-09-21T22:07:48.439 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011428009999999999 2024-09-21T22:07:48.439 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.439 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.439 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:48.440 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.067477+0000", 2024-09-21T22:07:48.440 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4059e-05 2024-09-21T22:07:48.440 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.440 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.440 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:48.440 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.067492+0000", 2024-09-21T22:07:48.440 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5054e-05 2024-09-21T22:07:48.440 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.440 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.440 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:48.440 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.067638+0000", 2024-09-21T22:07:48.440 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014583900000000001 2024-09-21T22:07:48.440 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.440 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.441 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:48.441 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.067882+0000", 2024-09-21T22:07:48.441 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024415500000000002 2024-09-21T22:07:48.441 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.441 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.441 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-09-21T22:07:48.441 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.068247+0000", 2024-09-21T22:07:48.441 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00036570800000000002 2024-09-21T22:07:48.441 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.441 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.441 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:48.441 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.119971+0000", 2024-09-21T22:07:48.441 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.051723866 2024-09-21T22:07:48.441 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.441 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.442 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:48.442 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.128849+0000", 2024-09-21T22:07:48.442 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0088779489999999996 2024-09-21T22:07:48.442 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.442 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.442 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:48.442 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.129000+0000", 2024-09-21T22:07:48.442 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015062499999999999 2024-09-21T22:07:48.442 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.442 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.442 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:48.442 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.129116+0000", 2024-09-21T22:07:48.442 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011560700000000001 2024-09-21T22:07:48.442 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:48.442 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:48.443 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:48.443 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.443 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.443 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5008.0:127 2.3 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-21T22:07:48.443 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:38.140308+0000", 2024-09-21T22:07:48.443 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.226125458, 2024-09-21T22:07:48.443 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.055290512999999999, 2024-09-21T22:07:48.443 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:48.443 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:48.443 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:48.443 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:48.443 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5008", 2024-09-21T22:07:48.443 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/1400114666", 2024-09-21T22:07:48.443 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 127 2024-09-21T22:07:48.443 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.444 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:48.444 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.444 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:48.444 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.140308+0000", 2024-09-21T22:07:48.444 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:48.444 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.444 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.444 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:48.444 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.140308+0000", 2024-09-21T22:07:48.444 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:48.444 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.444 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.444 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:48.444 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.140310+0000", 2024-09-21T22:07:48.445 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6509999999999999e-06 2024-09-21T22:07:48.445 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.445 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.445 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:48.445 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.141504+0000", 2024-09-21T22:07:48.445 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011935750000000001 2024-09-21T22:07:48.445 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.445 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.445 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:48.445 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.141522+0000", 2024-09-21T22:07:48.445 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8349e-05 2024-09-21T22:07:48.445 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.445 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.445 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:48.445 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.141531+0000", 2024-09-21T22:07:48.446 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3239999999999992e-06 2024-09-21T22:07:48.446 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.446 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.446 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:48.446 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.141661+0000", 2024-09-21T22:07:48.446 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013001000000000001 2024-09-21T22:07:48.446 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.446 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.446 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:48.446 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.141907+0000", 2024-09-21T22:07:48.446 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024547499999999999 2024-09-21T22:07:48.446 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.446 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.446 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-09-21T22:07:48.446 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.142274+0000", 2024-09-21T22:07:48.447 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00036716400000000001 2024-09-21T22:07:48.447 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.447 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.447 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:48.447 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.189864+0000", 2024-09-21T22:07:48.447 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047589617000000001 2024-09-21T22:07:48.447 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.447 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.447 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:48.447 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.195406+0000", 2024-09-21T22:07:48.447 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0055425860000000004 2024-09-21T22:07:48.447 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.447 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.447 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:48.447 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.195507+0000", 2024-09-21T22:07:48.448 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000100862 2024-09-21T22:07:48.448 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.448 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.448 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:48.448 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.195599+0000", 2024-09-21T22:07:48.448 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1899999999999998e-05 2024-09-21T22:07:48.448 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:48.448 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:48.448 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:48.448 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.448 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.448 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5008.0:129 2.3 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head [write 2228224~917504 in=917504b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-21T22:07:48.448 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:38.209297+0000", 2024-09-21T22:07:48.448 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.157136576999999, 2024-09-21T22:07:48.448 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.060441054000000001, 2024-09-21T22:07:48.449 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:48.449 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:48.449 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:48.449 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:48.449 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5008", 2024-09-21T22:07:48.449 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/1400114666", 2024-09-21T22:07:48.449 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 129 2024-09-21T22:07:48.449 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.449 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:48.449 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.449 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:48.449 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.209297+0000", 2024-09-21T22:07:48.449 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:48.449 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.449 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.450 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:48.450 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.209297+0000", 2024-09-21T22:07:48.450 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:48.450 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.450 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.450 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:48.450 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.209299+0000", 2024-09-21T22:07:48.450 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.274e-06 2024-09-21T22:07:48.450 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.450 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.450 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:48.450 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.210077+0000", 2024-09-21T22:07:48.450 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00077849799999999997 2024-09-21T22:07:48.450 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.450 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.451 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:48.451 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.210094+0000", 2024-09-21T22:07:48.451 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6772000000000001e-05 2024-09-21T22:07:48.451 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.451 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.451 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:48.451 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.210103+0000", 2024-09-21T22:07:48.451 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6780000000000006e-06 2024-09-21T22:07:48.451 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.451 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.451 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:48.451 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.210254+0000", 2024-09-21T22:07:48.451 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015141699999999999 2024-09-21T22:07:48.451 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.451 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.452 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:48.452 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.210501+0000", 2024-09-21T22:07:48.452 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000246498 2024-09-21T22:07:48.452 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.452 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.452 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-09-21T22:07:48.452 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.210872+0000", 2024-09-21T22:07:48.452 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00037147500000000002 2024-09-21T22:07:48.452 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.452 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.452 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:48.452 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.269019+0000", 2024-09-21T22:07:48.452 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058147114 2024-09-21T22:07:48.452 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.452 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.453 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:48.453 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.269638+0000", 2024-09-21T22:07:48.453 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00061896499999999997 2024-09-21T22:07:48.453 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.453 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.453 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:48.453 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.269693+0000", 2024-09-21T22:07:48.453 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4786e-05 2024-09-21T22:07:48.453 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.453 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.453 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:48.453 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.269738+0000", 2024-09-21T22:07:48.453 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5577000000000001e-05 2024-09-21T22:07:48.453 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:48.453 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:48.454 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:48.454 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.454 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.454 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5008.0:131 2.3 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-21T22:07:48.454 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:38.279879+0000", 2024-09-21T22:07:48.454 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 10.086554887, 2024-09-21T22:07:48.454 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.054184296999999999, 2024-09-21T22:07:48.454 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:48.454 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:48.454 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:48.454 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:48.454 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5008", 2024-09-21T22:07:48.454 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/1400114666", 2024-09-21T22:07:48.454 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 131 2024-09-21T22:07:48.455 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.455 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:48.455 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.455 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:48.455 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.279879+0000", 2024-09-21T22:07:48.455 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:48.455 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.455 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.455 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:48.455 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.279879+0000", 2024-09-21T22:07:48.455 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:48.455 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.455 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.455 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:48.455 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.279880+0000", 2024-09-21T22:07:48.456 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.136e-06 2024-09-21T22:07:48.456 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.456 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.456 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:48.456 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.280636+0000", 2024-09-21T22:07:48.456 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00075625900000000001 2024-09-21T22:07:48.456 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.456 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.456 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:48.456 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.280648+0000", 2024-09-21T22:07:48.456 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2026e-05 2024-09-21T22:07:48.456 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.456 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.456 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:48.456 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.280654+0000", 2024-09-21T22:07:48.457 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8839999999999999e-06 2024-09-21T22:07:48.457 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.457 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.457 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:48.457 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.280739+0000", 2024-09-21T22:07:48.457 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4196000000000005e-05 2024-09-21T22:07:48.457 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.457 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.457 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:48.457 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.280808+0000", 2024-09-21T22:07:48.457 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9685999999999997e-05 2024-09-21T22:07:48.457 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.457 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.457 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-09-21T22:07:48.457 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.280900+0000", 2024-09-21T22:07:48.457 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1407000000000001e-05 2024-09-21T22:07:48.458 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.458 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.458 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:48.458 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.307730+0000", 2024-09-21T22:07:48.458 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026830568999999999 2024-09-21T22:07:48.458 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.458 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.458 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:48.458 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.333901+0000", 2024-09-21T22:07:48.458 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026170787000000001 2024-09-21T22:07:48.458 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.458 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.458 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:48.458 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.333996+0000", 2024-09-21T22:07:48.458 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4723999999999996e-05 2024-09-21T22:07:48.459 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:48.459 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:48.459 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:48.459 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.334063+0000", 2024-09-21T22:07:48.459 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7623e-05 2024-09-21T22:07:48.459 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:48.459 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:48.459 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:48.459 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:48.459 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:48.459 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:07:48.461 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T22:07:48.461 INFO:tasks.workunit.client.0.smithi157.stderr:+ DEV=/dev/nbd1 2024-09-21T22:07:48.462 INFO:tasks.workunit.client.0.smithi157.stderr:+ get_pid rbd ns 2024-09-21T22:07:48.462 INFO:tasks.workunit.client.0.smithi157.stderr:+ local pool=rbd 2024-09-21T22:07:48.462 INFO:tasks.workunit.client.0.smithi157.stderr:+ local ns=ns 2024-09-21T22:07:48.462 INFO:tasks.workunit.client.0.smithi157.stderr:++ rbd device --device-type nbd --format xml list 2024-09-21T22:07:48.462 INFO:tasks.workunit.client.0.smithi157.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\''ns'\''][image='\''testrbdnbd21553'\''][device='\''/dev/nbd1'\'']/id' 2024-09-21T22:07:48.462 INFO:tasks.workunit.client.0.smithi157.stderr:+ PID=23296 2024-09-21T22:07:48.462 INFO:tasks.workunit.client.0.smithi157.stderr:+ test -n 23296 2024-09-21T22:07:48.462 INFO:tasks.workunit.client.0.smithi157.stderr:+ ps -p 23296 -C rbd-nbd 2024-09-21T22:07:48.462 INFO:tasks.workunit.client.0.smithi157.stdout: PID TTY TIME CMD 2024-09-21T22:07:48.462 INFO:tasks.workunit.client.0.smithi157.stdout: 23250 ? 00:00:00 rbd-nbd 2024-09-21T22:07:48.462 INFO:tasks.workunit.client.0.smithi157.stdout: 23296 ? 00:00:00 rbd-nbd 2024-09-21T22:07:48.464 INFO:tasks.workunit.client.0.smithi157.stderr:+ unmap_device rbd/ns/testrbdnbd21553 23296 2024-09-21T22:07:48.464 INFO:tasks.workunit.client.0.smithi157.stderr:+ local args=rbd/ns/testrbdnbd21553 2024-09-21T22:07:48.464 INFO:tasks.workunit.client.0.smithi157.stderr:+ local pid=23296 2024-09-21T22:07:48.464 INFO:tasks.workunit.client.0.smithi157.stderr:+ _sudo rbd device --device-type nbd unmap rbd/ns/testrbdnbd21553 2024-09-21T22:07:48.464 INFO:tasks.workunit.client.0.smithi157.stderr:+ local cmd 2024-09-21T22:07:48.464 INFO:tasks.workunit.client.0.smithi157.stderr:++ id -u 2024-09-21T22:07:48.464 INFO:tasks.workunit.client.0.smithi157.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T22:07:48.464 INFO:tasks.workunit.client.0.smithi157.stderr:++ which rbd 2024-09-21T22:07:48.464 INFO:tasks.workunit.client.0.smithi157.stderr:+ cmd=/usr/bin/rbd 2024-09-21T22:07:48.464 INFO:tasks.workunit.client.0.smithi157.stderr:+ shift 2024-09-21T22:07:48.464 INFO:tasks.workunit.client.0.smithi157.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap rbd/ns/testrbdnbd21553 2024-09-21T22:07:48.502 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:48.500+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:48.539 INFO:tasks.workunit.client.0.smithi157.stderr:+ rbd device --device-type nbd list 2024-09-21T22:07:48.540 INFO:tasks.workunit.client.0.smithi157.stderr:+ expect_false grep '^23296\b' 2024-09-21T22:07:48.540 INFO:tasks.workunit.client.0.smithi157.stderr:+ grep '^23296\b' 2024-09-21T22:07:48.566 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:48.564+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:48.598 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:07:48.598 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:07:48.598 INFO:teuthology.orchestra.run.smithi039.stdout: "num_ops": 0 2024-09-21T22:07:48.598 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:07:48.600 INFO:tasks.workunit.client.0.smithi157.stderr:+ return 0 2024-09-21T22:07:48.600 INFO:tasks.workunit.client.0.smithi157.stderr:+ ps -C rbd-nbd 2024-09-21T22:07:48.600 INFO:tasks.workunit.client.0.smithi157.stderr:+ expect_false grep '^ *23296\b' 2024-09-21T22:07:48.600 INFO:tasks.workunit.client.0.smithi157.stderr:+ grep '^ *23296\b' 2024-09-21T22:07:48.606 INFO:tasks.workunit.client.0.smithi157.stderr:+ return 0 2024-09-21T22:07:48.606 INFO:tasks.workunit.client.0.smithi157.stderr:+ sleep 0.5 2024-09-21T22:07:48.612 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T22:07:48.667 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:48.662+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:48.731 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:07:48.731 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:07:48.731 INFO:teuthology.orchestra.run.smithi039.stdout: "complaint_time": 30, 2024-09-21T22:07:48.731 INFO:teuthology.orchestra.run.smithi039.stdout: "num_blocked_ops": 0 2024-09-21T22:07:48.731 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:07:48.746 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T22:07:48.788 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:48.786+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:48.821 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-21T22:07:48.880 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:48.878+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:48.921 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:07:48.921 INFO:teuthology.orchestra.run.smithi039.stdout: "size": 20, 2024-09-21T22:07:48.921 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 600, 2024-09-21T22:07:48.921 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [ 2024-09-21T22:07:48.921 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.921 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:694 2.5 e63/37 2:b1d55633:::rbd_data.126483d75f2d.000000000000000b:head v 63'26, mlcod=63'24)", 2024-09-21T22:07:48.921 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.163199+0000", 2024-09-21T22:07:48.921 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 23.754061156999999, 2024-09-21T22:07:48.921 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.034676672999999998, 2024-09-21T22:07:48.921 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:48.922 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:48.922 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:48.922 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:48.922 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.922 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:48.922 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.163199+0000", 2024-09-21T22:07:48.922 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:48.922 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.922 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.922 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:48.922 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.163199+0000", 2024-09-21T22:07:48.922 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:48.922 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.922 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.923 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:48.923 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.163200+0000", 2024-09-21T22:07:48.923 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.703e-06 2024-09-21T22:07:48.923 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.923 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.923 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:48.923 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.164013+0000", 2024-09-21T22:07:48.923 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000812447 2024-09-21T22:07:48.923 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.923 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.923 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:48.923 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.164042+0000", 2024-09-21T22:07:48.923 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9006e-05 2024-09-21T22:07:48.923 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.924 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.924 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:48.924 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.164064+0000", 2024-09-21T22:07:48.924 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1719999999999999e-05 2024-09-21T22:07:48.924 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.924 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.924 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:48.924 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.164143+0000", 2024-09-21T22:07:48.924 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.9283999999999995e-05 2024-09-21T22:07:48.924 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.924 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.924 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:48.924 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.164165+0000", 2024-09-21T22:07:48.924 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.2379e-05 2024-09-21T22:07:48.925 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.925 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.925 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:48.925 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.197827+0000", 2024-09-21T22:07:48.925 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.033661782000000001 2024-09-21T22:07:48.925 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.925 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.925 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:48.925 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.197875+0000", 2024-09-21T22:07:48.925 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8352000000000001e-05 2024-09-21T22:07:48.925 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:48.925 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:48.925 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:48.926 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.926 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.926 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:696 2.5 e63/37 2:b1d55633:::rbd_data.126483d75f2d.000000000000000b:head v 63'28, mlcod=63'26)", 2024-09-21T22:07:48.926 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.218913+0000", 2024-09-21T22:07:48.926 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 23.698346710999999, 2024-09-21T22:07:48.926 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.056187599999999997, 2024-09-21T22:07:48.926 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:48.926 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:48.926 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:48.926 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:48.926 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.926 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:48.926 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.218913+0000", 2024-09-21T22:07:48.927 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:48.927 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.927 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.927 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:48.927 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.218913+0000", 2024-09-21T22:07:48.927 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:48.927 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.927 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.927 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:48.927 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.218914+0000", 2024-09-21T22:07:48.927 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.1930000000000001e-06 2024-09-21T22:07:48.927 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.927 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.927 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:48.927 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.219721+0000", 2024-09-21T22:07:48.928 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00080701500000000001 2024-09-21T22:07:48.928 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.928 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.928 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:48.928 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.219743+0000", 2024-09-21T22:07:48.928 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1364000000000001e-05 2024-09-21T22:07:48.928 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.928 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.928 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:48.928 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.219759+0000", 2024-09-21T22:07:48.928 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.611e-05 2024-09-21T22:07:48.928 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.928 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.929 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:48.929 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.219908+0000", 2024-09-21T22:07:48.929 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000149252 2024-09-21T22:07:48.929 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.929 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.929 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:48.929 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.219951+0000", 2024-09-21T22:07:48.929 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.2518999999999997e-05 2024-09-21T22:07:48.929 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.929 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.930 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:48.930 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.275031+0000", 2024-09-21T22:07:48.930 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.055080241000000002 2024-09-21T22:07:48.930 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.930 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.930 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:48.930 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.275101+0000", 2024-09-21T22:07:48.930 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.9906000000000005e-05 2024-09-21T22:07:48.930 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:48.930 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:48.930 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:48.931 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.931 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.931 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:698 2.5 e63/37 2:b1d55633:::rbd_data.126483d75f2d.000000000000000b:head v 63'30, mlcod=63'28)", 2024-09-21T22:07:48.931 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.289319+0000", 2024-09-21T22:07:48.931 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 23.627940502000001, 2024-09-21T22:07:48.931 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.053930944000000001, 2024-09-21T22:07:48.931 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:48.931 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:48.931 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:48.931 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:48.931 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.931 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:48.931 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.289319+0000", 2024-09-21T22:07:48.931 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:48.932 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.932 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.932 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:48.932 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.289319+0000", 2024-09-21T22:07:48.932 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:48.932 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.932 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.932 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:48.932 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.289321+0000", 2024-09-21T22:07:48.932 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.221e-06 2024-09-21T22:07:48.932 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.932 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.932 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:48.932 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.290091+0000", 2024-09-21T22:07:48.933 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00077069300000000003 2024-09-21T22:07:48.933 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.933 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.933 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:48.933 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.290109+0000", 2024-09-21T22:07:48.933 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7218999999999999e-05 2024-09-21T22:07:48.933 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.933 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.933 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:48.933 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.290122+0000", 2024-09-21T22:07:48.933 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3678e-05 2024-09-21T22:07:48.933 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.933 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.934 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:48.934 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.290274+0000", 2024-09-21T22:07:48.934 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00015205899999999999 2024-09-21T22:07:48.934 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.934 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.934 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:48.934 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.290313+0000", 2024-09-21T22:07:48.934 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.8636000000000001e-05 2024-09-21T22:07:48.934 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.934 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.934 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:48.934 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.343179+0000", 2024-09-21T22:07:48.934 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.052866199000000003 2024-09-21T22:07:48.934 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.935 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.935 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:48.935 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.343250+0000", 2024-09-21T22:07:48.935 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.1239e-05 2024-09-21T22:07:48.935 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:48.935 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:48.935 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:48.935 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.935 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.935 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:702 2.5 e63/37 2:a70ae99f:::rbd_data.126483d75f2d.000000000000000c:head v 63'34, mlcod=63'32)", 2024-09-21T22:07:48.935 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.392532+0000", 2024-09-21T22:07:48.935 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 23.524727914, 2024-09-21T22:07:48.935 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.035703990999999997, 2024-09-21T22:07:48.935 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:48.936 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:48.936 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:48.936 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:48.936 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.936 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:48.936 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.392532+0000", 2024-09-21T22:07:48.936 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:48.936 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.936 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.936 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:48.936 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.392532+0000", 2024-09-21T22:07:48.936 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:48.936 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.936 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.937 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:48.937 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.392534+0000", 2024-09-21T22:07:48.937 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.64e-06 2024-09-21T22:07:48.937 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.937 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.937 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:48.937 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.393321+0000", 2024-09-21T22:07:48.937 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00078705699999999997 2024-09-21T22:07:48.937 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.937 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.937 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:48.937 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.393347+0000", 2024-09-21T22:07:48.937 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.6735999999999999e-05 2024-09-21T22:07:48.937 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.938 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.938 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:48.938 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.393361+0000", 2024-09-21T22:07:48.938 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3346000000000001e-05 2024-09-21T22:07:48.938 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.938 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.938 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:48.938 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.393469+0000", 2024-09-21T22:07:48.938 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000108039 2024-09-21T22:07:48.938 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.938 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.938 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:48.938 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.393512+0000", 2024-09-21T22:07:48.938 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.2862999999999999e-05 2024-09-21T22:07:48.939 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.939 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.939 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:48.939 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.428170+0000", 2024-09-21T22:07:48.939 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.034658858000000001 2024-09-21T22:07:48.939 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.939 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.939 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:48.939 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.428236+0000", 2024-09-21T22:07:48.939 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.5451999999999996e-05 2024-09-21T22:07:48.939 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:48.939 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:48.939 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:48.939 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.939 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.940 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:708 2.1 e63/53 2:837ddd83:::rbd_data.126483d75f2d.000000000000000d:head v 63'6, mlcod=57'4)", 2024-09-21T22:07:48.940 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.517007+0000", 2024-09-21T22:07:48.940 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 23.400252987999998, 2024-09-21T22:07:48.940 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.032495584000000001, 2024-09-21T22:07:48.940 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:48.940 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:48.940 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:48.940 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:48.940 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.940 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:48.940 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.517007+0000", 2024-09-21T22:07:48.940 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:48.940 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.941 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.941 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:48.941 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.517007+0000", 2024-09-21T22:07:48.941 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:48.941 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.941 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.941 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:48.941 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.517008+0000", 2024-09-21T22:07:48.941 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.604e-06 2024-09-21T22:07:48.941 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.941 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.941 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:48.941 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.517603+0000", 2024-09-21T22:07:48.941 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00059461900000000001 2024-09-21T22:07:48.942 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.942 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.942 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:48.942 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.517632+0000", 2024-09-21T22:07:48.942 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8663e-05 2024-09-21T22:07:48.942 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.942 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.942 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:48.942 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.517647+0000", 2024-09-21T22:07:48.942 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4833e-05 2024-09-21T22:07:48.942 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.942 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.942 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:48.943 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.517714+0000", 2024-09-21T22:07:48.943 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.7669000000000006e-05 2024-09-21T22:07:48.943 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.943 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.943 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:48.943 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.517737+0000", 2024-09-21T22:07:48.943 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.2469000000000001e-05 2024-09-21T22:07:48.943 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.943 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.943 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:48.943 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.549370+0000", 2024-09-21T22:07:48.943 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.031633221000000003 2024-09-21T22:07:48.943 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.943 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.943 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:48.944 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.549502+0000", 2024-09-21T22:07:48.944 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013250600000000001 2024-09-21T22:07:48.944 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:48.944 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:48.944 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:48.944 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.944 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.944 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:712 2.1 e63/53 2:837ddd83:::rbd_data.126483d75f2d.000000000000000d:head v 63'10, mlcod=63'8)", 2024-09-21T22:07:48.944 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.604029+0000", 2024-09-21T22:07:48.944 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 23.313231048999999, 2024-09-21T22:07:48.944 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.037982371000000001, 2024-09-21T22:07:48.944 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:48.944 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:48.945 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:48.945 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:48.945 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.945 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:48.945 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.604029+0000", 2024-09-21T22:07:48.945 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:48.945 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.945 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.945 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:48.945 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.604029+0000", 2024-09-21T22:07:48.945 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:48.945 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.945 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.945 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:48.946 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.604031+0000", 2024-09-21T22:07:48.946 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8870000000000001e-06 2024-09-21T22:07:48.946 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.946 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.946 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:48.946 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.604395+0000", 2024-09-21T22:07:48.946 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00036422000000000001 2024-09-21T22:07:48.946 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.946 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.946 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:48.946 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.604424+0000", 2024-09-21T22:07:48.946 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8609e-05 2024-09-21T22:07:48.946 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.946 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.947 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:48.947 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.604437+0000", 2024-09-21T22:07:48.947 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3086e-05 2024-09-21T22:07:48.947 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.947 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.947 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:48.947 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.604509+0000", 2024-09-21T22:07:48.947 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.2448999999999997e-05 2024-09-21T22:07:48.947 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.947 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.947 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:48.947 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.604531+0000", 2024-09-21T22:07:48.947 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1602000000000001e-05 2024-09-21T22:07:48.947 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.947 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.948 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:48.948 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.641885+0000", 2024-09-21T22:07:48.948 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.037354545000000003 2024-09-21T22:07:48.948 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.948 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.948 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:48.948 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.642011+0000", 2024-09-21T22:07:48.948 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00012597300000000001 2024-09-21T22:07:48.948 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:48.948 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:48.948 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:48.948 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.948 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.948 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:720 2.1 e63/53 2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head v 63'18, mlcod=63'16)", 2024-09-21T22:07:48.949 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.805111+0000", 2024-09-21T22:07:48.949 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 23.112148989000001, 2024-09-21T22:07:48.949 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.046082864000000001, 2024-09-21T22:07:48.949 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:48.949 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:48.949 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:48.949 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:48.949 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.949 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:48.949 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.805111+0000", 2024-09-21T22:07:48.949 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:48.949 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.949 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.949 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:48.950 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.805111+0000", 2024-09-21T22:07:48.950 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:48.950 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.950 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.950 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:48.950 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.805112+0000", 2024-09-21T22:07:48.950 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.02e-06 2024-09-21T22:07:48.950 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.950 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.950 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:48.951 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.805429+0000", 2024-09-21T22:07:48.951 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00031740799999999998 2024-09-21T22:07:48.951 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.951 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.951 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:48.951 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.805459+0000", 2024-09-21T22:07:48.951 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.955e-05 2024-09-21T22:07:48.951 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.951 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.951 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:48.951 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.805472+0000", 2024-09-21T22:07:48.951 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3533000000000001e-05 2024-09-21T22:07:48.951 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.951 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.952 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:48.952 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.805581+0000", 2024-09-21T22:07:48.952 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000108194 2024-09-21T22:07:48.952 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.952 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.952 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:48.952 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.805615+0000", 2024-09-21T22:07:48.952 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.4310000000000002e-05 2024-09-21T22:07:48.952 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.952 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.952 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:48.952 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.851146+0000", 2024-09-21T22:07:48.952 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.045531092000000002 2024-09-21T22:07:48.952 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.953 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.953 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:48.953 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.851194+0000", 2024-09-21T22:07:48.953 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.7756999999999997e-05 2024-09-21T22:07:48.953 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:48.953 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:48.953 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:48.953 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.953 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.953 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:723 2.1 e63/53 2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head v 63'20, mlcod=63'18)", 2024-09-21T22:07:48.953 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.862341+0000", 2024-09-21T22:07:48.953 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 23.054919054999999, 2024-09-21T22:07:48.953 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.036454671000000001, 2024-09-21T22:07:48.954 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:48.954 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:48.954 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:48.954 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:48.954 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.954 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:48.954 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.862341+0000", 2024-09-21T22:07:48.954 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:48.954 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.954 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.954 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:48.954 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.862341+0000", 2024-09-21T22:07:48.954 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:48.955 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.955 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.955 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:48.955 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.862342+0000", 2024-09-21T22:07:48.955 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.0299999999999997e-07 2024-09-21T22:07:48.955 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.955 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.955 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:48.955 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.862721+0000", 2024-09-21T22:07:48.955 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00037889800000000002 2024-09-21T22:07:48.955 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.955 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.955 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:48.955 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.862738+0000", 2024-09-21T22:07:48.955 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6957999999999999e-05 2024-09-21T22:07:48.956 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.956 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.956 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:48.956 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.862747+0000", 2024-09-21T22:07:48.956 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.0729999999999994e-06 2024-09-21T22:07:48.956 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.956 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.956 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:48.956 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.862830+0000", 2024-09-21T22:07:48.956 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.3113000000000002e-05 2024-09-21T22:07:48.956 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.956 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.956 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:48.956 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.862860+0000", 2024-09-21T22:07:48.957 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9980000000000001e-05 2024-09-21T22:07:48.957 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.957 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.957 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:48.957 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.898705+0000", 2024-09-21T22:07:48.957 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.035845063000000003 2024-09-21T22:07:48.957 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.957 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.957 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:48.957 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.898795+0000", 2024-09-21T22:07:48.957 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.0682999999999999e-05 2024-09-21T22:07:48.957 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:48.957 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:48.957 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:48.958 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.958 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.958 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:136 2.5 e72/37 2:b1d55633:::rbd_data.126483d75f2d.000000000000000b:head v 72'44, mlcod=72'42)", 2024-09-21T22:07:48.958 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.386273+0000", 2024-09-21T22:07:48.958 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 10.53098694, 2024-09-21T22:07:48.958 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.033806744999999999, 2024-09-21T22:07:48.958 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:48.958 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:48.958 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:48.958 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:48.958 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.958 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:48.958 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.386273+0000", 2024-09-21T22:07:48.958 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:48.959 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.959 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.959 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:48.959 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.386273+0000", 2024-09-21T22:07:48.959 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:48.959 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.959 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.959 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:48.959 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.386274+0000", 2024-09-21T22:07:48.959 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.8299999999999996e-07 2024-09-21T22:07:48.959 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.959 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.959 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:48.960 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.387108+0000", 2024-09-21T22:07:48.960 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00083467800000000005 2024-09-21T22:07:48.960 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.960 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.960 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:48.960 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.387121+0000", 2024-09-21T22:07:48.960 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2823000000000001e-05 2024-09-21T22:07:48.960 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.960 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.960 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:48.960 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.387128+0000", 2024-09-21T22:07:48.960 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.404e-06 2024-09-21T22:07:48.960 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.961 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.961 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:48.961 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.387179+0000", 2024-09-21T22:07:48.961 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.1888000000000002e-05 2024-09-21T22:07:48.961 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.961 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.961 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:48.961 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.387201+0000", 2024-09-21T22:07:48.961 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.2019999999999999e-05 2024-09-21T22:07:48.961 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.961 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.961 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:48.961 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.419993+0000", 2024-09-21T22:07:48.962 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.032791182000000002 2024-09-21T22:07:48.962 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.962 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.962 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:48.962 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.420080+0000", 2024-09-21T22:07:48.962 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.7066999999999998e-05 2024-09-21T22:07:48.962 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:48.962 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:48.962 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:48.962 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.962 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.962 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:138 2.5 e72/37 2:b1d55633:::rbd_data.126483d75f2d.000000000000000b:head v 72'46, mlcod=72'44)", 2024-09-21T22:07:48.962 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.430104+0000", 2024-09-21T22:07:48.963 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 10.487155592000001, 2024-09-21T22:07:48.963 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.034087458000000001, 2024-09-21T22:07:48.963 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:48.963 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:48.963 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:48.963 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:48.963 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.963 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:48.963 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.430104+0000", 2024-09-21T22:07:48.963 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:48.963 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.963 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.963 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:48.964 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.430104+0000", 2024-09-21T22:07:48.964 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:48.964 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.964 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.964 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:48.964 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.430105+0000", 2024-09-21T22:07:48.964 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.1680000000000001e-06 2024-09-21T22:07:48.964 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.964 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.964 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:48.964 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.430864+0000", 2024-09-21T22:07:48.964 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00075819400000000001 2024-09-21T22:07:48.965 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.965 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.965 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:48.965 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.430881+0000", 2024-09-21T22:07:48.965 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7000999999999999e-05 2024-09-21T22:07:48.965 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.965 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.965 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:48.965 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.430889+0000", 2024-09-21T22:07:48.965 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.9540000000000004e-06 2024-09-21T22:07:48.965 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.965 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.965 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:48.966 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.430959+0000", 2024-09-21T22:07:48.966 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.0858999999999994e-05 2024-09-21T22:07:48.966 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.966 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.966 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:48.966 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.430996+0000", 2024-09-21T22:07:48.966 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.6443999999999999e-05 2024-09-21T22:07:48.966 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.966 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.966 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:48.966 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.464087+0000", 2024-09-21T22:07:48.966 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.033090799999999997 2024-09-21T22:07:48.966 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.967 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.967 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:48.967 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.464192+0000", 2024-09-21T22:07:48.967 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00010503800000000001 2024-09-21T22:07:48.967 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:48.967 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:48.967 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:48.967 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.967 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.967 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:140 2.5 e72/37 2:b1d55633:::rbd_data.126483d75f2d.000000000000000b:head v 72'48, mlcod=72'46)", 2024-09-21T22:07:48.967 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.475737+0000", 2024-09-21T22:07:48.967 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 10.441522644000001, 2024-09-21T22:07:48.967 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.037632025, 2024-09-21T22:07:48.967 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:48.968 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:48.968 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:48.968 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:48.968 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.968 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:48.968 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.475737+0000", 2024-09-21T22:07:48.968 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:48.968 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.968 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.968 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:48.968 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.475737+0000", 2024-09-21T22:07:48.968 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:48.968 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.969 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.969 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:48.969 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.475739+0000", 2024-09-21T22:07:48.969 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.666e-06 2024-09-21T22:07:48.969 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.969 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.969 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:48.969 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.476538+0000", 2024-09-21T22:07:48.969 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00079927399999999999 2024-09-21T22:07:48.969 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.969 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.970 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:48.970 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.476556+0000", 2024-09-21T22:07:48.970 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7666999999999999e-05 2024-09-21T22:07:48.970 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.970 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.970 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:48.970 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.476565+0000", 2024-09-21T22:07:48.970 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.0650000000000005e-06 2024-09-21T22:07:48.970 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.970 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.970 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:48.970 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.476651+0000", 2024-09-21T22:07:48.970 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.5916999999999997e-05 2024-09-21T22:07:48.971 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.971 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.971 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:48.971 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.476696+0000", 2024-09-21T22:07:48.971 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.5537000000000001e-05 2024-09-21T22:07:48.971 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.971 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.971 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:48.971 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.513300+0000", 2024-09-21T22:07:48.971 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.036603498999999998 2024-09-21T22:07:48.971 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.971 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.971 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:48.971 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.513369+0000", 2024-09-21T22:07:48.972 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.9400000000000006e-05 2024-09-21T22:07:48.972 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:48.972 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:48.972 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:48.972 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.972 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.972 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:145 2.5 e72/37 2:a70ae99f:::rbd_data.126483d75f2d.000000000000000c:head v 72'52, mlcod=72'50)", 2024-09-21T22:07:48.972 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.578171+0000", 2024-09-21T22:07:48.972 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 10.339088794, 2024-09-21T22:07:48.972 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.037181537000000001, 2024-09-21T22:07:48.972 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:48.972 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:48.972 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:48.973 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:48.973 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.973 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:48.973 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.578171+0000", 2024-09-21T22:07:48.973 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:48.973 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.973 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.973 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:48.973 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.578171+0000", 2024-09-21T22:07:48.973 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:48.973 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.973 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.973 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:48.974 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.578172+0000", 2024-09-21T22:07:48.974 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.017e-06 2024-09-21T22:07:48.974 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.974 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.974 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:48.974 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.579000+0000", 2024-09-21T22:07:48.974 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00082799499999999999 2024-09-21T22:07:48.974 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.974 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.974 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:48.974 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.579020+0000", 2024-09-21T22:07:48.974 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.0165000000000001e-05 2024-09-21T22:07:48.974 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.974 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.974 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:48.975 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.579036+0000", 2024-09-21T22:07:48.975 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5767e-05 2024-09-21T22:07:48.975 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.975 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.975 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:48.975 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.579106+0000", 2024-09-21T22:07:48.975 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.0091000000000004e-05 2024-09-21T22:07:48.975 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.975 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.975 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:48.975 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.579131+0000", 2024-09-21T22:07:48.975 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4978e-05 2024-09-21T22:07:48.975 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.976 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.976 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:48.976 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.615249+0000", 2024-09-21T22:07:48.976 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.036118355999999997 2024-09-21T22:07:48.976 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.976 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.976 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:48.976 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.615353+0000", 2024-09-21T22:07:48.976 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000103168 2024-09-21T22:07:48.976 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:48.976 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:48.976 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:48.976 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.977 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.977 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:147 2.5 e72/37 2:a70ae99f:::rbd_data.126483d75f2d.000000000000000c:head v 72'54, mlcod=72'52)", 2024-09-21T22:07:48.977 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.627203+0000", 2024-09-21T22:07:48.977 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 10.290056681999999, 2024-09-21T22:07:48.977 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.038203972000000003, 2024-09-21T22:07:48.977 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:48.977 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:48.977 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:48.977 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:48.977 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.977 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:48.977 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.627203+0000", 2024-09-21T22:07:48.977 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:48.977 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.978 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.978 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:48.978 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.627203+0000", 2024-09-21T22:07:48.978 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:48.978 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.978 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.978 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:48.978 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.627204+0000", 2024-09-21T22:07:48.978 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3069999999999999e-06 2024-09-21T22:07:48.978 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.978 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.978 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:48.978 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.628114+0000", 2024-09-21T22:07:48.978 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00090942099999999999 2024-09-21T22:07:48.979 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.979 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.979 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:48.979 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.628140+0000", 2024-09-21T22:07:48.979 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5882000000000001e-05 2024-09-21T22:07:48.979 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.979 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.979 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:48.979 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.628162+0000", 2024-09-21T22:07:48.979 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.2293000000000002e-05 2024-09-21T22:07:48.979 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.979 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.979 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:48.979 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.628250+0000", 2024-09-21T22:07:48.980 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.7973000000000006e-05 2024-09-21T22:07:48.980 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.980 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.980 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:48.980 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.628272+0000", 2024-09-21T22:07:48.980 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1682999999999999e-05 2024-09-21T22:07:48.980 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.980 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.980 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:48.980 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.665322+0000", 2024-09-21T22:07:48.980 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.037050650999999997 2024-09-21T22:07:48.980 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.980 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.980 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:48.981 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.665407+0000", 2024-09-21T22:07:48.981 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.4761999999999999e-05 2024-09-21T22:07:48.981 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:48.981 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:48.981 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:48.981 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.981 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.981 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:149 2.5 e72/37 2:a70ae99f:::rbd_data.126483d75f2d.000000000000000c:head v 72'56, mlcod=72'54)", 2024-09-21T22:07:48.981 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.690377+0000", 2024-09-21T22:07:48.981 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 10.226882938999999, 2024-09-21T22:07:48.981 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.049254799000000002, 2024-09-21T22:07:48.981 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:48.981 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:48.981 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:48.982 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:48.982 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.982 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:48.982 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.690377+0000", 2024-09-21T22:07:48.982 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:48.982 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.982 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.982 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:48.982 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.690377+0000", 2024-09-21T22:07:48.982 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:48.982 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.982 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.982 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:48.982 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.690378+0000", 2024-09-21T22:07:48.983 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.344e-06 2024-09-21T22:07:48.983 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.983 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.983 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:48.983 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.691302+0000", 2024-09-21T22:07:48.983 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00092322400000000005 2024-09-21T22:07:48.983 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.983 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.983 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:48.983 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.691321+0000", 2024-09-21T22:07:48.983 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9928999999999998e-05 2024-09-21T22:07:48.983 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.983 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.983 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:48.983 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.691345+0000", 2024-09-21T22:07:48.984 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.372e-05 2024-09-21T22:07:48.984 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.984 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.984 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:48.984 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.691442+0000", 2024-09-21T22:07:48.984 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.7270999999999998e-05 2024-09-21T22:07:48.984 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.984 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.984 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:48.984 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.691512+0000", 2024-09-21T22:07:48.984 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.9183000000000005e-05 2024-09-21T22:07:48.984 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.984 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.984 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:48.985 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.739545+0000", 2024-09-21T22:07:48.985 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.048033657 2024-09-21T22:07:48.985 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.985 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.985 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:48.985 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.739632+0000", 2024-09-21T22:07:48.985 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.6471000000000006e-05 2024-09-21T22:07:48.985 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:48.985 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:48.985 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:48.985 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.985 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.985 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:156 2.1 e72/53 2:837ddd83:::rbd_data.126483d75f2d.000000000000000d:head v 72'28, mlcod=72'26)", 2024-09-21T22:07:48.985 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.824856+0000", 2024-09-21T22:07:48.986 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 10.092403501, 2024-09-21T22:07:48.986 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.034320985999999998, 2024-09-21T22:07:48.986 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:48.986 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:48.986 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:48.986 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:48.986 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.986 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:48.986 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.824856+0000", 2024-09-21T22:07:48.986 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:48.986 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.986 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.986 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:48.986 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.824856+0000", 2024-09-21T22:07:48.987 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:48.987 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.987 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.987 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:48.987 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.824858+0000", 2024-09-21T22:07:48.987 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2979999999999999e-06 2024-09-21T22:07:48.987 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.987 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.987 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:48.987 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.825174+0000", 2024-09-21T22:07:48.987 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000316335 2024-09-21T22:07:48.987 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.987 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.987 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:48.988 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.825193+0000", 2024-09-21T22:07:48.988 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9436000000000002e-05 2024-09-21T22:07:48.988 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.988 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.988 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:48.988 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.825202+0000", 2024-09-21T22:07:48.988 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.8359999999999998e-06 2024-09-21T22:07:48.988 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.988 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.988 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:48.988 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.825286+0000", 2024-09-21T22:07:48.988 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.3683999999999994e-05 2024-09-21T22:07:48.988 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.988 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.989 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:48.989 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.825314+0000", 2024-09-21T22:07:48.989 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.7653000000000001e-05 2024-09-21T22:07:48.989 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.989 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.989 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:48.989 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.859122+0000", 2024-09-21T22:07:48.989 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.033808624000000002 2024-09-21T22:07:48.989 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.989 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.989 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:48.989 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.859177+0000", 2024-09-21T22:07:48.989 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.5120000000000001e-05 2024-09-21T22:07:48.989 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:48.989 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:48.990 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:48.990 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.990 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.990 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:158 2.1 e72/53 2:837ddd83:::rbd_data.126483d75f2d.000000000000000d:head v 72'30, mlcod=72'28)", 2024-09-21T22:07:48.990 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.869277+0000", 2024-09-21T22:07:48.990 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 10.047983304000001, 2024-09-21T22:07:48.990 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.040508390999999998, 2024-09-21T22:07:48.990 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:48.990 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:48.990 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:48.990 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:48.990 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.990 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:48.991 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.869277+0000", 2024-09-21T22:07:48.991 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:48.991 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.991 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.991 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:48.991 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.869277+0000", 2024-09-21T22:07:48.991 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:48.991 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.991 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.991 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:48.991 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.869278+0000", 2024-09-21T22:07:48.991 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5269999999999999e-06 2024-09-21T22:07:48.991 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.991 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.991 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:48.992 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.869902+0000", 2024-09-21T22:07:48.992 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00062399000000000005 2024-09-21T22:07:48.992 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.992 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.992 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:48.992 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.869924+0000", 2024-09-21T22:07:48.992 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1875999999999999e-05 2024-09-21T22:07:48.992 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.992 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.992 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:48.992 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.869937+0000", 2024-09-21T22:07:48.992 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2685e-05 2024-09-21T22:07:48.992 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.992 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.993 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:48.993 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.870058+0000", 2024-09-21T22:07:48.993 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000120862 2024-09-21T22:07:48.993 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.993 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.993 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:48.993 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.870093+0000", 2024-09-21T22:07:48.993 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.5896000000000003e-05 2024-09-21T22:07:48.993 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.993 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.993 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:48.993 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.909728+0000", 2024-09-21T22:07:48.993 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.039634461000000003 2024-09-21T22:07:48.993 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.994 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.994 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:48.994 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.909785+0000", 2024-09-21T22:07:48.994 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.7093999999999999e-05 2024-09-21T22:07:48.994 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:48.994 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:48.994 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:48.994 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.994 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.994 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:161 2.1 e72/53 2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head v 72'32, mlcod=72'30)", 2024-09-21T22:07:48.994 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.924455+0000", 2024-09-21T22:07:48.994 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 9.9928049419999994, 2024-09-21T22:07:48.994 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.038366863000000001, 2024-09-21T22:07:48.994 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:48.995 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:48.995 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:48.995 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:48.995 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.995 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:48.995 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.924455+0000", 2024-09-21T22:07:48.995 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:48.995 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.995 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.995 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:48.995 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.924455+0000", 2024-09-21T22:07:48.995 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:48.995 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.995 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.995 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:48.996 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.924456+0000", 2024-09-21T22:07:48.996 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.003e-06 2024-09-21T22:07:48.996 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.996 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.996 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:48.996 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.924811+0000", 2024-09-21T22:07:48.996 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00035519499999999999 2024-09-21T22:07:48.996 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.996 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.996 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:48.996 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.924826+0000", 2024-09-21T22:07:48.996 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5296999999999999e-05 2024-09-21T22:07:48.996 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.996 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.997 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:48.997 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.924834+0000", 2024-09-21T22:07:48.997 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.5379999999999997e-06 2024-09-21T22:07:48.997 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.997 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.997 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:48.997 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.924910+0000", 2024-09-21T22:07:48.997 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.6488000000000004e-05 2024-09-21T22:07:48.997 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.997 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.997 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:48.997 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.924934+0000", 2024-09-21T22:07:48.997 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3598999999999999e-05 2024-09-21T22:07:48.997 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.997 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.998 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:48.998 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.962739+0000", 2024-09-21T22:07:48.998 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.037805370999999997 2024-09-21T22:07:48.998 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.998 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.998 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:48.998 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.962822+0000", 2024-09-21T22:07:48.998 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.2372000000000003e-05 2024-09-21T22:07:48.998 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:48.998 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:48.998 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:48.998 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.998 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.998 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:163 2.1 e72/53 2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head v 72'34, mlcod=72'32)", 2024-09-21T22:07:48.999 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.971683+0000", 2024-09-21T22:07:48.999 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 9.9455764020000004, 2024-09-21T22:07:48.999 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.043171320999999999, 2024-09-21T22:07:48.999 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:48.999 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:48.999 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:48.999 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:48.999 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.999 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:48.999 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.971683+0000", 2024-09-21T22:07:48.999 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:48.999 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:48.999 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:48.999 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:49.000 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.971683+0000", 2024-09-21T22:07:49.000 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:49.000 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:49.000 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:49.000 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:49.000 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.971684+0000", 2024-09-21T22:07:49.000 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.6600000000000005e-07 2024-09-21T22:07:49.000 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:49.000 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:49.000 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:49.000 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.971945+0000", 2024-09-21T22:07:49.000 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00026101899999999998 2024-09-21T22:07:49.000 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:49.000 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:49.001 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:49.001 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.971959+0000", 2024-09-21T22:07:49.001 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4054e-05 2024-09-21T22:07:49.001 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:49.001 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:49.001 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:49.001 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.971969+0000", 2024-09-21T22:07:49.001 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.499e-06 2024-09-21T22:07:49.001 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:49.001 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:49.001 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:49.001 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.972037+0000", 2024-09-21T22:07:49.001 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.8448000000000006e-05 2024-09-21T22:07:49.001 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:49.001 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:49.002 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:49.002 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.972057+0000", 2024-09-21T22:07:49.002 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9817999999999999e-05 2024-09-21T22:07:49.002 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:49.002 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:49.002 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:49.002 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.014770+0000", 2024-09-21T22:07:49.002 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.042712464999999998 2024-09-21T22:07:49.002 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:49.002 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:49.002 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:49.002 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.014855+0000", 2024-09-21T22:07:49.002 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.5152000000000001e-05 2024-09-21T22:07:49.002 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:49.003 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:49.003 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:49.003 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:49.003 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:49.003 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:165 2.1 e72/53 2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head v 72'36, mlcod=72'34)", 2024-09-21T22:07:49.003 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:39.028631+0000", 2024-09-21T22:07:49.003 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 9.8886289250000008, 2024-09-21T22:07:49.003 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.033430561999999997, 2024-09-21T22:07:49.003 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:49.003 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:49.003 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:49.004 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:49.004 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:49.004 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:49.004 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.028631+0000", 2024-09-21T22:07:49.004 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:49.004 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:49.004 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:49.004 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:49.004 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.028631+0000", 2024-09-21T22:07:49.004 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:49.004 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:49.004 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:49.005 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:49.005 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.028633+0000", 2024-09-21T22:07:49.005 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5799999999999999e-06 2024-09-21T22:07:49.005 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:49.005 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:49.005 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:49.005 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.029209+0000", 2024-09-21T22:07:49.005 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00057670799999999999 2024-09-21T22:07:49.005 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:49.005 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:49.005 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:49.005 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.029238+0000", 2024-09-21T22:07:49.005 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8943000000000001e-05 2024-09-21T22:07:49.005 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:49.006 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:49.006 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:49.006 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.029252+0000", 2024-09-21T22:07:49.006 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3672999999999999e-05 2024-09-21T22:07:49.006 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:49.006 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:49.006 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:49.006 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.029328+0000", 2024-09-21T22:07:49.006 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.6497999999999998e-05 2024-09-21T22:07:49.006 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:49.006 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:49.006 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:49.006 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.029353+0000", 2024-09-21T22:07:49.006 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5060999999999999e-05 2024-09-21T22:07:49.007 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:49.007 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:49.007 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:49.007 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.061993+0000", 2024-09-21T22:07:49.007 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.032639286000000003 2024-09-21T22:07:49.007 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:49.007 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:49.007 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:49.007 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.062062+0000", 2024-09-21T22:07:49.007 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.8813000000000007e-05 2024-09-21T22:07:49.007 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:49.007 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:49.007 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:49.008 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:49.008 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:49.008 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:167 2.1 e72/53 2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head v 72'38, mlcod=72'36)", 2024-09-21T22:07:49.008 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:39.077324+0000", 2024-09-21T22:07:49.008 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 9.8399361110000001, 2024-09-21T22:07:49.008 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.036031961000000001, 2024-09-21T22:07:49.008 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:49.008 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:49.008 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:49.008 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:49.008 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:49.008 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:49.008 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.077324+0000", 2024-09-21T22:07:49.008 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:49.008 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:49.009 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:49.009 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:49.009 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.077324+0000", 2024-09-21T22:07:49.009 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:49.009 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:49.009 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:49.009 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:49.009 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.077326+0000", 2024-09-21T22:07:49.009 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.75e-06 2024-09-21T22:07:49.009 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:49.009 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:49.009 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:49.009 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.077719+0000", 2024-09-21T22:07:49.009 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00039322599999999999 2024-09-21T22:07:49.010 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:49.010 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:49.010 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:49.010 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.077749+0000", 2024-09-21T22:07:49.010 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9895000000000001e-05 2024-09-21T22:07:49.010 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:49.010 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:49.010 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:49.010 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.077763+0000", 2024-09-21T22:07:49.010 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4517e-05 2024-09-21T22:07:49.010 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:49.010 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:49.010 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:49.010 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.077843+0000", 2024-09-21T22:07:49.010 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.9994999999999996e-05 2024-09-21T22:07:49.011 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:49.011 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:49.011 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:49.011 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.077867+0000", 2024-09-21T22:07:49.011 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3640000000000001e-05 2024-09-21T22:07:49.011 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:49.011 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:49.011 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:49.011 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.113216+0000", 2024-09-21T22:07:49.011 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.035349168 2024-09-21T22:07:49.011 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:49.011 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:49.012 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:49.012 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.113356+0000", 2024-09-21T22:07:49.012 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013977 2024-09-21T22:07:49.012 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:49.012 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:49.012 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:49.012 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:49.012 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:49.012 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:07:49.013 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T22:07:49.014 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:49.008+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:49.014 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T22:07:49.108 INFO:tasks.workunit.client.0.smithi157.stderr:+ DEV= 2024-09-21T22:07:49.108 INFO:tasks.workunit.client.0.smithi157.stderr:+ unmap_device rbd/testrbdnbd21553 23250 2024-09-21T22:07:49.108 INFO:tasks.workunit.client.0.smithi157.stderr:+ local args=rbd/testrbdnbd21553 2024-09-21T22:07:49.108 INFO:tasks.workunit.client.0.smithi157.stderr:+ local pid=23250 2024-09-21T22:07:49.109 INFO:tasks.workunit.client.0.smithi157.stderr:+ _sudo rbd device --device-type nbd unmap rbd/testrbdnbd21553 2024-09-21T22:07:49.109 INFO:tasks.workunit.client.0.smithi157.stderr:+ local cmd 2024-09-21T22:07:49.109 INFO:tasks.workunit.client.0.smithi157.stderr:++ id -u 2024-09-21T22:07:49.110 INFO:tasks.workunit.client.0.smithi157.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T22:07:49.111 INFO:tasks.workunit.client.0.smithi157.stderr:++ which rbd 2024-09-21T22:07:49.113 INFO:tasks.workunit.client.0.smithi157.stderr:+ cmd=/usr/bin/rbd 2024-09-21T22:07:49.113 INFO:tasks.workunit.client.0.smithi157.stderr:+ shift 2024-09-21T22:07:49.113 INFO:tasks.workunit.client.0.smithi157.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap rbd/testrbdnbd21553 2024-09-21T22:07:49.114 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:49.110+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:49.177 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:07:49.177 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:07:49.177 INFO:teuthology.orchestra.run.smithi039.stdout: "num_ops": 0 2024-09-21T22:07:49.178 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:07:49.192 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T22:07:49.194 INFO:teuthology.orchestra.run.smithi039.stdout:osd.0: {} 2024-09-21T22:07:49.194 INFO:teuthology.orchestra.run.smithi039.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T22:07:49.207 INFO:tasks.workunit.client.0.smithi157.stderr:+ rbd device --device-type nbd list 2024-09-21T22:07:49.207 INFO:tasks.workunit.client.0.smithi157.stderr:+ expect_false grep '^23250\b' 2024-09-21T22:07:49.207 INFO:tasks.workunit.client.0.smithi157.stderr:+ grep '^23250\b' 2024-09-21T22:07:49.211 INFO:teuthology.orchestra.run.smithi039.stdout:osd.1: {} 2024-09-21T22:07:49.212 INFO:teuthology.orchestra.run.smithi039.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T22:07:49.215 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:49.212+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:49.229 INFO:teuthology.orchestra.run.smithi039.stdout:osd.2: {} 2024-09-21T22:07:49.230 INFO:teuthology.orchestra.run.smithi039.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T22:07:49.258 INFO:teuthology.orchestra.run.smithi039.stdout:osd.3: {} 2024-09-21T22:07:49.258 INFO:teuthology.orchestra.run.smithi039.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T22:07:49.263 INFO:tasks.workunit.client.0.smithi157.stderr:+ return 0 2024-09-21T22:07:49.263 INFO:tasks.workunit.client.0.smithi157.stderr:+ ps -C rbd-nbd 2024-09-21T22:07:49.263 INFO:tasks.workunit.client.0.smithi157.stderr:+ expect_false grep '^ *23250\b' 2024-09-21T22:07:49.263 INFO:tasks.workunit.client.0.smithi157.stderr:+ grep '^ *23250\b' 2024-09-21T22:07:49.268 INFO:tasks.workunit.client.0.smithi157.stderr:+ return 0 2024-09-21T22:07:49.268 INFO:tasks.workunit.client.0.smithi157.stderr:+ sleep 0.5 2024-09-21T22:07:49.270 INFO:teuthology.orchestra.run.smithi039.stdout:osd.4: {} 2024-09-21T22:07:49.271 INFO:teuthology.orchestra.run.smithi039.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T22:07:49.292 INFO:teuthology.orchestra.run.smithi039.stdout:osd.5: {} 2024-09-21T22:07:49.293 INFO:teuthology.orchestra.run.smithi039.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T22:07:49.312 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:49.312 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:49.315 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:49.312+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:49.326 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T22:07:49.425 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:49.422+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:49.499 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:49.499 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:49.513 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T22:07:49.563 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:49.558+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:49.622 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:49.620+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:49.692 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:49.692 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:49.704 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T22:07:49.754 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:49.750+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:49.773 INFO:tasks.workunit.client.0.smithi157.stderr:+ '[' rbd = rbd ']' 2024-09-21T22:07:49.773 INFO:tasks.workunit.client.0.smithi157.stderr:++ _sudo rbd device --device-type nbd map testrbdnbd21553 2024-09-21T22:07:49.773 INFO:tasks.workunit.client.0.smithi157.stderr:++ local cmd 2024-09-21T22:07:49.773 INFO:tasks.workunit.client.0.smithi157.stderr:+++ id -u 2024-09-21T22:07:49.773 INFO:tasks.workunit.client.0.smithi157.stderr:++ '[' 1000 -eq 0 ']' 2024-09-21T22:07:49.773 INFO:tasks.workunit.client.0.smithi157.stderr:+++ which rbd 2024-09-21T22:07:49.774 INFO:tasks.workunit.client.0.smithi157.stderr:++ cmd=/usr/bin/rbd 2024-09-21T22:07:49.774 INFO:tasks.workunit.client.0.smithi157.stderr:++ shift 2024-09-21T22:07:49.774 INFO:tasks.workunit.client.0.smithi157.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map testrbdnbd21553 2024-09-21T22:07:49.824 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:49.816+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:49.827 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:49.827 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:49.827 INFO:teuthology.orchestra.run.smithi039.stderr:noscrub is set 2024-09-21T22:07:49.845 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T22:07:49.921 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:07:49.916+0000 7f2075a06640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31720) UID: 0 2024-09-21T22:07:49.935 INFO:tasks.workunit.client.0.smithi157.stderr:+ DEV=/dev/nbd0 2024-09-21T22:07:49.935 INFO:tasks.workunit.client.0.smithi157.stderr:+ get_pid rbd 2024-09-21T22:07:49.935 INFO:tasks.workunit.client.0.smithi157.stderr:+ local pool=rbd 2024-09-21T22:07:49.935 INFO:tasks.workunit.client.0.smithi157.stderr:+ local ns= 2024-09-21T22:07:49.936 INFO:tasks.workunit.client.0.smithi157.stderr:++ rbd device --device-type nbd --format xml list 2024-09-21T22:07:49.936 INFO:tasks.workunit.client.0.smithi157.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd21553'\''][device='\''/dev/nbd0'\'']/id' 2024-09-21T22:07:50.009 INFO:tasks.workunit.client.0.smithi157.stderr:+ PID=23390 2024-09-21T22:07:50.009 INFO:tasks.workunit.client.0.smithi157.stderr:+ test -n 23390 2024-09-21T22:07:50.009 INFO:tasks.workunit.client.0.smithi157.stderr:+ ps -p 23390 -C rbd-nbd 2024-09-21T22:07:50.015 INFO:tasks.workunit.client.0.smithi157.stdout: PID TTY TIME CMD 2024-09-21T22:07:50.015 INFO:tasks.workunit.client.0.smithi157.stdout: 23390 ? 00:00:00 rbd-nbd 2024-09-21T22:07:50.015 INFO:tasks.workunit.client.0.smithi157.stderr:+ unmap_device testrbdnbd21553 23390 2024-09-21T22:07:50.015 INFO:tasks.workunit.client.0.smithi157.stderr:+ local args=testrbdnbd21553 2024-09-21T22:07:50.015 INFO:tasks.workunit.client.0.smithi157.stderr:+ local pid=23390 2024-09-21T22:07:50.015 INFO:tasks.workunit.client.0.smithi157.stderr:+ _sudo rbd device --device-type nbd unmap testrbdnbd21553 2024-09-21T22:07:50.016 INFO:tasks.workunit.client.0.smithi157.stderr:+ local cmd 2024-09-21T22:07:50.016 INFO:tasks.workunit.client.0.smithi157.stderr:++ id -u 2024-09-21T22:07:50.017 INFO:tasks.workunit.client.0.smithi157.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T22:07:50.018 INFO:tasks.workunit.client.0.smithi157.stderr:++ which rbd 2024-09-21T22:07:50.019 INFO:tasks.workunit.client.0.smithi157.stderr:+ cmd=/usr/bin/rbd 2024-09-21T22:07:50.019 INFO:tasks.workunit.client.0.smithi157.stderr:+ shift 2024-09-21T22:07:50.019 INFO:tasks.workunit.client.0.smithi157.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap testrbdnbd21553 2024-09-21T22:07:50.024 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:50.018+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:50.058 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:50.058 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:50.080 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T22:07:50.108 INFO:tasks.workunit.client.0.smithi157.stderr:+ rbd device --device-type nbd list 2024-09-21T22:07:50.109 INFO:tasks.workunit.client.0.smithi157.stderr:+ expect_false grep '^23390\b' 2024-09-21T22:07:50.109 INFO:tasks.workunit.client.0.smithi157.stderr:+ grep '^23390\b' 2024-09-21T22:07:50.129 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:50.126+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:50.156 INFO:tasks.workunit.client.0.smithi157.stderr:+ return 0 2024-09-21T22:07:50.157 INFO:tasks.workunit.client.0.smithi157.stderr:+ ps -C rbd-nbd 2024-09-21T22:07:50.157 INFO:tasks.workunit.client.0.smithi157.stderr:+ expect_false grep '^ *23390\b' 2024-09-21T22:07:50.157 INFO:tasks.workunit.client.0.smithi157.stderr:+ grep '^ *23390\b' 2024-09-21T22:07:50.162 INFO:tasks.workunit.client.0.smithi157.stderr:+ return 0 2024-09-21T22:07:50.162 INFO:tasks.workunit.client.0.smithi157.stderr:+ sleep 0.5 2024-09-21T22:07:50.214 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:50.214 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:50.224 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:50.220+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:50.229 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T22:07:50.326 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:50.324+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:50.398 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:50.398 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:50.413 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T22:07:50.428 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:50.426+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:50.527 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:50.522+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:50.594 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:50.595 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:50.609 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T22:07:50.654 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:50.652+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:50.663 INFO:tasks.workunit.client.0.smithi157.stderr:+ DEV= 2024-09-21T22:07:50.663 INFO:tasks.workunit.client.0.smithi157.stderr:+ rbd config global set global rbd_default_pool new_default_pool21553 2024-09-21T22:07:50.721 INFO:tasks.workunit.client.0.smithi157.stderr:+ rbd create --size 10M testrbdnbd21553 2024-09-21T22:07:50.729 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:07:50.724+0000 7f2075a06640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31720) UID: 0 2024-09-21T22:07:50.738 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:50.738 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:50.753 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T22:07:50.778 INFO:tasks.workunit.client.0.smithi157.stderr:++ _sudo rbd device --device-type nbd map testrbdnbd21553 2024-09-21T22:07:50.778 INFO:tasks.workunit.client.0.smithi157.stderr:++ local cmd 2024-09-21T22:07:50.778 INFO:tasks.workunit.client.0.smithi157.stderr:+++ id -u 2024-09-21T22:07:50.779 INFO:tasks.workunit.client.0.smithi157.stderr:++ '[' 1000 -eq 0 ']' 2024-09-21T22:07:50.779 INFO:tasks.workunit.client.0.smithi157.stderr:+++ which rbd 2024-09-21T22:07:50.780 INFO:tasks.workunit.client.0.smithi157.stderr:++ cmd=/usr/bin/rbd 2024-09-21T22:07:50.781 INFO:tasks.workunit.client.0.smithi157.stderr:++ shift 2024-09-21T22:07:50.781 INFO:tasks.workunit.client.0.smithi157.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map testrbdnbd21553 2024-09-21T22:07:50.846 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:07:50.844+0000 7f2075a06640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31720) UID: 0 2024-09-21T22:07:50.928 INFO:tasks.workunit.client.0.smithi157.stderr:+ DEV=/dev/nbd0 2024-09-21T22:07:50.928 INFO:tasks.workunit.client.0.smithi157.stderr:+ get_pid new_default_pool21553 2024-09-21T22:07:50.928 INFO:tasks.workunit.client.0.smithi157.stderr:+ local pool=new_default_pool21553 2024-09-21T22:07:50.928 INFO:tasks.workunit.client.0.smithi157.stderr:+ local ns= 2024-09-21T22:07:50.928 INFO:tasks.workunit.client.0.smithi157.stderr:++ rbd device --device-type nbd --format xml list 2024-09-21T22:07:50.929 INFO:tasks.workunit.client.0.smithi157.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''new_default_pool21553'\''][namespace='\'''\''][image='\''testrbdnbd21553'\''][device='\''/dev/nbd0'\'']/id' 2024-09-21T22:07:50.931 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:50.931 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:50.931 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:07:50.928+0000 7f2075a06640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31720) UID: 0 2024-09-21T22:07:50.945 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T22:07:50.999 INFO:tasks.workunit.client.0.smithi157.stderr:+ PID=23516 2024-09-21T22:07:50.999 INFO:tasks.workunit.client.0.smithi157.stderr:+ test -n 23516 2024-09-21T22:07:50.999 INFO:tasks.workunit.client.0.smithi157.stderr:+ ps -p 23516 -C rbd-nbd 2024-09-21T22:07:51.007 INFO:tasks.workunit.client.0.smithi157.stdout: PID TTY TIME CMD 2024-09-21T22:07:51.007 INFO:tasks.workunit.client.0.smithi157.stdout: 23516 ? 00:00:00 rbd-nbd 2024-09-21T22:07:51.008 INFO:tasks.workunit.client.0.smithi157.stderr:+ unmap_device testrbdnbd21553 23516 2024-09-21T22:07:51.008 INFO:tasks.workunit.client.0.smithi157.stderr:+ local args=testrbdnbd21553 2024-09-21T22:07:51.008 INFO:tasks.workunit.client.0.smithi157.stderr:+ local pid=23516 2024-09-21T22:07:51.008 INFO:tasks.workunit.client.0.smithi157.stderr:+ _sudo rbd device --device-type nbd unmap testrbdnbd21553 2024-09-21T22:07:51.008 INFO:tasks.workunit.client.0.smithi157.stderr:+ local cmd 2024-09-21T22:07:51.009 INFO:tasks.workunit.client.0.smithi157.stderr:++ id -u 2024-09-21T22:07:51.009 INFO:tasks.workunit.client.0.smithi157.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T22:07:51.010 INFO:tasks.workunit.client.0.smithi157.stderr:++ which rbd 2024-09-21T22:07:51.011 INFO:tasks.workunit.client.0.smithi157.stderr:+ cmd=/usr/bin/rbd 2024-09-21T22:07:51.011 INFO:tasks.workunit.client.0.smithi157.stderr:+ shift 2024-09-21T22:07:51.011 INFO:tasks.workunit.client.0.smithi157.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap testrbdnbd21553 2024-09-21T22:07:51.032 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:51.030+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:51.098 INFO:tasks.workunit.client.0.smithi157.stderr:+ rbd device --device-type nbd list 2024-09-21T22:07:51.098 INFO:tasks.workunit.client.0.smithi157.stderr:+ expect_false grep '^23516\b' 2024-09-21T22:07:51.099 INFO:tasks.workunit.client.0.smithi157.stderr:+ grep '^23516\b' 2024-09-21T22:07:51.111 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:51.112 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:51.126 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T22:07:51.134 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:51.130+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:51.154 INFO:tasks.workunit.client.0.smithi157.stderr:+ return 0 2024-09-21T22:07:51.155 INFO:tasks.workunit.client.0.smithi157.stderr:+ ps -C rbd-nbd 2024-09-21T22:07:51.155 INFO:tasks.workunit.client.0.smithi157.stderr:+ expect_false grep '^ *23516\b' 2024-09-21T22:07:51.155 INFO:tasks.workunit.client.0.smithi157.stderr:+ grep '^ *23516\b' 2024-09-21T22:07:51.161 INFO:tasks.workunit.client.0.smithi157.stderr:+ return 0 2024-09-21T22:07:51.161 INFO:tasks.workunit.client.0.smithi157.stderr:+ sleep 0.5 2024-09-21T22:07:51.234 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:51.232+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:51.245 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:51.245 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:51.259 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T22:07:51.311 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T22:07:51.346 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:51.344+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:51.427 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:51.428 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:51.435 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:51.432+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:51.442 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T22:07:51.471 INFO:teuthology.orchestra.run.smithi039.stdout:osd.0: {} 2024-09-21T22:07:51.471 INFO:teuthology.orchestra.run.smithi039.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T22:07:51.488 INFO:teuthology.orchestra.run.smithi039.stdout:osd.1: {} 2024-09-21T22:07:51.488 INFO:teuthology.orchestra.run.smithi039.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T22:07:51.507 INFO:teuthology.orchestra.run.smithi039.stdout:osd.2: {} 2024-09-21T22:07:51.507 INFO:teuthology.orchestra.run.smithi039.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T22:07:51.526 INFO:teuthology.orchestra.run.smithi039.stdout:osd.3: {} 2024-09-21T22:07:51.526 INFO:teuthology.orchestra.run.smithi039.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T22:07:51.536 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:51.534+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:51.544 INFO:teuthology.orchestra.run.smithi039.stdout:osd.4: {} 2024-09-21T22:07:51.544 INFO:teuthology.orchestra.run.smithi039.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T22:07:51.563 INFO:teuthology.orchestra.run.smithi039.stdout:osd.5: {} 2024-09-21T22:07:51.563 INFO:teuthology.orchestra.run.smithi039.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T22:07:51.615 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:07:51.615 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-21T22:07:51.615 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-21T22:07:51.615 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-21T22:07:51.615 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.615 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5031.0:3 2.2 2:53d5e6d0:::rbd_header.126483d75f2d:head [call rbd.get_size in=11b,call,call rbd.get_features in=15b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e65)", 2024-09-21T22:07:51.616 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:27.373436+0000", 2024-09-21T22:07:51.616 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 24.239678832999999, 2024-09-21T22:07:51.616 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.86524013700000002, 2024-09-21T22:07:51.616 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:51.616 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:51.616 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:07:51.616 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:51.616 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5031", 2024-09-21T22:07:51.616 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/3232705659", 2024-09-21T22:07:51.616 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3 2024-09-21T22:07:51.616 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.616 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:51.616 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.617 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:51.617 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:27.373436+0000", 2024-09-21T22:07:51.617 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:51.617 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.617 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.617 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:51.617 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:27.373436+0000", 2024-09-21T22:07:51.617 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:51.617 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.617 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.617 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:51.617 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:27.373439+0000", 2024-09-21T22:07:51.618 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5969999999999999e-06 2024-09-21T22:07:51.618 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.618 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.618 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:51.618 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:27.373450+0000", 2024-09-21T22:07:51.618 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1326e-05 2024-09-21T22:07:51.618 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.618 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.618 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:51.618 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:27.373471+0000", 2024-09-21T22:07:51.618 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0169999999999998e-05 2024-09-21T22:07:51.618 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.618 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.618 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:51.619 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:27.373486+0000", 2024-09-21T22:07:51.619 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5483000000000001e-05 2024-09-21T22:07:51.619 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.619 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.619 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:51.619 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:27.373602+0000", 2024-09-21T22:07:51.619 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000115603 2024-09-21T22:07:51.619 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.619 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.619 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-21T22:07:51.619 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:27.373617+0000", 2024-09-21T22:07:51.619 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4956999999999999e-05 2024-09-21T22:07:51.619 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.619 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.620 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:51.620 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.234578+0000", 2024-09-21T22:07:51.620 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.86096097900000002 2024-09-21T22:07:51.620 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.620 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.620 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for missing object", 2024-09-21T22:07:51.620 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.234771+0000", 2024-09-21T22:07:51.620 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019394700000000001 2024-09-21T22:07:51.620 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.620 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.620 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:51.620 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.237470+0000", 2024-09-21T22:07:51.620 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026981319999999998 2024-09-21T22:07:51.620 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.620 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.621 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:51.621 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.237791+0000", 2024-09-21T22:07:51.621 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00032138499999999999 2024-09-21T22:07:51.621 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.621 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.621 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:51.621 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.238677+0000", 2024-09-21T22:07:51.621 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00088555800000000005 2024-09-21T22:07:51.621 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:51.621 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:51.621 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:51.621 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.621 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.621 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.2 66/65 [PushOp(2:4481fad7:ns::rbd_data.127f79bc213d.0000000000000006:head, version: 57'9, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:4481fad7:ns::rbd_data.127f79bc213d.0000000000000006:head@57'9, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T22:07:51.622 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:28.247056+0000", 2024-09-21T22:07:51.622 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 23.36605917, 2024-09-21T22:07:51.622 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.045025012000000003, 2024-09-21T22:07:51.622 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:51.622 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:51.622 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:07:51.622 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:51.622 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.622 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:51.622 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.247056+0000", 2024-09-21T22:07:51.622 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:51.622 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.622 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.623 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:51.623 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.247056+0000", 2024-09-21T22:07:51.623 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:51.623 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.623 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.623 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:51.623 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.247057+0000", 2024-09-21T22:07:51.623 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2500000000000001e-06 2024-09-21T22:07:51.623 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.623 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.623 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:51.623 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.249384+0000", 2024-09-21T22:07:51.623 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0023265619999999999 2024-09-21T22:07:51.624 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.624 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.624 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:51.624 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.249429+0000", 2024-09-21T22:07:51.624 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5392999999999997e-05 2024-09-21T22:07:51.624 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.624 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.624 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:51.624 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.249476+0000", 2024-09-21T22:07:51.624 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6570000000000003e-05 2024-09-21T22:07:51.624 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.624 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.624 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:51.624 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.249611+0000", 2024-09-21T22:07:51.624 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000135143 2024-09-21T22:07:51.625 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.625 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.625 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:51.625 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.249626+0000", 2024-09-21T22:07:51.625 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.469e-05 2024-09-21T22:07:51.625 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.625 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.625 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:51.625 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.292081+0000", 2024-09-21T22:07:51.625 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.042455404000000002 2024-09-21T22:07:51.625 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:51.625 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:51.625 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:51.625 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.625 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.625 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5031.0:9 2.2 2:53d5e6d0:::rbd_header.126483d75f2d:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-21T22:07:51.626 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:28.250163+0000", 2024-09-21T22:07:51.626 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 23.362951775999999, 2024-09-21T22:07:51.626 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.042986430999999999, 2024-09-21T22:07:51.626 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:51.626 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:51.626 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:07:51.626 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:51.626 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5031", 2024-09-21T22:07:51.626 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/3232705659", 2024-09-21T22:07:51.626 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 9 2024-09-21T22:07:51.626 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.626 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:51.626 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.626 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:51.627 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.250163+0000", 2024-09-21T22:07:51.627 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:51.627 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.627 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.627 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:51.627 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.250163+0000", 2024-09-21T22:07:51.627 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:51.627 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.627 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.627 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:51.627 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.250166+0000", 2024-09-21T22:07:51.627 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7039999999999999e-06 2024-09-21T22:07:51.627 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.627 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.627 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:51.628 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.250175+0000", 2024-09-21T22:07:51.628 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7560000000000005e-06 2024-09-21T22:07:51.628 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.628 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.628 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:51.628 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.250199+0000", 2024-09-21T22:07:51.628 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4049999999999998e-05 2024-09-21T22:07:51.628 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.628 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.628 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:51.628 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.250218+0000", 2024-09-21T22:07:51.628 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9236e-05 2024-09-21T22:07:51.628 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.628 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.628 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:51.629 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.292193+0000", 2024-09-21T22:07:51.629 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.041974694 2024-09-21T22:07:51.629 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.629 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.629 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:51.629 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.292492+0000", 2024-09-21T22:07:51.629 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00029900100000000002 2024-09-21T22:07:51.629 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.629 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.629 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:51.629 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.293150+0000", 2024-09-21T22:07:51.629 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00065799000000000001 2024-09-21T22:07:51.629 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:51.629 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:51.629 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:51.629 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.630 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.630 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5031.0:12 2.2 2:53d5e6d0:::rbd_header.126483d75f2d:head [watch watch cookie 140495695137040] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-09-21T22:07:51.630 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:28.295580+0000", 2024-09-21T22:07:51.630 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 23.317534889000001, 2024-09-21T22:07:51.630 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.033090870000000001, 2024-09-21T22:07:51.630 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:51.630 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:51.630 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:51.630 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:51.630 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5031", 2024-09-21T22:07:51.630 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/3232705659", 2024-09-21T22:07:51.630 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 12 2024-09-21T22:07:51.630 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.630 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:51.630 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.631 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:51.631 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.295580+0000", 2024-09-21T22:07:51.631 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:51.631 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.631 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.631 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:51.631 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.295580+0000", 2024-09-21T22:07:51.631 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:51.631 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.631 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.631 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:51.631 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.295581+0000", 2024-09-21T22:07:51.631 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6799999999999999e-07 2024-09-21T22:07:51.631 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.631 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.632 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:51.632 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.295586+0000", 2024-09-21T22:07:51.632 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8659999999999997e-06 2024-09-21T22:07:51.632 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.632 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.632 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:51.632 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.295599+0000", 2024-09-21T22:07:51.632 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2897999999999999e-05 2024-09-21T22:07:51.632 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.632 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.632 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:51.632 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.295614+0000", 2024-09-21T22:07:51.632 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5406000000000001e-05 2024-09-21T22:07:51.632 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.632 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.632 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:51.633 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.295665+0000", 2024-09-21T22:07:51.633 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0501000000000003e-05 2024-09-21T22:07:51.633 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.633 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.633 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:51.633 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.295727+0000", 2024-09-21T22:07:51.633 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2073e-05 2024-09-21T22:07:51.633 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.633 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.633 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-09-21T22:07:51.633 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.295849+0000", 2024-09-21T22:07:51.633 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012190500000000001 2024-09-21T22:07:51.633 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.633 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.633 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:51.634 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.296429+0000", 2024-09-21T22:07:51.634 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00058019699999999996 2024-09-21T22:07:51.634 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.634 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.634 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:51.634 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.328308+0000", 2024-09-21T22:07:51.634 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.031878849000000001 2024-09-21T22:07:51.634 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.634 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.634 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:51.634 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.328421+0000", 2024-09-21T22:07:51.634 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000113322 2024-09-21T22:07:51.634 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.634 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.634 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:51.635 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.328671+0000", 2024-09-21T22:07:51.635 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024998499999999999 2024-09-21T22:07:51.635 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:51.635 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:51.635 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:51.635 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.635 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.635 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.2 66/65 [PushOp(2:4481fad7:ns::rbd_data.127f79bc213d.0000000000000006:head, version: 57'9, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:4481fad7:ns::rbd_data.127f79bc213d.0000000000000006:head@57'9, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:07:51.635 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:28.296301+0000", 2024-09-21T22:07:51.635 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 23.316813968000002, 2024-09-21T22:07:51.635 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.031905291000000002, 2024-09-21T22:07:51.635 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:51.635 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:51.635 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:07:51.635 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:51.636 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.636 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:51.636 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.296301+0000", 2024-09-21T22:07:51.636 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:51.636 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.636 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.636 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:51.636 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.296301+0000", 2024-09-21T22:07:51.636 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:51.636 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.636 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.636 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:51.636 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.296315+0000", 2024-09-21T22:07:51.636 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3867e-05 2024-09-21T22:07:51.636 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.636 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.637 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:51.637 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.297432+0000", 2024-09-21T22:07:51.637 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011164759999999999 2024-09-21T22:07:51.637 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.637 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.637 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:51.637 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.297454+0000", 2024-09-21T22:07:51.637 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2813e-05 2024-09-21T22:07:51.637 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.637 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.637 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:51.637 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.297479+0000", 2024-09-21T22:07:51.637 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4817e-05 2024-09-21T22:07:51.637 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.637 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.637 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:51.638 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.297582+0000", 2024-09-21T22:07:51.638 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010261500000000001 2024-09-21T22:07:51.638 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.638 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.638 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:51.638 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.297600+0000", 2024-09-21T22:07:51.638 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8323000000000001e-05 2024-09-21T22:07:51.638 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.638 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.638 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:51.638 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.328207+0000", 2024-09-21T22:07:51.639 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.030606379999999999 2024-09-21T22:07:51.639 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:51.639 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:51.639 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:51.639 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.639 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.639 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.5031.0:12 2.2 e66/65 ondisk, result = 0)", 2024-09-21T22:07:51.639 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:28.297512+0000", 2024-09-21T22:07:51.639 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 23.315603395, 2024-09-21T22:07:51.639 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.031170693999999999, 2024-09-21T22:07:51.639 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:51.639 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:51.639 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:07:51.639 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:51.639 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.640 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:51.640 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.297512+0000", 2024-09-21T22:07:51.640 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:51.640 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.640 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.640 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:51.640 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.297512+0000", 2024-09-21T22:07:51.640 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:51.640 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.640 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.640 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:51.640 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.297512+0000", 2024-09-21T22:07:51.640 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7700000000000005e-07 2024-09-21T22:07:51.640 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.640 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.641 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:51.641 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.297517+0000", 2024-09-21T22:07:51.641 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1039999999999998e-06 2024-09-21T22:07:51.641 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.641 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.641 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:51.641 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.297528+0000", 2024-09-21T22:07:51.641 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0633e-05 2024-09-21T22:07:51.641 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.641 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.641 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:51.641 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.297537+0000", 2024-09-21T22:07:51.641 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7199999999999995e-06 2024-09-21T22:07:51.641 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.641 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.641 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:51.642 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.328273+0000", 2024-09-21T22:07:51.642 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.030736574999999999 2024-09-21T22:07:51.642 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.642 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.642 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:51.642 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.328293+0000", 2024-09-21T22:07:51.642 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9304e-05 2024-09-21T22:07:51.642 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.642 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.642 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:51.642 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.328683+0000", 2024-09-21T22:07:51.642 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00038988100000000002 2024-09-21T22:07:51.643 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:51.643 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:51.643 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:51.643 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.643 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.643 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.2 66/65 [PushOp(2:4481fad7:ns::rbd_data.127f79bc213d.0000000000000006:head, version: 57'9, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:4481fad7:ns::rbd_data.127f79bc213d.0000000000000006:head@57'9, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:07:51.643 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:28.331166+0000", 2024-09-21T22:07:51.643 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 23.281949025999999, 2024-09-21T22:07:51.643 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.039995378999999998, 2024-09-21T22:07:51.643 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:51.643 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:51.643 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:07:51.643 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:51.643 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.643 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:51.644 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.331166+0000", 2024-09-21T22:07:51.644 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:51.644 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.644 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.644 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:51.644 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.331166+0000", 2024-09-21T22:07:51.644 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:51.644 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.644 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.644 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:51.644 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.331167+0000", 2024-09-21T22:07:51.644 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9199999999999999e-07 2024-09-21T22:07:51.644 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.644 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.644 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:51.645 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.332127+0000", 2024-09-21T22:07:51.645 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00095969800000000004 2024-09-21T22:07:51.645 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.645 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.645 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:51.645 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.332149+0000", 2024-09-21T22:07:51.645 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1974999999999999e-05 2024-09-21T22:07:51.645 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.645 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.645 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:51.645 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.332161+0000", 2024-09-21T22:07:51.645 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2276000000000001e-05 2024-09-21T22:07:51.645 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.645 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.645 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:51.646 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.332290+0000", 2024-09-21T22:07:51.646 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000129214 2024-09-21T22:07:51.646 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.646 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.646 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:51.646 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.332312+0000", 2024-09-21T22:07:51.646 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1308999999999999e-05 2024-09-21T22:07:51.646 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.646 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.646 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:51.646 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.371162+0000", 2024-09-21T22:07:51.646 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.038850015000000002 2024-09-21T22:07:51.646 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:51.646 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:51.647 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:51.647 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.647 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.647 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4811.0:3 2.3 2:c4c92e5a:::rbd_trash:head [call rbd.trash_list in=13b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-21T22:07:51.647 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:28.475306+0000", 2024-09-21T22:07:51.647 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 23.137809482000002, 2024-09-21T22:07:51.647 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.77996670099999998, 2024-09-21T22:07:51.647 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:51.647 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:51.647 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:07:51.647 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:51.647 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4811", 2024-09-21T22:07:51.648 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/2028736390", 2024-09-21T22:07:51.648 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3 2024-09-21T22:07:51.648 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.648 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:51.648 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.648 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:51.648 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.475306+0000", 2024-09-21T22:07:51.648 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:51.648 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.648 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.648 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:51.648 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.475306+0000", 2024-09-21T22:07:51.648 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:51.648 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.648 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.648 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:51.648 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.475308+0000", 2024-09-21T22:07:51.649 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8959999999999999e-06 2024-09-21T22:07:51.649 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.649 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.649 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:51.649 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.475314+0000", 2024-09-21T22:07:51.649 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2280000000000003e-06 2024-09-21T22:07:51.649 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.649 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.649 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:51.649 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.475329+0000", 2024-09-21T22:07:51.649 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5054e-05 2024-09-21T22:07:51.649 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.649 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.649 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:51.649 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.475338+0000", 2024-09-21T22:07:51.649 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3770000000000002e-06 2024-09-21T22:07:51.650 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.650 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.650 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:51.650 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.475484+0000", 2024-09-21T22:07:51.650 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014612300000000001 2024-09-21T22:07:51.650 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.650 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.650 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-21T22:07:51.650 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.475506+0000", 2024-09-21T22:07:51.650 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1319999999999999e-05 2024-09-21T22:07:51.650 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.650 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.650 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:51.650 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.254433+0000", 2024-09-21T22:07:51.650 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.778927708 2024-09-21T22:07:51.650 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.651 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.651 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:51.651 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.254979+0000", 2024-09-21T22:07:51.651 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00054573100000000004 2024-09-21T22:07:51.651 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.651 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.651 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:51.651 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.255272+0000", 2024-09-21T22:07:51.651 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000293264 2024-09-21T22:07:51.651 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:51.651 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:51.651 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:51.651 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.651 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.652 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.3 67/66 [PushOp(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head, version: 63'11, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head@63'11, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T22:07:51.652 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:29.262595+0000", 2024-09-21T22:07:51.652 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 22.350520298999999, 2024-09-21T22:07:51.652 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.036027272999999999, 2024-09-21T22:07:51.652 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:51.652 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:51.652 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:07:51.652 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:51.652 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.652 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:51.652 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.262595+0000", 2024-09-21T22:07:51.652 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:51.652 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.653 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.653 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:51.653 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.262595+0000", 2024-09-21T22:07:51.653 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:51.653 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.653 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.653 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:51.653 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.262596+0000", 2024-09-21T22:07:51.653 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4300000000000001e-06 2024-09-21T22:07:51.653 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.653 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.653 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:51.653 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.264505+0000", 2024-09-21T22:07:51.653 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019085650000000001 2024-09-21T22:07:51.653 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.654 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.654 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:51.654 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.264557+0000", 2024-09-21T22:07:51.654 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2040000000000003e-05 2024-09-21T22:07:51.654 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.654 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.654 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:51.654 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.264589+0000", 2024-09-21T22:07:51.654 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1636e-05 2024-09-21T22:07:51.654 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.654 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.654 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:51.655 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.264778+0000", 2024-09-21T22:07:51.655 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018935600000000001 2024-09-21T22:07:51.655 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.655 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.655 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:51.655 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.264803+0000", 2024-09-21T22:07:51.655 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4692999999999999e-05 2024-09-21T22:07:51.655 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.655 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.656 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:51.656 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.298622+0000", 2024-09-21T22:07:51.656 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.033819553000000002 2024-09-21T22:07:51.656 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:51.656 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:51.656 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:51.656 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.656 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.656 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.3 67/66 [PushOp(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head, version: 63'11, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head@63'11, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:07:51.656 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:29.301490+0000", 2024-09-21T22:07:51.656 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 22.311625512999999, 2024-09-21T22:07:51.656 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.041178293999999997, 2024-09-21T22:07:51.656 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:51.656 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:51.656 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:07:51.657 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:51.657 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.657 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:51.657 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.301490+0000", 2024-09-21T22:07:51.657 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:51.657 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.657 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.657 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:51.657 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.301490+0000", 2024-09-21T22:07:51.657 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:51.657 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.658 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.658 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:51.658 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.301491+0000", 2024-09-21T22:07:51.658 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.513e-06 2024-09-21T22:07:51.658 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.658 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.658 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:51.658 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.302815+0000", 2024-09-21T22:07:51.658 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001323722 2024-09-21T22:07:51.658 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.658 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.658 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:51.658 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.302863+0000", 2024-09-21T22:07:51.658 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8449999999999999e-05 2024-09-21T22:07:51.659 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.659 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.659 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:51.659 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.302895+0000", 2024-09-21T22:07:51.659 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1072999999999999e-05 2024-09-21T22:07:51.659 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.659 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.659 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:51.659 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.303014+0000", 2024-09-21T22:07:51.659 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011907400000000001 2024-09-21T22:07:51.659 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.659 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.659 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:51.659 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.303029+0000", 2024-09-21T22:07:51.660 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5322e-05 2024-09-21T22:07:51.660 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.660 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.660 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:51.660 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.342668+0000", 2024-09-21T22:07:51.660 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.039639140000000003 2024-09-21T22:07:51.660 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:51.660 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:51.660 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:51.660 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.660 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.660 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.3 67/66 [PushOp(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head, version: 63'11, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head@63'11, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:07:51.660 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:29.345626+0000", 2024-09-21T22:07:51.660 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 22.267488779000001, 2024-09-21T22:07:51.660 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.034820216000000001, 2024-09-21T22:07:51.661 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:51.661 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:51.661 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:07:51.661 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:51.661 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.661 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:51.661 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.345626+0000", 2024-09-21T22:07:51.661 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:51.661 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.661 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.661 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:51.661 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.345626+0000", 2024-09-21T22:07:51.661 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:51.661 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.661 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.662 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:51.662 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.345628+0000", 2024-09-21T22:07:51.662 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3799999999999999e-06 2024-09-21T22:07:51.662 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.662 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.662 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:51.662 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.347215+0000", 2024-09-21T22:07:51.662 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015868760000000001 2024-09-21T22:07:51.662 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.662 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.662 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:51.662 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.347283+0000", 2024-09-21T22:07:51.662 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8267000000000002e-05 2024-09-21T22:07:51.662 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.663 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.663 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:51.663 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.347336+0000", 2024-09-21T22:07:51.663 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3455999999999998e-05 2024-09-21T22:07:51.663 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.663 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.663 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:51.663 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.347463+0000", 2024-09-21T22:07:51.663 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012614399999999999 2024-09-21T22:07:51.663 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.663 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.663 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:51.663 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.347473+0000", 2024-09-21T22:07:51.663 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0462000000000001e-05 2024-09-21T22:07:51.664 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.664 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.664 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:51.664 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.380447+0000", 2024-09-21T22:07:51.664 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.032973631000000003 2024-09-21T22:07:51.664 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:51.664 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:51.664 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:51.664 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.664 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.664 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.3 67/66 [PushOp(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head, version: 63'11, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head@63'11, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 4194304, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:07:51.664 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:29.383677+0000", 2024-09-21T22:07:51.664 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 22.229437902000001, 2024-09-21T22:07:51.664 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.031566611000000001, 2024-09-21T22:07:51.665 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:51.665 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:51.665 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:07:51.665 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:51.665 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.665 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:51.665 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.383677+0000", 2024-09-21T22:07:51.665 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:51.665 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.665 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.665 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:51.665 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.383677+0000", 2024-09-21T22:07:51.665 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:51.666 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.666 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.666 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:51.666 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.383679+0000", 2024-09-21T22:07:51.666 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4500000000000001e-06 2024-09-21T22:07:51.666 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.666 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.666 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:51.666 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.384982+0000", 2024-09-21T22:07:51.666 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001303069 2024-09-21T22:07:51.666 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.667 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.667 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:51.667 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.385028+0000", 2024-09-21T22:07:51.667 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5655999999999998e-05 2024-09-21T22:07:51.667 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.667 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.667 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:51.667 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.385056+0000", 2024-09-21T22:07:51.667 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8877000000000002e-05 2024-09-21T22:07:51.667 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.667 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.667 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:51.667 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.385161+0000", 2024-09-21T22:07:51.668 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000104325 2024-09-21T22:07:51.668 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.668 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.668 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:51.668 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.385173+0000", 2024-09-21T22:07:51.668 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2493e-05 2024-09-21T22:07:51.668 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.668 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.668 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:51.668 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.415244+0000", 2024-09-21T22:07:51.668 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.030070741000000002 2024-09-21T22:07:51.668 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:51.668 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:51.668 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:51.669 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.669 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.669 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.3 67/66 [PushOp(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head, version: 63'19, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head@63'19, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T22:07:51.669 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:29.420098+0000", 2024-09-21T22:07:51.669 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 22.193016944, 2024-09-21T22:07:51.669 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.067283846999999994, 2024-09-21T22:07:51.669 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:51.669 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:51.669 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:07:51.669 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:51.669 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.669 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:51.669 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.420098+0000", 2024-09-21T22:07:51.669 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:51.669 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.670 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.670 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:51.670 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.420098+0000", 2024-09-21T22:07:51.670 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:51.670 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.670 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.670 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:51.670 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.420100+0000", 2024-09-21T22:07:51.670 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.716e-06 2024-09-21T22:07:51.670 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.670 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.670 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:51.670 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.421949+0000", 2024-09-21T22:07:51.671 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0018485190000000001 2024-09-21T22:07:51.671 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.671 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.671 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:51.671 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.421996+0000", 2024-09-21T22:07:51.671 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7089000000000003e-05 2024-09-21T22:07:51.671 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.671 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.671 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:51.671 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.422042+0000", 2024-09-21T22:07:51.671 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6010000000000002e-05 2024-09-21T22:07:51.671 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.671 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.671 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:51.671 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.422231+0000", 2024-09-21T22:07:51.672 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018960900000000001 2024-09-21T22:07:51.672 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.672 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.672 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:51.672 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.422254+0000", 2024-09-21T22:07:51.672 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2864e-05 2024-09-21T22:07:51.672 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.672 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.672 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:51.672 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.487382+0000", 2024-09-21T22:07:51.672 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.065128039999999998 2024-09-21T22:07:51.672 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:51.672 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:51.673 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:51.673 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.673 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.673 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.3 67/66 [PushOp(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head, version: 63'19, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head@63'19, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:07:51.673 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:29.491783+0000", 2024-09-21T22:07:51.673 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 22.121332598999999, 2024-09-21T22:07:51.673 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.04476596, 2024-09-21T22:07:51.673 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:51.673 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:51.673 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:07:51.673 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:51.673 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.673 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:51.673 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.491783+0000", 2024-09-21T22:07:51.673 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:51.674 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.674 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.674 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:51.674 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.491783+0000", 2024-09-21T22:07:51.674 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:51.674 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.674 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.674 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:51.674 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.491784+0000", 2024-09-21T22:07:51.674 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.17e-06 2024-09-21T22:07:51.674 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.674 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.675 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:51.675 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.493126+0000", 2024-09-21T22:07:51.675 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0013424489999999999 2024-09-21T22:07:51.675 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.675 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.675 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:51.675 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.493170+0000", 2024-09-21T22:07:51.675 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4153000000000002e-05 2024-09-21T22:07:51.675 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.675 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.675 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:51.675 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.493218+0000", 2024-09-21T22:07:51.676 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7392999999999998e-05 2024-09-21T22:07:51.676 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.676 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.676 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:51.676 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.493392+0000", 2024-09-21T22:07:51.676 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017392699999999999 2024-09-21T22:07:51.676 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.676 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.676 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:51.676 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.493416+0000", 2024-09-21T22:07:51.677 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4117e-05 2024-09-21T22:07:51.677 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.677 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.677 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:51.677 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.536549+0000", 2024-09-21T22:07:51.677 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.043132750999999997 2024-09-21T22:07:51.677 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:51.677 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:51.677 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:51.677 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.677 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.677 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.3 67/66 [PushOp(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head, version: 63'19, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head@63'19, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:07:51.677 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:29.539831+0000", 2024-09-21T22:07:51.677 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 22.073283838999998, 2024-09-21T22:07:51.678 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.036233222000000002, 2024-09-21T22:07:51.678 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:51.678 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:51.678 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:07:51.678 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:51.678 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.678 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:51.678 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.539831+0000", 2024-09-21T22:07:51.679 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:51.679 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.679 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.679 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:51.679 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.539831+0000", 2024-09-21T22:07:51.680 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:51.680 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.680 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.680 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:51.680 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.539833+0000", 2024-09-21T22:07:51.680 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5200000000000001e-06 2024-09-21T22:07:51.680 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.680 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.680 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:51.680 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.541130+0000", 2024-09-21T22:07:51.680 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0012972719999999999 2024-09-21T22:07:51.680 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.680 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.680 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:51.680 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.541179+0000", 2024-09-21T22:07:51.681 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8507000000000002e-05 2024-09-21T22:07:51.681 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.681 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.681 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:51.681 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.541226+0000", 2024-09-21T22:07:51.681 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7663000000000001e-05 2024-09-21T22:07:51.681 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.681 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.681 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:51.681 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.541361+0000", 2024-09-21T22:07:51.681 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000135065 2024-09-21T22:07:51.681 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.681 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.681 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:51.681 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.541374+0000", 2024-09-21T22:07:51.682 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2295e-05 2024-09-21T22:07:51.682 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.682 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.682 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:51.682 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.576065+0000", 2024-09-21T22:07:51.682 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.034690899999999997 2024-09-21T22:07:51.682 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:51.682 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:51.682 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:51.682 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.682 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.682 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.3 67/66 [PushOp(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head, version: 63'19, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head@63'19, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 4194304, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:07:51.682 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:29.579455+0000", 2024-09-21T22:07:51.682 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 22.033660530999999, 2024-09-21T22:07:51.683 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.034295594999999998, 2024-09-21T22:07:51.683 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:51.683 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:51.683 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:07:51.683 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:51.683 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.683 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:51.683 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.579455+0000", 2024-09-21T22:07:51.683 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:51.683 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.683 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.683 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:51.683 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.579455+0000", 2024-09-21T22:07:51.683 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:51.683 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.684 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.684 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:51.684 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.579456+0000", 2024-09-21T22:07:51.684 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.404e-06 2024-09-21T22:07:51.684 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.684 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.684 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:51.684 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.580763+0000", 2024-09-21T22:07:51.684 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001307325 2024-09-21T22:07:51.684 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.684 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.684 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:51.684 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.580811+0000", 2024-09-21T22:07:51.685 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7843000000000003e-05 2024-09-21T22:07:51.685 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.685 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.685 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:51.685 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.580860+0000", 2024-09-21T22:07:51.685 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9036999999999998e-05 2024-09-21T22:07:51.685 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.685 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.685 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:51.685 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.580988+0000", 2024-09-21T22:07:51.685 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000127529 2024-09-21T22:07:51.685 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.685 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.686 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:51.686 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.580998+0000", 2024-09-21T22:07:51.686 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0617e-05 2024-09-21T22:07:51.686 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.686 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.686 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:51.686 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.613750+0000", 2024-09-21T22:07:51.686 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.032751839999999997 2024-09-21T22:07:51.686 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:51.686 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:51.686 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:51.686 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.686 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.686 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5008.0:125 2.3 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-21T22:07:51.687 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:38.066298+0000", 2024-09-21T22:07:51.687 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 13.546816867, 2024-09-21T22:07:51.687 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.062817136999999995, 2024-09-21T22:07:51.687 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:51.687 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:51.687 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:51.687 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:51.687 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5008", 2024-09-21T22:07:51.687 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/1400114666", 2024-09-21T22:07:51.687 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 125 2024-09-21T22:07:51.687 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.687 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:51.687 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.687 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:51.688 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.066298+0000", 2024-09-21T22:07:51.688 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:51.688 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.688 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.688 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:51.688 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.066298+0000", 2024-09-21T22:07:51.688 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:51.688 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.688 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.688 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:51.688 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.066300+0000", 2024-09-21T22:07:51.688 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.474e-06 2024-09-21T22:07:51.688 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.688 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.688 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:51.689 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.067443+0000", 2024-09-21T22:07:51.689 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011428009999999999 2024-09-21T22:07:51.689 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.689 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.689 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:51.689 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.067477+0000", 2024-09-21T22:07:51.689 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4059e-05 2024-09-21T22:07:51.689 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.689 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.689 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:51.689 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.067492+0000", 2024-09-21T22:07:51.690 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5054e-05 2024-09-21T22:07:51.690 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.690 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.690 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:51.690 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.067638+0000", 2024-09-21T22:07:51.690 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014583900000000001 2024-09-21T22:07:51.690 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.690 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.690 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:51.690 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.067882+0000", 2024-09-21T22:07:51.690 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024415500000000002 2024-09-21T22:07:51.690 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.691 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.691 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-09-21T22:07:51.691 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.068247+0000", 2024-09-21T22:07:51.691 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00036570800000000002 2024-09-21T22:07:51.691 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.691 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.691 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:51.691 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.119971+0000", 2024-09-21T22:07:51.691 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.051723866 2024-09-21T22:07:51.691 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.691 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.691 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:51.691 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.128849+0000", 2024-09-21T22:07:51.692 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0088779489999999996 2024-09-21T22:07:51.692 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.692 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.692 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:51.692 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.129000+0000", 2024-09-21T22:07:51.692 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015062499999999999 2024-09-21T22:07:51.692 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.692 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.692 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:51.692 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.129116+0000", 2024-09-21T22:07:51.692 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011560700000000001 2024-09-21T22:07:51.692 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:51.692 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:51.692 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:51.692 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.693 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.693 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5008.0:127 2.3 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-21T22:07:51.693 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:38.140308+0000", 2024-09-21T22:07:51.693 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 13.472806765, 2024-09-21T22:07:51.693 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.055290512999999999, 2024-09-21T22:07:51.693 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:51.693 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:51.693 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:51.693 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:51.693 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5008", 2024-09-21T22:07:51.693 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/1400114666", 2024-09-21T22:07:51.693 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 127 2024-09-21T22:07:51.694 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.694 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:51.694 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.694 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:51.694 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.140308+0000", 2024-09-21T22:07:51.694 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:51.694 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.694 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.694 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:51.694 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.140308+0000", 2024-09-21T22:07:51.694 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:51.694 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.694 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.694 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:51.695 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.140310+0000", 2024-09-21T22:07:51.695 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6509999999999999e-06 2024-09-21T22:07:51.695 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.695 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.695 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:51.695 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.141504+0000", 2024-09-21T22:07:51.695 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011935750000000001 2024-09-21T22:07:51.695 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.695 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.695 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:51.695 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.141522+0000", 2024-09-21T22:07:51.696 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8349e-05 2024-09-21T22:07:51.696 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.696 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.696 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:51.696 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.141531+0000", 2024-09-21T22:07:51.696 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3239999999999992e-06 2024-09-21T22:07:51.696 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.696 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.696 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:51.696 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.141661+0000", 2024-09-21T22:07:51.696 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013001000000000001 2024-09-21T22:07:51.696 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.696 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.697 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:51.697 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.141907+0000", 2024-09-21T22:07:51.697 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024547499999999999 2024-09-21T22:07:51.697 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.697 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.697 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-09-21T22:07:51.697 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.142274+0000", 2024-09-21T22:07:51.697 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00036716400000000001 2024-09-21T22:07:51.697 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.697 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.697 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:51.697 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.189864+0000", 2024-09-21T22:07:51.697 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047589617000000001 2024-09-21T22:07:51.697 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.698 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.698 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:51.698 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.195406+0000", 2024-09-21T22:07:51.698 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0055425860000000004 2024-09-21T22:07:51.698 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.698 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.698 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:51.698 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.195507+0000", 2024-09-21T22:07:51.698 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000100862 2024-09-21T22:07:51.698 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.698 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.698 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:51.698 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.195599+0000", 2024-09-21T22:07:51.699 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1899999999999998e-05 2024-09-21T22:07:51.699 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:51.699 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:51.699 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:51.699 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.699 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.699 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5008.0:129 2.3 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head [write 2228224~917504 in=917504b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-21T22:07:51.699 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:38.209297+0000", 2024-09-21T22:07:51.699 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 13.403817884, 2024-09-21T22:07:51.699 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.060441054000000001, 2024-09-21T22:07:51.699 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:51.699 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:51.699 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:51.699 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:51.700 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5008", 2024-09-21T22:07:51.700 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/1400114666", 2024-09-21T22:07:51.700 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 129 2024-09-21T22:07:51.700 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.700 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:51.700 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.700 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:51.700 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.209297+0000", 2024-09-21T22:07:51.700 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:51.700 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.700 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.700 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:51.700 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.209297+0000", 2024-09-21T22:07:51.700 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:51.700 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.701 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.701 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:51.701 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.209299+0000", 2024-09-21T22:07:51.701 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.274e-06 2024-09-21T22:07:51.701 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.701 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.701 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:51.701 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.210077+0000", 2024-09-21T22:07:51.702 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00077849799999999997 2024-09-21T22:07:51.702 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.702 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.702 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:51.702 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.210094+0000", 2024-09-21T22:07:51.702 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6772000000000001e-05 2024-09-21T22:07:51.702 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.702 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.702 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:51.702 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.210103+0000", 2024-09-21T22:07:51.702 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6780000000000006e-06 2024-09-21T22:07:51.702 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.702 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.702 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:51.703 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.210254+0000", 2024-09-21T22:07:51.703 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015141699999999999 2024-09-21T22:07:51.703 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.703 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.703 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:51.703 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.210501+0000", 2024-09-21T22:07:51.703 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000246498 2024-09-21T22:07:51.703 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.703 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.703 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-09-21T22:07:51.703 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.210872+0000", 2024-09-21T22:07:51.703 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00037147500000000002 2024-09-21T22:07:51.703 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.703 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.703 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:51.704 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.269019+0000", 2024-09-21T22:07:51.704 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058147114 2024-09-21T22:07:51.704 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.704 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.704 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:51.704 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.269638+0000", 2024-09-21T22:07:51.704 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00061896499999999997 2024-09-21T22:07:51.704 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.704 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.704 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:51.704 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.269693+0000", 2024-09-21T22:07:51.704 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4786e-05 2024-09-21T22:07:51.704 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.704 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.705 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:51.705 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.269738+0000", 2024-09-21T22:07:51.705 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5577000000000001e-05 2024-09-21T22:07:51.705 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:51.705 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:51.705 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:51.705 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.705 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.705 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5008.0:131 2.3 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-21T22:07:51.705 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:38.279879+0000", 2024-09-21T22:07:51.705 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 13.333236193999999, 2024-09-21T22:07:51.705 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.054184296999999999, 2024-09-21T22:07:51.706 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:51.706 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:51.706 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:51.706 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:51.706 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5008", 2024-09-21T22:07:51.706 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/1400114666", 2024-09-21T22:07:51.706 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 131 2024-09-21T22:07:51.706 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.706 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:51.706 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.706 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:51.706 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.279879+0000", 2024-09-21T22:07:51.707 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:51.707 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.707 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.707 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:51.707 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.279879+0000", 2024-09-21T22:07:51.707 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:51.707 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.707 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.707 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:51.707 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.279880+0000", 2024-09-21T22:07:51.707 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.136e-06 2024-09-21T22:07:51.707 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.707 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.708 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:51.708 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.280636+0000", 2024-09-21T22:07:51.708 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00075625900000000001 2024-09-21T22:07:51.708 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.708 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.708 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:51.708 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.280648+0000", 2024-09-21T22:07:51.708 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2026e-05 2024-09-21T22:07:51.708 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.708 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.708 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:51.708 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.280654+0000", 2024-09-21T22:07:51.708 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8839999999999999e-06 2024-09-21T22:07:51.708 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.709 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.709 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:51.709 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.280739+0000", 2024-09-21T22:07:51.709 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4196000000000005e-05 2024-09-21T22:07:51.709 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.709 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.709 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:51.709 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.280808+0000", 2024-09-21T22:07:51.709 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9685999999999997e-05 2024-09-21T22:07:51.709 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.709 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.709 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-09-21T22:07:51.709 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.280900+0000", 2024-09-21T22:07:51.709 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1407000000000001e-05 2024-09-21T22:07:51.709 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.710 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.710 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:51.710 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.307730+0000", 2024-09-21T22:07:51.710 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026830568999999999 2024-09-21T22:07:51.710 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.710 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.710 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:51.710 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.333901+0000", 2024-09-21T22:07:51.710 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026170787000000001 2024-09-21T22:07:51.710 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.710 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.710 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:51.710 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.333996+0000", 2024-09-21T22:07:51.710 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4723999999999996e-05 2024-09-21T22:07:51.710 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:51.711 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:51.711 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:51.711 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.334063+0000", 2024-09-21T22:07:51.711 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7623e-05 2024-09-21T22:07:51.711 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:51.711 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:51.711 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:51.711 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:51.711 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:51.711 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:07:51.712 INFO:tasks.workunit.client.0.smithi157.stderr:+ DEV= 2024-09-21T22:07:51.712 INFO:tasks.workunit.client.0.smithi157.stderr:+ rbd config global rm global rbd_default_pool 2024-09-21T22:07:51.713 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:51.708+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:51.713 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T22:07:51.717 INFO:tasks.workunit.client.0.smithi157.stderr:++ _sudo rbd device --device-type nbd map rbd/testrbdnbd21553 2024-09-21T22:07:51.717 INFO:tasks.workunit.client.0.smithi157.stderr:++ local cmd 2024-09-21T22:07:51.717 INFO:tasks.workunit.client.0.smithi157.stderr:+++ id -u 2024-09-21T22:07:51.718 INFO:tasks.workunit.client.0.smithi157.stderr:++ '[' 1000 -eq 0 ']' 2024-09-21T22:07:51.718 INFO:tasks.workunit.client.0.smithi157.stderr:+++ which rbd 2024-09-21T22:07:51.720 INFO:tasks.workunit.client.0.smithi157.stderr:++ cmd=/usr/bin/rbd 2024-09-21T22:07:51.720 INFO:tasks.workunit.client.0.smithi157.stderr:++ shift 2024-09-21T22:07:51.720 INFO:tasks.workunit.client.0.smithi157.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map rbd/testrbdnbd21553 2024-09-21T22:07:51.814 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:51.812+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:51.846 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:07:51.846 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:07:51.846 INFO:teuthology.orchestra.run.smithi039.stdout: "num_ops": 0 2024-09-21T22:07:51.847 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:07:51.857 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-21T22:07:51.862 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T22:07:51.864 INFO:tasks.workunit.client.0.smithi157.stderr:+ DEV=/dev/nbd0 2024-09-21T22:07:51.864 INFO:tasks.workunit.client.0.smithi157.stderr:+ get_pid rbd 2024-09-21T22:07:51.865 INFO:tasks.workunit.client.0.smithi157.stderr:+ local pool=rbd 2024-09-21T22:07:51.865 INFO:tasks.workunit.client.0.smithi157.stderr:+ local ns= 2024-09-21T22:07:51.865 INFO:tasks.workunit.client.0.smithi157.stderr:++ rbd device --device-type nbd --format xml list 2024-09-21T22:07:51.865 INFO:tasks.workunit.client.0.smithi157.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd21553'\''][device='\''/dev/nbd0'\'']/id' 2024-09-21T22:07:51.915 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:51.912+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:51.932 INFO:tasks.workunit.client.0.smithi157.stderr:+ PID=23610 2024-09-21T22:07:51.933 INFO:tasks.workunit.client.0.smithi157.stderr:+ test -n 23610 2024-09-21T22:07:51.933 INFO:tasks.workunit.client.0.smithi157.stderr:+ ps -p 23610 -C rbd-nbd 2024-09-21T22:07:51.939 INFO:tasks.workunit.client.0.smithi157.stdout: PID TTY TIME CMD 2024-09-21T22:07:51.939 INFO:tasks.workunit.client.0.smithi157.stdout: 23610 ? 00:00:00 rbd-nbd 2024-09-21T22:07:51.939 INFO:tasks.workunit.client.0.smithi157.stderr:+ _sudo kill 23610 2024-09-21T22:07:51.940 INFO:tasks.workunit.client.0.smithi157.stderr:+ local cmd 2024-09-21T22:07:51.940 INFO:tasks.workunit.client.0.smithi157.stderr:++ id -u 2024-09-21T22:07:51.941 INFO:tasks.workunit.client.0.smithi157.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T22:07:51.942 INFO:tasks.workunit.client.0.smithi157.stderr:++ which kill 2024-09-21T22:07:51.943 INFO:tasks.workunit.client.0.smithi157.stderr:+ cmd=/usr/bin/kill 2024-09-21T22:07:51.943 INFO:tasks.workunit.client.0.smithi157.stderr:+ shift 2024-09-21T22:07:51.943 INFO:tasks.workunit.client.0.smithi157.stderr:+ sudo -nE /usr/bin/kill 23610 2024-09-21T22:07:52.015 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:52.010+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:52.033 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:07:52.033 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:07:52.033 INFO:teuthology.orchestra.run.smithi039.stdout: "complaint_time": 30, 2024-09-21T22:07:52.033 INFO:teuthology.orchestra.run.smithi039.stdout: "num_blocked_ops": 0 2024-09-21T22:07:52.033 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:07:52.048 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T22:07:52.056 INFO:tasks.workunit.client.0.smithi157.stderr:++ seq 10 2024-09-21T22:07:52.063 INFO:tasks.workunit.client.0.smithi157.stderr:+ for i in `seq 10` 2024-09-21T22:07:52.063 INFO:tasks.workunit.client.0.smithi157.stderr:+ rbd device --device-type nbd list 2024-09-21T22:07:52.066 INFO:tasks.workunit.client.0.smithi157.stderr:+ expect_false grep '^23610 *rbd *testrbdnbd21553' 2024-09-21T22:07:52.066 INFO:tasks.workunit.client.0.smithi157.stderr:+ grep '^23610 *rbd *testrbdnbd21553' 2024-09-21T22:07:52.136 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:52.134+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:52.158 INFO:tasks.workunit.client.0.smithi157.stderr:+ return 0 2024-09-21T22:07:52.158 INFO:tasks.workunit.client.0.smithi157.stderr:+ break 2024-09-21T22:07:52.158 INFO:tasks.workunit.client.0.smithi157.stderr:+ rbd device --device-type nbd list 2024-09-21T22:07:52.159 INFO:tasks.workunit.client.0.smithi157.stderr:+ expect_false grep '^23610 *rbd *testrbdnbd21553' 2024-09-21T22:07:52.159 INFO:tasks.workunit.client.0.smithi157.stderr:+ grep '^23610 *rbd *testrbdnbd21553' 2024-09-21T22:07:52.211 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:07:52.211 INFO:teuthology.orchestra.run.smithi039.stdout: "size": 20, 2024-09-21T22:07:52.211 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 600, 2024-09-21T22:07:52.211 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [ 2024-09-21T22:07:52.212 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.212 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:694 2.5 e63/37 2:b1d55633:::rbd_data.126483d75f2d.000000000000000b:head v 63'26, mlcod=63'24)", 2024-09-21T22:07:52.212 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.163199+0000", 2024-09-21T22:07:52.212 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 27.044819306000001, 2024-09-21T22:07:52.212 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.034676672999999998, 2024-09-21T22:07:52.212 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:52.212 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:52.212 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:52.212 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:52.212 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.212 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:52.212 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.163199+0000", 2024-09-21T22:07:52.212 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:52.213 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.213 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.213 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:52.213 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.163199+0000", 2024-09-21T22:07:52.213 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:52.213 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.213 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.213 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:52.213 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.163200+0000", 2024-09-21T22:07:52.213 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.703e-06 2024-09-21T22:07:52.213 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.213 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.213 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:52.213 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.164013+0000", 2024-09-21T22:07:52.214 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000812447 2024-09-21T22:07:52.214 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.214 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.214 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:52.214 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.164042+0000", 2024-09-21T22:07:52.214 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9006e-05 2024-09-21T22:07:52.214 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.214 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.214 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:52.214 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.164064+0000", 2024-09-21T22:07:52.214 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1719999999999999e-05 2024-09-21T22:07:52.214 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.214 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.215 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:52.215 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.164143+0000", 2024-09-21T22:07:52.215 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.9283999999999995e-05 2024-09-21T22:07:52.215 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.215 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.215 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:52.215 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.164165+0000", 2024-09-21T22:07:52.215 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.2379e-05 2024-09-21T22:07:52.215 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.215 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.215 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:52.215 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.197827+0000", 2024-09-21T22:07:52.215 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.033661782000000001 2024-09-21T22:07:52.215 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.216 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.216 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:52.216 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.197875+0000", 2024-09-21T22:07:52.216 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8352000000000001e-05 2024-09-21T22:07:52.216 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.216 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:52.216 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.216 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.216 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.216 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:696 2.5 e63/37 2:b1d55633:::rbd_data.126483d75f2d.000000000000000b:head v 63'28, mlcod=63'26)", 2024-09-21T22:07:52.216 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.218913+0000", 2024-09-21T22:07:52.216 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 26.989104860000001, 2024-09-21T22:07:52.216 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.056187599999999997, 2024-09-21T22:07:52.216 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:52.217 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:52.217 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:52.217 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:52.217 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.217 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:52.217 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.218913+0000", 2024-09-21T22:07:52.217 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:52.217 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.217 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.217 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:52.217 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.218913+0000", 2024-09-21T22:07:52.217 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:52.217 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.217 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.217 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:52.218 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.218914+0000", 2024-09-21T22:07:52.218 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.1930000000000001e-06 2024-09-21T22:07:52.218 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.218 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.218 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:52.218 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.219721+0000", 2024-09-21T22:07:52.218 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00080701500000000001 2024-09-21T22:07:52.218 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.218 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.218 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:52.218 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.219743+0000", 2024-09-21T22:07:52.218 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1364000000000001e-05 2024-09-21T22:07:52.218 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.218 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.219 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:52.219 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.219759+0000", 2024-09-21T22:07:52.219 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.611e-05 2024-09-21T22:07:52.219 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.219 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.219 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:52.219 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.219908+0000", 2024-09-21T22:07:52.219 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000149252 2024-09-21T22:07:52.219 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.219 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.219 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:52.219 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.219951+0000", 2024-09-21T22:07:52.219 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.2518999999999997e-05 2024-09-21T22:07:52.220 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.220 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.220 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:52.220 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.275031+0000", 2024-09-21T22:07:52.220 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.055080241000000002 2024-09-21T22:07:52.220 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.220 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.220 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:52.220 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.275101+0000", 2024-09-21T22:07:52.220 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.9906000000000005e-05 2024-09-21T22:07:52.220 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.220 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:52.220 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.220 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.220 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.221 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:698 2.5 e63/37 2:b1d55633:::rbd_data.126483d75f2d.000000000000000b:head v 63'30, mlcod=63'28)", 2024-09-21T22:07:52.221 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.289319+0000", 2024-09-21T22:07:52.221 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 26.918698651, 2024-09-21T22:07:52.221 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.053930944000000001, 2024-09-21T22:07:52.221 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:52.221 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:52.221 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:52.221 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:52.221 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.221 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:52.221 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.289319+0000", 2024-09-21T22:07:52.221 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:52.221 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.222 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.222 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:52.222 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.289319+0000", 2024-09-21T22:07:52.222 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:52.222 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.222 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.222 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:52.222 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.289321+0000", 2024-09-21T22:07:52.222 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.221e-06 2024-09-21T22:07:52.222 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.222 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.222 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:52.222 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.290091+0000", 2024-09-21T22:07:52.223 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00077069300000000003 2024-09-21T22:07:52.223 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.223 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.223 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:52.223 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.290109+0000", 2024-09-21T22:07:52.223 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7218999999999999e-05 2024-09-21T22:07:52.223 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.223 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.223 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:52.223 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.290122+0000", 2024-09-21T22:07:52.223 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3678e-05 2024-09-21T22:07:52.223 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.223 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.223 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:52.224 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.290274+0000", 2024-09-21T22:07:52.224 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00015205899999999999 2024-09-21T22:07:52.224 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.224 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.224 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:52.224 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.290313+0000", 2024-09-21T22:07:52.224 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.8636000000000001e-05 2024-09-21T22:07:52.224 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.224 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.224 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:52.224 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.343179+0000", 2024-09-21T22:07:52.224 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.052866199000000003 2024-09-21T22:07:52.224 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.224 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.225 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:52.225 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.343250+0000", 2024-09-21T22:07:52.225 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.1239e-05 2024-09-21T22:07:52.225 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.225 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:52.225 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.225 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.225 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.225 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:702 2.5 e63/37 2:a70ae99f:::rbd_data.126483d75f2d.000000000000000c:head v 63'34, mlcod=63'32)", 2024-09-21T22:07:52.225 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.392532+0000", 2024-09-21T22:07:52.225 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 26.815486063000002, 2024-09-21T22:07:52.225 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.035703990999999997, 2024-09-21T22:07:52.225 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:52.225 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:52.226 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:52.226 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:52.226 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.226 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:52.226 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.392532+0000", 2024-09-21T22:07:52.226 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:52.226 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.226 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.226 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:52.226 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.392532+0000", 2024-09-21T22:07:52.226 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:52.226 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.226 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.226 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:52.227 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.392534+0000", 2024-09-21T22:07:52.227 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.64e-06 2024-09-21T22:07:52.227 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.227 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.227 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:52.227 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.393321+0000", 2024-09-21T22:07:52.227 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00078705699999999997 2024-09-21T22:07:52.227 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.227 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.227 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:52.227 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.393347+0000", 2024-09-21T22:07:52.227 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.6735999999999999e-05 2024-09-21T22:07:52.228 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.228 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.228 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:52.228 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.393361+0000", 2024-09-21T22:07:52.228 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3346000000000001e-05 2024-09-21T22:07:52.228 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.228 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.228 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:52.228 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.393469+0000", 2024-09-21T22:07:52.228 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000108039 2024-09-21T22:07:52.228 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.228 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.228 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:52.229 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.393512+0000", 2024-09-21T22:07:52.229 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.2862999999999999e-05 2024-09-21T22:07:52.229 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.229 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.229 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:52.229 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.428170+0000", 2024-09-21T22:07:52.229 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.034658858000000001 2024-09-21T22:07:52.229 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.229 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.229 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:52.229 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.428236+0000", 2024-09-21T22:07:52.229 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.5451999999999996e-05 2024-09-21T22:07:52.230 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.230 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:52.230 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.230 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.230 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.230 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:708 2.1 e63/53 2:837ddd83:::rbd_data.126483d75f2d.000000000000000d:head v 63'6, mlcod=57'4)", 2024-09-21T22:07:52.230 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.517007+0000", 2024-09-21T22:07:52.230 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 26.691011137, 2024-09-21T22:07:52.230 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.032495584000000001, 2024-09-21T22:07:52.230 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:52.230 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:52.230 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:52.230 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:52.231 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.231 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:52.231 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.517007+0000", 2024-09-21T22:07:52.231 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:52.231 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.231 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.231 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:52.231 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.517007+0000", 2024-09-21T22:07:52.231 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:52.231 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.231 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.231 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:52.231 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.517008+0000", 2024-09-21T22:07:52.232 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.604e-06 2024-09-21T22:07:52.232 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.232 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.232 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:52.232 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.517603+0000", 2024-09-21T22:07:52.232 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00059461900000000001 2024-09-21T22:07:52.232 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.232 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.232 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:52.232 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.517632+0000", 2024-09-21T22:07:52.232 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8663e-05 2024-09-21T22:07:52.232 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.232 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.233 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:52.233 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.517647+0000", 2024-09-21T22:07:52.233 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4833e-05 2024-09-21T22:07:52.233 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.233 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.233 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:52.233 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.517714+0000", 2024-09-21T22:07:52.233 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.7669000000000006e-05 2024-09-21T22:07:52.233 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.233 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.233 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:52.233 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.517737+0000", 2024-09-21T22:07:52.233 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.2469000000000001e-05 2024-09-21T22:07:52.233 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.233 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.234 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:52.234 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.549370+0000", 2024-09-21T22:07:52.234 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.031633221000000003 2024-09-21T22:07:52.234 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.234 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.234 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:52.234 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.549502+0000", 2024-09-21T22:07:52.234 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013250600000000001 2024-09-21T22:07:52.234 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.234 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:52.234 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.234 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.234 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.235 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:712 2.1 e63/53 2:837ddd83:::rbd_data.126483d75f2d.000000000000000d:head v 63'10, mlcod=63'8)", 2024-09-21T22:07:52.235 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.604029+0000", 2024-09-21T22:07:52.235 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 26.603989198000001, 2024-09-21T22:07:52.235 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.037982371000000001, 2024-09-21T22:07:52.235 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:52.235 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:52.235 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:52.235 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:52.235 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.235 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:52.235 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.604029+0000", 2024-09-21T22:07:52.235 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:52.235 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.236 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.236 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:52.236 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.604029+0000", 2024-09-21T22:07:52.236 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:52.236 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.236 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.236 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:52.236 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.604031+0000", 2024-09-21T22:07:52.236 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8870000000000001e-06 2024-09-21T22:07:52.236 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.236 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.236 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:52.236 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.604395+0000", 2024-09-21T22:07:52.236 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00036422000000000001 2024-09-21T22:07:52.237 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.237 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.237 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:52.237 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.604424+0000", 2024-09-21T22:07:52.237 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8609e-05 2024-09-21T22:07:52.237 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.237 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.237 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:52.237 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.604437+0000", 2024-09-21T22:07:52.237 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3086e-05 2024-09-21T22:07:52.237 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.237 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.237 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:52.237 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.604509+0000", 2024-09-21T22:07:52.238 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.2448999999999997e-05 2024-09-21T22:07:52.238 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.238 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.238 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:52.238 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.604531+0000", 2024-09-21T22:07:52.238 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1602000000000001e-05 2024-09-21T22:07:52.238 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.238 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.238 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:52.238 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.641885+0000", 2024-09-21T22:07:52.238 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.037354545000000003 2024-09-21T22:07:52.238 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.239 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.239 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:52.239 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.642011+0000", 2024-09-21T22:07:52.239 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00012597300000000001 2024-09-21T22:07:52.239 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.239 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:52.239 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.239 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.239 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.239 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:720 2.1 e63/53 2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head v 63'18, mlcod=63'16)", 2024-09-21T22:07:52.239 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.805111+0000", 2024-09-21T22:07:52.239 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 26.402907138, 2024-09-21T22:07:52.239 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.046082864000000001, 2024-09-21T22:07:52.240 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:52.240 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:52.240 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:52.240 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:52.240 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.240 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:52.240 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.805111+0000", 2024-09-21T22:07:52.240 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:52.240 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.240 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.240 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:52.240 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.805111+0000", 2024-09-21T22:07:52.240 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:52.241 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.241 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.241 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:52.241 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.805112+0000", 2024-09-21T22:07:52.241 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.02e-06 2024-09-21T22:07:52.241 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.241 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.241 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:52.241 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.805429+0000", 2024-09-21T22:07:52.241 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00031740799999999998 2024-09-21T22:07:52.241 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.241 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.241 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:52.241 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.805459+0000", 2024-09-21T22:07:52.242 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.955e-05 2024-09-21T22:07:52.242 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.242 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.242 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:52.242 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.805472+0000", 2024-09-21T22:07:52.242 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3533000000000001e-05 2024-09-21T22:07:52.242 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.242 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.242 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:52.242 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.805581+0000", 2024-09-21T22:07:52.242 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000108194 2024-09-21T22:07:52.242 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.242 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.243 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:52.243 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.805615+0000", 2024-09-21T22:07:52.243 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.4310000000000002e-05 2024-09-21T22:07:52.243 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.243 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.243 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:52.243 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.851146+0000", 2024-09-21T22:07:52.243 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.045531092000000002 2024-09-21T22:07:52.243 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.243 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.243 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:52.243 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.851194+0000", 2024-09-21T22:07:52.243 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.7756999999999997e-05 2024-09-21T22:07:52.244 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.244 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:52.244 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.244 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.244 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.244 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:723 2.1 e63/53 2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head v 63'20, mlcod=63'18)", 2024-09-21T22:07:52.244 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.862341+0000", 2024-09-21T22:07:52.244 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 26.345677204000001, 2024-09-21T22:07:52.244 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.036454671000000001, 2024-09-21T22:07:52.244 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:52.244 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:52.244 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:52.244 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:52.245 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.245 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:52.245 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.862341+0000", 2024-09-21T22:07:52.245 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:52.245 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.245 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.245 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:52.245 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.862341+0000", 2024-09-21T22:07:52.245 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:52.245 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.245 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.245 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:52.245 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.862342+0000", 2024-09-21T22:07:52.245 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.0299999999999997e-07 2024-09-21T22:07:52.246 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.246 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.246 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:52.246 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.862721+0000", 2024-09-21T22:07:52.246 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00037889800000000002 2024-09-21T22:07:52.246 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.246 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.246 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:52.246 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.862738+0000", 2024-09-21T22:07:52.246 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6957999999999999e-05 2024-09-21T22:07:52.246 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.246 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.247 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:52.247 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.862747+0000", 2024-09-21T22:07:52.247 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.0729999999999994e-06 2024-09-21T22:07:52.247 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.247 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.247 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:52.247 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.862830+0000", 2024-09-21T22:07:52.247 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.3113000000000002e-05 2024-09-21T22:07:52.247 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.247 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.247 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:52.247 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.862860+0000", 2024-09-21T22:07:52.247 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9980000000000001e-05 2024-09-21T22:07:52.248 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.248 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.248 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:52.248 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.898705+0000", 2024-09-21T22:07:52.248 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.035845063000000003 2024-09-21T22:07:52.248 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.248 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.248 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:52.248 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.898795+0000", 2024-09-21T22:07:52.248 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.0682999999999999e-05 2024-09-21T22:07:52.248 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.248 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:52.248 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.248 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.249 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.249 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:136 2.5 e72/37 2:b1d55633:::rbd_data.126483d75f2d.000000000000000b:head v 72'44, mlcod=72'42)", 2024-09-21T22:07:52.249 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.386273+0000", 2024-09-21T22:07:52.249 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 13.821745089, 2024-09-21T22:07:52.249 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.033806744999999999, 2024-09-21T22:07:52.249 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:52.249 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:52.249 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:52.249 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:52.249 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.249 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:52.249 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.386273+0000", 2024-09-21T22:07:52.250 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:52.250 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.250 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.250 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:52.250 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.386273+0000", 2024-09-21T22:07:52.250 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:52.250 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.250 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.250 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:52.250 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.386274+0000", 2024-09-21T22:07:52.250 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.8299999999999996e-07 2024-09-21T22:07:52.250 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.250 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.251 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:52.251 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.387108+0000", 2024-09-21T22:07:52.251 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00083467800000000005 2024-09-21T22:07:52.251 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.251 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.251 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:52.251 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.387121+0000", 2024-09-21T22:07:52.251 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2823000000000001e-05 2024-09-21T22:07:52.251 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.251 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.251 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:52.251 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.387128+0000", 2024-09-21T22:07:52.251 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.404e-06 2024-09-21T22:07:52.251 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.252 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.252 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:52.252 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.387179+0000", 2024-09-21T22:07:52.252 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.1888000000000002e-05 2024-09-21T22:07:52.252 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.252 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.252 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:52.252 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.387201+0000", 2024-09-21T22:07:52.252 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.2019999999999999e-05 2024-09-21T22:07:52.252 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.252 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.252 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:52.252 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.419993+0000", 2024-09-21T22:07:52.252 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.032791182000000002 2024-09-21T22:07:52.253 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.253 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.253 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:52.253 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.420080+0000", 2024-09-21T22:07:52.253 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.7066999999999998e-05 2024-09-21T22:07:52.253 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.253 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:52.253 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.253 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.253 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.253 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:138 2.5 e72/37 2:b1d55633:::rbd_data.126483d75f2d.000000000000000b:head v 72'46, mlcod=72'44)", 2024-09-21T22:07:52.253 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.430104+0000", 2024-09-21T22:07:52.253 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 13.777913741000001, 2024-09-21T22:07:52.253 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.034087458000000001, 2024-09-21T22:07:52.254 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:52.254 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:52.254 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:52.254 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:52.254 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.254 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:52.254 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.430104+0000", 2024-09-21T22:07:52.254 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:52.254 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.254 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.254 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:52.254 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.430104+0000", 2024-09-21T22:07:52.254 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:52.254 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.255 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.255 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:52.255 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.430105+0000", 2024-09-21T22:07:52.255 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.1680000000000001e-06 2024-09-21T22:07:52.255 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.255 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.255 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:52.255 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.430864+0000", 2024-09-21T22:07:52.255 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00075819400000000001 2024-09-21T22:07:52.255 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.255 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.255 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:52.255 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.430881+0000", 2024-09-21T22:07:52.256 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7000999999999999e-05 2024-09-21T22:07:52.256 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.256 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.256 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:52.256 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.430889+0000", 2024-09-21T22:07:52.256 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.9540000000000004e-06 2024-09-21T22:07:52.256 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.256 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.256 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:52.256 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.430959+0000", 2024-09-21T22:07:52.256 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.0858999999999994e-05 2024-09-21T22:07:52.256 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.257 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.257 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:52.257 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.430996+0000", 2024-09-21T22:07:52.257 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.6443999999999999e-05 2024-09-21T22:07:52.257 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.257 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.257 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:52.257 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.464087+0000", 2024-09-21T22:07:52.257 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.033090799999999997 2024-09-21T22:07:52.257 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.257 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.257 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:52.257 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.464192+0000", 2024-09-21T22:07:52.257 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00010503800000000001 2024-09-21T22:07:52.258 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.258 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:52.258 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.258 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.258 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.258 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:140 2.5 e72/37 2:b1d55633:::rbd_data.126483d75f2d.000000000000000b:head v 72'48, mlcod=72'46)", 2024-09-21T22:07:52.258 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.475737+0000", 2024-09-21T22:07:52.258 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 13.732280792999999, 2024-09-21T22:07:52.258 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.037632025, 2024-09-21T22:07:52.258 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:52.258 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:52.258 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:52.259 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:52.259 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.259 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:52.259 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.475737+0000", 2024-09-21T22:07:52.259 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:52.259 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.259 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.259 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:52.259 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.475737+0000", 2024-09-21T22:07:52.259 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:52.259 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.259 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.259 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:52.260 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.475739+0000", 2024-09-21T22:07:52.260 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.666e-06 2024-09-21T22:07:52.260 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.260 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.260 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:52.260 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.476538+0000", 2024-09-21T22:07:52.260 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00079927399999999999 2024-09-21T22:07:52.260 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.260 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.260 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:52.260 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.476556+0000", 2024-09-21T22:07:52.260 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7666999999999999e-05 2024-09-21T22:07:52.260 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.260 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.260 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:52.261 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.476565+0000", 2024-09-21T22:07:52.261 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.0650000000000005e-06 2024-09-21T22:07:52.261 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.261 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.261 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:52.261 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.476651+0000", 2024-09-21T22:07:52.261 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.5916999999999997e-05 2024-09-21T22:07:52.261 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.261 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.261 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:52.261 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.476696+0000", 2024-09-21T22:07:52.261 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.5537000000000001e-05 2024-09-21T22:07:52.261 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.261 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.262 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:52.262 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.513300+0000", 2024-09-21T22:07:52.262 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.036603498999999998 2024-09-21T22:07:52.262 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.262 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.262 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:52.262 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.513369+0000", 2024-09-21T22:07:52.262 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.9400000000000006e-05 2024-09-21T22:07:52.262 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.262 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:52.262 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.262 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.262 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.263 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:145 2.5 e72/37 2:a70ae99f:::rbd_data.126483d75f2d.000000000000000c:head v 72'52, mlcod=72'50)", 2024-09-21T22:07:52.263 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.578171+0000", 2024-09-21T22:07:52.263 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 13.629846943, 2024-09-21T22:07:52.263 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.037181537000000001, 2024-09-21T22:07:52.263 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:52.263 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:52.263 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:52.263 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:52.263 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.263 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:52.263 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.578171+0000", 2024-09-21T22:07:52.263 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:52.264 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.264 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.264 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:52.264 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.578171+0000", 2024-09-21T22:07:52.264 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:52.264 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.264 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.264 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:52.264 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.578172+0000", 2024-09-21T22:07:52.264 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.017e-06 2024-09-21T22:07:52.264 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.264 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.264 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:52.264 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.579000+0000", 2024-09-21T22:07:52.264 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00082799499999999999 2024-09-21T22:07:52.265 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.265 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.265 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:52.265 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.579020+0000", 2024-09-21T22:07:52.265 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.0165000000000001e-05 2024-09-21T22:07:52.265 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.265 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.265 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:52.265 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.579036+0000", 2024-09-21T22:07:52.265 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5767e-05 2024-09-21T22:07:52.265 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.265 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.265 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:52.266 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.579106+0000", 2024-09-21T22:07:52.266 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.0091000000000004e-05 2024-09-21T22:07:52.266 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.266 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.266 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:52.266 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.579131+0000", 2024-09-21T22:07:52.266 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4978e-05 2024-09-21T22:07:52.266 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.266 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.266 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:52.266 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.615249+0000", 2024-09-21T22:07:52.266 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.036118355999999997 2024-09-21T22:07:52.266 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.266 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.267 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:52.267 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.615353+0000", 2024-09-21T22:07:52.267 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000103168 2024-09-21T22:07:52.267 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.267 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:52.267 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.267 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.267 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.267 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:147 2.5 e72/37 2:a70ae99f:::rbd_data.126483d75f2d.000000000000000c:head v 72'54, mlcod=72'52)", 2024-09-21T22:07:52.267 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.627203+0000", 2024-09-21T22:07:52.267 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 13.580814831, 2024-09-21T22:07:52.267 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.038203972000000003, 2024-09-21T22:07:52.267 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:52.267 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:52.267 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:52.268 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:52.268 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.268 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:52.268 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.627203+0000", 2024-09-21T22:07:52.268 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:52.268 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.268 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.268 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:52.268 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.627203+0000", 2024-09-21T22:07:52.268 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:52.268 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.268 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.268 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:52.268 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.627204+0000", 2024-09-21T22:07:52.268 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3069999999999999e-06 2024-09-21T22:07:52.268 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.269 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.269 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:52.269 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.628114+0000", 2024-09-21T22:07:52.269 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00090942099999999999 2024-09-21T22:07:52.269 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.269 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.269 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:52.269 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.628140+0000", 2024-09-21T22:07:52.269 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5882000000000001e-05 2024-09-21T22:07:52.269 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.269 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.269 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:52.269 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.628162+0000", 2024-09-21T22:07:52.269 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.2293000000000002e-05 2024-09-21T22:07:52.270 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.270 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.270 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:52.270 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.628250+0000", 2024-09-21T22:07:52.270 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.7973000000000006e-05 2024-09-21T22:07:52.270 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.270 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.270 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:52.270 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.628272+0000", 2024-09-21T22:07:52.270 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1682999999999999e-05 2024-09-21T22:07:52.270 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.270 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.270 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:52.271 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.665322+0000", 2024-09-21T22:07:52.271 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.037050650999999997 2024-09-21T22:07:52.271 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.271 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.271 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:52.271 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.665407+0000", 2024-09-21T22:07:52.271 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.4761999999999999e-05 2024-09-21T22:07:52.271 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.271 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:52.271 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.271 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.271 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.271 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:149 2.5 e72/37 2:a70ae99f:::rbd_data.126483d75f2d.000000000000000c:head v 72'56, mlcod=72'54)", 2024-09-21T22:07:52.271 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.690377+0000", 2024-09-21T22:07:52.271 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 13.517641088, 2024-09-21T22:07:52.272 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.049254799000000002, 2024-09-21T22:07:52.272 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:52.272 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:52.272 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:52.272 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:52.272 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.272 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:52.272 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.690377+0000", 2024-09-21T22:07:52.272 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:52.272 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.272 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.272 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:52.272 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.690377+0000", 2024-09-21T22:07:52.272 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:52.273 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.273 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.273 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:52.273 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.690378+0000", 2024-09-21T22:07:52.273 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.344e-06 2024-09-21T22:07:52.273 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.273 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.273 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:52.273 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.691302+0000", 2024-09-21T22:07:52.273 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00092322400000000005 2024-09-21T22:07:52.273 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.273 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.273 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:52.273 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.691321+0000", 2024-09-21T22:07:52.274 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9928999999999998e-05 2024-09-21T22:07:52.274 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.274 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.274 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:52.274 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.691345+0000", 2024-09-21T22:07:52.274 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.372e-05 2024-09-21T22:07:52.274 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.274 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.274 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:52.274 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.691442+0000", 2024-09-21T22:07:52.274 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.7270999999999998e-05 2024-09-21T22:07:52.274 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.274 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.274 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:52.275 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.691512+0000", 2024-09-21T22:07:52.275 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.9183000000000005e-05 2024-09-21T22:07:52.275 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.275 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.275 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:52.275 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.739545+0000", 2024-09-21T22:07:52.275 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.048033657 2024-09-21T22:07:52.275 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.275 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.275 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:52.275 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.739632+0000", 2024-09-21T22:07:52.275 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.6471000000000006e-05 2024-09-21T22:07:52.275 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.275 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:52.275 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.276 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.276 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.276 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:156 2.1 e72/53 2:837ddd83:::rbd_data.126483d75f2d.000000000000000d:head v 72'28, mlcod=72'26)", 2024-09-21T22:07:52.276 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.824856+0000", 2024-09-21T22:07:52.276 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 13.38316165, 2024-09-21T22:07:52.276 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.034320985999999998, 2024-09-21T22:07:52.276 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:52.276 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:52.276 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:52.276 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:52.276 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.276 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:52.276 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.824856+0000", 2024-09-21T22:07:52.276 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:52.277 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.277 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.277 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:52.277 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.824856+0000", 2024-09-21T22:07:52.277 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:52.277 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.277 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.277 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:52.277 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.824858+0000", 2024-09-21T22:07:52.277 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2979999999999999e-06 2024-09-21T22:07:52.277 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.277 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.277 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:52.277 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.825174+0000", 2024-09-21T22:07:52.277 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000316335 2024-09-21T22:07:52.278 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.278 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.278 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:52.278 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.825193+0000", 2024-09-21T22:07:52.278 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9436000000000002e-05 2024-09-21T22:07:52.278 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.278 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.278 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:52.278 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.825202+0000", 2024-09-21T22:07:52.278 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.8359999999999998e-06 2024-09-21T22:07:52.278 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.278 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.278 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:52.278 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.825286+0000", 2024-09-21T22:07:52.279 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.3683999999999994e-05 2024-09-21T22:07:52.279 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.279 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.279 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:52.279 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.825314+0000", 2024-09-21T22:07:52.279 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.7653000000000001e-05 2024-09-21T22:07:52.279 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.279 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.279 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:52.279 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.859122+0000", 2024-09-21T22:07:52.279 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.033808624000000002 2024-09-21T22:07:52.279 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.279 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.279 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:52.280 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.859177+0000", 2024-09-21T22:07:52.280 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.5120000000000001e-05 2024-09-21T22:07:52.280 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.280 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:52.280 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.280 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.280 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.280 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:158 2.1 e72/53 2:837ddd83:::rbd_data.126483d75f2d.000000000000000d:head v 72'30, mlcod=72'28)", 2024-09-21T22:07:52.280 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.869277+0000", 2024-09-21T22:07:52.280 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 13.338741453000001, 2024-09-21T22:07:52.280 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.040508390999999998, 2024-09-21T22:07:52.280 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:52.280 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:52.280 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:52.281 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:52.281 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.281 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:52.281 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.869277+0000", 2024-09-21T22:07:52.281 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:52.281 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.281 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.281 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:52.281 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.869277+0000", 2024-09-21T22:07:52.281 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:52.281 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.281 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.281 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:52.281 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.869278+0000", 2024-09-21T22:07:52.282 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5269999999999999e-06 2024-09-21T22:07:52.282 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.282 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.282 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:52.282 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.869902+0000", 2024-09-21T22:07:52.282 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00062399000000000005 2024-09-21T22:07:52.282 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.282 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.282 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:52.282 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.869924+0000", 2024-09-21T22:07:52.282 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1875999999999999e-05 2024-09-21T22:07:52.282 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.282 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.283 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:52.283 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.869937+0000", 2024-09-21T22:07:52.283 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2685e-05 2024-09-21T22:07:52.283 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.283 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.283 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:52.283 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.870058+0000", 2024-09-21T22:07:52.283 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000120862 2024-09-21T22:07:52.283 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.283 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.283 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:52.283 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.870093+0000", 2024-09-21T22:07:52.283 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.5896000000000003e-05 2024-09-21T22:07:52.283 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.284 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.284 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:52.284 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.909728+0000", 2024-09-21T22:07:52.284 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.039634461000000003 2024-09-21T22:07:52.284 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.284 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.284 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:52.284 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.909785+0000", 2024-09-21T22:07:52.284 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.7093999999999999e-05 2024-09-21T22:07:52.284 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.284 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:52.284 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.284 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.285 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.285 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:161 2.1 e72/53 2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head v 72'32, mlcod=72'30)", 2024-09-21T22:07:52.285 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.924455+0000", 2024-09-21T22:07:52.285 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 13.283563091, 2024-09-21T22:07:52.285 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.038366863000000001, 2024-09-21T22:07:52.285 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:52.285 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:52.285 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:52.285 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:52.285 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.285 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:52.285 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.924455+0000", 2024-09-21T22:07:52.285 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:52.285 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.286 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.286 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:52.286 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.924455+0000", 2024-09-21T22:07:52.286 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:52.286 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.286 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.286 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:52.286 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.924456+0000", 2024-09-21T22:07:52.286 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.003e-06 2024-09-21T22:07:52.286 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.286 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.286 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:52.286 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.924811+0000", 2024-09-21T22:07:52.287 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00035519499999999999 2024-09-21T22:07:52.287 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.287 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.287 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:52.287 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.924826+0000", 2024-09-21T22:07:52.287 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5296999999999999e-05 2024-09-21T22:07:52.287 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.287 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.287 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:52.287 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.924834+0000", 2024-09-21T22:07:52.287 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.5379999999999997e-06 2024-09-21T22:07:52.287 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.288 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.288 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:52.288 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.924910+0000", 2024-09-21T22:07:52.288 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.6488000000000004e-05 2024-09-21T22:07:52.288 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.288 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.288 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:52.288 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.924934+0000", 2024-09-21T22:07:52.288 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3598999999999999e-05 2024-09-21T22:07:52.288 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.288 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.288 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:52.288 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.962739+0000", 2024-09-21T22:07:52.289 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.037805370999999997 2024-09-21T22:07:52.289 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.289 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.289 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:52.289 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.962822+0000", 2024-09-21T22:07:52.289 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.2372000000000003e-05 2024-09-21T22:07:52.289 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.289 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:52.289 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.289 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.289 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.289 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:163 2.1 e72/53 2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head v 72'34, mlcod=72'32)", 2024-09-21T22:07:52.289 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.971683+0000", 2024-09-21T22:07:52.290 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 13.236334551000001, 2024-09-21T22:07:52.290 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.043171320999999999, 2024-09-21T22:07:52.290 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:52.290 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:52.290 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:52.290 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:52.290 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.290 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:52.290 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.971683+0000", 2024-09-21T22:07:52.290 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:52.290 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.290 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.290 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:52.291 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.971683+0000", 2024-09-21T22:07:52.291 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:52.291 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.291 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.291 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:52.291 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.971684+0000", 2024-09-21T22:07:52.291 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.6600000000000005e-07 2024-09-21T22:07:52.291 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.291 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.291 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:52.291 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.971945+0000", 2024-09-21T22:07:52.291 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00026101899999999998 2024-09-21T22:07:52.291 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.292 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.292 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:52.292 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.971959+0000", 2024-09-21T22:07:52.292 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4054e-05 2024-09-21T22:07:52.292 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.292 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.292 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:52.292 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.971969+0000", 2024-09-21T22:07:52.292 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.499e-06 2024-09-21T22:07:52.292 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.292 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.292 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:52.292 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.972037+0000", 2024-09-21T22:07:52.292 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.8448000000000006e-05 2024-09-21T22:07:52.293 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.293 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.293 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:52.293 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.972057+0000", 2024-09-21T22:07:52.293 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9817999999999999e-05 2024-09-21T22:07:52.293 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.293 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.293 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:52.293 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.014770+0000", 2024-09-21T22:07:52.293 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.042712464999999998 2024-09-21T22:07:52.293 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.293 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.293 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:52.293 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.014855+0000", 2024-09-21T22:07:52.294 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.5152000000000001e-05 2024-09-21T22:07:52.294 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.294 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:52.294 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.294 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.294 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.294 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:165 2.1 e72/53 2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head v 72'36, mlcod=72'34)", 2024-09-21T22:07:52.294 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:39.028631+0000", 2024-09-21T22:07:52.294 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 13.179387073999999, 2024-09-21T22:07:52.294 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.033430561999999997, 2024-09-21T22:07:52.294 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:52.294 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:52.294 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:52.294 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:52.295 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.295 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:52.295 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.028631+0000", 2024-09-21T22:07:52.295 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:52.295 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.295 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.295 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:52.295 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.028631+0000", 2024-09-21T22:07:52.295 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:52.295 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.295 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.295 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:52.295 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.028633+0000", 2024-09-21T22:07:52.295 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5799999999999999e-06 2024-09-21T22:07:52.296 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.296 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.296 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:52.296 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.029209+0000", 2024-09-21T22:07:52.296 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00057670799999999999 2024-09-21T22:07:52.296 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.296 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.296 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:52.296 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.029238+0000", 2024-09-21T22:07:52.296 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8943000000000001e-05 2024-09-21T22:07:52.296 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.296 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.296 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:52.296 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.029252+0000", 2024-09-21T22:07:52.297 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3672999999999999e-05 2024-09-21T22:07:52.297 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.297 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.297 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:52.297 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.029328+0000", 2024-09-21T22:07:52.297 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.6497999999999998e-05 2024-09-21T22:07:52.297 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.297 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.297 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:52.297 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.029353+0000", 2024-09-21T22:07:52.297 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5060999999999999e-05 2024-09-21T22:07:52.297 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.297 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.297 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:52.298 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.061993+0000", 2024-09-21T22:07:52.298 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.032639286000000003 2024-09-21T22:07:52.298 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.298 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.298 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:52.298 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.062062+0000", 2024-09-21T22:07:52.298 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.8813000000000007e-05 2024-09-21T22:07:52.298 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.298 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:52.298 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.298 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.298 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.298 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:167 2.1 e72/53 2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head v 72'38, mlcod=72'36)", 2024-09-21T22:07:52.299 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:39.077324+0000", 2024-09-21T22:07:52.299 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 13.13069426, 2024-09-21T22:07:52.299 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.036031961000000001, 2024-09-21T22:07:52.299 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:52.299 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:52.299 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:52.299 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:52.299 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.299 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:52.299 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.077324+0000", 2024-09-21T22:07:52.299 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:52.299 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.299 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.300 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:52.300 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.077324+0000", 2024-09-21T22:07:52.300 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:52.300 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.300 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.300 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:52.300 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.077326+0000", 2024-09-21T22:07:52.300 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.75e-06 2024-09-21T22:07:52.300 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.300 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.300 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:52.300 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.077719+0000", 2024-09-21T22:07:52.300 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00039322599999999999 2024-09-21T22:07:52.300 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.301 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.301 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:52.301 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.077749+0000", 2024-09-21T22:07:52.301 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9895000000000001e-05 2024-09-21T22:07:52.301 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.301 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.301 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:52.301 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.077763+0000", 2024-09-21T22:07:52.301 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4517e-05 2024-09-21T22:07:52.301 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.301 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.301 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:52.301 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.077843+0000", 2024-09-21T22:07:52.301 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.9994999999999996e-05 2024-09-21T22:07:52.302 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.302 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.302 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:52.302 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.077867+0000", 2024-09-21T22:07:52.302 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3640000000000001e-05 2024-09-21T22:07:52.302 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.302 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.302 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:52.302 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.113216+0000", 2024-09-21T22:07:52.302 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.035349168 2024-09-21T22:07:52.302 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.302 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.302 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:52.302 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.113356+0000", 2024-09-21T22:07:52.303 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013977 2024-09-21T22:07:52.303 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.303 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:52.303 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.303 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.303 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:52.303 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:07:52.303 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 5, 0] out_osds: [2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-09-21T22:07:52.304 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T22:07:52.304 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-21T22:07:52.304 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-09-21T22:07:52.305 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:07:52.300+0000 7f2075a06640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31720) UID: 0 2024-09-21T22:07:52.306 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T22:07:52.307 INFO:tasks.workunit.client.0.smithi157.stderr:+ return 0 2024-09-21T22:07:52.307 INFO:tasks.workunit.client.0.smithi157.stderr:+ QUIESCE_HOOK=/tmp/tmp.GqKuv7rsZi/quiesce.sh 2024-09-21T22:07:52.307 INFO:tasks.workunit.client.0.smithi157.stderr:++ _sudo rbd device --device-type nbd map --quiesce --quiesce-hook /tmp/tmp.GqKuv7rsZi/quiesce.sh rbd/testrbdnbd21553 2024-09-21T22:07:52.307 INFO:tasks.workunit.client.0.smithi157.stderr:++ local cmd 2024-09-21T22:07:52.307 INFO:tasks.workunit.client.0.smithi157.stderr:+++ id -u 2024-09-21T22:07:52.308 INFO:tasks.workunit.client.0.smithi157.stderr:++ '[' 1000 -eq 0 ']' 2024-09-21T22:07:52.308 INFO:tasks.workunit.client.0.smithi157.stderr:+++ which rbd 2024-09-21T22:07:52.308 INFO:tasks.workunit.client.0.smithi157.stderr:++ cmd=/usr/bin/rbd 2024-09-21T22:07:52.308 INFO:tasks.workunit.client.0.smithi157.stderr:++ shift 2024-09-21T22:07:52.308 INFO:tasks.workunit.client.0.smithi157.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map --quiesce --quiesce-hook /tmp/tmp.GqKuv7rsZi/quiesce.sh rbd/testrbdnbd21553 2024-09-21T22:07:52.374 INFO:tasks.workunit.client.0.smithi157.stderr:+ DEV=/dev/nbd1 2024-09-21T22:07:52.374 INFO:tasks.workunit.client.0.smithi157.stderr:+ get_pid rbd 2024-09-21T22:07:52.374 INFO:tasks.workunit.client.0.smithi157.stderr:+ local pool=rbd 2024-09-21T22:07:52.374 INFO:tasks.workunit.client.0.smithi157.stderr:+ local ns= 2024-09-21T22:07:52.374 INFO:tasks.workunit.client.0.smithi157.stderr:++ rbd device --device-type nbd --format xml list 2024-09-21T22:07:52.374 INFO:tasks.workunit.client.0.smithi157.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd21553'\''][device='\''/dev/nbd1'\'']/id' 2024-09-21T22:07:52.405 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:52.402+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:52.466 INFO:tasks.workunit.client.0.smithi157.stderr:+ PID=23672 2024-09-21T22:07:52.467 INFO:tasks.workunit.client.0.smithi157.stderr:+ test -n 23672 2024-09-21T22:07:52.467 INFO:tasks.workunit.client.0.smithi157.stderr:+ ps -p 23672 -C rbd-nbd 2024-09-21T22:07:52.471 INFO:tasks.workunit.client.0.smithi157.stdout: PID TTY TIME CMD 2024-09-21T22:07:52.471 INFO:tasks.workunit.client.0.smithi157.stdout: 23672 ? 00:00:00 rbd-nbd 2024-09-21T22:07:52.472 INFO:tasks.workunit.client.0.smithi157.stderr:+ test '!' -e /tmp/tmp.GqKuv7rsZi/quiesce.sh 2024-09-21T22:07:52.472 INFO:tasks.workunit.client.0.smithi157.stderr:+ expect_false rbd snap create rbd/testrbdnbd21553@quiesce1 2024-09-21T22:07:52.472 INFO:tasks.workunit.client.0.smithi157.stderr:+ rbd snap create rbd/testrbdnbd21553@quiesce1 2024-09-21T22:07:52.474 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:07:52.474 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:07:52.474 INFO:teuthology.orchestra.run.smithi039.stdout: "num_ops": 0 2024-09-21T22:07:52.474 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:07:52.489 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T22:07:52.532 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:52.530+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:52.547 INFO:tasks.workunit.client.0.smithi157.stderr: Creating snap: 10% complete...2024-09-21T22:07:52.541+0000 7efcbaf5e640 -1 librbd::ImageWatcher: 0x7efc98007590 failed to notify quiesce: (1) Operation not permitted 2024-09-21T22:07:52.547 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-21T22:07:52.541+0000 7efcbaf5e640 -1 librbd::SnapshotCreateRequest: failed to notify quiesce: (1) Operation not permitted 2024-09-21T22:07:52.548 INFO:tasks.workunit.client.0.smithi157.stderr: Creating snap: 10% complete...failed. 2024-09-21T22:07:52.548 INFO:tasks.workunit.client.0.smithi157.stderr:rbd: failed to create snapshot: (1) Operation not permitted 2024-09-21T22:07:52.562 INFO:tasks.workunit.client.0.smithi157.stderr:+ return 0 2024-09-21T22:07:52.562 INFO:tasks.workunit.client.0.smithi157.stderr:+ _sudo dd if=/tmp/tmp.GqKuv7rsZi/data of=/dev/nbd1 bs=1M count=1 oflag=direct 2024-09-21T22:07:52.563 INFO:tasks.workunit.client.0.smithi157.stderr:+ local cmd 2024-09-21T22:07:52.563 INFO:tasks.workunit.client.0.smithi157.stderr:++ id -u 2024-09-21T22:07:52.564 INFO:tasks.workunit.client.0.smithi157.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T22:07:52.564 INFO:tasks.workunit.client.0.smithi157.stderr:++ which dd 2024-09-21T22:07:52.565 INFO:tasks.workunit.client.0.smithi157.stderr:+ cmd=/usr/bin/dd 2024-09-21T22:07:52.566 INFO:tasks.workunit.client.0.smithi157.stderr:+ shift 2024-09-21T22:07:52.566 INFO:tasks.workunit.client.0.smithi157.stderr:+ sudo -nE /usr/bin/dd if=/tmp/tmp.GqKuv7rsZi/data of=/dev/nbd1 bs=1M count=1 oflag=direct 2024-09-21T22:07:52.607 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:52.602+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:52.649 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:07:52.649 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:07:52.649 INFO:teuthology.orchestra.run.smithi039.stdout: "complaint_time": 30, 2024-09-21T22:07:52.650 INFO:teuthology.orchestra.run.smithi039.stdout: "num_blocked_ops": 0 2024-09-21T22:07:52.650 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:07:52.663 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T22:07:52.670 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T22:07:52.670 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:07:52.670 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 78, 2024-09-21T22:07:52.670 INFO:teuthology.orchestra.run.smithi039.stdout: "fsid": "00bb09a3-e410-4e18-a2b2-08eaf61978e6", 2024-09-21T22:07:52.670 INFO:teuthology.orchestra.run.smithi039.stdout: "created": "2024-09-21T22:05:45.303341+0000", 2024-09-21T22:07:52.670 INFO:teuthology.orchestra.run.smithi039.stdout: "modified": "2024-09-21T22:07:49.823464+0000", 2024-09-21T22:07:52.670 INFO:teuthology.orchestra.run.smithi039.stdout: "last_up_change": "2024-09-21T22:07:27.209405+0000", 2024-09-21T22:07:52.670 INFO:teuthology.orchestra.run.smithi039.stdout: "last_in_change": "2024-09-21T22:07:10.044634+0000", 2024-09-21T22:07:52.670 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-21T22:07:52.670 INFO:teuthology.orchestra.run.smithi039.stdout: "flags_num": 5801984, 2024-09-21T22:07:52.670 INFO:teuthology.orchestra.run.smithi039.stdout: "flags_set": [ 2024-09-21T22:07:52.670 INFO:teuthology.orchestra.run.smithi039.stdout: "noscrub", 2024-09-21T22:07:52.671 INFO:teuthology.orchestra.run.smithi039.stdout: "pglog_hardlimit", 2024-09-21T22:07:52.671 INFO:teuthology.orchestra.run.smithi039.stdout: "purged_snapdirs", 2024-09-21T22:07:52.671 INFO:teuthology.orchestra.run.smithi039.stdout: "recovery_deletes", 2024-09-21T22:07:52.671 INFO:teuthology.orchestra.run.smithi039.stdout: "sortbitwise" 2024-09-21T22:07:52.671 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T22:07:52.671 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_version": 5, 2024-09-21T22:07:52.671 INFO:teuthology.orchestra.run.smithi039.stdout: "full_ratio": 0.94999998807907104, 2024-09-21T22:07:52.671 INFO:teuthology.orchestra.run.smithi039.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-21T22:07:52.671 INFO:teuthology.orchestra.run.smithi039.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-21T22:07:52.671 INFO:teuthology.orchestra.run.smithi039.stdout: "cluster_snapshot": "", 2024-09-21T22:07:52.671 INFO:teuthology.orchestra.run.smithi039.stdout: "pool_max": 3, 2024-09-21T22:07:52.671 INFO:teuthology.orchestra.run.smithi039.stdout: "max_osd": 6, 2024-09-21T22:07:52.671 INFO:teuthology.orchestra.run.smithi039.stdout: "require_min_compat_client": "luminous", 2024-09-21T22:07:52.671 INFO:teuthology.orchestra.run.smithi039.stdout: "min_compat_client": "luminous", 2024-09-21T22:07:52.672 INFO:teuthology.orchestra.run.smithi039.stdout: "require_osd_release": "squid", 2024-09-21T22:07:52.672 INFO:teuthology.orchestra.run.smithi039.stdout: "allow_crimson": false, 2024-09-21T22:07:52.672 INFO:teuthology.orchestra.run.smithi039.stdout: "pools": [ 2024-09-21T22:07:52.672 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.672 INFO:teuthology.orchestra.run.smithi039.stdout: "pool": 1, 2024-09-21T22:07:52.672 INFO:teuthology.orchestra.run.smithi039.stdout: "pool_name": ".mgr", 2024-09-21T22:07:52.672 INFO:teuthology.orchestra.run.smithi039.stdout: "create_time": "2024-09-21T22:06:00.868591+0000", 2024-09-21T22:07:52.672 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-09-21T22:07:52.672 INFO:teuthology.orchestra.run.smithi039.stdout: "flags_names": "hashpspool", 2024-09-21T22:07:52.672 INFO:teuthology.orchestra.run.smithi039.stdout: "type": 1, 2024-09-21T22:07:52.672 INFO:teuthology.orchestra.run.smithi039.stdout: "size": 2, 2024-09-21T22:07:52.672 INFO:teuthology.orchestra.run.smithi039.stdout: "min_size": 1, 2024-09-21T22:07:52.672 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_rule": 0, 2024-09-21T22:07:52.672 INFO:teuthology.orchestra.run.smithi039.stdout: "peering_crush_bucket_count": 0, 2024-09-21T22:07:52.673 INFO:teuthology.orchestra.run.smithi039.stdout: "peering_crush_bucket_target": 0, 2024-09-21T22:07:52.673 INFO:teuthology.orchestra.run.smithi039.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T22:07:52.673 INFO:teuthology.orchestra.run.smithi039.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T22:07:52.673 INFO:teuthology.orchestra.run.smithi039.stdout: "is_stretch_pool": false, 2024-09-21T22:07:52.673 INFO:teuthology.orchestra.run.smithi039.stdout: "object_hash": 2, 2024-09-21T22:07:52.673 INFO:teuthology.orchestra.run.smithi039.stdout: "pg_autoscale_mode": "off", 2024-09-21T22:07:52.673 INFO:teuthology.orchestra.run.smithi039.stdout: "pg_num": 1, 2024-09-21T22:07:52.673 INFO:teuthology.orchestra.run.smithi039.stdout: "pg_placement_num": 1, 2024-09-21T22:07:52.673 INFO:teuthology.orchestra.run.smithi039.stdout: "pg_placement_num_target": 1, 2024-09-21T22:07:52.673 INFO:teuthology.orchestra.run.smithi039.stdout: "pg_num_target": 1, 2024-09-21T22:07:52.673 INFO:teuthology.orchestra.run.smithi039.stdout: "pg_num_pending": 1, 2024-09-21T22:07:52.673 INFO:teuthology.orchestra.run.smithi039.stdout: "last_pg_merge_meta": { 2024-09-21T22:07:52.673 INFO:teuthology.orchestra.run.smithi039.stdout: "source_pgid": "0.0", 2024-09-21T22:07:52.674 INFO:teuthology.orchestra.run.smithi039.stdout: "ready_epoch": 0, 2024-09-21T22:07:52.674 INFO:teuthology.orchestra.run.smithi039.stdout: "last_epoch_started": 0, 2024-09-21T22:07:52.674 INFO:teuthology.orchestra.run.smithi039.stdout: "last_epoch_clean": 0, 2024-09-21T22:07:52.674 INFO:teuthology.orchestra.run.smithi039.stdout: "source_version": "0'0", 2024-09-21T22:07:52.674 INFO:teuthology.orchestra.run.smithi039.stdout: "target_version": "0'0" 2024-09-21T22:07:52.674 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.674 INFO:teuthology.orchestra.run.smithi039.stdout: "last_change": "15", 2024-09-21T22:07:52.674 INFO:teuthology.orchestra.run.smithi039.stdout: "last_force_op_resend": "0", 2024-09-21T22:07:52.674 INFO:teuthology.orchestra.run.smithi039.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T22:07:52.674 INFO:teuthology.orchestra.run.smithi039.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T22:07:52.674 INFO:teuthology.orchestra.run.smithi039.stdout: "auid": 0, 2024-09-21T22:07:52.674 INFO:teuthology.orchestra.run.smithi039.stdout: "snap_mode": "selfmanaged", 2024-09-21T22:07:52.674 INFO:teuthology.orchestra.run.smithi039.stdout: "snap_seq": 0, 2024-09-21T22:07:52.674 INFO:teuthology.orchestra.run.smithi039.stdout: "snap_epoch": 0, 2024-09-21T22:07:52.675 INFO:teuthology.orchestra.run.smithi039.stdout: "pool_snaps": [], 2024-09-21T22:07:52.675 INFO:teuthology.orchestra.run.smithi039.stdout: "removed_snaps": "[]", 2024-09-21T22:07:52.675 INFO:teuthology.orchestra.run.smithi039.stdout: "quota_max_bytes": 0, 2024-09-21T22:07:52.675 INFO:teuthology.orchestra.run.smithi039.stdout: "quota_max_objects": 0, 2024-09-21T22:07:52.675 INFO:teuthology.orchestra.run.smithi039.stdout: "tiers": [], 2024-09-21T22:07:52.675 INFO:teuthology.orchestra.run.smithi039.stdout: "tier_of": -1, 2024-09-21T22:07:52.675 INFO:teuthology.orchestra.run.smithi039.stdout: "read_tier": -1, 2024-09-21T22:07:52.675 INFO:teuthology.orchestra.run.smithi039.stdout: "write_tier": -1, 2024-09-21T22:07:52.675 INFO:teuthology.orchestra.run.smithi039.stdout: "cache_mode": "none", 2024-09-21T22:07:52.675 INFO:teuthology.orchestra.run.smithi039.stdout: "target_max_bytes": 0, 2024-09-21T22:07:52.675 INFO:teuthology.orchestra.run.smithi039.stdout: "target_max_objects": 0, 2024-09-21T22:07:52.675 INFO:teuthology.orchestra.run.smithi039.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T22:07:52.675 INFO:teuthology.orchestra.run.smithi039.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T22:07:52.676 INFO:teuthology.orchestra.run.smithi039.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T22:07:52.676 INFO:teuthology.orchestra.run.smithi039.stdout: "cache_min_flush_age": 0, 2024-09-21T22:07:52.676 INFO:teuthology.orchestra.run.smithi039.stdout: "cache_min_evict_age": 0, 2024-09-21T22:07:52.676 INFO:teuthology.orchestra.run.smithi039.stdout: "erasure_code_profile": "", 2024-09-21T22:07:52.676 INFO:teuthology.orchestra.run.smithi039.stdout: "hit_set_params": { 2024-09-21T22:07:52.676 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "none" 2024-09-21T22:07:52.676 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.676 INFO:teuthology.orchestra.run.smithi039.stdout: "hit_set_period": 0, 2024-09-21T22:07:52.676 INFO:teuthology.orchestra.run.smithi039.stdout: "hit_set_count": 0, 2024-09-21T22:07:52.676 INFO:teuthology.orchestra.run.smithi039.stdout: "use_gmt_hitset": true, 2024-09-21T22:07:52.676 INFO:teuthology.orchestra.run.smithi039.stdout: "min_read_recency_for_promote": 0, 2024-09-21T22:07:52.676 INFO:teuthology.orchestra.run.smithi039.stdout: "min_write_recency_for_promote": 0, 2024-09-21T22:07:52.676 INFO:teuthology.orchestra.run.smithi039.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T22:07:52.676 INFO:teuthology.orchestra.run.smithi039.stdout: "hit_set_search_last_n": 0, 2024-09-21T22:07:52.677 INFO:teuthology.orchestra.run.smithi039.stdout: "grade_table": [], 2024-09-21T22:07:52.677 INFO:teuthology.orchestra.run.smithi039.stdout: "stripe_width": 0, 2024-09-21T22:07:52.677 INFO:teuthology.orchestra.run.smithi039.stdout: "expected_num_objects": 0, 2024-09-21T22:07:52.677 INFO:teuthology.orchestra.run.smithi039.stdout: "fast_read": false, 2024-09-21T22:07:52.677 INFO:teuthology.orchestra.run.smithi039.stdout: "options": { 2024-09-21T22:07:52.677 INFO:teuthology.orchestra.run.smithi039.stdout: "pg_num_max": 32, 2024-09-21T22:07:52.677 INFO:teuthology.orchestra.run.smithi039.stdout: "pg_num_min": 1 2024-09-21T22:07:52.677 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.677 INFO:teuthology.orchestra.run.smithi039.stdout: "application_metadata": { 2024-09-21T22:07:52.677 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr": {} 2024-09-21T22:07:52.677 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.677 INFO:teuthology.orchestra.run.smithi039.stdout: "read_balance": { 2024-09-21T22:07:52.678 INFO:teuthology.orchestra.run.smithi039.stdout: "score_type": "Fair distribution", 2024-09-21T22:07:52.678 INFO:teuthology.orchestra.run.smithi039.stdout: "score_acting": 6.059999942779541, 2024-09-21T22:07:52.678 INFO:teuthology.orchestra.run.smithi039.stdout: "score_stable": 6.059999942779541, 2024-09-21T22:07:52.678 INFO:teuthology.orchestra.run.smithi039.stdout: "optimal_score": 0.33000001311302185, 2024-09-21T22:07:52.678 INFO:teuthology.orchestra.run.smithi039.stdout: "raw_score_acting": 2, 2024-09-21T22:07:52.678 INFO:teuthology.orchestra.run.smithi039.stdout: "raw_score_stable": 2, 2024-09-21T22:07:52.678 INFO:teuthology.orchestra.run.smithi039.stdout: "primary_affinity_weighted": 1, 2024-09-21T22:07:52.678 INFO:teuthology.orchestra.run.smithi039.stdout: "average_primary_affinity": 1, 2024-09-21T22:07:52.678 INFO:teuthology.orchestra.run.smithi039.stdout: "average_primary_affinity_weighted": 1 2024-09-21T22:07:52.678 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.678 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.678 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.679 INFO:teuthology.orchestra.run.smithi039.stdout: "pool": 2, 2024-09-21T22:07:52.679 INFO:teuthology.orchestra.run.smithi039.stdout: "pool_name": "rbd", 2024-09-21T22:07:52.679 INFO:teuthology.orchestra.run.smithi039.stdout: "create_time": "2024-09-21T22:06:03.535857+0000", 2024-09-21T22:07:52.679 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 8193, 2024-09-21T22:07:52.679 INFO:teuthology.orchestra.run.smithi039.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-21T22:07:52.679 INFO:teuthology.orchestra.run.smithi039.stdout: "type": 1, 2024-09-21T22:07:52.679 INFO:teuthology.orchestra.run.smithi039.stdout: "size": 2, 2024-09-21T22:07:52.679 INFO:teuthology.orchestra.run.smithi039.stdout: "min_size": 1, 2024-09-21T22:07:52.679 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_rule": 0, 2024-09-21T22:07:52.679 INFO:teuthology.orchestra.run.smithi039.stdout: "peering_crush_bucket_count": 0, 2024-09-21T22:07:52.679 INFO:teuthology.orchestra.run.smithi039.stdout: "peering_crush_bucket_target": 0, 2024-09-21T22:07:52.679 INFO:teuthology.orchestra.run.smithi039.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T22:07:52.679 INFO:teuthology.orchestra.run.smithi039.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T22:07:52.679 INFO:teuthology.orchestra.run.smithi039.stdout: "is_stretch_pool": false, 2024-09-21T22:07:52.680 INFO:teuthology.orchestra.run.smithi039.stdout: "object_hash": 2, 2024-09-21T22:07:52.680 INFO:teuthology.orchestra.run.smithi039.stdout: "pg_autoscale_mode": "off", 2024-09-21T22:07:52.680 INFO:teuthology.orchestra.run.smithi039.stdout: "pg_num": 8, 2024-09-21T22:07:52.680 INFO:teuthology.orchestra.run.smithi039.stdout: "pg_placement_num": 8, 2024-09-21T22:07:52.680 INFO:teuthology.orchestra.run.smithi039.stdout: "pg_placement_num_target": 8, 2024-09-21T22:07:52.680 INFO:teuthology.orchestra.run.smithi039.stdout: "pg_num_target": 8, 2024-09-21T22:07:52.680 INFO:teuthology.orchestra.run.smithi039.stdout: "pg_num_pending": 8, 2024-09-21T22:07:52.680 INFO:teuthology.orchestra.run.smithi039.stdout: "last_pg_merge_meta": { 2024-09-21T22:07:52.680 INFO:teuthology.orchestra.run.smithi039.stdout: "source_pgid": "0.0", 2024-09-21T22:07:52.680 INFO:teuthology.orchestra.run.smithi039.stdout: "ready_epoch": 0, 2024-09-21T22:07:52.680 INFO:teuthology.orchestra.run.smithi039.stdout: "last_epoch_started": 0, 2024-09-21T22:07:52.680 INFO:teuthology.orchestra.run.smithi039.stdout: "last_epoch_clean": 0, 2024-09-21T22:07:52.680 INFO:teuthology.orchestra.run.smithi039.stdout: "source_version": "0'0", 2024-09-21T22:07:52.680 INFO:teuthology.orchestra.run.smithi039.stdout: "target_version": "0'0" 2024-09-21T22:07:52.681 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.681 INFO:teuthology.orchestra.run.smithi039.stdout: "last_change": "76", 2024-09-21T22:07:52.681 INFO:teuthology.orchestra.run.smithi039.stdout: "last_force_op_resend": "0", 2024-09-21T22:07:52.681 INFO:teuthology.orchestra.run.smithi039.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T22:07:52.681 INFO:teuthology.orchestra.run.smithi039.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T22:07:52.681 INFO:teuthology.orchestra.run.smithi039.stdout: "auid": 0, 2024-09-21T22:07:52.681 INFO:teuthology.orchestra.run.smithi039.stdout: "snap_mode": "selfmanaged", 2024-09-21T22:07:52.681 INFO:teuthology.orchestra.run.smithi039.stdout: "snap_seq": 4, 2024-09-21T22:07:52.681 INFO:teuthology.orchestra.run.smithi039.stdout: "snap_epoch": 76, 2024-09-21T22:07:52.681 INFO:teuthology.orchestra.run.smithi039.stdout: "pool_snaps": [], 2024-09-21T22:07:52.681 INFO:teuthology.orchestra.run.smithi039.stdout: "removed_snaps": "[]", 2024-09-21T22:07:52.681 INFO:teuthology.orchestra.run.smithi039.stdout: "quota_max_bytes": 0, 2024-09-21T22:07:52.681 INFO:teuthology.orchestra.run.smithi039.stdout: "quota_max_objects": 0, 2024-09-21T22:07:52.681 INFO:teuthology.orchestra.run.smithi039.stdout: "tiers": [], 2024-09-21T22:07:52.682 INFO:teuthology.orchestra.run.smithi039.stdout: "tier_of": -1, 2024-09-21T22:07:52.682 INFO:teuthology.orchestra.run.smithi039.stdout: "read_tier": -1, 2024-09-21T22:07:52.682 INFO:teuthology.orchestra.run.smithi039.stdout: "write_tier": -1, 2024-09-21T22:07:52.682 INFO:teuthology.orchestra.run.smithi039.stdout: "cache_mode": "none", 2024-09-21T22:07:52.682 INFO:teuthology.orchestra.run.smithi039.stdout: "target_max_bytes": 0, 2024-09-21T22:07:52.682 INFO:teuthology.orchestra.run.smithi039.stdout: "target_max_objects": 0, 2024-09-21T22:07:52.682 INFO:teuthology.orchestra.run.smithi039.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T22:07:52.682 INFO:teuthology.orchestra.run.smithi039.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T22:07:52.682 INFO:teuthology.orchestra.run.smithi039.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T22:07:52.682 INFO:teuthology.orchestra.run.smithi039.stdout: "cache_min_flush_age": 0, 2024-09-21T22:07:52.682 INFO:teuthology.orchestra.run.smithi039.stdout: "cache_min_evict_age": 0, 2024-09-21T22:07:52.682 INFO:teuthology.orchestra.run.smithi039.stdout: "erasure_code_profile": "", 2024-09-21T22:07:52.682 INFO:teuthology.orchestra.run.smithi039.stdout: "hit_set_params": { 2024-09-21T22:07:52.682 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "none" 2024-09-21T22:07:52.682 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.683 INFO:teuthology.orchestra.run.smithi039.stdout: "hit_set_period": 0, 2024-09-21T22:07:52.683 INFO:teuthology.orchestra.run.smithi039.stdout: "hit_set_count": 0, 2024-09-21T22:07:52.683 INFO:teuthology.orchestra.run.smithi039.stdout: "use_gmt_hitset": true, 2024-09-21T22:07:52.683 INFO:teuthology.orchestra.run.smithi039.stdout: "min_read_recency_for_promote": 0, 2024-09-21T22:07:52.683 INFO:teuthology.orchestra.run.smithi039.stdout: "min_write_recency_for_promote": 0, 2024-09-21T22:07:52.683 INFO:teuthology.orchestra.run.smithi039.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T22:07:52.683 INFO:teuthology.orchestra.run.smithi039.stdout: "hit_set_search_last_n": 0, 2024-09-21T22:07:52.683 INFO:teuthology.orchestra.run.smithi039.stdout: "grade_table": [], 2024-09-21T22:07:52.683 INFO:teuthology.orchestra.run.smithi039.stdout: "stripe_width": 0, 2024-09-21T22:07:52.683 INFO:teuthology.orchestra.run.smithi039.stdout: "expected_num_objects": 0, 2024-09-21T22:07:52.683 INFO:teuthology.orchestra.run.smithi039.stdout: "fast_read": false, 2024-09-21T22:07:52.683 INFO:teuthology.orchestra.run.smithi039.stdout: "options": {}, 2024-09-21T22:07:52.683 INFO:teuthology.orchestra.run.smithi039.stdout: "application_metadata": { 2024-09-21T22:07:52.683 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd": {} 2024-09-21T22:07:52.684 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.684 INFO:teuthology.orchestra.run.smithi039.stdout: "read_balance": { 2024-09-21T22:07:52.684 INFO:teuthology.orchestra.run.smithi039.stdout: "score_type": "Fair distribution", 2024-09-21T22:07:52.684 INFO:teuthology.orchestra.run.smithi039.stdout: "score_acting": 2.2699999809265137, 2024-09-21T22:07:52.684 INFO:teuthology.orchestra.run.smithi039.stdout: "score_stable": 2.2699999809265137, 2024-09-21T22:07:52.684 INFO:teuthology.orchestra.run.smithi039.stdout: "optimal_score": 0.82999998331069946, 2024-09-21T22:07:52.684 INFO:teuthology.orchestra.run.smithi039.stdout: "raw_score_acting": 1.8799999952316284, 2024-09-21T22:07:52.684 INFO:teuthology.orchestra.run.smithi039.stdout: "raw_score_stable": 1.8799999952316284, 2024-09-21T22:07:52.684 INFO:teuthology.orchestra.run.smithi039.stdout: "primary_affinity_weighted": 1, 2024-09-21T22:07:52.684 INFO:teuthology.orchestra.run.smithi039.stdout: "average_primary_affinity": 1, 2024-09-21T22:07:52.684 INFO:teuthology.orchestra.run.smithi039.stdout: "average_primary_affinity_weighted": 1 2024-09-21T22:07:52.684 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.684 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.684 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.684 INFO:teuthology.orchestra.run.smithi039.stdout: "pool": 3, 2024-09-21T22:07:52.685 INFO:teuthology.orchestra.run.smithi039.stdout: "pool_name": "new_default_pool21553", 2024-09-21T22:07:52.685 INFO:teuthology.orchestra.run.smithi039.stdout: "create_time": "2024-09-21T22:07:15.499114+0000", 2024-09-21T22:07:52.685 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 8193, 2024-09-21T22:07:52.685 INFO:teuthology.orchestra.run.smithi039.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-21T22:07:52.685 INFO:teuthology.orchestra.run.smithi039.stdout: "type": 1, 2024-09-21T22:07:52.685 INFO:teuthology.orchestra.run.smithi039.stdout: "size": 2, 2024-09-21T22:07:52.685 INFO:teuthology.orchestra.run.smithi039.stdout: "min_size": 1, 2024-09-21T22:07:52.685 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_rule": 0, 2024-09-21T22:07:52.685 INFO:teuthology.orchestra.run.smithi039.stdout: "peering_crush_bucket_count": 0, 2024-09-21T22:07:52.685 INFO:teuthology.orchestra.run.smithi039.stdout: "peering_crush_bucket_target": 0, 2024-09-21T22:07:52.685 INFO:teuthology.orchestra.run.smithi039.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T22:07:52.685 INFO:teuthology.orchestra.run.smithi039.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T22:07:52.685 INFO:teuthology.orchestra.run.smithi039.stdout: "is_stretch_pool": false, 2024-09-21T22:07:52.685 INFO:teuthology.orchestra.run.smithi039.stdout: "object_hash": 2, 2024-09-21T22:07:52.685 INFO:teuthology.orchestra.run.smithi039.stdout: "pg_autoscale_mode": "off", 2024-09-21T22:07:52.686 INFO:teuthology.orchestra.run.smithi039.stdout: "pg_num": 8, 2024-09-21T22:07:52.686 INFO:teuthology.orchestra.run.smithi039.stdout: "pg_placement_num": 8, 2024-09-21T22:07:52.686 INFO:teuthology.orchestra.run.smithi039.stdout: "pg_placement_num_target": 8, 2024-09-21T22:07:52.686 INFO:teuthology.orchestra.run.smithi039.stdout: "pg_num_target": 8, 2024-09-21T22:07:52.686 INFO:teuthology.orchestra.run.smithi039.stdout: "pg_num_pending": 8, 2024-09-21T22:07:52.686 INFO:teuthology.orchestra.run.smithi039.stdout: "last_pg_merge_meta": { 2024-09-21T22:07:52.686 INFO:teuthology.orchestra.run.smithi039.stdout: "source_pgid": "0.0", 2024-09-21T22:07:52.686 INFO:teuthology.orchestra.run.smithi039.stdout: "ready_epoch": 0, 2024-09-21T22:07:52.686 INFO:teuthology.orchestra.run.smithi039.stdout: "last_epoch_started": 0, 2024-09-21T22:07:52.686 INFO:teuthology.orchestra.run.smithi039.stdout: "last_epoch_clean": 0, 2024-09-21T22:07:52.686 INFO:teuthology.orchestra.run.smithi039.stdout: "source_version": "0'0", 2024-09-21T22:07:52.686 INFO:teuthology.orchestra.run.smithi039.stdout: "target_version": "0'0" 2024-09-21T22:07:52.686 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.686 INFO:teuthology.orchestra.run.smithi039.stdout: "last_change": "61", 2024-09-21T22:07:52.687 INFO:teuthology.orchestra.run.smithi039.stdout: "last_force_op_resend": "0", 2024-09-21T22:07:52.687 INFO:teuthology.orchestra.run.smithi039.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T22:07:52.687 INFO:teuthology.orchestra.run.smithi039.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T22:07:52.687 INFO:teuthology.orchestra.run.smithi039.stdout: "auid": 0, 2024-09-21T22:07:52.687 INFO:teuthology.orchestra.run.smithi039.stdout: "snap_mode": "selfmanaged", 2024-09-21T22:07:52.687 INFO:teuthology.orchestra.run.smithi039.stdout: "snap_seq": 2, 2024-09-21T22:07:52.687 INFO:teuthology.orchestra.run.smithi039.stdout: "snap_epoch": 61, 2024-09-21T22:07:52.687 INFO:teuthology.orchestra.run.smithi039.stdout: "pool_snaps": [], 2024-09-21T22:07:52.687 INFO:teuthology.orchestra.run.smithi039.stdout: "removed_snaps": "[]", 2024-09-21T22:07:52.687 INFO:teuthology.orchestra.run.smithi039.stdout: "quota_max_bytes": 0, 2024-09-21T22:07:52.687 INFO:teuthology.orchestra.run.smithi039.stdout: "quota_max_objects": 0, 2024-09-21T22:07:52.687 INFO:teuthology.orchestra.run.smithi039.stdout: "tiers": [], 2024-09-21T22:07:52.687 INFO:teuthology.orchestra.run.smithi039.stdout: "tier_of": -1, 2024-09-21T22:07:52.687 INFO:teuthology.orchestra.run.smithi039.stdout: "read_tier": -1, 2024-09-21T22:07:52.688 INFO:teuthology.orchestra.run.smithi039.stdout: "write_tier": -1, 2024-09-21T22:07:52.688 INFO:teuthology.orchestra.run.smithi039.stdout: "cache_mode": "none", 2024-09-21T22:07:52.688 INFO:teuthology.orchestra.run.smithi039.stdout: "target_max_bytes": 0, 2024-09-21T22:07:52.688 INFO:teuthology.orchestra.run.smithi039.stdout: "target_max_objects": 0, 2024-09-21T22:07:52.688 INFO:teuthology.orchestra.run.smithi039.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T22:07:52.688 INFO:teuthology.orchestra.run.smithi039.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T22:07:52.688 INFO:teuthology.orchestra.run.smithi039.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T22:07:52.688 INFO:teuthology.orchestra.run.smithi039.stdout: "cache_min_flush_age": 0, 2024-09-21T22:07:52.688 INFO:teuthology.orchestra.run.smithi039.stdout: "cache_min_evict_age": 0, 2024-09-21T22:07:52.688 INFO:teuthology.orchestra.run.smithi039.stdout: "erasure_code_profile": "", 2024-09-21T22:07:52.688 INFO:teuthology.orchestra.run.smithi039.stdout: "hit_set_params": { 2024-09-21T22:07:52.688 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "none" 2024-09-21T22:07:52.688 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.688 INFO:teuthology.orchestra.run.smithi039.stdout: "hit_set_period": 0, 2024-09-21T22:07:52.688 INFO:teuthology.orchestra.run.smithi039.stdout: "hit_set_count": 0, 2024-09-21T22:07:52.689 INFO:teuthology.orchestra.run.smithi039.stdout: "use_gmt_hitset": true, 2024-09-21T22:07:52.689 INFO:teuthology.orchestra.run.smithi039.stdout: "min_read_recency_for_promote": 0, 2024-09-21T22:07:52.689 INFO:teuthology.orchestra.run.smithi039.stdout: "min_write_recency_for_promote": 0, 2024-09-21T22:07:52.689 INFO:teuthology.orchestra.run.smithi039.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T22:07:52.689 INFO:teuthology.orchestra.run.smithi039.stdout: "hit_set_search_last_n": 0, 2024-09-21T22:07:52.689 INFO:teuthology.orchestra.run.smithi039.stdout: "grade_table": [], 2024-09-21T22:07:52.689 INFO:teuthology.orchestra.run.smithi039.stdout: "stripe_width": 0, 2024-09-21T22:07:52.689 INFO:teuthology.orchestra.run.smithi039.stdout: "expected_num_objects": 0, 2024-09-21T22:07:52.689 INFO:teuthology.orchestra.run.smithi039.stdout: "fast_read": false, 2024-09-21T22:07:52.689 INFO:teuthology.orchestra.run.smithi039.stdout: "options": {}, 2024-09-21T22:07:52.689 INFO:teuthology.orchestra.run.smithi039.stdout: "application_metadata": { 2024-09-21T22:07:52.689 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd": {} 2024-09-21T22:07:52.689 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.689 INFO:teuthology.orchestra.run.smithi039.stdout: "read_balance": { 2024-09-21T22:07:52.690 INFO:teuthology.orchestra.run.smithi039.stdout: "score_type": "Fair distribution", 2024-09-21T22:07:52.690 INFO:teuthology.orchestra.run.smithi039.stdout: "score_acting": 3.0099999904632568, 2024-09-21T22:07:52.690 INFO:teuthology.orchestra.run.smithi039.stdout: "score_stable": 3.0099999904632568, 2024-09-21T22:07:52.690 INFO:teuthology.orchestra.run.smithi039.stdout: "optimal_score": 0.82999998331069946, 2024-09-21T22:07:52.690 INFO:teuthology.orchestra.run.smithi039.stdout: "raw_score_acting": 2.5, 2024-09-21T22:07:52.690 INFO:teuthology.orchestra.run.smithi039.stdout: "raw_score_stable": 2.5, 2024-09-21T22:07:52.690 INFO:teuthology.orchestra.run.smithi039.stdout: "primary_affinity_weighted": 1, 2024-09-21T22:07:52.690 INFO:teuthology.orchestra.run.smithi039.stdout: "average_primary_affinity": 1, 2024-09-21T22:07:52.690 INFO:teuthology.orchestra.run.smithi039.stdout: "average_primary_affinity_weighted": 1 2024-09-21T22:07:52.690 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.690 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.690 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T22:07:52.690 INFO:teuthology.orchestra.run.smithi039.stdout: "osds": [ 2024-09-21T22:07:52.690 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.690 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": 0, 2024-09-21T22:07:52.691 INFO:teuthology.orchestra.run.smithi039.stdout: "uuid": "8336c397-2018-4c75-ab31-2e5f6ad564b5", 2024-09-21T22:07:52.691 INFO:teuthology.orchestra.run.smithi039.stdout: "up": 1, 2024-09-21T22:07:52.691 INFO:teuthology.orchestra.run.smithi039.stdout: "in": 1, 2024-09-21T22:07:52.691 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 1, 2024-09-21T22:07:52.691 INFO:teuthology.orchestra.run.smithi039.stdout: "primary_affinity": 1, 2024-09-21T22:07:52.691 INFO:teuthology.orchestra.run.smithi039.stdout: "last_clean_begin": 0, 2024-09-21T22:07:52.691 INFO:teuthology.orchestra.run.smithi039.stdout: "last_clean_end": 0, 2024-09-21T22:07:52.691 INFO:teuthology.orchestra.run.smithi039.stdout: "up_from": 11, 2024-09-21T22:07:52.691 INFO:teuthology.orchestra.run.smithi039.stdout: "up_thru": 58, 2024-09-21T22:07:52.691 INFO:teuthology.orchestra.run.smithi039.stdout: "down_at": 0, 2024-09-21T22:07:52.691 INFO:teuthology.orchestra.run.smithi039.stdout: "lost_at": 0, 2024-09-21T22:07:52.691 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2024-09-21T22:07:52.691 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2024-09-21T22:07:52.691 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.692 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2024-09-21T22:07:52.692 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6800", 2024-09-21T22:07:52.692 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 3054061468 2024-09-21T22:07:52.692 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.692 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.692 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-09-21T22:07:52.692 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6801", 2024-09-21T22:07:52.692 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 3054061468 2024-09-21T22:07:52.692 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.692 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:52.692 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.692 INFO:teuthology.orchestra.run.smithi039.stdout: "cluster_addrs": { 2024-09-21T22:07:52.692 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2024-09-21T22:07:52.692 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.693 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2024-09-21T22:07:52.693 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6802", 2024-09-21T22:07:52.693 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 3054061468 2024-09-21T22:07:52.693 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.693 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.693 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-09-21T22:07:52.693 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6803", 2024-09-21T22:07:52.693 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 3054061468 2024-09-21T22:07:52.693 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.693 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:52.693 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.693 INFO:teuthology.orchestra.run.smithi039.stdout: "heartbeat_back_addrs": { 2024-09-21T22:07:52.693 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2024-09-21T22:07:52.693 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.693 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2024-09-21T22:07:52.694 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6808", 2024-09-21T22:07:52.694 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 3054061468 2024-09-21T22:07:52.694 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.694 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.694 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-09-21T22:07:52.694 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6810", 2024-09-21T22:07:52.694 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 3054061468 2024-09-21T22:07:52.694 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.694 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:52.694 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.694 INFO:teuthology.orchestra.run.smithi039.stdout: "heartbeat_front_addrs": { 2024-09-21T22:07:52.694 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2024-09-21T22:07:52.694 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.694 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2024-09-21T22:07:52.695 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6804", 2024-09-21T22:07:52.695 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 3054061468 2024-09-21T22:07:52.695 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.695 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.695 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-09-21T22:07:52.695 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6806", 2024-09-21T22:07:52.695 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 3054061468 2024-09-21T22:07:52.695 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.695 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:52.695 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.695 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6801/3054061468", 2024-09-21T22:07:52.695 INFO:teuthology.orchestra.run.smithi039.stdout: "cluster_addr": "172.21.15.39:6803/3054061468", 2024-09-21T22:07:52.695 INFO:teuthology.orchestra.run.smithi039.stdout: "heartbeat_back_addr": "172.21.15.39:6810/3054061468", 2024-09-21T22:07:52.695 INFO:teuthology.orchestra.run.smithi039.stdout: "heartbeat_front_addr": "172.21.15.39:6806/3054061468", 2024-09-21T22:07:52.695 INFO:teuthology.orchestra.run.smithi039.stdout: "state": [ 2024-09-21T22:07:52.696 INFO:teuthology.orchestra.run.smithi039.stdout: "exists", 2024-09-21T22:07:52.696 INFO:teuthology.orchestra.run.smithi039.stdout: "up" 2024-09-21T22:07:52.696 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:52.696 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.696 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.696 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": 1, 2024-09-21T22:07:52.696 INFO:teuthology.orchestra.run.smithi039.stdout: "uuid": "bca64af6-7427-4c5d-97a4-a183b1f3a7d9", 2024-09-21T22:07:52.696 INFO:teuthology.orchestra.run.smithi039.stdout: "up": 1, 2024-09-21T22:07:52.696 INFO:teuthology.orchestra.run.smithi039.stdout: "in": 1, 2024-09-21T22:07:52.696 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 1, 2024-09-21T22:07:52.696 INFO:teuthology.orchestra.run.smithi039.stdout: "primary_affinity": 1, 2024-09-21T22:07:52.696 INFO:teuthology.orchestra.run.smithi039.stdout: "last_clean_begin": 0, 2024-09-21T22:07:52.696 INFO:teuthology.orchestra.run.smithi039.stdout: "last_clean_end": 0, 2024-09-21T22:07:52.696 INFO:teuthology.orchestra.run.smithi039.stdout: "up_from": 11, 2024-09-21T22:07:52.696 INFO:teuthology.orchestra.run.smithi039.stdout: "up_thru": 65, 2024-09-21T22:07:52.697 INFO:teuthology.orchestra.run.smithi039.stdout: "down_at": 0, 2024-09-21T22:07:52.697 INFO:teuthology.orchestra.run.smithi039.stdout: "lost_at": 0, 2024-09-21T22:07:52.697 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2024-09-21T22:07:52.697 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2024-09-21T22:07:52.697 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.697 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2024-09-21T22:07:52.697 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6816", 2024-09-21T22:07:52.697 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 2876495139 2024-09-21T22:07:52.697 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.697 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.697 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-09-21T22:07:52.697 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6817", 2024-09-21T22:07:52.697 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 2876495139 2024-09-21T22:07:52.697 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.698 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:52.698 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.698 INFO:teuthology.orchestra.run.smithi039.stdout: "cluster_addrs": { 2024-09-21T22:07:52.698 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2024-09-21T22:07:52.698 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.698 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2024-09-21T22:07:52.698 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6818", 2024-09-21T22:07:52.698 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 2876495139 2024-09-21T22:07:52.698 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.698 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.698 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-09-21T22:07:52.698 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6819", 2024-09-21T22:07:52.698 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 2876495139 2024-09-21T22:07:52.698 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.698 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:52.699 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.699 INFO:teuthology.orchestra.run.smithi039.stdout: "heartbeat_back_addrs": { 2024-09-21T22:07:52.699 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2024-09-21T22:07:52.699 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.699 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2024-09-21T22:07:52.699 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6822", 2024-09-21T22:07:52.699 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 2876495139 2024-09-21T22:07:52.699 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.699 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.699 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-09-21T22:07:52.699 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6823", 2024-09-21T22:07:52.699 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 2876495139 2024-09-21T22:07:52.699 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.699 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:52.700 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.700 INFO:teuthology.orchestra.run.smithi039.stdout: "heartbeat_front_addrs": { 2024-09-21T22:07:52.700 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2024-09-21T22:07:52.700 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.700 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2024-09-21T22:07:52.700 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6820", 2024-09-21T22:07:52.700 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 2876495139 2024-09-21T22:07:52.700 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.700 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.700 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-09-21T22:07:52.700 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6821", 2024-09-21T22:07:52.700 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 2876495139 2024-09-21T22:07:52.700 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.700 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:52.700 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.701 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6817/2876495139", 2024-09-21T22:07:52.701 INFO:teuthology.orchestra.run.smithi039.stdout: "cluster_addr": "172.21.15.39:6819/2876495139", 2024-09-21T22:07:52.701 INFO:teuthology.orchestra.run.smithi039.stdout: "heartbeat_back_addr": "172.21.15.39:6823/2876495139", 2024-09-21T22:07:52.701 INFO:teuthology.orchestra.run.smithi039.stdout: "heartbeat_front_addr": "172.21.15.39:6821/2876495139", 2024-09-21T22:07:52.701 INFO:teuthology.orchestra.run.smithi039.stdout: "state": [ 2024-09-21T22:07:52.701 INFO:teuthology.orchestra.run.smithi039.stdout: "exists", 2024-09-21T22:07:52.701 INFO:teuthology.orchestra.run.smithi039.stdout: "up" 2024-09-21T22:07:52.701 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:52.701 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.701 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.701 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": 2, 2024-09-21T22:07:52.701 INFO:teuthology.orchestra.run.smithi039.stdout: "uuid": "5f2678ed-6a23-4a67-96a4-c0ca91254bd7", 2024-09-21T22:07:52.701 INFO:teuthology.orchestra.run.smithi039.stdout: "up": 1, 2024-09-21T22:07:52.701 INFO:teuthology.orchestra.run.smithi039.stdout: "in": 0, 2024-09-21T22:07:52.701 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 0, 2024-09-21T22:07:52.702 INFO:teuthology.orchestra.run.smithi039.stdout: "primary_affinity": 1, 2024-09-21T22:07:52.702 INFO:teuthology.orchestra.run.smithi039.stdout: "last_clean_begin": 0, 2024-09-21T22:07:52.702 INFO:teuthology.orchestra.run.smithi039.stdout: "last_clean_end": 0, 2024-09-21T22:07:52.702 INFO:teuthology.orchestra.run.smithi039.stdout: "up_from": 11, 2024-09-21T22:07:52.702 INFO:teuthology.orchestra.run.smithi039.stdout: "up_thru": 42, 2024-09-21T22:07:52.702 INFO:teuthology.orchestra.run.smithi039.stdout: "down_at": 0, 2024-09-21T22:07:52.702 INFO:teuthology.orchestra.run.smithi039.stdout: "lost_at": 0, 2024-09-21T22:07:52.702 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2024-09-21T22:07:52.702 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2024-09-21T22:07:52.702 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.702 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2024-09-21T22:07:52.702 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6805", 2024-09-21T22:07:52.702 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 1262038957 2024-09-21T22:07:52.702 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.702 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.703 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-09-21T22:07:52.703 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6807", 2024-09-21T22:07:52.703 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 1262038957 2024-09-21T22:07:52.703 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.703 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:52.703 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.703 INFO:teuthology.orchestra.run.smithi039.stdout: "cluster_addrs": { 2024-09-21T22:07:52.703 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2024-09-21T22:07:52.703 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.703 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2024-09-21T22:07:52.703 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6809", 2024-09-21T22:07:52.703 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 1262038957 2024-09-21T22:07:52.703 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.703 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.704 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-09-21T22:07:52.704 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6811", 2024-09-21T22:07:52.704 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 1262038957 2024-09-21T22:07:52.704 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.704 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:52.704 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.704 INFO:teuthology.orchestra.run.smithi039.stdout: "heartbeat_back_addrs": { 2024-09-21T22:07:52.704 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2024-09-21T22:07:52.704 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.704 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2024-09-21T22:07:52.704 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6814", 2024-09-21T22:07:52.704 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 1262038957 2024-09-21T22:07:52.704 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.704 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.705 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-09-21T22:07:52.705 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6815", 2024-09-21T22:07:52.705 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 1262038957 2024-09-21T22:07:52.705 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.705 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:52.705 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.705 INFO:teuthology.orchestra.run.smithi039.stdout: "heartbeat_front_addrs": { 2024-09-21T22:07:52.705 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2024-09-21T22:07:52.705 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.705 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2024-09-21T22:07:52.705 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6812", 2024-09-21T22:07:52.705 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 1262038957 2024-09-21T22:07:52.705 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.705 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.705 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-09-21T22:07:52.706 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6813", 2024-09-21T22:07:52.706 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 1262038957 2024-09-21T22:07:52.706 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.706 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:52.706 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.706 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.39:6807/1262038957", 2024-09-21T22:07:52.706 INFO:teuthology.orchestra.run.smithi039.stdout: "cluster_addr": "172.21.15.39:6811/1262038957", 2024-09-21T22:07:52.706 INFO:teuthology.orchestra.run.smithi039.stdout: "heartbeat_back_addr": "172.21.15.39:6815/1262038957", 2024-09-21T22:07:52.706 INFO:teuthology.orchestra.run.smithi039.stdout: "heartbeat_front_addr": "172.21.15.39:6813/1262038957", 2024-09-21T22:07:52.706 INFO:teuthology.orchestra.run.smithi039.stdout: "state": [ 2024-09-21T22:07:52.706 INFO:teuthology.orchestra.run.smithi039.stdout: "exists", 2024-09-21T22:07:52.706 INFO:teuthology.orchestra.run.smithi039.stdout: "up" 2024-09-21T22:07:52.706 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:52.706 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.707 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.707 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": 3, 2024-09-21T22:07:52.707 INFO:teuthology.orchestra.run.smithi039.stdout: "uuid": "aba30555-ef9e-47c9-a38b-ee1a4b8092f5", 2024-09-21T22:07:52.707 INFO:teuthology.orchestra.run.smithi039.stdout: "up": 1, 2024-09-21T22:07:52.707 INFO:teuthology.orchestra.run.smithi039.stdout: "in": 1, 2024-09-21T22:07:52.707 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 1, 2024-09-21T22:07:52.707 INFO:teuthology.orchestra.run.smithi039.stdout: "primary_affinity": 1, 2024-09-21T22:07:52.707 INFO:teuthology.orchestra.run.smithi039.stdout: "last_clean_begin": 0, 2024-09-21T22:07:52.707 INFO:teuthology.orchestra.run.smithi039.stdout: "last_clean_end": 0, 2024-09-21T22:07:52.707 INFO:teuthology.orchestra.run.smithi039.stdout: "up_from": 11, 2024-09-21T22:07:52.707 INFO:teuthology.orchestra.run.smithi039.stdout: "up_thru": 58, 2024-09-21T22:07:52.707 INFO:teuthology.orchestra.run.smithi039.stdout: "down_at": 0, 2024-09-21T22:07:52.707 INFO:teuthology.orchestra.run.smithi039.stdout: "lost_at": 0, 2024-09-21T22:07:52.707 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2024-09-21T22:07:52.708 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2024-09-21T22:07:52.708 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.708 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2024-09-21T22:07:52.708 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.64:6800", 2024-09-21T22:07:52.708 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 1132198485 2024-09-21T22:07:52.708 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.708 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.708 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-09-21T22:07:52.708 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.64:6801", 2024-09-21T22:07:52.708 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 1132198485 2024-09-21T22:07:52.708 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.708 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:52.708 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.708 INFO:teuthology.orchestra.run.smithi039.stdout: "cluster_addrs": { 2024-09-21T22:07:52.709 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2024-09-21T22:07:52.709 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.709 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2024-09-21T22:07:52.709 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.64:6802", 2024-09-21T22:07:52.709 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 1132198485 2024-09-21T22:07:52.709 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.709 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.709 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-09-21T22:07:52.709 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.64:6803", 2024-09-21T22:07:52.709 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 1132198485 2024-09-21T22:07:52.709 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.709 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:52.709 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.709 INFO:teuthology.orchestra.run.smithi039.stdout: "heartbeat_back_addrs": { 2024-09-21T22:07:52.710 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2024-09-21T22:07:52.710 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.710 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2024-09-21T22:07:52.710 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.64:6806", 2024-09-21T22:07:52.710 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 1132198485 2024-09-21T22:07:52.710 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.710 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.710 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-09-21T22:07:52.710 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.64:6807", 2024-09-21T22:07:52.710 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 1132198485 2024-09-21T22:07:52.710 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.710 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:52.710 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.710 INFO:teuthology.orchestra.run.smithi039.stdout: "heartbeat_front_addrs": { 2024-09-21T22:07:52.711 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2024-09-21T22:07:52.711 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.711 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2024-09-21T22:07:52.711 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.64:6804", 2024-09-21T22:07:52.711 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 1132198485 2024-09-21T22:07:52.711 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.711 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.711 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-09-21T22:07:52.711 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.64:6805", 2024-09-21T22:07:52.711 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 1132198485 2024-09-21T22:07:52.711 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.711 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:52.711 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.711 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.64:6801/1132198485", 2024-09-21T22:07:52.712 INFO:teuthology.orchestra.run.smithi039.stdout: "cluster_addr": "172.21.15.64:6803/1132198485", 2024-09-21T22:07:52.712 INFO:teuthology.orchestra.run.smithi039.stdout: "heartbeat_back_addr": "172.21.15.64:6807/1132198485", 2024-09-21T22:07:52.712 INFO:teuthology.orchestra.run.smithi039.stdout: "heartbeat_front_addr": "172.21.15.64:6805/1132198485", 2024-09-21T22:07:52.712 INFO:teuthology.orchestra.run.smithi039.stdout: "state": [ 2024-09-21T22:07:52.712 INFO:teuthology.orchestra.run.smithi039.stdout: "exists", 2024-09-21T22:07:52.712 INFO:teuthology.orchestra.run.smithi039.stdout: "up" 2024-09-21T22:07:52.712 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:52.712 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.712 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.712 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": 4, 2024-09-21T22:07:52.712 INFO:teuthology.orchestra.run.smithi039.stdout: "uuid": "2a1288d7-618d-4f44-8346-930cf56d4e80", 2024-09-21T22:07:52.712 INFO:teuthology.orchestra.run.smithi039.stdout: "up": 1, 2024-09-21T22:07:52.712 INFO:teuthology.orchestra.run.smithi039.stdout: "in": 1, 2024-09-21T22:07:52.712 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 1, 2024-09-21T22:07:52.713 INFO:teuthology.orchestra.run.smithi039.stdout: "primary_affinity": 1, 2024-09-21T22:07:52.713 INFO:teuthology.orchestra.run.smithi039.stdout: "last_clean_begin": 0, 2024-09-21T22:07:52.713 INFO:teuthology.orchestra.run.smithi039.stdout: "last_clean_end": 0, 2024-09-21T22:07:52.713 INFO:teuthology.orchestra.run.smithi039.stdout: "up_from": 11, 2024-09-21T22:07:52.713 INFO:teuthology.orchestra.run.smithi039.stdout: "up_thru": 58, 2024-09-21T22:07:52.713 INFO:teuthology.orchestra.run.smithi039.stdout: "down_at": 0, 2024-09-21T22:07:52.713 INFO:teuthology.orchestra.run.smithi039.stdout: "lost_at": 0, 2024-09-21T22:07:52.713 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2024-09-21T22:07:52.713 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2024-09-21T22:07:52.713 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.713 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2024-09-21T22:07:52.713 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.64:6808", 2024-09-21T22:07:52.713 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 3500786459 2024-09-21T22:07:52.713 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.714 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.714 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-09-21T22:07:52.714 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.64:6809", 2024-09-21T22:07:52.714 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 3500786459 2024-09-21T22:07:52.714 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.714 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:52.714 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.714 INFO:teuthology.orchestra.run.smithi039.stdout: "cluster_addrs": { 2024-09-21T22:07:52.714 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2024-09-21T22:07:52.714 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.714 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2024-09-21T22:07:52.714 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.64:6810", 2024-09-21T22:07:52.714 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 3500786459 2024-09-21T22:07:52.714 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.715 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.715 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-09-21T22:07:52.715 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.64:6811", 2024-09-21T22:07:52.715 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 3500786459 2024-09-21T22:07:52.715 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.715 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:52.715 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.715 INFO:teuthology.orchestra.run.smithi039.stdout: "heartbeat_back_addrs": { 2024-09-21T22:07:52.715 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2024-09-21T22:07:52.715 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.715 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2024-09-21T22:07:52.715 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.64:6814", 2024-09-21T22:07:52.715 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 3500786459 2024-09-21T22:07:52.715 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.716 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.716 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-09-21T22:07:52.716 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.64:6815", 2024-09-21T22:07:52.716 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 3500786459 2024-09-21T22:07:52.716 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.716 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:52.716 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.716 INFO:teuthology.orchestra.run.smithi039.stdout: "heartbeat_front_addrs": { 2024-09-21T22:07:52.716 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2024-09-21T22:07:52.716 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.716 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2024-09-21T22:07:52.716 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.64:6812", 2024-09-21T22:07:52.716 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 3500786459 2024-09-21T22:07:52.716 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.717 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.717 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-09-21T22:07:52.717 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.64:6813", 2024-09-21T22:07:52.717 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 3500786459 2024-09-21T22:07:52.717 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.717 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:52.717 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.717 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.64:6809/3500786459", 2024-09-21T22:07:52.717 INFO:teuthology.orchestra.run.smithi039.stdout: "cluster_addr": "172.21.15.64:6811/3500786459", 2024-09-21T22:07:52.717 INFO:teuthology.orchestra.run.smithi039.stdout: "heartbeat_back_addr": "172.21.15.64:6815/3500786459", 2024-09-21T22:07:52.717 INFO:teuthology.orchestra.run.smithi039.stdout: "heartbeat_front_addr": "172.21.15.64:6813/3500786459", 2024-09-21T22:07:52.717 INFO:teuthology.orchestra.run.smithi039.stdout: "state": [ 2024-09-21T22:07:52.717 INFO:teuthology.orchestra.run.smithi039.stdout: "exists", 2024-09-21T22:07:52.717 INFO:teuthology.orchestra.run.smithi039.stdout: "up" 2024-09-21T22:07:52.717 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:52.718 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.718 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.718 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": 5, 2024-09-21T22:07:52.718 INFO:teuthology.orchestra.run.smithi039.stdout: "uuid": "3ce719a8-34e9-4890-bd40-d22201e7bad8", 2024-09-21T22:07:52.718 INFO:teuthology.orchestra.run.smithi039.stdout: "up": 1, 2024-09-21T22:07:52.718 INFO:teuthology.orchestra.run.smithi039.stdout: "in": 1, 2024-09-21T22:07:52.718 INFO:teuthology.orchestra.run.smithi039.stdout: "weight": 1, 2024-09-21T22:07:52.718 INFO:teuthology.orchestra.run.smithi039.stdout: "primary_affinity": 1, 2024-09-21T22:07:52.718 INFO:teuthology.orchestra.run.smithi039.stdout: "last_clean_begin": 11, 2024-09-21T22:07:52.718 INFO:teuthology.orchestra.run.smithi039.stdout: "last_clean_end": 41, 2024-09-21T22:07:52.718 INFO:teuthology.orchestra.run.smithi039.stdout: "up_from": 65, 2024-09-21T22:07:52.718 INFO:teuthology.orchestra.run.smithi039.stdout: "up_thru": 66, 2024-09-21T22:07:52.718 INFO:teuthology.orchestra.run.smithi039.stdout: "down_at": 42, 2024-09-21T22:07:52.718 INFO:teuthology.orchestra.run.smithi039.stdout: "lost_at": 0, 2024-09-21T22:07:52.718 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addrs": { 2024-09-21T22:07:52.719 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2024-09-21T22:07:52.719 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.719 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2024-09-21T22:07:52.719 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.64:6816", 2024-09-21T22:07:52.719 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 1547734524 2024-09-21T22:07:52.719 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.719 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.719 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-09-21T22:07:52.719 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.64:6817", 2024-09-21T22:07:52.719 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 1547734524 2024-09-21T22:07:52.719 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.719 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:52.719 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.719 INFO:teuthology.orchestra.run.smithi039.stdout: "cluster_addrs": { 2024-09-21T22:07:52.720 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2024-09-21T22:07:52.720 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.720 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2024-09-21T22:07:52.720 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.64:6818", 2024-09-21T22:07:52.720 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 1547734524 2024-09-21T22:07:52.720 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.720 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.720 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-09-21T22:07:52.720 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.64:6819", 2024-09-21T22:07:52.720 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 1547734524 2024-09-21T22:07:52.720 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.720 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:52.720 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.720 INFO:teuthology.orchestra.run.smithi039.stdout: "heartbeat_back_addrs": { 2024-09-21T22:07:52.720 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2024-09-21T22:07:52.721 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.721 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2024-09-21T22:07:52.721 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.64:6822", 2024-09-21T22:07:52.721 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 1547734524 2024-09-21T22:07:52.721 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.721 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.721 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-09-21T22:07:52.721 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.64:6823", 2024-09-21T22:07:52.721 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 1547734524 2024-09-21T22:07:52.721 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.721 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:52.721 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.721 INFO:teuthology.orchestra.run.smithi039.stdout: "heartbeat_front_addrs": { 2024-09-21T22:07:52.721 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2024-09-21T22:07:52.721 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.722 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2024-09-21T22:07:52.722 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.64:6820", 2024-09-21T22:07:52.722 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 1547734524 2024-09-21T22:07:52.722 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.722 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.722 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-09-21T22:07:52.722 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.64:6821", 2024-09-21T22:07:52.722 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 1547734524 2024-09-21T22:07:52.722 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.722 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:52.722 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.722 INFO:teuthology.orchestra.run.smithi039.stdout: "public_addr": "172.21.15.64:6817/1547734524", 2024-09-21T22:07:52.722 INFO:teuthology.orchestra.run.smithi039.stdout: "cluster_addr": "172.21.15.64:6819/1547734524", 2024-09-21T22:07:52.722 INFO:teuthology.orchestra.run.smithi039.stdout: "heartbeat_back_addr": "172.21.15.64:6823/1547734524", 2024-09-21T22:07:52.723 INFO:teuthology.orchestra.run.smithi039.stdout: "heartbeat_front_addr": "172.21.15.64:6821/1547734524", 2024-09-21T22:07:52.723 INFO:teuthology.orchestra.run.smithi039.stdout: "state": [ 2024-09-21T22:07:52.723 INFO:teuthology.orchestra.run.smithi039.stdout: "exists", 2024-09-21T22:07:52.723 INFO:teuthology.orchestra.run.smithi039.stdout: "up" 2024-09-21T22:07:52.723 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:52.723 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.723 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T22:07:52.723 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_xinfo": [ 2024-09-21T22:07:52.723 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.723 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": 0, 2024-09-21T22:07:52.723 INFO:teuthology.orchestra.run.smithi039.stdout: "down_stamp": "0.000000", 2024-09-21T22:07:52.723 INFO:teuthology.orchestra.run.smithi039.stdout: "laggy_probability": 0, 2024-09-21T22:07:52.723 INFO:teuthology.orchestra.run.smithi039.stdout: "laggy_interval": 0, 2024-09-21T22:07:52.723 INFO:teuthology.orchestra.run.smithi039.stdout: "features": 4540701547738038271, 2024-09-21T22:07:52.723 INFO:teuthology.orchestra.run.smithi039.stdout: "old_weight": 0, 2024-09-21T22:07:52.724 INFO:teuthology.orchestra.run.smithi039.stdout: "last_purged_snaps_scrub": "2024-09-21T22:05:54.224881+0000", 2024-09-21T22:07:52.724 INFO:teuthology.orchestra.run.smithi039.stdout: "dead_epoch": 0 2024-09-21T22:07:52.724 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.724 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.724 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": 1, 2024-09-21T22:07:52.724 INFO:teuthology.orchestra.run.smithi039.stdout: "down_stamp": "0.000000", 2024-09-21T22:07:52.724 INFO:teuthology.orchestra.run.smithi039.stdout: "laggy_probability": 0, 2024-09-21T22:07:52.724 INFO:teuthology.orchestra.run.smithi039.stdout: "laggy_interval": 0, 2024-09-21T22:07:52.724 INFO:teuthology.orchestra.run.smithi039.stdout: "features": 4540701547738038271, 2024-09-21T22:07:52.724 INFO:teuthology.orchestra.run.smithi039.stdout: "old_weight": 0, 2024-09-21T22:07:52.724 INFO:teuthology.orchestra.run.smithi039.stdout: "last_purged_snaps_scrub": "2024-09-21T22:05:54.668978+0000", 2024-09-21T22:07:52.724 INFO:teuthology.orchestra.run.smithi039.stdout: "dead_epoch": 0 2024-09-21T22:07:52.724 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.724 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.725 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": 2, 2024-09-21T22:07:52.725 INFO:teuthology.orchestra.run.smithi039.stdout: "down_stamp": "0.000000", 2024-09-21T22:07:52.725 INFO:teuthology.orchestra.run.smithi039.stdout: "laggy_probability": 0, 2024-09-21T22:07:52.725 INFO:teuthology.orchestra.run.smithi039.stdout: "laggy_interval": 0, 2024-09-21T22:07:52.725 INFO:teuthology.orchestra.run.smithi039.stdout: "features": 4540701547738038271, 2024-09-21T22:07:52.725 INFO:teuthology.orchestra.run.smithi039.stdout: "old_weight": 65536, 2024-09-21T22:07:52.725 INFO:teuthology.orchestra.run.smithi039.stdout: "last_purged_snaps_scrub": "2024-09-21T22:05:54.315876+0000", 2024-09-21T22:07:52.725 INFO:teuthology.orchestra.run.smithi039.stdout: "dead_epoch": 0 2024-09-21T22:07:52.725 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.725 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.725 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": 3, 2024-09-21T22:07:52.725 INFO:teuthology.orchestra.run.smithi039.stdout: "down_stamp": "0.000000", 2024-09-21T22:07:52.725 INFO:teuthology.orchestra.run.smithi039.stdout: "laggy_probability": 0, 2024-09-21T22:07:52.725 INFO:teuthology.orchestra.run.smithi039.stdout: "laggy_interval": 0, 2024-09-21T22:07:52.726 INFO:teuthology.orchestra.run.smithi039.stdout: "features": 4540701547738038271, 2024-09-21T22:07:52.726 INFO:teuthology.orchestra.run.smithi039.stdout: "old_weight": 0, 2024-09-21T22:07:52.726 INFO:teuthology.orchestra.run.smithi039.stdout: "last_purged_snaps_scrub": "2024-09-21T22:05:54.314795+0000", 2024-09-21T22:07:52.726 INFO:teuthology.orchestra.run.smithi039.stdout: "dead_epoch": 0 2024-09-21T22:07:52.726 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.726 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.726 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": 4, 2024-09-21T22:07:52.726 INFO:teuthology.orchestra.run.smithi039.stdout: "down_stamp": "0.000000", 2024-09-21T22:07:52.726 INFO:teuthology.orchestra.run.smithi039.stdout: "laggy_probability": 0, 2024-09-21T22:07:52.726 INFO:teuthology.orchestra.run.smithi039.stdout: "laggy_interval": 0, 2024-09-21T22:07:52.726 INFO:teuthology.orchestra.run.smithi039.stdout: "features": 4540701547738038271, 2024-09-21T22:07:52.726 INFO:teuthology.orchestra.run.smithi039.stdout: "old_weight": 0, 2024-09-21T22:07:52.726 INFO:teuthology.orchestra.run.smithi039.stdout: "last_purged_snaps_scrub": "2024-09-21T22:05:54.322403+0000", 2024-09-21T22:07:52.726 INFO:teuthology.orchestra.run.smithi039.stdout: "dead_epoch": 0 2024-09-21T22:07:52.727 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.727 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.727 INFO:teuthology.orchestra.run.smithi039.stdout: "osd": 5, 2024-09-21T22:07:52.727 INFO:teuthology.orchestra.run.smithi039.stdout: "down_stamp": "0.000000", 2024-09-21T22:07:52.727 INFO:teuthology.orchestra.run.smithi039.stdout: "laggy_probability": 0, 2024-09-21T22:07:52.727 INFO:teuthology.orchestra.run.smithi039.stdout: "laggy_interval": 0, 2024-09-21T22:07:52.727 INFO:teuthology.orchestra.run.smithi039.stdout: "features": 4540701547738038271, 2024-09-21T22:07:52.727 INFO:teuthology.orchestra.run.smithi039.stdout: "old_weight": 0, 2024-09-21T22:07:52.727 INFO:teuthology.orchestra.run.smithi039.stdout: "last_purged_snaps_scrub": "2024-09-21T22:05:54.750633+0000", 2024-09-21T22:07:52.727 INFO:teuthology.orchestra.run.smithi039.stdout: "dead_epoch": 42 2024-09-21T22:07:52.727 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.727 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T22:07:52.727 INFO:teuthology.orchestra.run.smithi039.stdout: "pg_upmap": [ 2024-09-21T22:07:52.727 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.728 INFO:teuthology.orchestra.run.smithi039.stdout: "pgid": "1.0", 2024-09-21T22:07:52.728 INFO:teuthology.orchestra.run.smithi039.stdout: "osds": [ 2024-09-21T22:07:52.728 INFO:teuthology.orchestra.run.smithi039.stdout: 3, 2024-09-21T22:07:52.728 INFO:teuthology.orchestra.run.smithi039.stdout: 2 2024-09-21T22:07:52.728 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:52.728 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.728 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-09-21T22:07:52.728 INFO:teuthology.orchestra.run.smithi039.stdout: "pg_upmap_items": [], 2024-09-21T22:07:52.728 INFO:teuthology.orchestra.run.smithi039.stdout: "pg_upmap_primaries": [], 2024-09-21T22:07:52.728 INFO:teuthology.orchestra.run.smithi039.stdout: "pg_temp": [], 2024-09-21T22:07:52.728 INFO:teuthology.orchestra.run.smithi039.stdout: "primary_temp": [], 2024-09-21T22:07:52.728 INFO:teuthology.orchestra.run.smithi039.stdout: "blocklist": {}, 2024-09-21T22:07:52.728 INFO:teuthology.orchestra.run.smithi039.stdout: "range_blocklist": {}, 2024-09-21T22:07:52.728 INFO:teuthology.orchestra.run.smithi039.stdout: "erasure_code_profiles": { 2024-09-21T22:07:52.729 INFO:teuthology.orchestra.run.smithi039.stdout: "default": { 2024-09-21T22:07:52.729 INFO:teuthology.orchestra.run.smithi039.stdout: "crush-failure-domain": "osd", 2024-09-21T22:07:52.729 INFO:teuthology.orchestra.run.smithi039.stdout: "k": "2", 2024-09-21T22:07:52.729 INFO:teuthology.orchestra.run.smithi039.stdout: "m": "1", 2024-09-21T22:07:52.729 INFO:teuthology.orchestra.run.smithi039.stdout: "plugin": "jerasure", 2024-09-21T22:07:52.729 INFO:teuthology.orchestra.run.smithi039.stdout: "technique": "reed_sol_van" 2024-09-21T22:07:52.729 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.729 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.729 INFO:teuthology.orchestra.run.smithi039.stdout: "removed_snaps_queue": [], 2024-09-21T22:07:52.729 INFO:teuthology.orchestra.run.smithi039.stdout: "new_removed_snaps": [], 2024-09-21T22:07:52.729 INFO:teuthology.orchestra.run.smithi039.stdout: "new_purged_snaps": [], 2024-09-21T22:07:52.729 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_node_flags": {}, 2024-09-21T22:07:52.729 INFO:teuthology.orchestra.run.smithi039.stdout: "device_class_flags": {}, 2024-09-21T22:07:52.729 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode": { 2024-09-21T22:07:52.729 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode_enabled": false, 2024-09-21T22:07:52.730 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_bucket_count": 0, 2024-09-21T22:07:52.730 INFO:teuthology.orchestra.run.smithi039.stdout: "degraded_stretch_mode": 0, 2024-09-21T22:07:52.730 INFO:teuthology.orchestra.run.smithi039.stdout: "recovering_stretch_mode": 0, 2024-09-21T22:07:52.730 INFO:teuthology.orchestra.run.smithi039.stdout: "stretch_mode_bucket": 0 2024-09-21T22:07:52.730 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.730 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:07:52.732 INFO:tasks.workunit.client.0.smithi157.stderr:1+0 records in 2024-09-21T22:07:52.732 INFO:tasks.workunit.client.0.smithi157.stderr:1+0 records out 2024-09-21T22:07:52.732 INFO:tasks.workunit.client.0.smithi157.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102921 s, 10.2 MB/s 2024-09-21T22:07:52.732 INFO:tasks.workunit.client.0.smithi157.stderr:+ touch /tmp/tmp.GqKuv7rsZi/quiesce.sh 2024-09-21T22:07:52.732 INFO:tasks.workunit.client.0.smithi157.stderr:+ chmod +x /tmp/tmp.GqKuv7rsZi/quiesce.sh 2024-09-21T22:07:52.732 INFO:tasks.workunit.client.0.smithi157.stderr:+ cat 2024-09-21T22:07:52.732 INFO:tasks.workunit.client.0.smithi157.stderr:+ rbd snap create rbd/testrbdnbd21553@quiesce1 2024-09-21T22:07:52.733 INFO:tasks.thrashosds.thrasher:j is {'epoch': 78, 'fsid': '00bb09a3-e410-4e18-a2b2-08eaf61978e6', 'created': '2024-09-21T22:05:45.303341+0000', 'modified': '2024-09-21T22:07:49.823464+0000', 'last_up_change': '2024-09-21T22:07:27.209405+0000', 'last_in_change': '2024-09-21T22:07:10.044634+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 5, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 6, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-21T22:06:00.868591+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6.059999942779541, 'score_stable': 6.059999942779541, 'optimal_score': 0.33000001311302185, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-21T22:06:03.535857+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '76', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 4, 'snap_epoch': 76, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.2699999809265137, 'score_stable': 2.2699999809265137, 'optimal_score': 0.8299999833106995, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'new_default_pool21553', 'create_time': '2024-09-21T22:07:15.499114+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '61', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 61, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3.009999990463257, 'score_stable': 3.009999990463257, 'optimal_score': 0.8299999833106995, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '8336c397-2018-4c75-ab31-2e5f6ad564b5', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 58, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6800', 'nonce': 3054061468}, {'type': 'v1', 'addr': '172.21.15.39:6801', 'nonce': 3054061468}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6802', 'nonce': 3054061468}, {'type': 'v1', 'addr': '172.21.15.39:6803', 'nonce': 3054061468}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6808', 'nonce': 3054061468}, {'type': 'v1', 'addr': '172.21.15.39:6810', 'nonce': 3054061468}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6804', 'nonce': 3054061468}, {'type': 'v1', 'addr': '172.21.15.39:6806', 'nonce': 3054061468}]}, 'public_addr': '172.21.15.39:6801/3054061468', 'cluster_addr': '172.21.15.39:6803/3054061468', 'heartbeat_back_addr': '172.21.15.39:6810/3054061468', 'heartbeat_front_addr': '172.21.15.39:6806/3054061468', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'bca64af6-7427-4c5d-97a4-a183b1f3a7d9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 65, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6816', 'nonce': 2876495139}, {'type': 'v1', 'addr': '172.21.15.39:6817', 'nonce': 2876495139}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6818', 'nonce': 2876495139}, {'type': 'v1', 'addr': '172.21.15.39:6819', 'nonce': 2876495139}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6822', 'nonce': 2876495139}, {'type': 'v1', 'addr': '172.21.15.39:6823', 'nonce': 2876495139}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6820', 'nonce': 2876495139}, {'type': 'v1', 'addr': '172.21.15.39:6821', 'nonce': 2876495139}]}, 'public_addr': '172.21.15.39:6817/2876495139', 'cluster_addr': '172.21.15.39:6819/2876495139', 'heartbeat_back_addr': '172.21.15.39:6823/2876495139', 'heartbeat_front_addr': '172.21.15.39:6821/2876495139', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '5f2678ed-6a23-4a67-96a4-c0ca91254bd7', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 42, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6805', 'nonce': 1262038957}, {'type': 'v1', 'addr': '172.21.15.39:6807', 'nonce': 1262038957}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6809', 'nonce': 1262038957}, {'type': 'v1', 'addr': '172.21.15.39:6811', 'nonce': 1262038957}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6814', 'nonce': 1262038957}, {'type': 'v1', 'addr': '172.21.15.39:6815', 'nonce': 1262038957}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6812', 'nonce': 1262038957}, {'type': 'v1', 'addr': '172.21.15.39:6813', 'nonce': 1262038957}]}, 'public_addr': '172.21.15.39:6807/1262038957', 'cluster_addr': '172.21.15.39:6811/1262038957', 'heartbeat_back_addr': '172.21.15.39:6815/1262038957', 'heartbeat_front_addr': '172.21.15.39:6813/1262038957', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'aba30555-ef9e-47c9-a38b-ee1a4b8092f5', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 58, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6800', 'nonce': 1132198485}, {'type': 'v1', 'addr': '172.21.15.64:6801', 'nonce': 1132198485}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6802', 'nonce': 1132198485}, {'type': 'v1', 'addr': '172.21.15.64:6803', 'nonce': 1132198485}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6806', 'nonce': 1132198485}, {'type': 'v1', 'addr': '172.21.15.64:6807', 'nonce': 1132198485}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6804', 'nonce': 1132198485}, {'type': 'v1', 'addr': '172.21.15.64:6805', 'nonce': 1132198485}]}, 'public_addr': '172.21.15.64:6801/1132198485', 'cluster_addr': '172.21.15.64:6803/1132198485', 'heartbeat_back_addr': '172.21.15.64:6807/1132198485', 'heartbeat_front_addr': '172.21.15.64:6805/1132198485', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '2a1288d7-618d-4f44-8346-930cf56d4e80', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 58, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6808', 'nonce': 3500786459}, {'type': 'v1', 'addr': '172.21.15.64:6809', 'nonce': 3500786459}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6810', 'nonce': 3500786459}, {'type': 'v1', 'addr': '172.21.15.64:6811', 'nonce': 3500786459}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6814', 'nonce': 3500786459}, {'type': 'v1', 'addr': '172.21.15.64:6815', 'nonce': 3500786459}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6812', 'nonce': 3500786459}, {'type': 'v1', 'addr': '172.21.15.64:6813', 'nonce': 3500786459}]}, 'public_addr': '172.21.15.64:6809/3500786459', 'cluster_addr': '172.21.15.64:6811/3500786459', 'heartbeat_back_addr': '172.21.15.64:6815/3500786459', 'heartbeat_front_addr': '172.21.15.64:6813/3500786459', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '3ce719a8-34e9-4890-bd40-d22201e7bad8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 11, 'last_clean_end': 41, 'up_from': 65, 'up_thru': 66, 'down_at': 42, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6816', 'nonce': 1547734524}, {'type': 'v1', 'addr': '172.21.15.64:6817', 'nonce': 1547734524}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6818', 'nonce': 1547734524}, {'type': 'v1', 'addr': '172.21.15.64:6819', 'nonce': 1547734524}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6822', 'nonce': 1547734524}, {'type': 'v1', 'addr': '172.21.15.64:6823', 'nonce': 1547734524}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6820', 'nonce': 1547734524}, {'type': 'v1', 'addr': '172.21.15.64:6821', 'nonce': 1547734524}]}, 'public_addr': '172.21.15.64:6817/1547734524', 'cluster_addr': '172.21.15.64:6819/1547734524', 'heartbeat_back_addr': '172.21.15.64:6823/1547734524', 'heartbeat_front_addr': '172.21.15.64:6821/1547734524', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T22:05:54.224881+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T22:05:54.668978+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-21T22:05:54.315876+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T22:05:54.314795+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T22:05:54.322403+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T22:05:54.750633+0000', 'dead_epoch': 42}], 'pg_upmap': [{'pgid': '1.0', 'osds': [3, 2]}], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-21T22:07:52.733 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2024-09-21T22:07:52.734 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:52.726+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:52.832 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:52.830+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:52.857 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:07:52.857 INFO:teuthology.orchestra.run.smithi039.stdout: "size": 20, 2024-09-21T22:07:52.858 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 600, 2024-09-21T22:07:52.858 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [ 2024-09-21T22:07:52.858 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.858 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.3 55/54 [PushOp(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head, version: 52'2, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head@52'2, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T22:07:52.858 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.084379+0000", 2024-09-21T22:07:52.858 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 40.768737731000002, 2024-09-21T22:07:52.858 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.051020674000000002, 2024-09-21T22:07:52.858 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:52.858 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:52.858 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:52.858 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:52.858 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.858 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:52.859 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.084379+0000", 2024-09-21T22:07:52.859 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:52.859 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.859 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.859 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:52.859 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.084379+0000", 2024-09-21T22:07:52.859 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:52.859 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.859 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.859 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:52.859 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.084381+0000", 2024-09-21T22:07:52.859 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.73e-06 2024-09-21T22:07:52.859 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.860 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.860 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:52.860 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.085494+0000", 2024-09-21T22:07:52.860 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011131559999999999 2024-09-21T22:07:52.860 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.860 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.860 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:52.860 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.085542+0000", 2024-09-21T22:07:52.860 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8334999999999997e-05 2024-09-21T22:07:52.860 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.860 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.860 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:52.860 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.085591+0000", 2024-09-21T22:07:52.860 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.9165000000000001e-05 2024-09-21T22:07:52.860 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.861 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.861 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:52.861 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.085777+0000", 2024-09-21T22:07:52.861 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000186152 2024-09-21T22:07:52.861 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.861 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.861 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:52.861 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.085796+0000", 2024-09-21T22:07:52.861 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8612999999999999e-05 2024-09-21T22:07:52.861 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.861 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.861 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:52.861 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.135399+0000", 2024-09-21T22:07:52.861 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.049603522999999997 2024-09-21T22:07:52.862 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.862 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:52.862 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.862 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.862 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.862 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.3 55/54 [PushOp(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head, version: 52'2, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head@52'2, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:07:52.862 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.139554+0000", 2024-09-21T22:07:52.862 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 40.713562627999998, 2024-09-21T22:07:52.862 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.056215389999999997, 2024-09-21T22:07:52.862 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:52.862 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:52.862 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:52.862 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:52.863 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.863 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:52.863 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.139554+0000", 2024-09-21T22:07:52.863 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:52.863 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.863 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.863 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:52.863 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.139554+0000", 2024-09-21T22:07:52.863 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:52.863 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.863 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.863 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:52.863 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.139556+0000", 2024-09-21T22:07:52.863 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7999999999999999e-06 2024-09-21T22:07:52.864 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.864 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.864 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:52.864 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.140671+0000", 2024-09-21T22:07:52.864 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001115612 2024-09-21T22:07:52.864 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.864 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.864 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:52.864 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.140720+0000", 2024-09-21T22:07:52.864 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8631999999999998e-05 2024-09-21T22:07:52.864 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.864 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.864 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:52.864 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.140752+0000", 2024-09-21T22:07:52.865 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.2199000000000001e-05 2024-09-21T22:07:52.865 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.865 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.865 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:52.865 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.140900+0000", 2024-09-21T22:07:52.865 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00014786000000000001 2024-09-21T22:07:52.865 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.865 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.865 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:52.865 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.140917+0000", 2024-09-21T22:07:52.865 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6753999999999999e-05 2024-09-21T22:07:52.865 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.865 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.865 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:52.866 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.195769+0000", 2024-09-21T22:07:52.866 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.054852533000000002 2024-09-21T22:07:52.866 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.866 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:52.866 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.866 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.866 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.866 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.3 55/54 [PushOp(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head, version: 52'2, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head@52'2, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:07:52.866 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.199187+0000", 2024-09-21T22:07:52.866 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 40.653929083999998, 2024-09-21T22:07:52.866 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.051543975999999998, 2024-09-21T22:07:52.866 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:52.866 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:52.866 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:52.867 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:52.867 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.867 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:52.867 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.199187+0000", 2024-09-21T22:07:52.867 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:52.867 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.867 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.867 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:52.867 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.199187+0000", 2024-09-21T22:07:52.867 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:52.867 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.867 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.867 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:52.867 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.199189+0000", 2024-09-21T22:07:52.868 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7969999999999999e-06 2024-09-21T22:07:52.868 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.868 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.868 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:52.868 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.200355+0000", 2024-09-21T22:07:52.868 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011660139999999999 2024-09-21T22:07:52.868 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.868 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.868 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:52.868 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.200405+0000", 2024-09-21T22:07:52.868 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.0124000000000003e-05 2024-09-21T22:07:52.868 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.868 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.868 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:52.869 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.200457+0000", 2024-09-21T22:07:52.869 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.1177999999999997e-05 2024-09-21T22:07:52.869 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.869 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.869 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:52.869 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.200635+0000", 2024-09-21T22:07:52.869 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00017841299999999999 2024-09-21T22:07:52.869 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.869 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.869 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:52.869 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.200653+0000", 2024-09-21T22:07:52.869 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7944999999999998e-05 2024-09-21T22:07:52.869 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.869 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.870 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:52.870 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.250731+0000", 2024-09-21T22:07:52.870 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.050078505000000002 2024-09-21T22:07:52.870 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.870 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:52.870 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.870 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.870 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.870 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.3 55/54 [PushOp(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head, version: 52'3, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head@52'3, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:07:52.870 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.343754+0000", 2024-09-21T22:07:52.870 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 40.509362373000002, 2024-09-21T22:07:52.870 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.063560607000000005, 2024-09-21T22:07:52.870 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:52.871 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:52.871 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:52.871 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:52.871 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.871 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:52.871 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.343754+0000", 2024-09-21T22:07:52.871 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:52.871 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.871 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.871 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:52.871 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.343754+0000", 2024-09-21T22:07:52.871 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:52.871 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.871 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.872 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:52.872 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.343756+0000", 2024-09-21T22:07:52.872 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7069999999999999e-06 2024-09-21T22:07:52.872 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.872 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.872 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:52.872 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.344864+0000", 2024-09-21T22:07:52.872 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011085159999999999 2024-09-21T22:07:52.872 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.872 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.872 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:52.872 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.344914+0000", 2024-09-21T22:07:52.872 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.9462000000000002e-05 2024-09-21T22:07:52.872 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.873 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.873 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:52.873 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.344961+0000", 2024-09-21T22:07:52.873 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.7633999999999998e-05 2024-09-21T22:07:52.873 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.873 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.873 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:52.873 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.345171+0000", 2024-09-21T22:07:52.873 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000209652 2024-09-21T22:07:52.873 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.873 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.873 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:52.873 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.345193+0000", 2024-09-21T22:07:52.873 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1985999999999999e-05 2024-09-21T22:07:52.873 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.874 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.874 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:52.874 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.407315+0000", 2024-09-21T22:07:52.874 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.06212165 2024-09-21T22:07:52.874 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.874 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:52.874 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.874 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.874 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.874 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.3 55/54 [PushOp(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head, version: 52'3, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head@52'3, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:07:52.874 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.410551+0000", 2024-09-21T22:07:52.874 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 40.442565242000001, 2024-09-21T22:07:52.874 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.054517522999999998, 2024-09-21T22:07:52.874 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:52.875 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:52.875 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:52.875 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:52.875 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.875 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:52.875 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.410551+0000", 2024-09-21T22:07:52.875 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:52.875 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.875 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.875 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:52.875 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.410551+0000", 2024-09-21T22:07:52.875 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:52.875 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.875 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.876 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:52.876 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.410553+0000", 2024-09-21T22:07:52.876 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5200000000000001e-06 2024-09-21T22:07:52.876 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.876 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.876 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:52.876 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.411747+0000", 2024-09-21T22:07:52.876 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011946089999999999 2024-09-21T22:07:52.876 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.876 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.876 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:52.876 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.411796+0000", 2024-09-21T22:07:52.876 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8717999999999997e-05 2024-09-21T22:07:52.876 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.877 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.877 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:52.877 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.411844+0000", 2024-09-21T22:07:52.877 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8291999999999998e-05 2024-09-21T22:07:52.877 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.877 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.877 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:52.877 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.412066+0000", 2024-09-21T22:07:52.877 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00022115899999999999 2024-09-21T22:07:52.877 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.877 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.877 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:52.877 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.412081+0000", 2024-09-21T22:07:52.877 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5654e-05 2024-09-21T22:07:52.877 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.878 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.878 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:52.878 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.465069+0000", 2024-09-21T22:07:52.878 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.052987570999999997 2024-09-21T22:07:52.878 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.878 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:52.878 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.878 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.878 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.878 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.3 55/54 [PushOp(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head, version: 52'3, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head@52'3, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 4194304, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:07:52.878 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.468339+0000", 2024-09-21T22:07:52.878 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 40.384777577999998, 2024-09-21T22:07:52.878 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.064010024999999998, 2024-09-21T22:07:52.878 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:52.879 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:52.879 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:52.879 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:52.879 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.879 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:52.879 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.468339+0000", 2024-09-21T22:07:52.879 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:52.879 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.879 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.879 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:52.879 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.468339+0000", 2024-09-21T22:07:52.879 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:52.879 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.879 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.880 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:52.880 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.468340+0000", 2024-09-21T22:07:52.880 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4330000000000001e-06 2024-09-21T22:07:52.880 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.880 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.880 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:52.880 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.469484+0000", 2024-09-21T22:07:52.880 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001143535 2024-09-21T22:07:52.880 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.880 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.880 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:52.880 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.469533+0000", 2024-09-21T22:07:52.880 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8980999999999997e-05 2024-09-21T22:07:52.880 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.880 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.881 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:52.881 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.469581+0000", 2024-09-21T22:07:52.881 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8551999999999999e-05 2024-09-21T22:07:52.881 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.881 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.881 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:52.881 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.469804+0000", 2024-09-21T22:07:52.881 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00022287699999999999 2024-09-21T22:07:52.881 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.881 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.881 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:52.881 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.469829+0000", 2024-09-21T22:07:52.881 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4766e-05 2024-09-21T22:07:52.881 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.882 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.882 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:52.882 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.532349+0000", 2024-09-21T22:07:52.882 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.062519880999999999 2024-09-21T22:07:52.882 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.882 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:52.882 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.882 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.882 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.882 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4840.0:668 2.3 2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:07:52.882 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.409114+0000", 2024-09-21T22:07:52.882 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 28.444002372, 2024-09-21T22:07:52.882 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.048703758999999999, 2024-09-21T22:07:52.882 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:52.883 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:52.883 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:52.883 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-21T22:07:52.883 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4840", 2024-09-21T22:07:52.883 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:52.883 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 668 2024-09-21T22:07:52.883 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.883 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:52.883 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.883 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:52.883 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.409114+0000", 2024-09-21T22:07:52.883 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:52.883 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.883 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.884 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:52.884 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.409114+0000", 2024-09-21T22:07:52.884 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:52.884 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.884 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.884 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:52.884 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.409115+0000", 2024-09-21T22:07:52.884 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.114e-06 2024-09-21T22:07:52.884 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.884 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.884 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:52.884 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.410308+0000", 2024-09-21T22:07:52.884 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011930949999999999 2024-09-21T22:07:52.884 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.885 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.885 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:52.885 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.410325+0000", 2024-09-21T22:07:52.885 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7088e-05 2024-09-21T22:07:52.885 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.885 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.885 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:52.885 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.410334+0000", 2024-09-21T22:07:52.885 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.2549999999999997e-06 2024-09-21T22:07:52.885 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.885 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.885 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:52.885 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.410393+0000", 2024-09-21T22:07:52.885 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.9327999999999998e-05 2024-09-21T22:07:52.886 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.886 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.886 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:52.886 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.410482+0000", 2024-09-21T22:07:52.886 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.9019999999999998e-05 2024-09-21T22:07:52.886 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.886 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.886 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-21T22:07:52.886 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.410617+0000", 2024-09-21T22:07:52.886 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000135232 2024-09-21T22:07:52.886 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.886 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.886 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:52.886 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.457051+0000", 2024-09-21T22:07:52.887 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.04643415 2024-09-21T22:07:52.887 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.887 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.887 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-21T22:07:52.887 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.457675+0000", 2024-09-21T22:07:52.887 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00062332700000000002 2024-09-21T22:07:52.887 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.887 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.887 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:52.887 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.457735+0000", 2024-09-21T22:07:52.887 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.029e-05 2024-09-21T22:07:52.887 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.887 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.887 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:52.888 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.457818+0000", 2024-09-21T22:07:52.888 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.2860000000000003e-05 2024-09-21T22:07:52.888 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.888 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:52.888 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.888 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.888 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.888 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4840.0:670 2.3 2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:07:52.888 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.466980+0000", 2024-09-21T22:07:52.888 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 28.386136304000001, 2024-09-21T22:07:52.888 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.050913930000000003, 2024-09-21T22:07:52.888 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:52.888 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:52.889 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:52.889 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-21T22:07:52.889 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4840", 2024-09-21T22:07:52.889 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:52.889 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 670 2024-09-21T22:07:52.889 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.889 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:52.889 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.889 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:52.889 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.466980+0000", 2024-09-21T22:07:52.889 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:52.889 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.889 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.889 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:52.890 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.466980+0000", 2024-09-21T22:07:52.890 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:52.890 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.890 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.890 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:52.890 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.466981+0000", 2024-09-21T22:07:52.890 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0839999999999999e-06 2024-09-21T22:07:52.890 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.890 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.890 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:52.890 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.467815+0000", 2024-09-21T22:07:52.890 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00083330399999999999 2024-09-21T22:07:52.890 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.890 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.891 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:52.891 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.467832+0000", 2024-09-21T22:07:52.891 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7601e-05 2024-09-21T22:07:52.891 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.891 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.891 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:52.891 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.467840+0000", 2024-09-21T22:07:52.891 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.2220000000000002e-06 2024-09-21T22:07:52.891 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.891 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.891 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:52.891 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.467974+0000", 2024-09-21T22:07:52.891 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000133411 2024-09-21T22:07:52.892 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.892 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.892 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:52.892 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.468223+0000", 2024-09-21T22:07:52.892 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00024906400000000001 2024-09-21T22:07:52.892 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.892 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.892 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-21T22:07:52.892 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.468616+0000", 2024-09-21T22:07:52.892 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000393433 2024-09-21T22:07:52.892 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.892 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.892 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:52.892 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.517016+0000", 2024-09-21T22:07:52.893 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.048399708 2024-09-21T22:07:52.893 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.893 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.893 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-21T22:07:52.893 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.517693+0000", 2024-09-21T22:07:52.893 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00067681000000000002 2024-09-21T22:07:52.893 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.893 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.893 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:52.893 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.517789+0000", 2024-09-21T22:07:52.893 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.5778000000000004e-05 2024-09-21T22:07:52.893 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.893 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.893 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:52.894 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.517894+0000", 2024-09-21T22:07:52.894 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000105515 2024-09-21T22:07:52.894 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.894 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:52.894 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.894 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.894 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.894 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4840.0:672 2.3 2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:07:52.894 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.527127+0000", 2024-09-21T22:07:52.894 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 28.325989841999998, 2024-09-21T22:07:52.894 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.047345602000000001, 2024-09-21T22:07:52.894 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:52.894 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:52.894 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:52.895 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-21T22:07:52.895 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4840", 2024-09-21T22:07:52.895 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:52.895 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 672 2024-09-21T22:07:52.895 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.895 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:52.895 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.895 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:52.895 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.527127+0000", 2024-09-21T22:07:52.895 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:52.895 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.895 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.895 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:52.895 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.527127+0000", 2024-09-21T22:07:52.896 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:52.896 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.896 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.896 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:52.896 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.527128+0000", 2024-09-21T22:07:52.896 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2810000000000001e-06 2024-09-21T22:07:52.896 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.896 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.896 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:52.896 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.527962+0000", 2024-09-21T22:07:52.896 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00083445199999999996 2024-09-21T22:07:52.896 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.896 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.896 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:52.897 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.527981+0000", 2024-09-21T22:07:52.897 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9009e-05 2024-09-21T22:07:52.897 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.897 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.897 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:52.897 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.527990+0000", 2024-09-21T22:07:52.897 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.8619999999999992e-06 2024-09-21T22:07:52.897 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.897 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.897 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:52.897 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.528061+0000", 2024-09-21T22:07:52.897 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.0226000000000002e-05 2024-09-21T22:07:52.897 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.897 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.897 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:52.898 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.528156+0000", 2024-09-21T22:07:52.898 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.5724999999999996e-05 2024-09-21T22:07:52.898 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.898 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.898 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-21T22:07:52.898 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.528303+0000", 2024-09-21T22:07:52.898 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000146374 2024-09-21T22:07:52.898 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.898 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.898 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:52.898 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.573772+0000", 2024-09-21T22:07:52.898 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.045469745999999998 2024-09-21T22:07:52.898 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.898 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.899 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-21T22:07:52.899 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.574299+0000", 2024-09-21T22:07:52.899 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00052649200000000004 2024-09-21T22:07:52.899 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.899 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.899 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:52.899 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.574366+0000", 2024-09-21T22:07:52.899 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.6872000000000006e-05 2024-09-21T22:07:52.899 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.899 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.899 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:52.899 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.574472+0000", 2024-09-21T22:07:52.899 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000106563 2024-09-21T22:07:52.899 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.899 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:52.900 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.900 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.900 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.900 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4840.0:686 2.3 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:07:52.900 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.943561+0000", 2024-09-21T22:07:52.900 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 27.909555566000002, 2024-09-21T22:07:52.900 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.049021493999999999, 2024-09-21T22:07:52.900 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:52.900 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:52.900 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:52.900 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-21T22:07:52.900 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4840", 2024-09-21T22:07:52.900 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:52.900 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 686 2024-09-21T22:07:52.901 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.901 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:52.901 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.901 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:52.901 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.943561+0000", 2024-09-21T22:07:52.901 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:52.901 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.901 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.901 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:52.901 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.943561+0000", 2024-09-21T22:07:52.901 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:52.901 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.901 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.901 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:52.901 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.943563+0000", 2024-09-21T22:07:52.902 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9420000000000002e-06 2024-09-21T22:07:52.902 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.902 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.902 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:52.902 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.944546+0000", 2024-09-21T22:07:52.902 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00098357099999999992 2024-09-21T22:07:52.902 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.902 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.902 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:52.902 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.944565+0000", 2024-09-21T22:07:52.902 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8621e-05 2024-09-21T22:07:52.902 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.902 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.902 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:52.903 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.944575+0000", 2024-09-21T22:07:52.903 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.4639999999999995e-06 2024-09-21T22:07:52.903 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.903 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.903 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:52.903 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.944647+0000", 2024-09-21T22:07:52.903 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.2786000000000004e-05 2024-09-21T22:07:52.903 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.903 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.903 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:52.903 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.944746+0000", 2024-09-21T22:07:52.903 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.8196999999999995e-05 2024-09-21T22:07:52.903 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.903 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.904 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-21T22:07:52.904 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.944890+0000", 2024-09-21T22:07:52.904 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000144804 2024-09-21T22:07:52.904 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.904 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.904 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:52.904 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.992404+0000", 2024-09-21T22:07:52.904 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.047513211999999999 2024-09-21T22:07:52.904 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.904 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.904 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-21T22:07:52.904 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.992480+0000", 2024-09-21T22:07:52.904 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.6491999999999999e-05 2024-09-21T22:07:52.904 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.904 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.905 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:52.905 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.992521+0000", 2024-09-21T22:07:52.905 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.0608000000000002e-05 2024-09-21T22:07:52.905 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.905 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.905 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:52.905 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.992582+0000", 2024-09-21T22:07:52.905 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.1797000000000005e-05 2024-09-21T22:07:52.905 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.905 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:52.905 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.905 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.905 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.905 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4840.0:720 2.1 2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:07:52.906 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.803827+0000", 2024-09-21T22:07:52.906 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 27.049289290000001, 2024-09-21T22:07:52.906 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.049890510999999998, 2024-09-21T22:07:52.906 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:52.906 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:52.906 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:52.906 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-21T22:07:52.906 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4840", 2024-09-21T22:07:52.906 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:52.906 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 720 2024-09-21T22:07:52.906 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.906 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:52.906 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.906 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:52.907 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.803827+0000", 2024-09-21T22:07:52.907 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:52.907 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.907 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.907 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:52.907 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.803827+0000", 2024-09-21T22:07:52.907 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:52.907 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.907 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.907 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:52.907 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.803828+0000", 2024-09-21T22:07:52.907 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.569999999999999e-07 2024-09-21T22:07:52.907 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.907 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.907 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:52.908 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.804689+0000", 2024-09-21T22:07:52.908 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00086121799999999997 2024-09-21T22:07:52.908 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.908 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.908 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:52.908 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.804706+0000", 2024-09-21T22:07:52.908 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6266999999999999e-05 2024-09-21T22:07:52.908 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.908 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.908 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:52.908 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.804713+0000", 2024-09-21T22:07:52.908 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.1010000000000003e-06 2024-09-21T22:07:52.908 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.908 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.909 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:52.909 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.804781+0000", 2024-09-21T22:07:52.909 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.7871000000000001e-05 2024-09-21T22:07:52.909 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.909 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.909 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:52.909 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.804859+0000", 2024-09-21T22:07:52.909 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.7891000000000004e-05 2024-09-21T22:07:52.909 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.909 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.909 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:52.909 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.804971+0000", 2024-09-21T22:07:52.909 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00011275100000000001 2024-09-21T22:07:52.909 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.909 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.910 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:52.910 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.852704+0000", 2024-09-21T22:07:52.910 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.047732361000000001 2024-09-21T22:07:52.910 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.910 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.910 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-21T22:07:52.910 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.853446+0000", 2024-09-21T22:07:52.910 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00074258799999999997 2024-09-21T22:07:52.910 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.910 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.910 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:52.910 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.853608+0000", 2024-09-21T22:07:52.910 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00016190900000000001 2024-09-21T22:07:52.910 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.911 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.911 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:52.911 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.853718+0000", 2024-09-21T22:07:52.911 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000109597 2024-09-21T22:07:52.911 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.911 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:52.911 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.911 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.911 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.911 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4840.0:723 2.1 2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:07:52.911 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.860543+0000", 2024-09-21T22:07:52.911 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 26.992573856, 2024-09-21T22:07:52.911 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.062354957000000003, 2024-09-21T22:07:52.911 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:52.912 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:52.912 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:52.912 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-21T22:07:52.912 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4840", 2024-09-21T22:07:52.912 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:52.912 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 723 2024-09-21T22:07:52.912 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.912 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:52.912 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.912 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:52.912 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.860543+0000", 2024-09-21T22:07:52.912 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:52.912 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.912 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.913 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:52.913 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.860543+0000", 2024-09-21T22:07:52.913 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:52.913 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.913 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.913 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:52.913 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.860544+0000", 2024-09-21T22:07:52.913 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.1119999999999999e-06 2024-09-21T22:07:52.913 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.913 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.913 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:52.913 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.861400+0000", 2024-09-21T22:07:52.913 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000856393 2024-09-21T22:07:52.913 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.914 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.914 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:52.914 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.861419+0000", 2024-09-21T22:07:52.914 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8992e-05 2024-09-21T22:07:52.914 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.914 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.914 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:52.914 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.861428+0000", 2024-09-21T22:07:52.914 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.551e-06 2024-09-21T22:07:52.914 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.914 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.914 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:52.914 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.861555+0000", 2024-09-21T22:07:52.914 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000127286 2024-09-21T22:07:52.915 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.915 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.915 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:52.915 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.861816+0000", 2024-09-21T22:07:52.915 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00026100899999999998 2024-09-21T22:07:52.915 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.915 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.915 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:52.915 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.862189+0000", 2024-09-21T22:07:52.915 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00037261899999999998 2024-09-21T22:07:52.915 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.915 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.915 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:52.915 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.921514+0000", 2024-09-21T22:07:52.916 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.059325210000000003 2024-09-21T22:07:52.916 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.916 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.916 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-21T22:07:52.916 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.922721+0000", 2024-09-21T22:07:52.916 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0012066819999999999 2024-09-21T22:07:52.916 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.916 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.916 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:52.916 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.922797+0000", 2024-09-21T22:07:52.916 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.6162999999999996e-05 2024-09-21T22:07:52.916 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.916 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.916 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:52.917 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.922898+0000", 2024-09-21T22:07:52.917 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00010094000000000001 2024-09-21T22:07:52.917 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.917 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:52.917 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.917 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.917 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.917 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:725 2.0 e63/16 2:08b4f96a:::rbd_data.126483d75f2d.000000000000000f:head v 63'33, mlcod=63'31)", 2024-09-21T22:07:52.917 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.936778+0000", 2024-09-21T22:07:52.917 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 26.916338113999998, 2024-09-21T22:07:52.917 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.047671937999999997, 2024-09-21T22:07:52.917 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:52.917 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:52.917 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:52.918 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:52.918 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.918 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:52.918 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.936778+0000", 2024-09-21T22:07:52.918 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:52.918 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.918 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.918 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:52.918 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.936778+0000", 2024-09-21T22:07:52.918 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:52.918 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.918 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.918 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:52.918 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.936780+0000", 2024-09-21T22:07:52.919 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7799999999999999e-06 2024-09-21T22:07:52.919 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.919 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.919 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:52.919 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.938475+0000", 2024-09-21T22:07:52.919 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0016951749999999999 2024-09-21T22:07:52.919 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.919 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.919 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:52.919 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.938506+0000", 2024-09-21T22:07:52.919 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.0865999999999999e-05 2024-09-21T22:07:52.919 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.919 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.919 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:52.920 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.938520+0000", 2024-09-21T22:07:52.920 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3825999999999999e-05 2024-09-21T22:07:52.920 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.920 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.920 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:52.920 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.938624+0000", 2024-09-21T22:07:52.920 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00010391900000000001 2024-09-21T22:07:52.920 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.920 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.920 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:52.920 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.938651+0000", 2024-09-21T22:07:52.920 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.7481999999999999e-05 2024-09-21T22:07:52.920 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.920 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.921 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:52.921 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.984369+0000", 2024-09-21T22:07:52.921 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.045717595 2024-09-21T22:07:52.921 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.921 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.921 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:52.921 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.984450+0000", 2024-09-21T22:07:52.921 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.1295e-05 2024-09-21T22:07:52.921 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.921 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:52.921 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.921 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.921 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.922 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:727 2.0 e63/16 2:08b4f96a:::rbd_data.126483d75f2d.000000000000000f:head v 63'35, mlcod=63'33)", 2024-09-21T22:07:52.922 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.998090+0000", 2024-09-21T22:07:52.922 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 26.855026938000002, 2024-09-21T22:07:52.922 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.052205704999999998, 2024-09-21T22:07:52.922 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:52.922 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:52.922 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:52.922 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:52.922 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.922 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:52.922 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.998090+0000", 2024-09-21T22:07:52.922 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:52.922 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.923 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.923 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:52.923 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.998090+0000", 2024-09-21T22:07:52.923 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:52.923 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.923 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.923 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:52.923 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.998091+0000", 2024-09-21T22:07:52.923 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6929999999999999e-06 2024-09-21T22:07:52.923 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.923 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.923 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:52.923 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.998940+0000", 2024-09-21T22:07:52.923 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00084875200000000004 2024-09-21T22:07:52.924 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.924 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.924 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:52.924 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.998970+0000", 2024-09-21T22:07:52.924 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9709999999999998e-05 2024-09-21T22:07:52.924 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.924 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.924 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:52.924 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.998984+0000", 2024-09-21T22:07:52.924 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3739e-05 2024-09-21T22:07:52.924 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.924 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.924 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:52.924 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.999097+0000", 2024-09-21T22:07:52.925 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000113484 2024-09-21T22:07:52.925 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.925 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.925 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:52.925 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.999135+0000", 2024-09-21T22:07:52.925 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.8405999999999998e-05 2024-09-21T22:07:52.925 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.925 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.925 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:52.925 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.050245+0000", 2024-09-21T22:07:52.925 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.051109935000000002 2024-09-21T22:07:52.925 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.925 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.925 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:52.926 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.050295+0000", 2024-09-21T22:07:52.926 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.9985999999999999e-05 2024-09-21T22:07:52.926 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.926 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:52.926 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.926 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.926 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.926 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:729 2.0 e63/16 2:08b4f96a:::rbd_data.126483d75f2d.000000000000000f:head v 63'37, mlcod=63'35)", 2024-09-21T22:07:52.926 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:26.062424+0000", 2024-09-21T22:07:52.926 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 26.790692316000001, 2024-09-21T22:07:52.926 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.047312186999999999, 2024-09-21T22:07:52.926 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:52.926 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:52.926 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:52.927 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:52.927 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.927 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:52.927 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.062424+0000", 2024-09-21T22:07:52.927 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:52.927 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.927 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.927 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:52.927 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.062424+0000", 2024-09-21T22:07:52.927 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:52.927 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.927 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.927 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:52.928 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.062426+0000", 2024-09-21T22:07:52.928 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7129999999999999e-06 2024-09-21T22:07:52.928 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.928 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.928 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:52.928 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.063211+0000", 2024-09-21T22:07:52.928 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00078488500000000003 2024-09-21T22:07:52.928 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.928 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.928 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:52.928 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.063241+0000", 2024-09-21T22:07:52.928 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.0056000000000002e-05 2024-09-21T22:07:52.928 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.928 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.929 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:52.929 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.063255+0000", 2024-09-21T22:07:52.929 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4022999999999999e-05 2024-09-21T22:07:52.929 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.929 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.929 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:52.929 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.063384+0000", 2024-09-21T22:07:52.929 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00012920600000000001 2024-09-21T22:07:52.929 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.929 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.929 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:52.929 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.063421+0000", 2024-09-21T22:07:52.929 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.7242e-05 2024-09-21T22:07:52.930 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.930 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.930 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:52.930 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.109666+0000", 2024-09-21T22:07:52.930 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.046244446000000002 2024-09-21T22:07:52.930 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.930 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.930 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:52.930 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.109736+0000", 2024-09-21T22:07:52.930 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.0616000000000003e-05 2024-09-21T22:07:52.930 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.930 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:52.930 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.930 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.931 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.931 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:731 2.0 e63/16 2:08b4f96a:::rbd_data.126483d75f2d.000000000000000f:head v 63'39, mlcod=63'37)", 2024-09-21T22:07:52.931 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:26.135946+0000", 2024-09-21T22:07:52.931 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 26.717170808999999, 2024-09-21T22:07:52.931 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.054290359000000003, 2024-09-21T22:07:52.931 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:52.931 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:52.931 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:52.931 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:52.931 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.931 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:52.931 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.135946+0000", 2024-09-21T22:07:52.931 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:52.931 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.932 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.932 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:52.932 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.135946+0000", 2024-09-21T22:07:52.932 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:52.932 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.932 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.932 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:52.932 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.135948+0000", 2024-09-21T22:07:52.932 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7999999999999999e-06 2024-09-21T22:07:52.932 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.932 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.932 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:52.932 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.136730+0000", 2024-09-21T22:07:52.932 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00078271899999999997 2024-09-21T22:07:52.933 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.933 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.933 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:52.933 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.136760+0000", 2024-09-21T22:07:52.933 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9876e-05 2024-09-21T22:07:52.933 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.933 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.933 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:52.933 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.136774+0000", 2024-09-21T22:07:52.933 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3478999999999999e-05 2024-09-21T22:07:52.933 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.933 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.933 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:52.933 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.136944+0000", 2024-09-21T22:07:52.933 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000170626 2024-09-21T22:07:52.934 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.934 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.934 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:52.934 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.136985+0000", 2024-09-21T22:07:52.934 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.0377999999999999e-05 2024-09-21T22:07:52.934 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.934 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.934 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:52.934 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.190137+0000", 2024-09-21T22:07:52.934 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.053152438000000003 2024-09-21T22:07:52.934 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.934 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.934 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:52.934 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.190236+0000", 2024-09-21T22:07:52.935 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.9042999999999993e-05 2024-09-21T22:07:52.935 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.935 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:52.935 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.935 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.935 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.935 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:125 2.3 e71/66 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head v 71'31, mlcod=71'29)", 2024-09-21T22:07:52.935 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.066855+0000", 2024-09-21T22:07:52.935 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 14.786261374, 2024-09-21T22:07:52.935 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.059856059000000003, 2024-09-21T22:07:52.935 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:52.935 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:52.935 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:52.935 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:52.936 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.936 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:52.936 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.066855+0000", 2024-09-21T22:07:52.936 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:52.936 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.936 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.936 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:52.936 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.066855+0000", 2024-09-21T22:07:52.936 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:52.936 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.936 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.936 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:52.936 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.066857+0000", 2024-09-21T22:07:52.936 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.807e-06 2024-09-21T22:07:52.937 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.937 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.937 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:52.937 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.068365+0000", 2024-09-21T22:07:52.937 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0015077910000000001 2024-09-21T22:07:52.937 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.937 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.937 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:52.937 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.068395+0000", 2024-09-21T22:07:52.937 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.0326000000000001e-05 2024-09-21T22:07:52.937 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.937 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.937 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:52.937 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.068415+0000", 2024-09-21T22:07:52.938 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9871999999999999e-05 2024-09-21T22:07:52.938 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.938 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.938 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:52.938 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.068580+0000", 2024-09-21T22:07:52.938 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000165317 2024-09-21T22:07:52.938 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.938 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.938 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:52.938 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.068618+0000", 2024-09-21T22:07:52.938 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.7753000000000003e-05 2024-09-21T22:07:52.938 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.938 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.938 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:52.939 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.126623+0000", 2024-09-21T22:07:52.939 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.058004642000000002 2024-09-21T22:07:52.939 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.939 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.939 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:52.939 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.126711+0000", 2024-09-21T22:07:52.939 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.8551e-05 2024-09-21T22:07:52.939 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.939 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:52.939 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.939 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.939 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.939 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:127 2.3 e71/66 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head v 71'33, mlcod=71'31)", 2024-09-21T22:07:52.939 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.140864+0000", 2024-09-21T22:07:52.939 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 14.712252346, 2024-09-21T22:07:52.940 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.052512380999999997, 2024-09-21T22:07:52.940 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:52.940 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:52.940 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:52.940 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:52.940 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.940 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:52.940 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.140864+0000", 2024-09-21T22:07:52.940 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:52.940 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.940 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.940 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:52.940 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.140864+0000", 2024-09-21T22:07:52.940 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:52.941 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.941 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.941 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:52.941 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.140866+0000", 2024-09-21T22:07:52.941 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.694e-06 2024-09-21T22:07:52.941 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.941 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.941 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:52.941 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.141712+0000", 2024-09-21T22:07:52.941 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00084581499999999998 2024-09-21T22:07:52.941 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.941 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.941 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:52.941 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.141742+0000", 2024-09-21T22:07:52.942 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9865999999999998e-05 2024-09-21T22:07:52.942 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.942 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.942 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:52.942 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.141761+0000", 2024-09-21T22:07:52.942 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9473000000000001e-05 2024-09-21T22:07:52.942 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.942 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.942 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:52.942 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.141919+0000", 2024-09-21T22:07:52.942 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00015758300000000001 2024-09-21T22:07:52.942 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.942 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.943 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:52.943 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.141955+0000", 2024-09-21T22:07:52.943 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.6613000000000003e-05 2024-09-21T22:07:52.943 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.943 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.943 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:52.943 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.193333+0000", 2024-09-21T22:07:52.943 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.051377260000000001 2024-09-21T22:07:52.943 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.943 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.943 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:52.943 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.193377+0000", 2024-09-21T22:07:52.943 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.4076999999999998e-05 2024-09-21T22:07:52.943 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.944 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:52.944 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.944 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.944 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.944 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:129 2.3 e71/66 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head v 71'35, mlcod=71'33)", 2024-09-21T22:07:52.944 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.209458+0000", 2024-09-21T22:07:52.944 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 14.643658597, 2024-09-21T22:07:52.944 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.056595578000000001, 2024-09-21T22:07:52.944 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:52.944 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:52.944 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:52.944 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:52.944 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.944 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:52.945 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.209458+0000", 2024-09-21T22:07:52.945 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:52.945 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.945 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.945 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:52.945 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.209458+0000", 2024-09-21T22:07:52.945 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:52.945 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.945 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.945 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:52.945 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.209460+0000", 2024-09-21T22:07:52.945 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7600000000000001e-06 2024-09-21T22:07:52.945 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.945 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.946 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:52.946 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.210317+0000", 2024-09-21T22:07:52.946 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00085771700000000003 2024-09-21T22:07:52.946 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.946 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.946 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:52.946 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.210346+0000", 2024-09-21T22:07:52.946 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8334e-05 2024-09-21T22:07:52.946 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.946 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.946 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:52.946 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.210365+0000", 2024-09-21T22:07:52.946 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9439000000000001e-05 2024-09-21T22:07:52.946 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.947 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.947 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:52.947 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.210544+0000", 2024-09-21T22:07:52.947 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00017845599999999999 2024-09-21T22:07:52.947 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.947 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.947 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:52.947 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.210584+0000", 2024-09-21T22:07:52.947 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.0423000000000002e-05 2024-09-21T22:07:52.947 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.947 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.947 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:52.947 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.265970+0000", 2024-09-21T22:07:52.948 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.055385765000000003 2024-09-21T22:07:52.948 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.948 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.948 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:52.948 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.266054+0000", 2024-09-21T22:07:52.948 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.3683999999999994e-05 2024-09-21T22:07:52.948 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.948 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:52.948 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.948 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.948 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.948 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:131 2.3 e71/66 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head v 71'37, mlcod=71'35)", 2024-09-21T22:07:52.948 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.279550+0000", 2024-09-21T22:07:52.948 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 14.573566759, 2024-09-21T22:07:52.948 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.052290419999999997, 2024-09-21T22:07:52.949 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:52.949 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:52.949 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:52.949 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:52.949 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.949 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:52.949 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.279550+0000", 2024-09-21T22:07:52.949 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:52.949 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.949 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.949 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:52.949 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.279550+0000", 2024-09-21T22:07:52.949 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:52.949 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.950 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.950 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:52.950 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.279552+0000", 2024-09-21T22:07:52.950 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9439999999999999e-06 2024-09-21T22:07:52.950 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.950 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.950 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:52.950 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.280296+0000", 2024-09-21T22:07:52.950 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00074417000000000005 2024-09-21T22:07:52.950 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.950 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.950 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:52.950 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.280324+0000", 2024-09-21T22:07:52.950 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8045999999999999e-05 2024-09-21T22:07:52.951 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.951 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.951 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:52.951 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.280345+0000", 2024-09-21T22:07:52.951 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.0849999999999999e-05 2024-09-21T22:07:52.951 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.951 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.951 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:52.951 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.280497+0000", 2024-09-21T22:07:52.951 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00015252500000000001 2024-09-21T22:07:52.951 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.951 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.951 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:52.951 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.280531+0000", 2024-09-21T22:07:52.951 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.4034e-05 2024-09-21T22:07:52.952 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.952 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.952 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:52.952 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.331769+0000", 2024-09-21T22:07:52.952 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.051237999999999999 2024-09-21T22:07:52.952 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:52.952 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:52.952 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:52.952 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.331840+0000", 2024-09-21T22:07:52.952 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.0851000000000003e-05 2024-09-21T22:07:52.952 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.952 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:52.952 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.952 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:52.952 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:52.953 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:07:52.955 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T22:07:52.955 INFO:tasks.workunit.client.0.smithi157.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-09-21T22:07:52.955 INFO:tasks.workunit.client.0.smithi157.stderr:+ _sudo dd if=/tmp/tmp.GqKuv7rsZi/data of=/dev/nbd1 bs=1M count=1 oflag=direct 2024-09-21T22:07:52.955 INFO:tasks.workunit.client.0.smithi157.stderr:+ local cmd 2024-09-21T22:07:52.956 INFO:tasks.workunit.client.0.smithi157.stderr:++ id -u 2024-09-21T22:07:52.956 INFO:tasks.workunit.client.0.smithi157.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T22:07:52.956 INFO:tasks.workunit.client.0.smithi157.stderr:++ which dd 2024-09-21T22:07:52.956 INFO:tasks.workunit.client.0.smithi157.stderr:+ cmd=/usr/bin/dd 2024-09-21T22:07:52.956 INFO:tasks.workunit.client.0.smithi157.stderr:+ shift 2024-09-21T22:07:52.956 INFO:tasks.workunit.client.0.smithi157.stderr:+ sudo -nE /usr/bin/dd if=/tmp/tmp.GqKuv7rsZi/data of=/dev/nbd1 bs=1M count=1 oflag=direct 2024-09-21T22:07:52.956 INFO:teuthology.orchestra.run.smithi039.stderr:nodeep-scrub is set 2024-09-21T22:07:52.957 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:52.952+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:53.025 INFO:tasks.workunit.client.0.smithi157.stderr:1+0 records in 2024-09-21T22:07:53.025 INFO:tasks.workunit.client.0.smithi157.stderr:1+0 records out 2024-09-21T22:07:53.025 INFO:tasks.workunit.client.0.smithi157.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0948812 s, 11.1 MB/s 2024-09-21T22:07:53.027 INFO:tasks.workunit.client.0.smithi157.stderr:++ cat /tmp/tmp.GqKuv7rsZi/quiesce 2024-09-21T22:07:53.029 INFO:tasks.workunit.client.0.smithi157.stderr:+ test /dev/nbd1 = /dev/nbd1 2024-09-21T22:07:53.029 INFO:tasks.workunit.client.0.smithi157.stderr:++ cat /tmp/tmp.GqKuv7rsZi/unquiesce 2024-09-21T22:07:53.030 INFO:tasks.workunit.client.0.smithi157.stderr:+ test /dev/nbd1 = /dev/nbd1 2024-09-21T22:07:53.030 INFO:tasks.workunit.client.0.smithi157.stderr:+ touch /tmp/tmp.GqKuv7rsZi/quiesce.sh 2024-09-21T22:07:53.032 INFO:tasks.workunit.client.0.smithi157.stderr:+ chmod +x /tmp/tmp.GqKuv7rsZi/quiesce.sh 2024-09-21T22:07:53.033 INFO:tasks.workunit.client.0.smithi157.stderr:+ cat 2024-09-21T22:07:53.034 INFO:tasks.workunit.client.0.smithi157.stderr:+ expect_false rbd snap create rbd/testrbdnbd21553@quiesce2 2024-09-21T22:07:53.035 INFO:tasks.workunit.client.0.smithi157.stderr:+ rbd snap create rbd/testrbdnbd21553@quiesce2 2024-09-21T22:07:53.055 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:53.050+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:53.113 INFO:tasks.workunit.client.0.smithi157.stderr: Creating snap: 10% complete... Creating snap: 10% complete...failed. 2024-09-21T22:07:53.113 INFO:tasks.workunit.client.0.smithi157.stderr:rbd: failed to create snapshot: (22) Invalid argument 2024-09-21T22:07:53.122 INFO:tasks.workunit.client.0.smithi157.stderr:+ return 0 2024-09-21T22:07:53.122 INFO:tasks.workunit.client.0.smithi157.stderr:+ _sudo dd if=/tmp/tmp.GqKuv7rsZi/data of=/dev/nbd1 bs=1M count=1 oflag=direct 2024-09-21T22:07:53.122 INFO:tasks.workunit.client.0.smithi157.stderr:+ local cmd 2024-09-21T22:07:53.122 INFO:tasks.workunit.client.0.smithi157.stderr:++ id -u 2024-09-21T22:07:53.124 INFO:tasks.workunit.client.0.smithi157.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T22:07:53.124 INFO:tasks.workunit.client.0.smithi157.stderr:++ which dd 2024-09-21T22:07:53.125 INFO:tasks.workunit.client.0.smithi157.stderr:+ cmd=/usr/bin/dd 2024-09-21T22:07:53.125 INFO:tasks.workunit.client.0.smithi157.stderr:+ shift 2024-09-21T22:07:53.125 INFO:tasks.workunit.client.0.smithi157.stderr:+ sudo -nE /usr/bin/dd if=/tmp/tmp.GqKuv7rsZi/data of=/dev/nbd1 bs=1M count=1 oflag=direct 2024-09-21T22:07:53.129 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:07:53.129 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:07:53.129 INFO:teuthology.orchestra.run.smithi039.stdout: "num_ops": 0 2024-09-21T22:07:53.129 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:07:53.143 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T22:07:53.188 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:53.186+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:53.209 INFO:tasks.workunit.client.0.smithi157.stderr:1+0 records in 2024-09-21T22:07:53.209 INFO:tasks.workunit.client.0.smithi157.stderr:1+0 records out 2024-09-21T22:07:53.209 INFO:tasks.workunit.client.0.smithi157.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0769992 s, 13.6 MB/s 2024-09-21T22:07:53.210 INFO:tasks.workunit.client.0.smithi157.stderr:+ cat 2024-09-21T22:07:53.213 INFO:tasks.workunit.client.0.smithi157.stderr:+ rbd snap create rbd/testrbdnbd21553@quiesce2 2024-09-21T22:07:53.257 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:07:53.252+0000 7f2075a06640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31720) UID: 0 2024-09-21T22:07:53.313 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:07:53.313 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:07:53.313 INFO:teuthology.orchestra.run.smithi039.stdout: "complaint_time": 30, 2024-09-21T22:07:53.313 INFO:teuthology.orchestra.run.smithi039.stdout: "num_blocked_ops": 0 2024-09-21T22:07:53.314 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:07:53.328 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T22:07:53.358 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:07:53.352+0000 7f2075a06640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31720) UID: 0 2024-09-21T22:07:53.450 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:07:53.450 INFO:teuthology.orchestra.run.smithi039.stdout: "size": 20, 2024-09-21T22:07:53.450 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 600, 2024-09-21T22:07:53.450 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [ 2024-09-21T22:07:53.450 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.450 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPull(2.3 e55/54 cost 1048576)", 2024-09-21T22:07:53.450 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.081116+0000", 2024-09-21T22:07:53.450 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 41.365325284999997, 2024-09-21T22:07:53.450 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0030214159999999999, 2024-09-21T22:07:53.451 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:53.451 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:53.451 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-21T22:07:53.451 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:53.451 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.451 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:53.451 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.081116+0000", 2024-09-21T22:07:53.451 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:53.451 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.451 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.451 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:53.451 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.081116+0000", 2024-09-21T22:07:53.451 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:53.452 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.452 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.452 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:53.452 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.081118+0000", 2024-09-21T22:07:53.452 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.545e-06 2024-09-21T22:07:53.452 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.452 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.452 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:53.452 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.081133+0000", 2024-09-21T22:07:53.452 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5642e-05 2024-09-21T22:07:53.452 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.452 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.452 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:53.452 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.081155+0000", 2024-09-21T22:07:53.453 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1858e-05 2024-09-21T22:07:53.453 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.453 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.453 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:53.453 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.081171+0000", 2024-09-21T22:07:53.453 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5804e-05 2024-09-21T22:07:53.453 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.453 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.453 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:53.453 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.081291+0000", 2024-09-21T22:07:53.453 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000120553 2024-09-21T22:07:53.453 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.453 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.453 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:53.453 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.084137+0000", 2024-09-21T22:07:53.454 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.002846014 2024-09-21T22:07:53.454 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:53.454 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:53.454 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:53.454 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.454 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.454 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPull(2.3 e55/54 cost 1048576)", 2024-09-21T22:07:53.454 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.136357+0000", 2024-09-21T22:07:53.454 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 41.310084218999997, 2024-09-21T22:07:53.454 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.002843904, 2024-09-21T22:07:53.455 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:53.455 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:53.455 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-21T22:07:53.455 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:53.455 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.455 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:53.455 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.136357+0000", 2024-09-21T22:07:53.455 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:53.455 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.455 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.455 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:53.455 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.136357+0000", 2024-09-21T22:07:53.455 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:53.455 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.456 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.456 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:53.456 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.136359+0000", 2024-09-21T22:07:53.456 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.0669999999999999e-06 2024-09-21T22:07:53.456 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.456 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.456 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:53.456 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.136381+0000", 2024-09-21T22:07:53.456 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.2104999999999999e-05 2024-09-21T22:07:53.456 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.456 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.457 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:53.457 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.136412+0000", 2024-09-21T22:07:53.457 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.0602999999999998e-05 2024-09-21T22:07:53.457 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.457 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.457 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:53.457 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.136427+0000", 2024-09-21T22:07:53.457 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4832e-05 2024-09-21T22:07:53.457 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.457 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.457 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:53.458 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.136569+0000", 2024-09-21T22:07:53.458 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000142702 2024-09-21T22:07:53.458 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.458 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.458 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:53.458 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.139201+0000", 2024-09-21T22:07:53.458 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.002631595 2024-09-21T22:07:53.458 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:53.458 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:53.458 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:53.458 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.458 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.458 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:667 2.3 e63/54 2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head v 63'4, mlcod=0'0)", 2024-09-21T22:07:53.458 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.402454+0000", 2024-09-21T22:07:53.459 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 29.043987700999999, 2024-09-21T22:07:53.459 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0058375959999999996, 2024-09-21T22:07:53.459 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:53.459 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:53.459 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:53.459 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:53.459 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.459 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:53.459 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.402454+0000", 2024-09-21T22:07:53.459 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:53.459 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.459 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.459 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:53.459 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.402454+0000", 2024-09-21T22:07:53.460 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:53.460 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.460 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.460 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:53.460 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.402456+0000", 2024-09-21T22:07:53.460 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1179999999999999e-06 2024-09-21T22:07:53.460 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.460 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.460 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:53.460 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.402618+0000", 2024-09-21T22:07:53.460 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00016243 2024-09-21T22:07:53.460 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.460 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.460 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:53.461 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.402643+0000", 2024-09-21T22:07:53.461 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4638999999999999e-05 2024-09-21T22:07:53.461 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.461 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.461 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:53.461 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.402658+0000", 2024-09-21T22:07:53.461 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5211000000000001e-05 2024-09-21T22:07:53.461 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.461 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.461 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:53.461 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.402756+0000", 2024-09-21T22:07:53.461 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.7880000000000005e-05 2024-09-21T22:07:53.461 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.462 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.462 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:53.462 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.402779+0000", 2024-09-21T22:07:53.462 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3524999999999999e-05 2024-09-21T22:07:53.462 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.462 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.462 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:53.462 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.408233+0000", 2024-09-21T22:07:53.462 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0054534700000000002 2024-09-21T22:07:53.462 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.462 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.462 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:53.462 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.408291+0000", 2024-09-21T22:07:53.462 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.8322999999999997e-05 2024-09-21T22:07:53.463 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:53.463 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:53.463 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:53.463 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.463 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.463 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:668 2.3 e63/54 2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head v 63'5, mlcod=52'3)", 2024-09-21T22:07:53.463 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.410724+0000", 2024-09-21T22:07:53.463 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 29.035717284, 2024-09-21T22:07:53.463 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.031657809000000002, 2024-09-21T22:07:53.463 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:53.463 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:53.463 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:53.463 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:53.463 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.464 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:53.464 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.410724+0000", 2024-09-21T22:07:53.464 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:53.464 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.464 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.464 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:53.464 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.410724+0000", 2024-09-21T22:07:53.464 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:53.464 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.464 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.464 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:53.464 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.410725+0000", 2024-09-21T22:07:53.464 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.4300000000000001e-07 2024-09-21T22:07:53.464 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.465 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.465 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:53.465 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.411436+0000", 2024-09-21T22:07:53.465 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00071118600000000004 2024-09-21T22:07:53.465 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.465 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.465 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:53.465 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.411455+0000", 2024-09-21T22:07:53.465 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8357999999999999e-05 2024-09-21T22:07:53.465 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.466 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.466 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:53.466 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.411470+0000", 2024-09-21T22:07:53.466 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5753999999999999e-05 2024-09-21T22:07:53.466 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.466 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.466 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:53.466 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.411537+0000", 2024-09-21T22:07:53.466 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.6766000000000004e-05 2024-09-21T22:07:53.466 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.466 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.466 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:53.466 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.411571+0000", 2024-09-21T22:07:53.466 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.3555e-05 2024-09-21T22:07:53.467 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.467 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.467 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:53.467 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.442317+0000", 2024-09-21T22:07:53.467 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.030746393 2024-09-21T22:07:53.467 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.467 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.467 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:53.467 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.442382+0000", 2024-09-21T22:07:53.467 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.4853999999999999e-05 2024-09-21T22:07:53.467 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:53.467 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:53.467 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:53.467 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.467 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.468 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:669 2.3 e63/54 2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head v 63'6, mlcod=63'4)", 2024-09-21T22:07:53.468 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.459924+0000", 2024-09-21T22:07:53.468 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 28.986517725999999, 2024-09-21T22:07:53.468 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0062657900000000002, 2024-09-21T22:07:53.468 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:53.468 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:53.468 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:53.468 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:53.468 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.468 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:53.468 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.459924+0000", 2024-09-21T22:07:53.468 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:53.468 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.468 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.469 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:53.469 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.459924+0000", 2024-09-21T22:07:53.469 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:53.469 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.469 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.469 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:53.469 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.459925+0000", 2024-09-21T22:07:53.469 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8169999999999999e-06 2024-09-21T22:07:53.469 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.469 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.469 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:53.469 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.459983+0000", 2024-09-21T22:07:53.469 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.7567e-05 2024-09-21T22:07:53.469 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.470 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.470 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:53.470 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.460011+0000", 2024-09-21T22:07:53.470 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8455000000000001e-05 2024-09-21T22:07:53.470 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.470 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.470 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:53.470 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.460021+0000", 2024-09-21T22:07:53.470 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.9410000000000002e-06 2024-09-21T22:07:53.470 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.470 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.470 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:53.470 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.460093+0000", 2024-09-21T22:07:53.470 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.1231999999999998e-05 2024-09-21T22:07:53.471 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.471 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.471 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:53.471 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.460124+0000", 2024-09-21T22:07:53.471 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.1335999999999999e-05 2024-09-21T22:07:53.471 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.471 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.471 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:53.471 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.466130+0000", 2024-09-21T22:07:53.471 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0060055519999999999 2024-09-21T22:07:53.471 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.471 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.471 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:53.471 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.466189+0000", 2024-09-21T22:07:53.472 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.9889999999999997e-05 2024-09-21T22:07:53.472 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:53.472 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:53.472 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:53.472 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.472 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.472 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:670 2.3 e63/54 2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head v 63'7, mlcod=63'5)", 2024-09-21T22:07:53.472 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.468764+0000", 2024-09-21T22:07:53.472 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 28.977677745000001, 2024-09-21T22:07:53.472 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.037963433999999997, 2024-09-21T22:07:53.472 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:53.472 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:53.472 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:53.472 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:53.472 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.473 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:53.473 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.468764+0000", 2024-09-21T22:07:53.473 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:53.473 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.473 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.473 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:53.473 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.468764+0000", 2024-09-21T22:07:53.473 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:53.473 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.473 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.473 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:53.473 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.468765+0000", 2024-09-21T22:07:53.473 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.9699999999999994e-07 2024-09-21T22:07:53.473 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.474 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.474 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:53.474 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.469295+0000", 2024-09-21T22:07:53.474 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00053003600000000001 2024-09-21T22:07:53.474 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.474 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.474 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:53.474 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.469311+0000", 2024-09-21T22:07:53.474 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6841999999999999e-05 2024-09-21T22:07:53.474 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.474 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.474 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:53.474 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.469324+0000", 2024-09-21T22:07:53.474 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2373e-05 2024-09-21T22:07:53.475 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.475 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.475 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:53.475 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.469391+0000", 2024-09-21T22:07:53.475 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.6880999999999999e-05 2024-09-21T22:07:53.475 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.475 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.475 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:53.475 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.469419+0000", 2024-09-21T22:07:53.475 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8742999999999999e-05 2024-09-21T22:07:53.475 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.475 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.475 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:53.475 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.506662+0000", 2024-09-21T22:07:53.475 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.037242994000000001 2024-09-21T22:07:53.476 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.476 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.476 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:53.476 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.506727+0000", 2024-09-21T22:07:53.476 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.4567999999999996e-05 2024-09-21T22:07:53.476 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:53.476 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:53.476 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:53.476 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.476 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.476 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:671 2.3 e63/54 2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head v 63'8, mlcod=63'6)", 2024-09-21T22:07:53.476 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.519938+0000", 2024-09-21T22:07:53.476 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 28.926503206, 2024-09-21T22:07:53.476 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0058323419999999999, 2024-09-21T22:07:53.476 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:53.477 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:53.477 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:53.477 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:53.477 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.477 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:53.477 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.519938+0000", 2024-09-21T22:07:53.477 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:53.477 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.477 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.477 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:53.477 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.519938+0000", 2024-09-21T22:07:53.477 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:53.477 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.477 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.478 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:53.478 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.519940+0000", 2024-09-21T22:07:53.478 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.666e-06 2024-09-21T22:07:53.478 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.478 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.478 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:53.478 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.520031+0000", 2024-09-21T22:07:53.478 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.1051999999999995e-05 2024-09-21T22:07:53.478 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.478 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.478 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:53.478 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.520056+0000", 2024-09-21T22:07:53.478 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5459000000000001e-05 2024-09-21T22:07:53.478 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.479 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.479 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:53.479 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.520072+0000", 2024-09-21T22:07:53.479 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5801e-05 2024-09-21T22:07:53.479 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.479 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.479 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:53.479 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.520178+0000", 2024-09-21T22:07:53.479 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000105388 2024-09-21T22:07:53.479 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.479 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.479 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:53.479 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.520204+0000", 2024-09-21T22:07:53.479 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.6208e-05 2024-09-21T22:07:53.480 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.480 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.480 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:53.480 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.525735+0000", 2024-09-21T22:07:53.480 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0055308520000000002 2024-09-21T22:07:53.480 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.480 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.480 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:53.480 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.525770+0000", 2024-09-21T22:07:53.480 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.5916e-05 2024-09-21T22:07:53.480 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:53.480 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:53.480 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:53.481 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.481 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.481 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:672 2.3 e63/54 2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head v 63'9, mlcod=63'7)", 2024-09-21T22:07:53.481 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.528432+0000", 2024-09-21T22:07:53.481 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 28.918009829999999, 2024-09-21T22:07:53.481 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.029763412999999999, 2024-09-21T22:07:53.481 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:53.481 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:53.481 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:53.481 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:53.481 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.481 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:53.481 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.528432+0000", 2024-09-21T22:07:53.481 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:53.482 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.482 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.482 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:53.482 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.528432+0000", 2024-09-21T22:07:53.482 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:53.482 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.482 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.482 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:53.482 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.528433+0000", 2024-09-21T22:07:53.482 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7179999999999999e-06 2024-09-21T22:07:53.482 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.482 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.482 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:53.483 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.528926+0000", 2024-09-21T22:07:53.483 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00049238500000000002 2024-09-21T22:07:53.483 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.483 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.483 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:53.483 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.528955+0000", 2024-09-21T22:07:53.483 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9037e-05 2024-09-21T22:07:53.483 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.483 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.483 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:53.483 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.528971+0000", 2024-09-21T22:07:53.483 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6622000000000001e-05 2024-09-21T22:07:53.483 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.483 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.483 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:53.484 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.529047+0000", 2024-09-21T22:07:53.484 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.5246000000000004e-05 2024-09-21T22:07:53.484 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.484 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.484 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:53.484 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.529074+0000", 2024-09-21T22:07:53.484 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.7532e-05 2024-09-21T22:07:53.484 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.484 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.484 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:53.484 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.558149+0000", 2024-09-21T22:07:53.484 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.029075192999999999 2024-09-21T22:07:53.484 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.484 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.485 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:53.485 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.558195+0000", 2024-09-21T22:07:53.485 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.5680000000000003e-05 2024-09-21T22:07:53.485 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:53.485 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:53.485 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:53.485 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.485 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.485 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:673 2.3 e63/54 2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head v 63'10, mlcod=63'8)", 2024-09-21T22:07:53.485 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.576513+0000", 2024-09-21T22:07:53.485 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 28.869928394999999, 2024-09-21T22:07:53.485 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.003674666, 2024-09-21T22:07:53.485 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:53.485 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:53.486 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:53.486 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:53.486 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.486 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:53.486 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.576513+0000", 2024-09-21T22:07:53.486 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:53.486 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.486 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.486 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:53.486 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.576513+0000", 2024-09-21T22:07:53.486 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:53.486 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.486 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.487 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:53.487 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.576514+0000", 2024-09-21T22:07:53.487 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.1299999999999998e-07 2024-09-21T22:07:53.487 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.487 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.487 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:53.487 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.576567+0000", 2024-09-21T22:07:53.487 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.2710000000000002e-05 2024-09-21T22:07:53.487 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.487 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.487 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:53.487 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.576579+0000", 2024-09-21T22:07:53.487 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2035000000000001e-05 2024-09-21T22:07:53.487 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.487 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.488 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:53.488 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.576585+0000", 2024-09-21T22:07:53.488 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.6050000000000003e-06 2024-09-21T22:07:53.488 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.488 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.488 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:53.488 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.576638+0000", 2024-09-21T22:07:53.488 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.2726999999999998e-05 2024-09-21T22:07:53.488 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.488 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.488 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:53.488 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.576656+0000", 2024-09-21T22:07:53.488 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8084000000000002e-05 2024-09-21T22:07:53.488 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.489 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.489 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:53.489 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.580156+0000", 2024-09-21T22:07:53.489 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0035002700000000002 2024-09-21T22:07:53.489 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.489 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.489 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:53.489 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.580188+0000", 2024-09-21T22:07:53.489 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.1322000000000003e-05 2024-09-21T22:07:53.489 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:53.489 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:53.489 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:53.489 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.490 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.490 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:674 2.3 e63/54 2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head v 63'11, mlcod=63'9)", 2024-09-21T22:07:53.490 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.584683+0000", 2024-09-21T22:07:53.490 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 28.861757954000002, 2024-09-21T22:07:53.490 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.028028352999999999, 2024-09-21T22:07:53.490 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:53.490 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:53.490 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:53.490 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:53.490 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.490 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:53.490 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.584683+0000", 2024-09-21T22:07:53.490 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:53.490 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.491 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.491 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:53.491 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.584683+0000", 2024-09-21T22:07:53.491 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:53.491 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.491 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.491 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:53.491 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.584684+0000", 2024-09-21T22:07:53.491 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.4600000000000004e-07 2024-09-21T22:07:53.491 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.491 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.491 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:53.491 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.585041+0000", 2024-09-21T22:07:53.491 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000356752 2024-09-21T22:07:53.491 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.492 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.492 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:53.492 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.585054+0000", 2024-09-21T22:07:53.492 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3387000000000001e-05 2024-09-21T22:07:53.492 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.492 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.492 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:53.492 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.585065+0000", 2024-09-21T22:07:53.492 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.1124999999999999e-05 2024-09-21T22:07:53.492 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.492 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.492 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:53.492 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.585142+0000", 2024-09-21T22:07:53.492 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.6743999999999996e-05 2024-09-21T22:07:53.492 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.493 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.493 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:53.493 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.585170+0000", 2024-09-21T22:07:53.493 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.7569999999999999e-05 2024-09-21T22:07:53.493 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.493 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.493 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:53.493 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.612637+0000", 2024-09-21T22:07:53.493 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.027467803999999998 2024-09-21T22:07:53.493 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.493 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.493 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:53.493 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.612712+0000", 2024-09-21T22:07:53.493 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.4325000000000004e-05 2024-09-21T22:07:53.494 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:53.494 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:53.494 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:53.494 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.494 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.494 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:683 2.3 e63/54 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head v 63'12, mlcod=63'10)", 2024-09-21T22:07:53.494 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.888242+0000", 2024-09-21T22:07:53.494 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 28.558199336000001, 2024-09-21T22:07:53.494 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0085158290000000008, 2024-09-21T22:07:53.494 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:53.494 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:53.494 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:53.494 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:53.494 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.495 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:53.495 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.888242+0000", 2024-09-21T22:07:53.495 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:53.495 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.495 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.495 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:53.495 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.888242+0000", 2024-09-21T22:07:53.495 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:53.495 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.495 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.495 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:53.495 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.888244+0000", 2024-09-21T22:07:53.495 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.553e-06 2024-09-21T22:07:53.495 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.495 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.496 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:53.496 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.888295+0000", 2024-09-21T22:07:53.496 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.1916000000000002e-05 2024-09-21T22:07:53.496 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.496 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.496 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:53.496 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.888319+0000", 2024-09-21T22:07:53.496 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3139999999999999e-05 2024-09-21T22:07:53.496 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.496 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.496 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:53.496 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.888332+0000", 2024-09-21T22:07:53.496 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2993000000000001e-05 2024-09-21T22:07:53.496 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.497 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.497 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:53.497 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.888441+0000", 2024-09-21T22:07:53.497 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000109039 2024-09-21T22:07:53.497 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.497 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.497 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:53.497 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.888481+0000", 2024-09-21T22:07:53.497 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.9950000000000002e-05 2024-09-21T22:07:53.497 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.497 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.497 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:53.497 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.896688+0000", 2024-09-21T22:07:53.497 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0082078729999999992 2024-09-21T22:07:53.498 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.498 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.498 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:53.498 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.896758+0000", 2024-09-21T22:07:53.498 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.9364999999999997e-05 2024-09-21T22:07:53.498 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:53.498 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:53.498 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:53.498 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.498 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.498 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:684 2.3 e63/54 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head v 63'13, mlcod=63'11)", 2024-09-21T22:07:53.498 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.899243+0000", 2024-09-21T22:07:53.498 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 28.547198559000002, 2024-09-21T22:07:53.498 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.033436192000000003, 2024-09-21T22:07:53.499 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:53.499 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:53.499 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:53.499 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:53.499 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.499 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:53.499 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.899243+0000", 2024-09-21T22:07:53.499 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:53.499 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.499 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.499 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:53.499 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.899243+0000", 2024-09-21T22:07:53.499 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:53.499 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.500 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.500 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:53.500 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.899244+0000", 2024-09-21T22:07:53.500 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.705e-06 2024-09-21T22:07:53.500 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.500 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.500 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:53.500 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.899604+0000", 2024-09-21T22:07:53.500 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00035930599999999999 2024-09-21T22:07:53.500 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.500 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.500 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:53.500 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.899622+0000", 2024-09-21T22:07:53.500 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8487000000000001e-05 2024-09-21T22:07:53.501 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.501 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.501 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:53.501 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.899634+0000", 2024-09-21T22:07:53.501 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2116999999999999e-05 2024-09-21T22:07:53.501 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.501 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.501 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:53.501 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.899721+0000", 2024-09-21T22:07:53.501 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.6718999999999994e-05 2024-09-21T22:07:53.501 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.501 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.501 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:53.501 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.899755+0000", 2024-09-21T22:07:53.501 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.4292000000000003e-05 2024-09-21T22:07:53.502 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.502 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.502 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:53.502 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.932545+0000", 2024-09-21T22:07:53.502 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.03278959 2024-09-21T22:07:53.502 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.502 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.502 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:53.502 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.932679+0000", 2024-09-21T22:07:53.502 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013397600000000001 2024-09-21T22:07:53.502 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:53.502 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:53.502 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:53.502 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.503 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.503 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:685 2.3 e63/54 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head v 63'14, mlcod=63'12)", 2024-09-21T22:07:53.503 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.935246+0000", 2024-09-21T22:07:53.503 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 28.511195722, 2024-09-21T22:07:53.503 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0073590590000000003, 2024-09-21T22:07:53.503 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:53.503 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:53.503 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:53.503 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:53.503 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.503 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:53.503 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.935246+0000", 2024-09-21T22:07:53.503 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:53.503 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.504 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.504 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:53.504 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.935246+0000", 2024-09-21T22:07:53.504 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:53.504 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.504 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.504 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:53.504 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.935247+0000", 2024-09-21T22:07:53.504 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4470000000000001e-06 2024-09-21T22:07:53.504 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.504 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.504 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:53.504 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.935295+0000", 2024-09-21T22:07:53.504 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.7963000000000001e-05 2024-09-21T22:07:53.504 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.505 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.505 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:53.505 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.935313+0000", 2024-09-21T22:07:53.505 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8317000000000001e-05 2024-09-21T22:07:53.505 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.505 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.505 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:53.505 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.935352+0000", 2024-09-21T22:07:53.505 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.8552e-05 2024-09-21T22:07:53.505 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.505 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.505 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:53.505 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.935438+0000", 2024-09-21T22:07:53.505 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.6405999999999999e-05 2024-09-21T22:07:53.505 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.506 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.506 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:53.506 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.935471+0000", 2024-09-21T22:07:53.506 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.3142999999999998e-05 2024-09-21T22:07:53.506 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.506 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.506 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:53.506 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.942552+0000", 2024-09-21T22:07:53.506 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0070809510000000003 2024-09-21T22:07:53.506 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.506 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.506 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:53.506 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.942605+0000", 2024-09-21T22:07:53.506 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.2280000000000001e-05 2024-09-21T22:07:53.507 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:53.507 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:53.507 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:53.507 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.507 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.507 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:686 2.3 e63/54 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head v 63'15, mlcod=63'13)", 2024-09-21T22:07:53.507 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.945043+0000", 2024-09-21T22:07:53.507 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 28.501398409, 2024-09-21T22:07:53.507 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.04716302, 2024-09-21T22:07:53.507 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:53.507 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:53.507 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:53.507 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:53.507 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.508 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:53.508 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.945043+0000", 2024-09-21T22:07:53.508 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:53.508 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.508 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.508 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:53.508 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.945043+0000", 2024-09-21T22:07:53.508 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:53.508 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.508 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.508 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:53.508 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.945045+0000", 2024-09-21T22:07:53.508 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8700000000000001e-06 2024-09-21T22:07:53.508 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.508 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.509 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:53.509 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.945899+0000", 2024-09-21T22:07:53.509 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00085401900000000002 2024-09-21T22:07:53.509 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.509 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.509 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:53.509 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.945928+0000", 2024-09-21T22:07:53.509 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9193999999999999e-05 2024-09-21T22:07:53.509 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.509 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.509 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:53.509 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.945948+0000", 2024-09-21T22:07:53.509 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9876000000000001e-05 2024-09-21T22:07:53.509 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.510 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.510 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:53.510 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.946060+0000", 2024-09-21T22:07:53.510 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00011175099999999999 2024-09-21T22:07:53.510 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.510 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.510 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:53.510 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.946103+0000", 2024-09-21T22:07:53.510 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.3090999999999998e-05 2024-09-21T22:07:53.510 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.510 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.510 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:53.510 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.992151+0000", 2024-09-21T22:07:53.510 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.046047948999999998 2024-09-21T22:07:53.511 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.511 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.511 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:53.511 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.992206+0000", 2024-09-21T22:07:53.511 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.5269999999999998e-05 2024-09-21T22:07:53.511 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:53.511 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:53.511 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:53.511 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.511 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.511 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:687 2.3 e63/54 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head v 63'16, mlcod=63'14)", 2024-09-21T22:07:53.511 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.994473+0000", 2024-09-21T22:07:53.511 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 28.451968785999998, 2024-09-21T22:07:53.511 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0054660990000000003, 2024-09-21T22:07:53.512 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:53.512 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:53.512 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:53.512 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:53.512 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.512 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:53.512 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.994473+0000", 2024-09-21T22:07:53.512 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:53.512 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.512 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.512 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:53.512 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.994473+0000", 2024-09-21T22:07:53.512 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:53.512 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.513 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.513 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:53.513 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.994474+0000", 2024-09-21T22:07:53.513 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.463e-06 2024-09-21T22:07:53.513 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.513 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.513 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:53.513 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.994546+0000", 2024-09-21T22:07:53.513 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.2260000000000003e-05 2024-09-21T22:07:53.513 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.513 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.513 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:53.513 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.994564+0000", 2024-09-21T22:07:53.513 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7983999999999999e-05 2024-09-21T22:07:53.514 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.514 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.514 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:53.514 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.994573+0000", 2024-09-21T22:07:53.514 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.4710000000000007e-06 2024-09-21T22:07:53.514 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.514 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.514 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:53.514 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.994634+0000", 2024-09-21T22:07:53.514 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.0872000000000003e-05 2024-09-21T22:07:53.514 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.514 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.514 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:53.514 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.994658+0000", 2024-09-21T22:07:53.515 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4573e-05 2024-09-21T22:07:53.515 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.515 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.515 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:53.515 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.999894+0000", 2024-09-21T22:07:53.515 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0052362390000000002 2024-09-21T22:07:53.515 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.515 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.515 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:53.515 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.999939+0000", 2024-09-21T22:07:53.515 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.4236999999999997e-05 2024-09-21T22:07:53.515 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:53.515 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:53.515 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:53.516 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.516 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.516 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:688 2.3 e63/54 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head v 63'17, mlcod=63'15)", 2024-09-21T22:07:53.516 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.001812+0000", 2024-09-21T22:07:53.516 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 28.444629705000001, 2024-09-21T22:07:53.516 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.028785011999999999, 2024-09-21T22:07:53.516 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:53.516 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:53.516 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:53.516 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:53.516 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.516 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:53.516 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.001812+0000", 2024-09-21T22:07:53.516 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:53.517 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.517 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.517 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:53.517 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.001812+0000", 2024-09-21T22:07:53.517 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:53.517 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.517 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.517 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:53.517 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.001813+0000", 2024-09-21T22:07:53.517 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2699999999999999e-06 2024-09-21T22:07:53.517 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.517 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.517 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:53.517 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.002252+0000", 2024-09-21T22:07:53.517 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00043878599999999998 2024-09-21T22:07:53.518 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.518 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.518 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:53.518 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.002271+0000", 2024-09-21T22:07:53.518 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9346000000000001e-05 2024-09-21T22:07:53.518 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.518 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.518 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:53.518 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.002282+0000", 2024-09-21T22:07:53.518 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0951000000000001e-05 2024-09-21T22:07:53.518 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.518 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.518 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:53.518 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.002398+0000", 2024-09-21T22:07:53.519 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000116412 2024-09-21T22:07:53.519 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.519 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.519 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:53.519 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.002425+0000", 2024-09-21T22:07:53.519 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.6291999999999998e-05 2024-09-21T22:07:53.519 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.519 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.519 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:53.519 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.030528+0000", 2024-09-21T22:07:53.519 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.028103390999999998 2024-09-21T22:07:53.519 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.519 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.519 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:53.519 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.030597+0000", 2024-09-21T22:07:53.520 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.8564000000000003e-05 2024-09-21T22:07:53.520 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:53.520 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:53.520 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:53.520 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.520 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.520 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:689 2.3 e63/54 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head v 63'18, mlcod=63'16)", 2024-09-21T22:07:53.520 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.032891+0000", 2024-09-21T22:07:53.520 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 28.413550123, 2024-09-21T22:07:53.520 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.005061032, 2024-09-21T22:07:53.520 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:53.520 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:53.520 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:53.521 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:53.521 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.521 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:53.521 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.032891+0000", 2024-09-21T22:07:53.521 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:53.521 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.521 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.521 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:53.521 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.032891+0000", 2024-09-21T22:07:53.521 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:53.521 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.521 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.521 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:53.521 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.032893+0000", 2024-09-21T22:07:53.522 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3030000000000001e-06 2024-09-21T22:07:53.522 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.522 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.522 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:53.522 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.032952+0000", 2024-09-21T22:07:53.522 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.9070000000000002e-05 2024-09-21T22:07:53.522 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.522 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.522 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:53.522 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.032969+0000", 2024-09-21T22:07:53.522 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7839e-05 2024-09-21T22:07:53.522 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.522 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.523 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:53.523 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.032979+0000", 2024-09-21T22:07:53.523 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.5370000000000003e-06 2024-09-21T22:07:53.523 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.523 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.523 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:53.523 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.033041+0000", 2024-09-21T22:07:53.523 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.1562000000000005e-05 2024-09-21T22:07:53.523 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.523 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.523 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:53.523 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.033068+0000", 2024-09-21T22:07:53.523 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.7226999999999999e-05 2024-09-21T22:07:53.523 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.523 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.524 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:53.524 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.037908+0000", 2024-09-21T22:07:53.524 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0048401770000000002 2024-09-21T22:07:53.524 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.524 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.524 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:53.524 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.037952+0000", 2024-09-21T22:07:53.524 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.4317000000000003e-05 2024-09-21T22:07:53.524 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:53.524 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:53.524 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:53.524 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.524 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.524 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:690 2.3 e63/54 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head v 63'19, mlcod=63'17)", 2024-09-21T22:07:53.525 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.039813+0000", 2024-09-21T22:07:53.525 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 28.406627906000001, 2024-09-21T22:07:53.525 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.035351891000000003, 2024-09-21T22:07:53.525 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:53.525 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:53.525 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:53.525 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:53.525 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.525 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:53.525 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.039813+0000", 2024-09-21T22:07:53.525 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:53.525 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.525 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.525 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:53.526 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.039813+0000", 2024-09-21T22:07:53.526 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:53.526 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.526 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.526 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:53.526 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.039814+0000", 2024-09-21T22:07:53.526 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.5300000000000002e-07 2024-09-21T22:07:53.526 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.526 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.526 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:53.526 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.040166+0000", 2024-09-21T22:07:53.526 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00035177 2024-09-21T22:07:53.526 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.526 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.527 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:53.527 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.040185+0000", 2024-09-21T22:07:53.527 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9000000000000001e-05 2024-09-21T22:07:53.527 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.527 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.527 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:53.527 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.040195+0000", 2024-09-21T22:07:53.527 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0254999999999999e-05 2024-09-21T22:07:53.527 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.527 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.527 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:53.527 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.040262+0000", 2024-09-21T22:07:53.527 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.6198000000000006e-05 2024-09-21T22:07:53.527 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.527 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.528 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:53.528 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.040288+0000", 2024-09-21T22:07:53.528 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5931999999999998e-05 2024-09-21T22:07:53.528 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.528 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.528 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:53.528 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.075106+0000", 2024-09-21T22:07:53.528 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.034818546999999998 2024-09-21T22:07:53.528 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.528 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.528 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:53.528 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.075165+0000", 2024-09-21T22:07:53.528 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.9236e-05 2024-09-21T22:07:53.528 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:53.529 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:53.529 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:53.529 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.529 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.529 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPull(2.3 e67/66 cost 1048576)", 2024-09-21T22:07:53.529 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:29.414957+0000", 2024-09-21T22:07:53.529 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 24.031484527, 2024-09-21T22:07:53.529 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0032647269999999998, 2024-09-21T22:07:53.529 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:53.529 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:53.529 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-21T22:07:53.529 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:53.529 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.529 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:53.529 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:29.414957+0000", 2024-09-21T22:07:53.530 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:53.530 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.530 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.530 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:53.530 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:29.414957+0000", 2024-09-21T22:07:53.530 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:53.530 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.530 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.530 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:53.530 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:29.414959+0000", 2024-09-21T22:07:53.530 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.396e-06 2024-09-21T22:07:53.530 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.530 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.530 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:53.531 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:29.414986+0000", 2024-09-21T22:07:53.531 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.72e-05 2024-09-21T22:07:53.531 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.531 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.531 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:53.531 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:29.415018+0000", 2024-09-21T22:07:53.531 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.2079000000000002e-05 2024-09-21T22:07:53.531 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.531 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.531 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:53.531 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:29.415035+0000", 2024-09-21T22:07:53.531 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6316000000000001e-05 2024-09-21T22:07:53.531 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.531 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.532 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:53.532 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:29.415201+0000", 2024-09-21T22:07:53.532 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00016583900000000001 2024-09-21T22:07:53.532 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.532 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.532 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:53.532 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:29.418222+0000", 2024-09-21T22:07:53.532 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0030208969999999998 2024-09-21T22:07:53.532 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:53.532 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:53.532 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:53.532 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.532 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.532 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPull(2.3 e67/66 cost 1048576)", 2024-09-21T22:07:53.532 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:29.486922+0000", 2024-09-21T22:07:53.533 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 23.959519589999999, 2024-09-21T22:07:53.533 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0029880940000000002, 2024-09-21T22:07:53.533 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:53.533 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:53.533 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-21T22:07:53.533 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:53.533 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.533 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:53.533 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:29.486922+0000", 2024-09-21T22:07:53.533 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:53.533 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.533 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.533 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:53.533 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:29.486922+0000", 2024-09-21T22:07:53.534 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:53.534 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.534 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.534 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:53.534 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:29.486924+0000", 2024-09-21T22:07:53.534 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.897e-06 2024-09-21T22:07:53.534 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.534 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.534 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:53.534 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:29.486975+0000", 2024-09-21T22:07:53.534 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.1275e-05 2024-09-21T22:07:53.534 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.534 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.534 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:53.535 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:29.487002+0000", 2024-09-21T22:07:53.535 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.7469000000000001e-05 2024-09-21T22:07:53.535 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.535 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.535 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:53.535 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:29.487016+0000", 2024-09-21T22:07:53.535 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3587000000000001e-05 2024-09-21T22:07:53.535 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.535 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.535 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:53.535 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:29.487158+0000", 2024-09-21T22:07:53.535 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00014215899999999999 2024-09-21T22:07:53.535 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:53.535 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:53.535 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:53.536 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:29.489910+0000", 2024-09-21T22:07:53.536 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0027517069999999999 2024-09-21T22:07:53.536 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:53.536 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:53.536 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:53.536 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:53.536 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:53.536 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:07:53.537 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T22:07:53.538 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:07:53.536+0000 7f2075a06640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31720) UID: 0 2024-09-21T22:07:53.581 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T22:07:53.638 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:53.634+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:53.671 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:07:53.672 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:07:53.672 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-09-21T22:07:53.672 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:07:53.686 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T22:07:53.735 INFO:teuthology.orchestra.run.smithi039.stdout:osd.0: {} 2024-09-21T22:07:53.735 INFO:teuthology.orchestra.run.smithi039.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T22:07:53.738 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:53.734+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:53.752 INFO:teuthology.orchestra.run.smithi039.stdout:osd.1: {} 2024-09-21T22:07:53.752 INFO:teuthology.orchestra.run.smithi039.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T22:07:53.768 INFO:teuthology.orchestra.run.smithi039.stdout:osd.2: {} 2024-09-21T22:07:53.769 INFO:teuthology.orchestra.run.smithi039.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T22:07:53.787 INFO:teuthology.orchestra.run.smithi039.stdout:osd.3: {} 2024-09-21T22:07:53.787 INFO:teuthology.orchestra.run.smithi039.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T22:07:53.804 INFO:teuthology.orchestra.run.smithi039.stdout:osd.4: {} 2024-09-21T22:07:53.805 INFO:teuthology.orchestra.run.smithi039.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T22:07:53.814 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:53.814 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:53.824 INFO:teuthology.orchestra.run.smithi039.stdout:osd.5: {} 2024-09-21T22:07:53.825 INFO:teuthology.orchestra.run.smithi039.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T22:07:53.829 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T22:07:53.874 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:07:53.872+0000 7f2075a06640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31720) UID: 0 2024-09-21T22:07:53.941 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:53.938+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:53.957 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:53.957 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:53.972 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T22:07:54.041 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:54.038+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:54.139 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:54.139 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:54.142 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:54.138+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:54.153 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T22:07:54.243 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:54.240+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:54.273 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:54.273 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:54.287 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T22:07:54.374 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:54.372+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:54.445 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:54.442+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:54.459 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:54.459 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:54.475 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T22:07:54.562 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:54.560+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:54.647 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:07:54.644+0000 7f2075a06640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31720) UID: 0 2024-09-21T22:07:54.647 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:54.647 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:54.662 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T22:07:54.784 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:54.784+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:54.884 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:54.882+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:54.914 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:54.914 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:54.929 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T22:07:54.957 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-21T22:07:54.985 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:07:54.980+0000 7f2075a06640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31720) UID: 0 2024-09-21T22:07:55.056 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:55.056 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:55.071 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T22:07:55.114 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:55.112+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:55.187 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:55.184+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:55.193 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:55.193 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:55.207 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T22:07:55.287 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:55.282+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:55.324 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:55.324 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:55.337 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T22:07:55.428 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:55.426+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:55.489 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:55.486+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:55.510 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:55.510 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:55.523 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T22:07:55.590 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:55.588+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:55.687 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:55.687 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:55.690 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:55.686+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:55.701 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T22:07:55.791 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:55.788+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:55.842 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T22:07:55.863 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:55.863 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:55.878 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T22:07:55.879 INFO:teuthology.orchestra.run.smithi039.stderr:noscrub is unset 2024-09-21T22:07:55.892 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:07:55.888+0000 7f2075a06640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31720) UID: 0 2024-09-21T22:07:55.988 INFO:teuthology.orchestra.run.smithi039.stdout:osd.0: {} 2024-09-21T22:07:55.988 INFO:teuthology.orchestra.run.smithi039.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T22:07:55.994 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:55.988+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:56.005 INFO:teuthology.orchestra.run.smithi039.stdout:osd.1: {} 2024-09-21T22:07:56.005 INFO:teuthology.orchestra.run.smithi039.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T22:07:56.021 INFO:teuthology.orchestra.run.smithi039.stdout:osd.2: {} 2024-09-21T22:07:56.021 INFO:teuthology.orchestra.run.smithi039.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T22:07:56.042 INFO:teuthology.orchestra.run.smithi039.stdout:osd.3: {} 2024-09-21T22:07:56.042 INFO:teuthology.orchestra.run.smithi039.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T22:07:56.063 INFO:teuthology.orchestra.run.smithi039.stdout:osd.4: {} 2024-09-21T22:07:56.063 INFO:teuthology.orchestra.run.smithi039.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T22:07:56.064 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:07:56.064 INFO:teuthology.orchestra.run.smithi039.stdout: "size": 20, 2024-09-21T22:07:56.064 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 600, 2024-09-21T22:07:56.064 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [ 2024-09-21T22:07:56.065 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.065 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.3 55/54 [PushOp(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head, version: 52'2, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head@52'2, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T22:07:56.065 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.084379+0000", 2024-09-21T22:07:56.065 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 43.975200872999999, 2024-09-21T22:07:56.065 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.051020674000000002, 2024-09-21T22:07:56.065 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:56.066 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:56.066 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:56.066 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:56.066 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.066 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:56.066 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.084379+0000", 2024-09-21T22:07:56.066 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:56.066 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.067 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.067 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:56.067 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.084379+0000", 2024-09-21T22:07:56.067 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:56.067 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.067 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.067 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:56.067 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.084381+0000", 2024-09-21T22:07:56.067 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.73e-06 2024-09-21T22:07:56.068 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.068 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.068 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:56.068 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.085494+0000", 2024-09-21T22:07:56.068 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011131559999999999 2024-09-21T22:07:56.068 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.068 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.068 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:56.068 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.085542+0000", 2024-09-21T22:07:56.069 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8334999999999997e-05 2024-09-21T22:07:56.069 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.069 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.069 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:56.069 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.085591+0000", 2024-09-21T22:07:56.069 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.9165000000000001e-05 2024-09-21T22:07:56.069 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.069 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.069 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:56.070 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.085777+0000", 2024-09-21T22:07:56.070 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000186152 2024-09-21T22:07:56.070 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.070 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.070 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:56.070 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.085796+0000", 2024-09-21T22:07:56.070 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8612999999999999e-05 2024-09-21T22:07:56.070 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.070 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.070 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:56.070 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.135399+0000", 2024-09-21T22:07:56.071 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.049603522999999997 2024-09-21T22:07:56.071 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:56.071 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:56.071 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:56.071 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.071 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.071 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.3 55/54 [PushOp(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head, version: 52'2, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head@52'2, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:07:56.071 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.139554+0000", 2024-09-21T22:07:56.071 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 43.920025770000002, 2024-09-21T22:07:56.071 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.056215389999999997, 2024-09-21T22:07:56.071 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:56.071 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:56.072 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:56.072 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:56.072 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.072 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:56.072 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.139554+0000", 2024-09-21T22:07:56.072 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:56.072 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.072 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.072 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:56.072 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.139554+0000", 2024-09-21T22:07:56.072 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:56.072 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.073 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.073 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:56.073 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.139556+0000", 2024-09-21T22:07:56.073 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7999999999999999e-06 2024-09-21T22:07:56.073 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.073 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.073 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:56.073 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.140671+0000", 2024-09-21T22:07:56.073 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001115612 2024-09-21T22:07:56.073 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.073 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.073 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:56.073 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.140720+0000", 2024-09-21T22:07:56.074 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8631999999999998e-05 2024-09-21T22:07:56.074 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.074 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.074 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:56.074 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.140752+0000", 2024-09-21T22:07:56.074 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.2199000000000001e-05 2024-09-21T22:07:56.074 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.074 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.074 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:56.074 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.140900+0000", 2024-09-21T22:07:56.074 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00014786000000000001 2024-09-21T22:07:56.074 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.075 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.075 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:56.075 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.140917+0000", 2024-09-21T22:07:56.075 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6753999999999999e-05 2024-09-21T22:07:56.075 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.075 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.075 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:56.075 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.195769+0000", 2024-09-21T22:07:56.075 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.054852533000000002 2024-09-21T22:07:56.075 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:56.075 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:56.075 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:56.075 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.075 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.076 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.3 55/54 [PushOp(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head, version: 52'2, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head@52'2, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:07:56.076 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.199187+0000", 2024-09-21T22:07:56.076 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 43.860392226000002, 2024-09-21T22:07:56.076 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.051543975999999998, 2024-09-21T22:07:56.076 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:56.076 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:56.076 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:56.076 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:56.076 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.077 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:56.077 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.199187+0000", 2024-09-21T22:07:56.077 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:56.077 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.077 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.077 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:56.077 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.199187+0000", 2024-09-21T22:07:56.077 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:56.077 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.077 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.077 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:56.077 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.199189+0000", 2024-09-21T22:07:56.077 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7969999999999999e-06 2024-09-21T22:07:56.077 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.077 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.078 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:56.078 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.200355+0000", 2024-09-21T22:07:56.078 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011660139999999999 2024-09-21T22:07:56.078 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.078 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.078 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:56.078 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.200405+0000", 2024-09-21T22:07:56.078 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.0124000000000003e-05 2024-09-21T22:07:56.078 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.078 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.078 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:56.078 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.200457+0000", 2024-09-21T22:07:56.078 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.1177999999999997e-05 2024-09-21T22:07:56.078 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.079 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.079 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:56.079 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.200635+0000", 2024-09-21T22:07:56.079 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00017841299999999999 2024-09-21T22:07:56.079 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.079 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.079 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:56.079 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.200653+0000", 2024-09-21T22:07:56.079 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7944999999999998e-05 2024-09-21T22:07:56.079 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.079 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.079 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:56.079 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.250731+0000", 2024-09-21T22:07:56.079 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.050078505000000002 2024-09-21T22:07:56.079 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:56.080 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:56.080 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:56.080 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.080 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.080 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.3 55/54 [PushOp(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head, version: 52'3, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head@52'3, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:07:56.080 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.343754+0000", 2024-09-21T22:07:56.080 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 43.715825514999999, 2024-09-21T22:07:56.080 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.063560607000000005, 2024-09-21T22:07:56.080 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:56.080 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:56.080 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:56.080 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:56.080 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.080 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:56.081 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.343754+0000", 2024-09-21T22:07:56.081 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:56.081 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.081 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.081 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:56.081 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.343754+0000", 2024-09-21T22:07:56.081 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:56.081 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.081 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.081 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:56.081 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.343756+0000", 2024-09-21T22:07:56.081 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7069999999999999e-06 2024-09-21T22:07:56.081 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.081 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.081 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:56.082 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.344864+0000", 2024-09-21T22:07:56.082 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011085159999999999 2024-09-21T22:07:56.082 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.082 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.082 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:56.082 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.344914+0000", 2024-09-21T22:07:56.082 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.9462000000000002e-05 2024-09-21T22:07:56.082 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.082 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.082 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:56.082 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.344961+0000", 2024-09-21T22:07:56.082 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.7633999999999998e-05 2024-09-21T22:07:56.082 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.082 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.083 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:56.083 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.345171+0000", 2024-09-21T22:07:56.083 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000209652 2024-09-21T22:07:56.083 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.083 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.083 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:56.083 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.345193+0000", 2024-09-21T22:07:56.083 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1985999999999999e-05 2024-09-21T22:07:56.083 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.083 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.083 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:56.083 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.407315+0000", 2024-09-21T22:07:56.083 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.06212165 2024-09-21T22:07:56.083 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:56.084 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:56.084 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:56.084 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.084 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.084 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.3 55/54 [PushOp(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head, version: 52'3, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head@52'3, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:07:56.084 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.410551+0000", 2024-09-21T22:07:56.084 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 43.649028383999998, 2024-09-21T22:07:56.084 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.054517522999999998, 2024-09-21T22:07:56.084 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:56.084 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:56.084 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:56.084 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:56.084 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.084 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:56.085 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.410551+0000", 2024-09-21T22:07:56.085 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:56.085 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.085 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.085 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:56.085 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.410551+0000", 2024-09-21T22:07:56.085 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:56.085 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.085 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.085 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:56.085 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.410553+0000", 2024-09-21T22:07:56.085 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5200000000000001e-06 2024-09-21T22:07:56.085 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.086 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.086 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:56.086 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.411747+0000", 2024-09-21T22:07:56.086 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011946089999999999 2024-09-21T22:07:56.086 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.086 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.086 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:56.086 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.411796+0000", 2024-09-21T22:07:56.086 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8717999999999997e-05 2024-09-21T22:07:56.086 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.086 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.086 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:56.086 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.411844+0000", 2024-09-21T22:07:56.087 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8291999999999998e-05 2024-09-21T22:07:56.087 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.087 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.087 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:56.087 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.412066+0000", 2024-09-21T22:07:56.087 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00022115899999999999 2024-09-21T22:07:56.087 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.087 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.087 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:56.087 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.412081+0000", 2024-09-21T22:07:56.087 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5654e-05 2024-09-21T22:07:56.088 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.088 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.088 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:56.088 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.465069+0000", 2024-09-21T22:07:56.088 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.052987570999999997 2024-09-21T22:07:56.088 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:56.088 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:56.088 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:56.088 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.088 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.088 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.3 55/54 [PushOp(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head, version: 52'3, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head@52'3, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 4194304, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:07:56.088 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.468339+0000", 2024-09-21T22:07:56.088 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 43.591240720000002, 2024-09-21T22:07:56.089 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.064010024999999998, 2024-09-21T22:07:56.089 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:56.089 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:56.089 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:07:56.089 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:56.089 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.089 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:56.089 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.468339+0000", 2024-09-21T22:07:56.089 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:56.089 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.089 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.089 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:56.089 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.468339+0000", 2024-09-21T22:07:56.090 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:56.090 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.090 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.090 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:56.090 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.468340+0000", 2024-09-21T22:07:56.090 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4330000000000001e-06 2024-09-21T22:07:56.090 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.090 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.090 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:56.090 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.469484+0000", 2024-09-21T22:07:56.090 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001143535 2024-09-21T22:07:56.090 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.091 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.091 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:56.091 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.469533+0000", 2024-09-21T22:07:56.091 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8980999999999997e-05 2024-09-21T22:07:56.091 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.091 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.091 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:56.091 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.469581+0000", 2024-09-21T22:07:56.091 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8551999999999999e-05 2024-09-21T22:07:56.091 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.091 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.091 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:56.091 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.469804+0000", 2024-09-21T22:07:56.092 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00022287699999999999 2024-09-21T22:07:56.092 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.092 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.092 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:56.092 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.469829+0000", 2024-09-21T22:07:56.092 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4766e-05 2024-09-21T22:07:56.092 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.092 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.092 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:56.092 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.532349+0000", 2024-09-21T22:07:56.092 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.062519880999999999 2024-09-21T22:07:56.092 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:56.092 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:56.092 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:56.092 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.093 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.093 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4840.0:668 2.3 2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:07:56.093 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.409114+0000", 2024-09-21T22:07:56.093 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 31.650465514, 2024-09-21T22:07:56.093 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.048703758999999999, 2024-09-21T22:07:56.093 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:56.093 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:56.093 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:56.093 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-21T22:07:56.093 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4840", 2024-09-21T22:07:56.093 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:56.093 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 668 2024-09-21T22:07:56.093 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.093 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:56.094 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.094 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:56.094 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.409114+0000", 2024-09-21T22:07:56.094 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:56.094 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.094 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.094 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:56.094 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.409114+0000", 2024-09-21T22:07:56.094 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:56.094 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.094 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.094 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:56.094 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.409115+0000", 2024-09-21T22:07:56.095 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.114e-06 2024-09-21T22:07:56.095 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.095 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.095 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:56.095 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.410308+0000", 2024-09-21T22:07:56.095 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011930949999999999 2024-09-21T22:07:56.095 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.095 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.095 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:56.095 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.410325+0000", 2024-09-21T22:07:56.095 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7088e-05 2024-09-21T22:07:56.095 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.095 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.096 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:56.096 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.410334+0000", 2024-09-21T22:07:56.096 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.2549999999999997e-06 2024-09-21T22:07:56.096 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.096 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.096 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:56.096 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.410393+0000", 2024-09-21T22:07:56.096 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.9327999999999998e-05 2024-09-21T22:07:56.096 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.096 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.096 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:56.096 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.410482+0000", 2024-09-21T22:07:56.096 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.9019999999999998e-05 2024-09-21T22:07:56.097 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.097 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.097 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-21T22:07:56.097 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.410617+0000", 2024-09-21T22:07:56.097 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000135232 2024-09-21T22:07:56.097 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.097 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.097 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:56.097 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.457051+0000", 2024-09-21T22:07:56.097 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.04643415 2024-09-21T22:07:56.097 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.097 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.097 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-21T22:07:56.098 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.457675+0000", 2024-09-21T22:07:56.098 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00062332700000000002 2024-09-21T22:07:56.098 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.098 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.098 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:56.098 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.457735+0000", 2024-09-21T22:07:56.098 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.029e-05 2024-09-21T22:07:56.098 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.098 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.098 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:56.098 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.457818+0000", 2024-09-21T22:07:56.098 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.2860000000000003e-05 2024-09-21T22:07:56.099 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:56.099 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:56.099 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:56.099 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.099 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.099 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4840.0:670 2.3 2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:07:56.099 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.466980+0000", 2024-09-21T22:07:56.099 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 31.592599446000001, 2024-09-21T22:07:56.099 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.050913930000000003, 2024-09-21T22:07:56.099 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:56.099 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:56.099 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:56.099 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-21T22:07:56.100 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4840", 2024-09-21T22:07:56.100 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:56.100 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 670 2024-09-21T22:07:56.100 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.100 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:56.100 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.100 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:56.100 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.466980+0000", 2024-09-21T22:07:56.100 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:56.100 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.100 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.100 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:56.101 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.466980+0000", 2024-09-21T22:07:56.101 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:56.101 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.101 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.101 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:56.101 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.466981+0000", 2024-09-21T22:07:56.101 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0839999999999999e-06 2024-09-21T22:07:56.101 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.101 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.101 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:56.101 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.467815+0000", 2024-09-21T22:07:56.102 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00083330399999999999 2024-09-21T22:07:56.102 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.102 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.102 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:56.102 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.467832+0000", 2024-09-21T22:07:56.102 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7601e-05 2024-09-21T22:07:56.102 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.102 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.102 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:56.102 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.467840+0000", 2024-09-21T22:07:56.103 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.2220000000000002e-06 2024-09-21T22:07:56.103 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.103 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.103 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:56.103 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.467974+0000", 2024-09-21T22:07:56.103 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000133411 2024-09-21T22:07:56.103 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.103 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.103 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:56.103 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.468223+0000", 2024-09-21T22:07:56.104 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00024906400000000001 2024-09-21T22:07:56.104 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.104 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.104 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-21T22:07:56.104 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.468616+0000", 2024-09-21T22:07:56.104 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000393433 2024-09-21T22:07:56.104 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.104 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.104 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:56.104 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.517016+0000", 2024-09-21T22:07:56.105 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.048399708 2024-09-21T22:07:56.105 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.105 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.105 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-21T22:07:56.105 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.517693+0000", 2024-09-21T22:07:56.105 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00067681000000000002 2024-09-21T22:07:56.105 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.105 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.105 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:56.105 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.517789+0000", 2024-09-21T22:07:56.106 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.5778000000000004e-05 2024-09-21T22:07:56.106 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.106 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.106 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:56.106 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.517894+0000", 2024-09-21T22:07:56.106 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000105515 2024-09-21T22:07:56.106 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:56.106 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:56.106 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:56.106 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.106 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.106 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4840.0:672 2.3 2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:07:56.106 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.527127+0000", 2024-09-21T22:07:56.107 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 31.532452983999999, 2024-09-21T22:07:56.107 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.047345602000000001, 2024-09-21T22:07:56.107 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:56.107 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:56.107 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:56.107 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-21T22:07:56.107 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4840", 2024-09-21T22:07:56.107 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:56.107 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 672 2024-09-21T22:07:56.107 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.107 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:56.107 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.107 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:56.107 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.527127+0000", 2024-09-21T22:07:56.108 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:56.108 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.108 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.108 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:56.108 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.527127+0000", 2024-09-21T22:07:56.108 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:56.108 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.108 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.108 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:56.108 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.527128+0000", 2024-09-21T22:07:56.108 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2810000000000001e-06 2024-09-21T22:07:56.108 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.108 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.108 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:56.108 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.527962+0000", 2024-09-21T22:07:56.109 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00083445199999999996 2024-09-21T22:07:56.109 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.109 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.109 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:56.109 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.527981+0000", 2024-09-21T22:07:56.109 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9009e-05 2024-09-21T22:07:56.109 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.109 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.109 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:56.109 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.527990+0000", 2024-09-21T22:07:56.109 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.8619999999999992e-06 2024-09-21T22:07:56.109 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.109 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.109 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:56.110 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.528061+0000", 2024-09-21T22:07:56.110 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.0226000000000002e-05 2024-09-21T22:07:56.110 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.110 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.110 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:56.110 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.528156+0000", 2024-09-21T22:07:56.110 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.5724999999999996e-05 2024-09-21T22:07:56.110 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.110 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.110 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-21T22:07:56.110 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.528303+0000", 2024-09-21T22:07:56.110 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000146374 2024-09-21T22:07:56.110 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.110 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.111 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:56.111 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.573772+0000", 2024-09-21T22:07:56.111 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.045469745999999998 2024-09-21T22:07:56.111 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.111 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.111 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-21T22:07:56.111 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.574299+0000", 2024-09-21T22:07:56.111 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00052649200000000004 2024-09-21T22:07:56.111 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.111 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.111 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:56.111 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.574366+0000", 2024-09-21T22:07:56.111 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.6872000000000006e-05 2024-09-21T22:07:56.112 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.112 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.112 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:56.112 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.574472+0000", 2024-09-21T22:07:56.112 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000106563 2024-09-21T22:07:56.112 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:56.112 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:56.112 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:56.112 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.112 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.112 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4840.0:686 2.3 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:07:56.112 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.943561+0000", 2024-09-21T22:07:56.112 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 31.116018707999999, 2024-09-21T22:07:56.112 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.049021493999999999, 2024-09-21T22:07:56.112 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:56.113 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:56.113 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:56.113 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-21T22:07:56.113 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4840", 2024-09-21T22:07:56.113 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:56.113 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 686 2024-09-21T22:07:56.113 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.113 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:56.113 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.113 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:56.113 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.943561+0000", 2024-09-21T22:07:56.113 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:56.113 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.113 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.113 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:56.114 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.943561+0000", 2024-09-21T22:07:56.114 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:56.114 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.114 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.114 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:56.114 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.943563+0000", 2024-09-21T22:07:56.114 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9420000000000002e-06 2024-09-21T22:07:56.114 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.114 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.114 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:56.114 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.944546+0000", 2024-09-21T22:07:56.114 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00098357099999999992 2024-09-21T22:07:56.114 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.114 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.115 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:56.115 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.944565+0000", 2024-09-21T22:07:56.115 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8621e-05 2024-09-21T22:07:56.115 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.115 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.115 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:56.115 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.944575+0000", 2024-09-21T22:07:56.115 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.4639999999999995e-06 2024-09-21T22:07:56.115 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.115 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.115 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:56.115 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.944647+0000", 2024-09-21T22:07:56.115 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.2786000000000004e-05 2024-09-21T22:07:56.115 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.115 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.116 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:56.116 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.944746+0000", 2024-09-21T22:07:56.116 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.8196999999999995e-05 2024-09-21T22:07:56.116 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.116 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.116 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-21T22:07:56.116 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.944890+0000", 2024-09-21T22:07:56.116 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000144804 2024-09-21T22:07:56.116 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.116 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.116 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:56.116 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.992404+0000", 2024-09-21T22:07:56.116 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.047513211999999999 2024-09-21T22:07:56.117 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.117 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.117 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-21T22:07:56.117 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.992480+0000", 2024-09-21T22:07:56.117 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.6491999999999999e-05 2024-09-21T22:07:56.117 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.117 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.117 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:56.117 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.992521+0000", 2024-09-21T22:07:56.117 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.0608000000000002e-05 2024-09-21T22:07:56.117 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.118 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.118 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:56.118 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.992582+0000", 2024-09-21T22:07:56.118 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.1797000000000005e-05 2024-09-21T22:07:56.118 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:56.118 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:56.118 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:56.118 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.118 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.118 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4840.0:720 2.1 2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:07:56.118 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.803827+0000", 2024-09-21T22:07:56.118 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 30.255752432000001, 2024-09-21T22:07:56.119 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.049890510999999998, 2024-09-21T22:07:56.119 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:56.119 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:56.119 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:56.119 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-21T22:07:56.119 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4840", 2024-09-21T22:07:56.119 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:56.119 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 720 2024-09-21T22:07:56.119 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.119 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:56.119 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.119 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:56.119 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.803827+0000", 2024-09-21T22:07:56.120 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:56.120 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.120 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.120 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:56.120 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.803827+0000", 2024-09-21T22:07:56.120 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:56.120 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.120 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.120 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:56.120 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.803828+0000", 2024-09-21T22:07:56.120 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.569999999999999e-07 2024-09-21T22:07:56.120 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.120 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.120 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:56.121 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.804689+0000", 2024-09-21T22:07:56.121 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00086121799999999997 2024-09-21T22:07:56.121 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.121 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.121 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:56.121 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.804706+0000", 2024-09-21T22:07:56.121 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6266999999999999e-05 2024-09-21T22:07:56.121 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.121 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.121 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:56.121 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.804713+0000", 2024-09-21T22:07:56.121 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.1010000000000003e-06 2024-09-21T22:07:56.122 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.122 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.122 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:56.122 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.804781+0000", 2024-09-21T22:07:56.122 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.7871000000000001e-05 2024-09-21T22:07:56.122 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.122 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.122 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:56.122 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.804859+0000", 2024-09-21T22:07:56.122 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.7891000000000004e-05 2024-09-21T22:07:56.122 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.122 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.123 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:56.123 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.804971+0000", 2024-09-21T22:07:56.123 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00011275100000000001 2024-09-21T22:07:56.123 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.123 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.123 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:56.123 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.852704+0000", 2024-09-21T22:07:56.123 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.047732361000000001 2024-09-21T22:07:56.123 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.123 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.123 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-21T22:07:56.123 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.853446+0000", 2024-09-21T22:07:56.124 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00074258799999999997 2024-09-21T22:07:56.124 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.124 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.124 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:56.124 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.853608+0000", 2024-09-21T22:07:56.124 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00016190900000000001 2024-09-21T22:07:56.124 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.124 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.124 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:56.124 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.853718+0000", 2024-09-21T22:07:56.124 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000109597 2024-09-21T22:07:56.124 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:56.124 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:56.125 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:56.125 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.125 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.125 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4840.0:723 2.1 2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:07:56.125 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.860543+0000", 2024-09-21T22:07:56.125 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 30.199036998, 2024-09-21T22:07:56.125 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.062354957000000003, 2024-09-21T22:07:56.125 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:56.125 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:56.125 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:56.125 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-21T22:07:56.125 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4840", 2024-09-21T22:07:56.126 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:56.126 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 723 2024-09-21T22:07:56.126 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.126 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:56.126 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.126 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:56.126 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.860543+0000", 2024-09-21T22:07:56.126 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:56.126 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.126 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.126 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:56.126 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.860543+0000", 2024-09-21T22:07:56.127 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:56.127 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.127 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.127 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:56.127 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.860544+0000", 2024-09-21T22:07:56.127 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.1119999999999999e-06 2024-09-21T22:07:56.127 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.127 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.127 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:56.127 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.861400+0000", 2024-09-21T22:07:56.127 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000856393 2024-09-21T22:07:56.127 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.128 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.128 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:56.128 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.861419+0000", 2024-09-21T22:07:56.128 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8992e-05 2024-09-21T22:07:56.128 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.128 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.128 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:56.128 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.861428+0000", 2024-09-21T22:07:56.128 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.551e-06 2024-09-21T22:07:56.128 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.128 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.128 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:56.129 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.861555+0000", 2024-09-21T22:07:56.129 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000127286 2024-09-21T22:07:56.129 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.129 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.129 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:56.129 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.861816+0000", 2024-09-21T22:07:56.129 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00026100899999999998 2024-09-21T22:07:56.129 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.129 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.129 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:56.129 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.862189+0000", 2024-09-21T22:07:56.129 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00037261899999999998 2024-09-21T22:07:56.129 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.130 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.130 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:56.130 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.921514+0000", 2024-09-21T22:07:56.130 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.059325210000000003 2024-09-21T22:07:56.130 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.130 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.130 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-21T22:07:56.130 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.922721+0000", 2024-09-21T22:07:56.130 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0012066819999999999 2024-09-21T22:07:56.130 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.130 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.130 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:56.131 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.922797+0000", 2024-09-21T22:07:56.131 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.6162999999999996e-05 2024-09-21T22:07:56.131 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.131 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.131 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:56.131 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.922898+0000", 2024-09-21T22:07:56.131 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00010094000000000001 2024-09-21T22:07:56.131 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:56.131 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:56.131 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:56.131 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.131 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.132 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:725 2.0 e63/16 2:08b4f96a:::rbd_data.126483d75f2d.000000000000000f:head v 63'33, mlcod=63'31)", 2024-09-21T22:07:56.132 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.936778+0000", 2024-09-21T22:07:56.132 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 30.122801255999999, 2024-09-21T22:07:56.132 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.047671937999999997, 2024-09-21T22:07:56.132 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:56.132 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:56.132 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:56.132 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:56.132 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.132 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:56.132 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.936778+0000", 2024-09-21T22:07:56.132 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:56.133 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.133 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.133 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:56.133 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.936778+0000", 2024-09-21T22:07:56.133 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:56.133 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.133 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.133 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:56.133 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.936780+0000", 2024-09-21T22:07:56.133 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7799999999999999e-06 2024-09-21T22:07:56.133 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.133 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.134 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:56.134 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.938475+0000", 2024-09-21T22:07:56.134 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0016951749999999999 2024-09-21T22:07:56.134 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.134 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.134 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:56.134 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.938506+0000", 2024-09-21T22:07:56.134 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.0865999999999999e-05 2024-09-21T22:07:56.134 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.134 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.134 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:56.134 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.938520+0000", 2024-09-21T22:07:56.134 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3825999999999999e-05 2024-09-21T22:07:56.134 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.134 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.135 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:56.135 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.938624+0000", 2024-09-21T22:07:56.135 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00010391900000000001 2024-09-21T22:07:56.135 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.135 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.135 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:56.135 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.938651+0000", 2024-09-21T22:07:56.135 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.7481999999999999e-05 2024-09-21T22:07:56.135 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.135 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.135 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:56.135 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.984369+0000", 2024-09-21T22:07:56.135 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.045717595 2024-09-21T22:07:56.135 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.136 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.136 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:56.136 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.984450+0000", 2024-09-21T22:07:56.136 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.1295e-05 2024-09-21T22:07:56.136 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:56.136 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:56.136 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:56.136 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.136 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.136 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:727 2.0 e63/16 2:08b4f96a:::rbd_data.126483d75f2d.000000000000000f:head v 63'35, mlcod=63'33)", 2024-09-21T22:07:56.136 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.998090+0000", 2024-09-21T22:07:56.136 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 30.061490079999999, 2024-09-21T22:07:56.136 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.052205704999999998, 2024-09-21T22:07:56.136 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:56.136 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:56.137 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:56.137 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:56.137 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.137 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:56.137 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.998090+0000", 2024-09-21T22:07:56.137 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:56.137 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.137 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.137 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:56.137 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.998090+0000", 2024-09-21T22:07:56.137 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:56.137 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.137 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.137 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:56.137 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.998091+0000", 2024-09-21T22:07:56.138 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6929999999999999e-06 2024-09-21T22:07:56.138 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.138 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.138 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:56.138 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.998940+0000", 2024-09-21T22:07:56.138 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00084875200000000004 2024-09-21T22:07:56.138 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.138 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.138 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:56.138 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.998970+0000", 2024-09-21T22:07:56.138 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9709999999999998e-05 2024-09-21T22:07:56.138 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.138 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.138 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:56.138 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.998984+0000", 2024-09-21T22:07:56.139 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3739e-05 2024-09-21T22:07:56.139 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.139 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.139 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:56.139 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.999097+0000", 2024-09-21T22:07:56.139 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000113484 2024-09-21T22:07:56.139 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.139 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.139 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:56.139 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.999135+0000", 2024-09-21T22:07:56.139 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.8405999999999998e-05 2024-09-21T22:07:56.139 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.139 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.139 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:56.139 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.050245+0000", 2024-09-21T22:07:56.140 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.051109935000000002 2024-09-21T22:07:56.140 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.140 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.140 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:56.140 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.050295+0000", 2024-09-21T22:07:56.140 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.9985999999999999e-05 2024-09-21T22:07:56.140 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:56.140 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:56.140 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:56.140 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.140 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.140 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:729 2.0 e63/16 2:08b4f96a:::rbd_data.126483d75f2d.000000000000000f:head v 63'37, mlcod=63'35)", 2024-09-21T22:07:56.140 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:26.062424+0000", 2024-09-21T22:07:56.140 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 29.997155458000002, 2024-09-21T22:07:56.140 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.047312186999999999, 2024-09-21T22:07:56.141 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:56.141 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:56.141 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:56.141 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:56.141 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.141 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:56.141 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.062424+0000", 2024-09-21T22:07:56.141 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:56.141 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.141 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.141 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:56.141 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.062424+0000", 2024-09-21T22:07:56.141 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:56.141 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.141 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.141 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:56.142 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.062426+0000", 2024-09-21T22:07:56.142 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7129999999999999e-06 2024-09-21T22:07:56.142 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.142 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.142 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:56.142 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.063211+0000", 2024-09-21T22:07:56.142 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00078488500000000003 2024-09-21T22:07:56.142 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.142 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.142 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:56.142 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.063241+0000", 2024-09-21T22:07:56.142 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.0056000000000002e-05 2024-09-21T22:07:56.142 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.143 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.143 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:56.143 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.063255+0000", 2024-09-21T22:07:56.143 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4022999999999999e-05 2024-09-21T22:07:56.143 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.143 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.143 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:56.143 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.063384+0000", 2024-09-21T22:07:56.143 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00012920600000000001 2024-09-21T22:07:56.143 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.143 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.143 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:56.143 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.063421+0000", 2024-09-21T22:07:56.143 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.7242e-05 2024-09-21T22:07:56.143 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.144 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.144 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:56.144 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.109666+0000", 2024-09-21T22:07:56.144 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.046244446000000002 2024-09-21T22:07:56.144 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.144 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.144 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:56.144 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.109736+0000", 2024-09-21T22:07:56.144 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.0616000000000003e-05 2024-09-21T22:07:56.144 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:56.144 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:56.144 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:56.144 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.145 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.145 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:731 2.0 e63/16 2:08b4f96a:::rbd_data.126483d75f2d.000000000000000f:head v 63'39, mlcod=63'37)", 2024-09-21T22:07:56.145 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:26.135946+0000", 2024-09-21T22:07:56.145 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 29.923633950999999, 2024-09-21T22:07:56.145 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.054290359000000003, 2024-09-21T22:07:56.145 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:56.145 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:56.145 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:56.145 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:56.145 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.145 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:56.145 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.135946+0000", 2024-09-21T22:07:56.146 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:56.146 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.146 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.146 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:56.146 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.135946+0000", 2024-09-21T22:07:56.146 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:56.146 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.146 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.146 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:56.146 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.135948+0000", 2024-09-21T22:07:56.146 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7999999999999999e-06 2024-09-21T22:07:56.146 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.146 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.147 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:56.147 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.136730+0000", 2024-09-21T22:07:56.147 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00078271899999999997 2024-09-21T22:07:56.147 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.147 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.147 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:56.147 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.136760+0000", 2024-09-21T22:07:56.147 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9876e-05 2024-09-21T22:07:56.147 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.147 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.147 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:56.147 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.136774+0000", 2024-09-21T22:07:56.148 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3478999999999999e-05 2024-09-21T22:07:56.148 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.148 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.148 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:56.148 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.136944+0000", 2024-09-21T22:07:56.148 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000170626 2024-09-21T22:07:56.148 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.148 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.148 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:56.148 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.136985+0000", 2024-09-21T22:07:56.148 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.0377999999999999e-05 2024-09-21T22:07:56.148 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.148 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.149 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:56.149 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.190137+0000", 2024-09-21T22:07:56.149 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.053152438000000003 2024-09-21T22:07:56.149 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.149 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.149 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:56.149 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.190236+0000", 2024-09-21T22:07:56.149 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.9042999999999993e-05 2024-09-21T22:07:56.149 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:56.149 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:56.149 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:56.149 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.150 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.150 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:125 2.3 e71/66 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head v 71'31, mlcod=71'29)", 2024-09-21T22:07:56.150 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.066855+0000", 2024-09-21T22:07:56.150 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 17.992724515999999, 2024-09-21T22:07:56.150 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.059856059000000003, 2024-09-21T22:07:56.150 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:56.150 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:56.150 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:56.151 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:56.151 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.151 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:56.151 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.066855+0000", 2024-09-21T22:07:56.151 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:56.151 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.151 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.151 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:56.151 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.066855+0000", 2024-09-21T22:07:56.151 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:56.151 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.151 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.151 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:56.151 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.066857+0000", 2024-09-21T22:07:56.152 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.807e-06 2024-09-21T22:07:56.152 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.152 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.152 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:56.152 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.068365+0000", 2024-09-21T22:07:56.152 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0015077910000000001 2024-09-21T22:07:56.152 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.152 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.152 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:56.152 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.068395+0000", 2024-09-21T22:07:56.152 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.0326000000000001e-05 2024-09-21T22:07:56.152 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.152 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.152 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:56.152 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.068415+0000", 2024-09-21T22:07:56.153 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9871999999999999e-05 2024-09-21T22:07:56.153 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.153 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.153 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:56.153 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.068580+0000", 2024-09-21T22:07:56.153 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000165317 2024-09-21T22:07:56.153 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.153 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.153 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:56.153 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.068618+0000", 2024-09-21T22:07:56.153 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.7753000000000003e-05 2024-09-21T22:07:56.153 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.153 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.153 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:56.154 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.126623+0000", 2024-09-21T22:07:56.154 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.058004642000000002 2024-09-21T22:07:56.154 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.154 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.154 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:56.154 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.126711+0000", 2024-09-21T22:07:56.154 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.8551e-05 2024-09-21T22:07:56.154 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:56.154 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:56.154 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:56.154 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.154 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.154 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:127 2.3 e71/66 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head v 71'33, mlcod=71'31)", 2024-09-21T22:07:56.154 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.140864+0000", 2024-09-21T22:07:56.155 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 17.918715488, 2024-09-21T22:07:56.155 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.052512380999999997, 2024-09-21T22:07:56.155 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:56.155 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:56.155 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:56.155 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:56.155 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.155 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:56.155 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.140864+0000", 2024-09-21T22:07:56.155 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:56.155 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.155 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.155 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:56.155 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.140864+0000", 2024-09-21T22:07:56.155 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:56.156 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.156 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.156 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:56.156 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.140866+0000", 2024-09-21T22:07:56.156 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.694e-06 2024-09-21T22:07:56.156 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.156 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.156 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:56.156 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.141712+0000", 2024-09-21T22:07:56.156 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00084581499999999998 2024-09-21T22:07:56.156 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.156 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.156 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:56.156 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.141742+0000", 2024-09-21T22:07:56.156 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9865999999999998e-05 2024-09-21T22:07:56.157 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.157 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.157 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:56.157 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.141761+0000", 2024-09-21T22:07:56.157 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9473000000000001e-05 2024-09-21T22:07:56.157 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.157 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.157 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:56.157 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.141919+0000", 2024-09-21T22:07:56.157 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00015758300000000001 2024-09-21T22:07:56.157 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.157 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.157 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:56.157 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.141955+0000", 2024-09-21T22:07:56.158 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.6613000000000003e-05 2024-09-21T22:07:56.158 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.158 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.158 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:56.158 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.193333+0000", 2024-09-21T22:07:56.158 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.051377260000000001 2024-09-21T22:07:56.158 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.158 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.158 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:56.158 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.193377+0000", 2024-09-21T22:07:56.158 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.4076999999999998e-05 2024-09-21T22:07:56.158 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:56.158 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:56.158 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:56.158 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.159 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.159 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:129 2.3 e71/66 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head v 71'35, mlcod=71'33)", 2024-09-21T22:07:56.159 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.209458+0000", 2024-09-21T22:07:56.159 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 17.850121738999999, 2024-09-21T22:07:56.159 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.056595578000000001, 2024-09-21T22:07:56.159 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:56.159 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:56.159 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:56.159 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:56.159 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.159 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:56.159 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.209458+0000", 2024-09-21T22:07:56.159 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:56.160 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.160 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.160 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:56.160 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.209458+0000", 2024-09-21T22:07:56.160 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:56.160 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.160 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.160 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:56.160 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.209460+0000", 2024-09-21T22:07:56.160 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7600000000000001e-06 2024-09-21T22:07:56.160 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.160 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.160 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:56.160 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.210317+0000", 2024-09-21T22:07:56.161 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00085771700000000003 2024-09-21T22:07:56.161 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.161 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.161 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:56.161 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.210346+0000", 2024-09-21T22:07:56.161 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8334e-05 2024-09-21T22:07:56.161 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.161 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.161 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:56.161 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.210365+0000", 2024-09-21T22:07:56.161 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9439000000000001e-05 2024-09-21T22:07:56.161 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.161 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.161 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:56.161 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.210544+0000", 2024-09-21T22:07:56.162 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00017845599999999999 2024-09-21T22:07:56.162 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.162 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.162 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:56.162 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.210584+0000", 2024-09-21T22:07:56.162 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.0423000000000002e-05 2024-09-21T22:07:56.162 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.162 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.162 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:56.162 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.265970+0000", 2024-09-21T22:07:56.162 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.055385765000000003 2024-09-21T22:07:56.162 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.162 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.162 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:56.163 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.266054+0000", 2024-09-21T22:07:56.163 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.3683999999999994e-05 2024-09-21T22:07:56.163 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:56.163 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:56.163 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:56.163 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.163 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.163 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:131 2.3 e71/66 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head v 71'37, mlcod=71'35)", 2024-09-21T22:07:56.163 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.279550+0000", 2024-09-21T22:07:56.163 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 17.780029900999999, 2024-09-21T22:07:56.163 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.052290419999999997, 2024-09-21T22:07:56.163 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:56.163 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:56.163 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:56.163 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:56.164 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.164 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:56.164 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.279550+0000", 2024-09-21T22:07:56.164 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:56.164 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.164 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.164 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:56.164 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.279550+0000", 2024-09-21T22:07:56.164 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:56.164 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.164 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.164 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:56.164 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.279552+0000", 2024-09-21T22:07:56.164 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9439999999999999e-06 2024-09-21T22:07:56.164 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.165 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.165 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:56.165 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.280296+0000", 2024-09-21T22:07:56.165 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00074417000000000005 2024-09-21T22:07:56.165 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.165 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.165 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:56.165 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.280324+0000", 2024-09-21T22:07:56.165 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8045999999999999e-05 2024-09-21T22:07:56.165 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.165 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.165 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:56.165 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.280345+0000", 2024-09-21T22:07:56.166 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.0849999999999999e-05 2024-09-21T22:07:56.166 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.166 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.166 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:56.166 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.280497+0000", 2024-09-21T22:07:56.166 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00015252500000000001 2024-09-21T22:07:56.166 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.166 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.166 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:56.166 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.280531+0000", 2024-09-21T22:07:56.166 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.4034e-05 2024-09-21T22:07:56.166 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.166 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.166 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:56.167 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.331769+0000", 2024-09-21T22:07:56.167 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.051237999999999999 2024-09-21T22:07:56.167 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.167 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.167 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:56.167 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.331840+0000", 2024-09-21T22:07:56.167 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.0851000000000003e-05 2024-09-21T22:07:56.167 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:56.167 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:56.167 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:56.167 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:56.167 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:56.167 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:07:56.169 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T22:07:56.169 INFO:teuthology.orchestra.run.smithi039.stdout:osd.5: {} 2024-09-21T22:07:56.170 INFO:teuthology.orchestra.run.smithi039.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T22:07:56.170 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:56.166+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:56.270 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:56.268+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:56.343 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:07:56.343 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:07:56.343 INFO:teuthology.orchestra.run.smithi039.stdout: "num_ops": 0 2024-09-21T22:07:56.343 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:07:56.357 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T22:07:56.404 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:07:56.402+0000 7fb1388c1640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25159) UID: 0 2024-09-21T22:07:56.472 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:56.468+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:56.476 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:07:56.476 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:07:56.476 INFO:teuthology.orchestra.run.smithi039.stdout: "complaint_time": 30, 2024-09-21T22:07:56.476 INFO:teuthology.orchestra.run.smithi039.stdout: "num_blocked_ops": 0 2024-09-21T22:07:56.477 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:07:56.491 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T22:07:56.580 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:56.578+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:56.666 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:07:56.666 INFO:teuthology.orchestra.run.smithi039.stdout: "size": 20, 2024-09-21T22:07:56.666 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 600, 2024-09-21T22:07:56.666 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [ 2024-09-21T22:07:56.666 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.666 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPull(2.3 e55/54 cost 1048576)", 2024-09-21T22:07:56.666 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.081116+0000", 2024-09-21T22:07:56.666 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 44.581196315, 2024-09-21T22:07:56.667 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0030214159999999999, 2024-09-21T22:07:56.667 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:56.667 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:56.667 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-21T22:07:56.667 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:56.667 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.667 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:56.667 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.081116+0000", 2024-09-21T22:07:56.667 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:56.667 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.667 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.667 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:56.667 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.081116+0000", 2024-09-21T22:07:56.667 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:56.668 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.668 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.668 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:56.668 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.081118+0000", 2024-09-21T22:07:56.668 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.545e-06 2024-09-21T22:07:56.668 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.668 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.668 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:56.668 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.081133+0000", 2024-09-21T22:07:56.668 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5642e-05 2024-09-21T22:07:56.668 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.668 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.668 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:56.668 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.081155+0000", 2024-09-21T22:07:56.669 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1858e-05 2024-09-21T22:07:56.669 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.669 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.669 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:56.669 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.081171+0000", 2024-09-21T22:07:56.669 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5804e-05 2024-09-21T22:07:56.669 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.669 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.669 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:56.669 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.081291+0000", 2024-09-21T22:07:56.669 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000120553 2024-09-21T22:07:56.669 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.669 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.669 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:56.670 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.084137+0000", 2024-09-21T22:07:56.670 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.002846014 2024-09-21T22:07:56.670 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:56.670 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:56.670 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:56.670 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.670 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.670 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPull(2.3 e55/54 cost 1048576)", 2024-09-21T22:07:56.670 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.136357+0000", 2024-09-21T22:07:56.670 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 44.525955248999999, 2024-09-21T22:07:56.670 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.002843904, 2024-09-21T22:07:56.670 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:56.670 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:56.670 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-21T22:07:56.671 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:56.671 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.671 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:56.671 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.136357+0000", 2024-09-21T22:07:56.671 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:56.671 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.671 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.671 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:56.671 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.136357+0000", 2024-09-21T22:07:56.671 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:56.671 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.671 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.671 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:56.672 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.136359+0000", 2024-09-21T22:07:56.672 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.0669999999999999e-06 2024-09-21T22:07:56.672 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.672 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.672 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:56.672 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.136381+0000", 2024-09-21T22:07:56.672 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.2104999999999999e-05 2024-09-21T22:07:56.672 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.672 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.672 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:56.672 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.136412+0000", 2024-09-21T22:07:56.672 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.0602999999999998e-05 2024-09-21T22:07:56.672 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.672 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.672 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:56.673 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.136427+0000", 2024-09-21T22:07:56.673 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4832e-05 2024-09-21T22:07:56.673 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.673 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.673 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:56.673 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.136569+0000", 2024-09-21T22:07:56.673 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000142702 2024-09-21T22:07:56.673 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.673 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.673 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:56.673 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.139201+0000", 2024-09-21T22:07:56.673 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.002631595 2024-09-21T22:07:56.673 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:56.673 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:56.674 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:56.674 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.674 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.674 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:667 2.3 e63/54 2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head v 63'4, mlcod=0'0)", 2024-09-21T22:07:56.674 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.402454+0000", 2024-09-21T22:07:56.674 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 32.259858731000001, 2024-09-21T22:07:56.674 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0058375959999999996, 2024-09-21T22:07:56.674 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:56.674 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:56.674 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:56.674 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:56.674 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.674 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:56.675 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.402454+0000", 2024-09-21T22:07:56.675 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:56.675 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.675 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.675 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:56.675 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.402454+0000", 2024-09-21T22:07:56.675 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:56.675 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.675 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.675 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:56.675 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.402456+0000", 2024-09-21T22:07:56.675 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1179999999999999e-06 2024-09-21T22:07:56.675 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.675 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.676 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:56.676 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.402618+0000", 2024-09-21T22:07:56.676 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00016243 2024-09-21T22:07:56.676 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.676 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.676 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:56.676 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.402643+0000", 2024-09-21T22:07:56.676 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4638999999999999e-05 2024-09-21T22:07:56.676 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.676 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.676 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:56.676 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.402658+0000", 2024-09-21T22:07:56.676 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5211000000000001e-05 2024-09-21T22:07:56.676 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.677 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.677 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:56.677 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.402756+0000", 2024-09-21T22:07:56.677 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.7880000000000005e-05 2024-09-21T22:07:56.677 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.677 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.677 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:56.677 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.402779+0000", 2024-09-21T22:07:56.677 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3524999999999999e-05 2024-09-21T22:07:56.677 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.677 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.677 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:56.677 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.408233+0000", 2024-09-21T22:07:56.677 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0054534700000000002 2024-09-21T22:07:56.677 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.678 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.678 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:56.678 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.408291+0000", 2024-09-21T22:07:56.678 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.8322999999999997e-05 2024-09-21T22:07:56.678 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:56.678 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:56.678 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:56.678 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.678 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.678 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:668 2.3 e63/54 2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head v 63'5, mlcod=52'3)", 2024-09-21T22:07:56.678 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.410724+0000", 2024-09-21T22:07:56.678 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 32.251588314000003, 2024-09-21T22:07:56.678 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.031657809000000002, 2024-09-21T22:07:56.678 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:56.678 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:56.679 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:56.679 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:56.679 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.679 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:56.679 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.410724+0000", 2024-09-21T22:07:56.679 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:56.679 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.679 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.679 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:56.679 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.410724+0000", 2024-09-21T22:07:56.679 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:56.679 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.679 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.679 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:56.679 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.410725+0000", 2024-09-21T22:07:56.680 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.4300000000000001e-07 2024-09-21T22:07:56.680 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.680 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.680 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:56.680 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.411436+0000", 2024-09-21T22:07:56.680 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00071118600000000004 2024-09-21T22:07:56.680 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.680 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.680 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:56.680 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.411455+0000", 2024-09-21T22:07:56.680 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8357999999999999e-05 2024-09-21T22:07:56.680 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.680 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.680 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:56.680 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.411470+0000", 2024-09-21T22:07:56.681 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5753999999999999e-05 2024-09-21T22:07:56.681 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.681 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.681 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:56.681 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.411537+0000", 2024-09-21T22:07:56.681 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.6766000000000004e-05 2024-09-21T22:07:56.681 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.681 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.681 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:56.681 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.411571+0000", 2024-09-21T22:07:56.681 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.3555e-05 2024-09-21T22:07:56.681 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.681 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.681 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:56.682 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.442317+0000", 2024-09-21T22:07:56.682 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.030746393 2024-09-21T22:07:56.682 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.682 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.682 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:56.682 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.442382+0000", 2024-09-21T22:07:56.682 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.4853999999999999e-05 2024-09-21T22:07:56.682 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:56.682 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:56.682 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:56.682 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.682 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.682 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:669 2.3 e63/54 2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head v 63'6, mlcod=63'4)", 2024-09-21T22:07:56.682 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.459924+0000", 2024-09-21T22:07:56.682 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 32.202388755999998, 2024-09-21T22:07:56.683 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0062657900000000002, 2024-09-21T22:07:56.683 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:56.683 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:56.683 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:56.683 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:56.683 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.683 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:56.683 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.459924+0000", 2024-09-21T22:07:56.683 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:56.683 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.683 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.683 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:56.683 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.459924+0000", 2024-09-21T22:07:56.683 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:56.683 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.684 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.684 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:56.684 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.459925+0000", 2024-09-21T22:07:56.684 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8169999999999999e-06 2024-09-21T22:07:56.684 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.684 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.684 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:56.684 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.459983+0000", 2024-09-21T22:07:56.684 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.7567e-05 2024-09-21T22:07:56.684 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.684 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.684 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:56.684 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.460011+0000", 2024-09-21T22:07:56.684 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8455000000000001e-05 2024-09-21T22:07:56.685 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.685 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.685 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:56.685 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.460021+0000", 2024-09-21T22:07:56.685 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.9410000000000002e-06 2024-09-21T22:07:56.685 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.685 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.685 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:56.685 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.460093+0000", 2024-09-21T22:07:56.685 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.1231999999999998e-05 2024-09-21T22:07:56.685 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.685 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.685 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:56.685 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.460124+0000", 2024-09-21T22:07:56.685 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.1335999999999999e-05 2024-09-21T22:07:56.686 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.686 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.686 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:56.686 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.466130+0000", 2024-09-21T22:07:56.686 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0060055519999999999 2024-09-21T22:07:56.686 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.686 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.686 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:56.686 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.466189+0000", 2024-09-21T22:07:56.686 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.9889999999999997e-05 2024-09-21T22:07:56.686 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:56.686 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:56.686 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:56.686 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.687 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.687 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:670 2.3 e63/54 2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head v 63'7, mlcod=63'5)", 2024-09-21T22:07:56.687 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.468764+0000", 2024-09-21T22:07:56.687 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 32.193548774999996, 2024-09-21T22:07:56.687 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.037963433999999997, 2024-09-21T22:07:56.687 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:56.687 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:56.687 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:56.687 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:56.687 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.687 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:56.687 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.468764+0000", 2024-09-21T22:07:56.687 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:56.687 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.688 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.688 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:56.688 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.468764+0000", 2024-09-21T22:07:56.688 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:56.688 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.688 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.688 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:56.688 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.468765+0000", 2024-09-21T22:07:56.688 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.9699999999999994e-07 2024-09-21T22:07:56.688 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.688 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.688 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:56.688 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.469295+0000", 2024-09-21T22:07:56.688 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00053003600000000001 2024-09-21T22:07:56.688 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.689 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.689 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:56.689 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.469311+0000", 2024-09-21T22:07:56.689 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6841999999999999e-05 2024-09-21T22:07:56.689 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.689 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.689 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:56.689 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.469324+0000", 2024-09-21T22:07:56.689 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2373e-05 2024-09-21T22:07:56.689 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.689 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.689 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:56.689 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.469391+0000", 2024-09-21T22:07:56.689 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.6880999999999999e-05 2024-09-21T22:07:56.689 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.690 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.690 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:56.690 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.469419+0000", 2024-09-21T22:07:56.690 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8742999999999999e-05 2024-09-21T22:07:56.690 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.690 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.690 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:56.690 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.506662+0000", 2024-09-21T22:07:56.690 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.037242994000000001 2024-09-21T22:07:56.690 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.690 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.690 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:56.690 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.506727+0000", 2024-09-21T22:07:56.690 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.4567999999999996e-05 2024-09-21T22:07:56.691 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:56.691 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:56.691 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:56.691 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.691 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.691 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:671 2.3 e63/54 2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head v 63'8, mlcod=63'6)", 2024-09-21T22:07:56.691 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.519938+0000", 2024-09-21T22:07:56.691 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 32.142374236000002, 2024-09-21T22:07:56.691 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0058323419999999999, 2024-09-21T22:07:56.691 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:56.691 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:56.691 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:56.691 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:56.691 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.691 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:56.692 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.519938+0000", 2024-09-21T22:07:56.692 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:56.692 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.692 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.692 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:56.692 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.519938+0000", 2024-09-21T22:07:56.692 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:56.692 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.692 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.692 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:56.692 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.519940+0000", 2024-09-21T22:07:56.692 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.666e-06 2024-09-21T22:07:56.692 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.692 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.693 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:56.693 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.520031+0000", 2024-09-21T22:07:56.693 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.1051999999999995e-05 2024-09-21T22:07:56.693 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.693 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.693 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:56.693 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.520056+0000", 2024-09-21T22:07:56.693 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5459000000000001e-05 2024-09-21T22:07:56.693 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.693 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.693 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:56.693 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.520072+0000", 2024-09-21T22:07:56.693 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5801e-05 2024-09-21T22:07:56.693 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.693 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.694 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:56.694 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.520178+0000", 2024-09-21T22:07:56.694 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000105388 2024-09-21T22:07:56.694 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.694 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.694 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:56.694 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.520204+0000", 2024-09-21T22:07:56.694 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.6208e-05 2024-09-21T22:07:56.694 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.694 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.694 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:56.694 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.525735+0000", 2024-09-21T22:07:56.694 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0055308520000000002 2024-09-21T22:07:56.695 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.695 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.695 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:56.695 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.525770+0000", 2024-09-21T22:07:56.695 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.5916e-05 2024-09-21T22:07:56.695 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:56.695 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:56.695 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:56.695 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.695 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.695 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:672 2.3 e63/54 2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head v 63'9, mlcod=63'7)", 2024-09-21T22:07:56.695 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.528432+0000", 2024-09-21T22:07:56.695 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 32.133880859999998, 2024-09-21T22:07:56.695 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.029763412999999999, 2024-09-21T22:07:56.695 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:56.696 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:56.696 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:56.696 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:56.696 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.696 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:56.696 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.528432+0000", 2024-09-21T22:07:56.696 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:56.696 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.696 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.696 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:56.696 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.528432+0000", 2024-09-21T22:07:56.696 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:56.696 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.696 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.697 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:56.697 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.528433+0000", 2024-09-21T22:07:56.697 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7179999999999999e-06 2024-09-21T22:07:56.697 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.697 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.697 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:56.697 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.528926+0000", 2024-09-21T22:07:56.697 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00049238500000000002 2024-09-21T22:07:56.697 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.697 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.697 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:56.697 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.528955+0000", 2024-09-21T22:07:56.697 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9037e-05 2024-09-21T22:07:56.697 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.697 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.698 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:56.698 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.528971+0000", 2024-09-21T22:07:56.698 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6622000000000001e-05 2024-09-21T22:07:56.698 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.698 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.698 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:56.698 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.529047+0000", 2024-09-21T22:07:56.698 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.5246000000000004e-05 2024-09-21T22:07:56.698 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.698 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.698 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:56.698 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.529074+0000", 2024-09-21T22:07:56.698 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.7532e-05 2024-09-21T22:07:56.698 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.699 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.699 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:56.699 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.558149+0000", 2024-09-21T22:07:56.699 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.029075192999999999 2024-09-21T22:07:56.699 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.699 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.699 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:56.699 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.558195+0000", 2024-09-21T22:07:56.699 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.5680000000000003e-05 2024-09-21T22:07:56.699 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:56.699 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:56.699 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:56.699 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.699 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.700 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:673 2.3 e63/54 2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head v 63'10, mlcod=63'8)", 2024-09-21T22:07:56.700 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.576513+0000", 2024-09-21T22:07:56.700 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 32.085799424999998, 2024-09-21T22:07:56.700 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.003674666, 2024-09-21T22:07:56.700 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:56.700 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:56.700 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:56.700 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:56.700 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.700 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:56.700 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.576513+0000", 2024-09-21T22:07:56.700 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:56.700 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.700 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.700 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:56.701 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.576513+0000", 2024-09-21T22:07:56.701 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:56.701 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.701 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.701 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:56.701 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.576514+0000", 2024-09-21T22:07:56.701 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.1299999999999998e-07 2024-09-21T22:07:56.701 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.701 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.701 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:56.701 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.576567+0000", 2024-09-21T22:07:56.701 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.2710000000000002e-05 2024-09-21T22:07:56.701 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.701 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.702 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:56.702 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.576579+0000", 2024-09-21T22:07:56.702 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2035000000000001e-05 2024-09-21T22:07:56.702 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.702 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.702 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:56.702 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.576585+0000", 2024-09-21T22:07:56.702 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.6050000000000003e-06 2024-09-21T22:07:56.702 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.702 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.702 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:56.702 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.576638+0000", 2024-09-21T22:07:56.702 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.2726999999999998e-05 2024-09-21T22:07:56.703 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.703 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.703 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:56.703 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.576656+0000", 2024-09-21T22:07:56.703 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8084000000000002e-05 2024-09-21T22:07:56.703 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.703 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.703 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:56.703 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.580156+0000", 2024-09-21T22:07:56.703 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0035002700000000002 2024-09-21T22:07:56.703 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.703 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.703 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:56.703 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.580188+0000", 2024-09-21T22:07:56.703 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.1322000000000003e-05 2024-09-21T22:07:56.704 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:56.704 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:56.704 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:56.704 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.704 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.704 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:674 2.3 e63/54 2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head v 63'11, mlcod=63'9)", 2024-09-21T22:07:56.704 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.584683+0000", 2024-09-21T22:07:56.704 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 32.077628984, 2024-09-21T22:07:56.704 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.028028352999999999, 2024-09-21T22:07:56.704 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:56.704 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:56.704 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:56.704 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:56.704 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.705 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:56.705 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.584683+0000", 2024-09-21T22:07:56.705 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:56.705 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.705 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.705 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:56.705 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.584683+0000", 2024-09-21T22:07:56.705 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:56.705 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.705 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.705 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:56.705 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.584684+0000", 2024-09-21T22:07:56.705 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.4600000000000004e-07 2024-09-21T22:07:56.705 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.705 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.706 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:56.706 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.585041+0000", 2024-09-21T22:07:56.706 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000356752 2024-09-21T22:07:56.706 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.706 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.706 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:56.706 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.585054+0000", 2024-09-21T22:07:56.706 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3387000000000001e-05 2024-09-21T22:07:56.706 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.706 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.706 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:56.706 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.585065+0000", 2024-09-21T22:07:56.706 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.1124999999999999e-05 2024-09-21T22:07:56.706 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.707 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.707 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:56.707 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.585142+0000", 2024-09-21T22:07:56.707 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.6743999999999996e-05 2024-09-21T22:07:56.707 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.707 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.707 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:56.707 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.585170+0000", 2024-09-21T22:07:56.707 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.7569999999999999e-05 2024-09-21T22:07:56.707 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.707 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.707 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:56.707 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.612637+0000", 2024-09-21T22:07:56.707 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.027467803999999998 2024-09-21T22:07:56.708 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.708 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.708 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:56.708 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.612712+0000", 2024-09-21T22:07:56.708 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.4325000000000004e-05 2024-09-21T22:07:56.708 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:56.708 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:56.708 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:56.708 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.708 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.708 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:683 2.3 e63/54 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head v 63'12, mlcod=63'10)", 2024-09-21T22:07:56.708 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.888242+0000", 2024-09-21T22:07:56.708 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 31.774070366, 2024-09-21T22:07:56.708 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0085158290000000008, 2024-09-21T22:07:56.708 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:56.709 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:56.709 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:56.709 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:56.709 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.709 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:56.709 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.888242+0000", 2024-09-21T22:07:56.709 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:56.709 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.709 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.709 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:56.709 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.888242+0000", 2024-09-21T22:07:56.709 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:56.709 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.709 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.709 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:56.710 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.888244+0000", 2024-09-21T22:07:56.710 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.553e-06 2024-09-21T22:07:56.710 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.710 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.710 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:56.710 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.888295+0000", 2024-09-21T22:07:56.710 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.1916000000000002e-05 2024-09-21T22:07:56.710 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.710 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.710 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:56.710 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.888319+0000", 2024-09-21T22:07:56.710 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3139999999999999e-05 2024-09-21T22:07:56.710 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.710 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.711 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:56.711 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.888332+0000", 2024-09-21T22:07:56.711 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2993000000000001e-05 2024-09-21T22:07:56.711 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.711 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.711 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:56.711 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.888441+0000", 2024-09-21T22:07:56.711 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000109039 2024-09-21T22:07:56.711 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.711 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.711 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:56.711 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.888481+0000", 2024-09-21T22:07:56.711 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.9950000000000002e-05 2024-09-21T22:07:56.711 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.711 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.712 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:56.712 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.896688+0000", 2024-09-21T22:07:56.712 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0082078729999999992 2024-09-21T22:07:56.712 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.712 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.712 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:56.712 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.896758+0000", 2024-09-21T22:07:56.712 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.9364999999999997e-05 2024-09-21T22:07:56.712 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:56.712 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:56.712 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:56.712 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.712 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.712 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:684 2.3 e63/54 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head v 63'13, mlcod=63'11)", 2024-09-21T22:07:56.713 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.899243+0000", 2024-09-21T22:07:56.713 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 31.763069589000001, 2024-09-21T22:07:56.713 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.033436192000000003, 2024-09-21T22:07:56.713 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:56.713 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:56.713 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:56.713 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:56.713 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.713 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:56.713 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.899243+0000", 2024-09-21T22:07:56.713 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:56.713 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.713 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.713 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:56.714 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.899243+0000", 2024-09-21T22:07:56.714 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:56.714 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.714 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.714 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:56.714 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.899244+0000", 2024-09-21T22:07:56.714 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.705e-06 2024-09-21T22:07:56.714 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.714 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.714 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:56.714 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.899604+0000", 2024-09-21T22:07:56.714 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00035930599999999999 2024-09-21T22:07:56.714 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.715 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.715 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:56.715 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.899622+0000", 2024-09-21T22:07:56.715 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8487000000000001e-05 2024-09-21T22:07:56.715 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.715 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.715 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:56.715 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.899634+0000", 2024-09-21T22:07:56.715 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2116999999999999e-05 2024-09-21T22:07:56.715 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.715 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.715 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:56.715 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.899721+0000", 2024-09-21T22:07:56.715 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.6718999999999994e-05 2024-09-21T22:07:56.715 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.716 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.716 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:56.716 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.899755+0000", 2024-09-21T22:07:56.716 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.4292000000000003e-05 2024-09-21T22:07:56.716 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.716 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.716 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:56.716 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.932545+0000", 2024-09-21T22:07:56.716 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.03278959 2024-09-21T22:07:56.716 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.716 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.716 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:56.716 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.932679+0000", 2024-09-21T22:07:56.716 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013397600000000001 2024-09-21T22:07:56.717 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:56.717 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:56.717 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:56.717 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.717 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.717 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:685 2.3 e63/54 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head v 63'14, mlcod=63'12)", 2024-09-21T22:07:56.717 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.935246+0000", 2024-09-21T22:07:56.717 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 31.727066751999999, 2024-09-21T22:07:56.717 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0073590590000000003, 2024-09-21T22:07:56.717 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:56.717 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:56.717 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:56.717 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:56.717 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.717 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:56.718 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.935246+0000", 2024-09-21T22:07:56.718 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:56.718 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.718 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.718 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:56.718 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.935246+0000", 2024-09-21T22:07:56.718 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:56.718 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.718 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.718 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:56.718 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.935247+0000", 2024-09-21T22:07:56.718 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4470000000000001e-06 2024-09-21T22:07:56.718 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.718 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.719 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:56.719 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.935295+0000", 2024-09-21T22:07:56.719 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.7963000000000001e-05 2024-09-21T22:07:56.719 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.719 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.719 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:56.719 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.935313+0000", 2024-09-21T22:07:56.719 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8317000000000001e-05 2024-09-21T22:07:56.719 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.719 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.719 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:56.719 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.935352+0000", 2024-09-21T22:07:56.719 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.8552e-05 2024-09-21T22:07:56.719 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.719 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.720 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:56.720 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.935438+0000", 2024-09-21T22:07:56.720 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.6405999999999999e-05 2024-09-21T22:07:56.720 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.720 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.720 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:56.720 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.935471+0000", 2024-09-21T22:07:56.720 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.3142999999999998e-05 2024-09-21T22:07:56.720 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.720 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.720 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:56.720 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.942552+0000", 2024-09-21T22:07:56.720 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0070809510000000003 2024-09-21T22:07:56.720 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.720 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.721 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:56.721 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.942605+0000", 2024-09-21T22:07:56.721 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.2280000000000001e-05 2024-09-21T22:07:56.721 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:56.721 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:56.721 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:56.721 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.721 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.721 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:686 2.3 e63/54 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head v 63'15, mlcod=63'13)", 2024-09-21T22:07:56.721 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.945043+0000", 2024-09-21T22:07:56.721 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 31.717269438999999, 2024-09-21T22:07:56.721 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.04716302, 2024-09-21T22:07:56.721 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:56.721 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:56.721 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:56.722 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:56.722 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.722 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:56.722 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.945043+0000", 2024-09-21T22:07:56.722 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:56.722 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.722 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.722 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:56.722 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.945043+0000", 2024-09-21T22:07:56.722 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:56.722 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.722 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.722 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:56.723 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.945045+0000", 2024-09-21T22:07:56.723 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8700000000000001e-06 2024-09-21T22:07:56.723 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.723 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.723 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:56.723 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.945899+0000", 2024-09-21T22:07:56.723 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00085401900000000002 2024-09-21T22:07:56.723 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.723 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.723 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:56.723 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.945928+0000", 2024-09-21T22:07:56.723 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9193999999999999e-05 2024-09-21T22:07:56.723 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.723 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.724 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:56.724 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.945948+0000", 2024-09-21T22:07:56.724 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9876000000000001e-05 2024-09-21T22:07:56.724 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.724 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.724 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:56.724 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.946060+0000", 2024-09-21T22:07:56.724 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00011175099999999999 2024-09-21T22:07:56.724 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.724 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.724 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:56.724 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.946103+0000", 2024-09-21T22:07:56.724 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.3090999999999998e-05 2024-09-21T22:07:56.724 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.724 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.725 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:56.725 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.992151+0000", 2024-09-21T22:07:56.725 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.046047948999999998 2024-09-21T22:07:56.725 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.725 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.725 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:56.725 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.992206+0000", 2024-09-21T22:07:56.725 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.5269999999999998e-05 2024-09-21T22:07:56.725 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:56.725 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:56.725 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:56.725 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.725 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.725 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:687 2.3 e63/54 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head v 63'16, mlcod=63'14)", 2024-09-21T22:07:56.726 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.994473+0000", 2024-09-21T22:07:56.726 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 31.667839816000001, 2024-09-21T22:07:56.726 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0054660990000000003, 2024-09-21T22:07:56.726 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:56.726 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:56.726 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:56.726 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:56.726 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.726 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:56.726 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.994473+0000", 2024-09-21T22:07:56.726 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:56.726 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.726 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.726 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:56.727 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.994473+0000", 2024-09-21T22:07:56.727 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:56.727 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.727 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.727 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:56.727 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.994474+0000", 2024-09-21T22:07:56.727 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.463e-06 2024-09-21T22:07:56.727 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.727 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.727 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:56.727 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.994546+0000", 2024-09-21T22:07:56.727 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.2260000000000003e-05 2024-09-21T22:07:56.727 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.727 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.727 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:56.728 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.994564+0000", 2024-09-21T22:07:56.728 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7983999999999999e-05 2024-09-21T22:07:56.728 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.728 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.728 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:56.728 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.994573+0000", 2024-09-21T22:07:56.728 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.4710000000000007e-06 2024-09-21T22:07:56.728 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.728 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.728 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:56.728 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.994634+0000", 2024-09-21T22:07:56.728 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.0872000000000003e-05 2024-09-21T22:07:56.728 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.728 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.729 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:56.729 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.994658+0000", 2024-09-21T22:07:56.729 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4573e-05 2024-09-21T22:07:56.729 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.729 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.729 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:56.729 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.999894+0000", 2024-09-21T22:07:56.729 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0052362390000000002 2024-09-21T22:07:56.729 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.729 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.729 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:56.729 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.999939+0000", 2024-09-21T22:07:56.729 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.4236999999999997e-05 2024-09-21T22:07:56.729 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:56.729 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:56.730 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:56.730 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.730 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.730 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:688 2.3 e63/54 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head v 63'17, mlcod=63'15)", 2024-09-21T22:07:56.730 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.001812+0000", 2024-09-21T22:07:56.730 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 31.660500734999999, 2024-09-21T22:07:56.730 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.028785011999999999, 2024-09-21T22:07:56.730 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:56.730 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:56.730 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:56.730 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:56.730 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.730 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:56.730 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.001812+0000", 2024-09-21T22:07:56.731 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:56.731 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.731 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.731 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:56.731 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.001812+0000", 2024-09-21T22:07:56.731 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:56.731 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.731 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.731 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:56.731 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.001813+0000", 2024-09-21T22:07:56.731 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2699999999999999e-06 2024-09-21T22:07:56.731 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.731 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.731 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:56.731 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.002252+0000", 2024-09-21T22:07:56.732 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00043878599999999998 2024-09-21T22:07:56.732 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.732 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.732 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:56.732 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.002271+0000", 2024-09-21T22:07:56.732 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9346000000000001e-05 2024-09-21T22:07:56.732 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.732 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.732 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:56.732 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.002282+0000", 2024-09-21T22:07:56.732 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0951000000000001e-05 2024-09-21T22:07:56.732 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.732 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.732 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:56.732 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.002398+0000", 2024-09-21T22:07:56.733 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000116412 2024-09-21T22:07:56.733 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.733 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.733 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:56.733 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.002425+0000", 2024-09-21T22:07:56.733 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.6291999999999998e-05 2024-09-21T22:07:56.733 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.733 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.733 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:56.733 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.030528+0000", 2024-09-21T22:07:56.733 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.028103390999999998 2024-09-21T22:07:56.733 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.733 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.733 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:56.733 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.030597+0000", 2024-09-21T22:07:56.734 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.8564000000000003e-05 2024-09-21T22:07:56.734 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:56.734 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:56.734 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:56.734 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.734 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.734 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:689 2.3 e63/54 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head v 63'18, mlcod=63'16)", 2024-09-21T22:07:56.734 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.032891+0000", 2024-09-21T22:07:56.734 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 31.629421152999999, 2024-09-21T22:07:56.734 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.005061032, 2024-09-21T22:07:56.734 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:56.734 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:56.734 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:56.735 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:56.735 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.735 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:56.735 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.032891+0000", 2024-09-21T22:07:56.735 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:56.735 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.735 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.735 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:56.735 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.032891+0000", 2024-09-21T22:07:56.735 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:56.735 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.735 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.735 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:56.735 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.032893+0000", 2024-09-21T22:07:56.736 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3030000000000001e-06 2024-09-21T22:07:56.736 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.736 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.736 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:56.736 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.032952+0000", 2024-09-21T22:07:56.736 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.9070000000000002e-05 2024-09-21T22:07:56.736 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.736 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.736 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:56.736 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.032969+0000", 2024-09-21T22:07:56.736 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7839e-05 2024-09-21T22:07:56.736 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.736 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.736 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:56.736 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.032979+0000", 2024-09-21T22:07:56.737 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.5370000000000003e-06 2024-09-21T22:07:56.737 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.737 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.737 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:56.737 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.033041+0000", 2024-09-21T22:07:56.737 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.1562000000000005e-05 2024-09-21T22:07:56.737 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.737 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.737 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:56.737 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.033068+0000", 2024-09-21T22:07:56.737 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.7226999999999999e-05 2024-09-21T22:07:56.737 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.737 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.737 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:56.738 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.037908+0000", 2024-09-21T22:07:56.738 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0048401770000000002 2024-09-21T22:07:56.738 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.738 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.738 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:56.738 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.037952+0000", 2024-09-21T22:07:56.738 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.4317000000000003e-05 2024-09-21T22:07:56.738 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:56.738 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:56.738 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:56.738 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.738 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.738 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:690 2.3 e63/54 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head v 63'19, mlcod=63'17)", 2024-09-21T22:07:56.738 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.039813+0000", 2024-09-21T22:07:56.739 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 31.622498936, 2024-09-21T22:07:56.739 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.035351891000000003, 2024-09-21T22:07:56.739 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:56.739 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:56.739 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:56.739 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:56.739 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.739 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:56.739 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.039813+0000", 2024-09-21T22:07:56.739 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:56.739 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.739 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.739 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:56.739 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.039813+0000", 2024-09-21T22:07:56.739 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:56.740 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.740 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.740 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:56.740 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.039814+0000", 2024-09-21T22:07:56.740 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.5300000000000002e-07 2024-09-21T22:07:56.740 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.740 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.740 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:56.740 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.040166+0000", 2024-09-21T22:07:56.740 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00035177 2024-09-21T22:07:56.740 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.740 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.740 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:56.740 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.040185+0000", 2024-09-21T22:07:56.740 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9000000000000001e-05 2024-09-21T22:07:56.741 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.741 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.741 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:56.741 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.040195+0000", 2024-09-21T22:07:56.741 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0254999999999999e-05 2024-09-21T22:07:56.741 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.741 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.741 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:56.741 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.040262+0000", 2024-09-21T22:07:56.741 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.6198000000000006e-05 2024-09-21T22:07:56.741 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.741 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.742 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:07:56.742 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.040288+0000", 2024-09-21T22:07:56.742 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5931999999999998e-05 2024-09-21T22:07:56.742 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.742 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.742 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:07:56.742 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.075106+0000", 2024-09-21T22:07:56.742 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.034818546999999998 2024-09-21T22:07:56.742 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.742 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.742 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:56.742 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.075165+0000", 2024-09-21T22:07:56.742 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.9236e-05 2024-09-21T22:07:56.742 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:56.742 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:56.743 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:56.743 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.743 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.743 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPull(2.3 e67/66 cost 1048576)", 2024-09-21T22:07:56.743 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:29.414957+0000", 2024-09-21T22:07:56.743 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 27.247355556999999, 2024-09-21T22:07:56.743 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0032647269999999998, 2024-09-21T22:07:56.743 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:56.743 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:56.743 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-21T22:07:56.743 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:56.743 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.743 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:56.743 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:29.414957+0000", 2024-09-21T22:07:56.744 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:56.744 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.744 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.744 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:56.744 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:29.414957+0000", 2024-09-21T22:07:56.744 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:56.744 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.744 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.744 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:56.744 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:29.414959+0000", 2024-09-21T22:07:56.744 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.396e-06 2024-09-21T22:07:56.744 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.744 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.744 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:56.744 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:29.414986+0000", 2024-09-21T22:07:56.745 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.72e-05 2024-09-21T22:07:56.745 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.745 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.745 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:56.745 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:29.415018+0000", 2024-09-21T22:07:56.745 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.2079000000000002e-05 2024-09-21T22:07:56.745 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.745 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.745 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:56.745 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:29.415035+0000", 2024-09-21T22:07:56.745 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6316000000000001e-05 2024-09-21T22:07:56.745 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.745 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.745 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:56.745 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:29.415201+0000", 2024-09-21T22:07:56.746 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00016583900000000001 2024-09-21T22:07:56.746 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.746 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.746 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:56.746 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:29.418222+0000", 2024-09-21T22:07:56.746 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0030208969999999998 2024-09-21T22:07:56.746 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:56.746 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:56.746 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:56.746 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.746 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.746 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPull(2.3 e67/66 cost 1048576)", 2024-09-21T22:07:56.746 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:29.486922+0000", 2024-09-21T22:07:56.746 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 27.175390620000002, 2024-09-21T22:07:56.747 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0029880940000000002, 2024-09-21T22:07:56.747 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:07:56.747 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:07:56.747 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "reached pg", 2024-09-21T22:07:56.747 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:07:56.747 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.747 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:07:56.747 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:29.486922+0000", 2024-09-21T22:07:56.747 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:56.747 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.747 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.747 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:07:56.747 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:29.486922+0000", 2024-09-21T22:07:56.747 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:07:56.748 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.748 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.748 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:07:56.748 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:29.486924+0000", 2024-09-21T22:07:56.748 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.897e-06 2024-09-21T22:07:56.748 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.748 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.748 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:07:56.748 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:29.486975+0000", 2024-09-21T22:07:56.748 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.1275e-05 2024-09-21T22:07:56.748 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.748 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.748 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:07:56.748 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:29.487002+0000", 2024-09-21T22:07:56.749 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.7469000000000001e-05 2024-09-21T22:07:56.749 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.749 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.749 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:07:56.749 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:29.487016+0000", 2024-09-21T22:07:56.749 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3587000000000001e-05 2024-09-21T22:07:56.749 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.749 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.749 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:07:56.749 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:29.487158+0000", 2024-09-21T22:07:56.749 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00014215899999999999 2024-09-21T22:07:56.749 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:07:56.749 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:07:56.749 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:07:56.750 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:29.489910+0000", 2024-09-21T22:07:56.750 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0027517069999999999 2024-09-21T22:07:56.750 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:56.750 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:56.750 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:56.750 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:07:56.750 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:07:56.750 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:07:56.751 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T22:07:56.752 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:07:56.752+0000 7f2075a06640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31720) UID: 0 2024-09-21T22:07:56.852 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:56.848+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:56.887 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:07:56.887 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:07:56.887 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-09-21T22:07:56.887 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:07:56.901 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T22:07:56.953 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:56.950+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:57.054 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:57.050+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:57.072 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:07:57.072 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:07:57.072 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-21T22:07:57.072 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-21T22:07:57.072 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:07:57.086 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T22:07:57.154 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:57.150+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:57.217 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:07:57.217 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-21T22:07:57.217 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-21T22:07:57.217 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-21T22:07:57.217 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.217 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:07:57.218 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.036355+0000", 2024-09-21T22:07:57.218 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 115.177752283, 2024-09-21T22:07:57.218 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.062441934999999997, 2024-09-21T22:07:57.218 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:57.218 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:57.218 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:57.218 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:57.218 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:07:57.218 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:07:57.218 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1 2024-09-21T22:07:57.218 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.218 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:57.218 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.218 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:57.219 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036355+0000", 2024-09-21T22:07:57.219 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:57.219 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.219 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.219 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:57.219 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036355+0000", 2024-09-21T22:07:57.219 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:57.219 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.219 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.219 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:57.219 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036357+0000", 2024-09-21T22:07:57.219 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.446e-06 2024-09-21T22:07:57.219 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.219 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.220 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:57.220 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036373+0000", 2024-09-21T22:07:57.220 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5447e-05 2024-09-21T22:07:57.220 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.220 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.220 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:57.220 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036399+0000", 2024-09-21T22:07:57.220 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6423000000000001e-05 2024-09-21T22:07:57.220 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.220 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.220 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:57.220 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036420+0000", 2024-09-21T22:07:57.220 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1129999999999999e-05 2024-09-21T22:07:57.220 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.220 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.221 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:57.221 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036557+0000", 2024-09-21T22:07:57.221 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013686899999999999 2024-09-21T22:07:57.221 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.221 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.221 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-21T22:07:57.221 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036585+0000", 2024-09-21T22:07:57.221 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8362999999999999e-05 2024-09-21T22:07:57.221 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.221 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.221 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:57.221 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.095068+0000", 2024-09-21T22:07:57.221 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058482972000000001 2024-09-21T22:07:57.221 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.221 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.222 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:57.222 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.095516+0000", 2024-09-21T22:07:57.222 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00044802999999999998 2024-09-21T22:07:57.222 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.222 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.222 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:57.222 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.095869+0000", 2024-09-21T22:07:57.222 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000352104 2024-09-21T22:07:57.222 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.222 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.222 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:57.222 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.097055+0000", 2024-09-21T22:07:57.222 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011865599999999999 2024-09-21T22:07:57.222 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.223 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.223 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:57.223 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.098637+0000", 2024-09-21T22:07:57.223 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001581458 2024-09-21T22:07:57.223 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.223 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.223 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:57.223 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.098728+0000", 2024-09-21T22:07:57.223 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1043000000000002e-05 2024-09-21T22:07:57.223 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.223 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.223 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:57.223 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.098797+0000", 2024-09-21T22:07:57.223 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9090000000000004e-05 2024-09-21T22:07:57.224 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:57.224 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:57.224 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:57.224 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.224 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.224 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4840.0:680 2.7 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:07:57.224 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.744506+0000", 2024-09-21T22:07:57.224 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 32.469601429000001, 2024-09-21T22:07:57.224 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.063865055000000004, 2024-09-21T22:07:57.224 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:57.224 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:57.224 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:57.224 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:57.224 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4840", 2024-09-21T22:07:57.224 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:57.225 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 680 2024-09-21T22:07:57.225 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.225 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:57.225 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.225 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:57.225 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.744506+0000", 2024-09-21T22:07:57.225 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:57.225 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.225 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.225 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:57.225 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.744506+0000", 2024-09-21T22:07:57.225 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:57.225 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.225 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.226 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:57.226 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.744508+0000", 2024-09-21T22:07:57.226 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.103e-06 2024-09-21T22:07:57.226 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.226 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.226 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:57.226 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.745371+0000", 2024-09-21T22:07:57.226 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00086322199999999997 2024-09-21T22:07:57.226 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.226 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.226 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:57.226 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.745401+0000", 2024-09-21T22:07:57.226 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9955999999999999e-05 2024-09-21T22:07:57.226 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.226 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.227 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:57.227 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.745416+0000", 2024-09-21T22:07:57.227 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4873e-05 2024-09-21T22:07:57.227 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.227 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.227 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:57.227 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.745509+0000", 2024-09-21T22:07:57.227 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3171000000000006e-05 2024-09-21T22:07:57.227 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.227 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.227 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:57.227 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.745632+0000", 2024-09-21T22:07:57.227 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012272699999999999 2024-09-21T22:07:57.227 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.227 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.228 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-09-21T22:07:57.228 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.745789+0000", 2024-09-21T22:07:57.228 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015781399999999999 2024-09-21T22:07:57.228 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.228 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.228 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:57.228 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.785494+0000", 2024-09-21T22:07:57.228 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.039704401 2024-09-21T22:07:57.228 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.228 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.228 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:57.228 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.808090+0000", 2024-09-21T22:07:57.228 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022596207 2024-09-21T22:07:57.228 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.229 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.229 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:57.229 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.808246+0000", 2024-09-21T22:07:57.229 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015567200000000001 2024-09-21T22:07:57.229 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.229 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.229 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:57.229 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.808371+0000", 2024-09-21T22:07:57.229 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012490900000000001 2024-09-21T22:07:57.229 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:57.229 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:57.229 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:57.229 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.229 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.229 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4840.0:682 2.7 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:07:57.230 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.822415+0000", 2024-09-21T22:07:57.230 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 32.391691866000002, 2024-09-21T22:07:57.230 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.065044633000000004, 2024-09-21T22:07:57.230 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:57.230 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:57.230 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:57.230 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:57.230 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4840", 2024-09-21T22:07:57.230 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:57.230 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 682 2024-09-21T22:07:57.230 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.230 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:57.230 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.230 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:57.230 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.822415+0000", 2024-09-21T22:07:57.231 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:57.231 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.231 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.231 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:57.231 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.822415+0000", 2024-09-21T22:07:57.231 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:57.231 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.231 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.231 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:57.231 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.822417+0000", 2024-09-21T22:07:57.231 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8029999999999999e-06 2024-09-21T22:07:57.231 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.231 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.231 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:57.232 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.823308+0000", 2024-09-21T22:07:57.232 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00089146500000000003 2024-09-21T22:07:57.232 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.232 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.232 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:57.232 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.823339+0000", 2024-09-21T22:07:57.232 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0493000000000001e-05 2024-09-21T22:07:57.232 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.233 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.233 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:57.233 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.823354+0000", 2024-09-21T22:07:57.233 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5472999999999999e-05 2024-09-21T22:07:57.233 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.233 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.233 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:57.233 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.823476+0000", 2024-09-21T22:07:57.233 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000121209 2024-09-21T22:07:57.233 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.233 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.233 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:57.233 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.823650+0000", 2024-09-21T22:07:57.233 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017394599999999999 2024-09-21T22:07:57.234 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.234 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.234 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-09-21T22:07:57.234 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.823873+0000", 2024-09-21T22:07:57.234 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022314899999999999 2024-09-21T22:07:57.234 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.234 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.234 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:57.234 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.884396+0000", 2024-09-21T22:07:57.234 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.060523674999999999 2024-09-21T22:07:57.234 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.235 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.235 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:57.235 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.887238+0000", 2024-09-21T22:07:57.235 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028420450000000001 2024-09-21T22:07:57.235 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.235 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.235 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:57.235 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.887369+0000", 2024-09-21T22:07:57.235 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013010699999999999 2024-09-21T22:07:57.235 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.235 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.235 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:57.235 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.887460+0000", 2024-09-21T22:07:57.235 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1267999999999994e-05 2024-09-21T22:07:57.236 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:57.236 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:57.236 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:57.236 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.236 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.236 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4840.0:692 2.5 2:b1d55633:::rbd_data.126483d75f2d.000000000000000b:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:07:57.236 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:25.091048+0000", 2024-09-21T22:07:57.236 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 32.123059331999997, 2024-09-21T22:07:57.236 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.060917092999999999, 2024-09-21T22:07:57.236 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:57.236 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:57.236 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:57.236 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:57.236 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4840", 2024-09-21T22:07:57.236 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:57.237 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 692 2024-09-21T22:07:57.237 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.237 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:57.237 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.237 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:57.237 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.091048+0000", 2024-09-21T22:07:57.237 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:57.237 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.237 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.237 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:57.237 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.091048+0000", 2024-09-21T22:07:57.237 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:57.237 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.238 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.238 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:57.238 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.091050+0000", 2024-09-21T22:07:57.238 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.03e-06 2024-09-21T22:07:57.238 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.238 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.238 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:57.238 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.091989+0000", 2024-09-21T22:07:57.238 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00093932100000000001 2024-09-21T22:07:57.238 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.238 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.238 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:57.238 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.092023+0000", 2024-09-21T22:07:57.238 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4310000000000002e-05 2024-09-21T22:07:57.239 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.239 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.239 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:57.239 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.092040+0000", 2024-09-21T22:07:57.239 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6426000000000002e-05 2024-09-21T22:07:57.239 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.239 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.239 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:57.239 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.092175+0000", 2024-09-21T22:07:57.239 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013500299999999999 2024-09-21T22:07:57.240 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.240 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.240 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:57.240 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.092307+0000", 2024-09-21T22:07:57.240 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000132562 2024-09-21T22:07:57.240 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.240 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.240 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:57.240 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.092499+0000", 2024-09-21T22:07:57.240 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000191972 2024-09-21T22:07:57.240 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.240 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.240 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:57.240 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.151065+0000", 2024-09-21T22:07:57.241 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058565514999999999 2024-09-21T22:07:57.241 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.241 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.241 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:57.241 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.151844+0000", 2024-09-21T22:07:57.241 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00077930800000000004 2024-09-21T22:07:57.241 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.241 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.241 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:57.241 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.151916+0000", 2024-09-21T22:07:57.241 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1921999999999993e-05 2024-09-21T22:07:57.241 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.241 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.241 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:57.242 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.151965+0000", 2024-09-21T22:07:57.242 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8724000000000003e-05 2024-09-21T22:07:57.242 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:57.242 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:57.242 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:57.242 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.242 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.242 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4840.0:696 2.5 2:b1d55633:::rbd_data.126483d75f2d.000000000000000b:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:07:57.242 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:25.218641+0000", 2024-09-21T22:07:57.242 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 31.995466063999999, 2024-09-21T22:07:57.242 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058363669999999999, 2024-09-21T22:07:57.242 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:57.242 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:57.242 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:57.243 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:57.243 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4840", 2024-09-21T22:07:57.243 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:57.243 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 696 2024-09-21T22:07:57.243 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.243 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:57.243 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.243 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:57.243 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.218641+0000", 2024-09-21T22:07:57.243 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:57.243 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.243 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.243 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:57.243 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.218641+0000", 2024-09-21T22:07:57.243 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:57.244 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.244 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.244 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:57.244 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.218642+0000", 2024-09-21T22:07:57.244 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.266e-06 2024-09-21T22:07:57.244 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.244 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.244 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:57.244 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.219432+0000", 2024-09-21T22:07:57.244 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00078986300000000002 2024-09-21T22:07:57.244 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.244 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.244 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:57.244 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.219450+0000", 2024-09-21T22:07:57.244 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7526000000000001e-05 2024-09-21T22:07:57.245 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.245 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.245 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:57.245 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.219459+0000", 2024-09-21T22:07:57.245 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3000000000000007e-06 2024-09-21T22:07:57.245 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.245 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.245 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:57.245 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.219655+0000", 2024-09-21T22:07:57.245 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019599099999999999 2024-09-21T22:07:57.245 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.245 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.245 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:57.245 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.219917+0000", 2024-09-21T22:07:57.246 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026198200000000001 2024-09-21T22:07:57.246 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.246 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.246 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:57.246 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.220198+0000", 2024-09-21T22:07:57.246 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00028154500000000002 2024-09-21T22:07:57.246 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.246 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.246 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:57.246 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.268856+0000", 2024-09-21T22:07:57.246 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.048657687999999998 2024-09-21T22:07:57.246 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.246 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.246 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:57.246 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.276903+0000", 2024-09-21T22:07:57.247 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0080472249999999999 2024-09-21T22:07:57.247 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.247 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.247 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:57.247 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.276958+0000", 2024-09-21T22:07:57.247 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4184999999999997e-05 2024-09-21T22:07:57.247 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.247 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.247 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:57.247 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.277005+0000", 2024-09-21T22:07:57.247 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7098999999999997e-05 2024-09-21T22:07:57.247 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:57.247 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:57.247 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:57.248 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.248 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.248 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4840.0:698 2.5 2:b1d55633:::rbd_data.126483d75f2d.000000000000000b:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:07:57.248 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:25.289380+0000", 2024-09-21T22:07:57.248 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 31.924727517000001, 2024-09-21T22:07:57.248 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.055691072000000001, 2024-09-21T22:07:57.248 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:57.248 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:57.248 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:57.248 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:57.248 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4840", 2024-09-21T22:07:57.248 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:57.248 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 698 2024-09-21T22:07:57.248 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.248 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:57.249 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.249 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:57.249 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.289380+0000", 2024-09-21T22:07:57.249 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:57.249 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.249 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.249 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:57.249 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.289380+0000", 2024-09-21T22:07:57.249 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:57.249 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.249 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.249 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:57.249 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.289381+0000", 2024-09-21T22:07:57.249 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3540000000000001e-06 2024-09-21T22:07:57.250 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.250 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.250 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:57.250 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.290221+0000", 2024-09-21T22:07:57.250 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00083999900000000002 2024-09-21T22:07:57.250 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.250 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.250 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:57.250 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.290238+0000", 2024-09-21T22:07:57.250 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7130999999999999e-05 2024-09-21T22:07:57.250 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.250 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.250 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:57.250 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.290247+0000", 2024-09-21T22:07:57.251 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6500000000000002e-06 2024-09-21T22:07:57.251 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.251 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.251 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:57.251 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.290320+0000", 2024-09-21T22:07:57.251 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3530999999999998e-05 2024-09-21T22:07:57.251 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.251 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.251 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:57.251 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.290433+0000", 2024-09-21T22:07:57.251 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000112744 2024-09-21T22:07:57.251 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.251 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.251 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:07:57.252 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.290605+0000", 2024-09-21T22:07:57.252 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017253700000000001 2024-09-21T22:07:57.252 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.252 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.252 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:57.252 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.320545+0000", 2024-09-21T22:07:57.252 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029939582999999999 2024-09-21T22:07:57.252 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.252 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.252 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:57.252 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.344991+0000", 2024-09-21T22:07:57.252 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024445956000000001 2024-09-21T22:07:57.252 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.252 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.253 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:57.253 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.345034+0000", 2024-09-21T22:07:57.253 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.2806000000000003e-05 2024-09-21T22:07:57.253 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.253 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.253 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:57.253 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.345071+0000", 2024-09-21T22:07:57.253 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6780999999999999e-05 2024-09-21T22:07:57.253 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:57.253 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:57.253 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:57.253 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.253 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.253 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4840.0:729 2.0 2:08b4f96a:::rbd_data.126483d75f2d.000000000000000f:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:07:57.253 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:26.062012+0000", 2024-09-21T22:07:57.254 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 31.152094830999999, 2024-09-21T22:07:57.254 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.060656823999999998, 2024-09-21T22:07:57.254 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:57.254 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:57.254 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:57.254 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:57.254 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4840", 2024-09-21T22:07:57.254 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:57.254 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 729 2024-09-21T22:07:57.254 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.254 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:57.254 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.254 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:57.254 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.062012+0000", 2024-09-21T22:07:57.254 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:57.255 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.255 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.255 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:57.255 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.062012+0000", 2024-09-21T22:07:57.255 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:57.255 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.255 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.255 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:57.255 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.062013+0000", 2024-09-21T22:07:57.255 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.243e-06 2024-09-21T22:07:57.255 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.255 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.255 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:57.255 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.062827+0000", 2024-09-21T22:07:57.255 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00081374799999999999 2024-09-21T22:07:57.256 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.256 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.256 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:57.256 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.062845+0000", 2024-09-21T22:07:57.256 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.808e-05 2024-09-21T22:07:57.256 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.256 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.256 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:57.256 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.062854+0000", 2024-09-21T22:07:57.256 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1039999999999996e-06 2024-09-21T22:07:57.256 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.256 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.256 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:57.256 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.062970+0000", 2024-09-21T22:07:57.256 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011601 2024-09-21T22:07:57.257 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.257 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.257 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:57.257 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.063230+0000", 2024-09-21T22:07:57.257 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025918799999999999 2024-09-21T22:07:57.257 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.257 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.257 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-09-21T22:07:57.257 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.063641+0000", 2024-09-21T22:07:57.257 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00041152799999999998 2024-09-21T22:07:57.257 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.257 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.257 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:57.257 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.122015+0000", 2024-09-21T22:07:57.257 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058374246999999997 2024-09-21T22:07:57.258 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.258 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.258 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:57.258 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.122451+0000", 2024-09-21T22:07:57.258 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000435771 2024-09-21T22:07:57.258 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.258 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.258 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:57.258 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.122576+0000", 2024-09-21T22:07:57.258 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012469099999999999 2024-09-21T22:07:57.258 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.258 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.258 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:57.258 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.122669+0000", 2024-09-21T22:07:57.258 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3214000000000006e-05 2024-09-21T22:07:57.259 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:57.259 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:57.259 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:57.259 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.259 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.259 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4840.0:731 2.0 2:08b4f96a:::rbd_data.126483d75f2d.000000000000000f:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:07:57.259 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:26.135603+0000", 2024-09-21T22:07:57.259 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 31.078504331000001, 2024-09-21T22:07:57.259 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.056838474999999999, 2024-09-21T22:07:57.259 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:57.259 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:57.259 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:57.259 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:57.259 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4840", 2024-09-21T22:07:57.260 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:07:57.260 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 731 2024-09-21T22:07:57.260 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.260 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:57.260 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.260 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:57.260 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.135603+0000", 2024-09-21T22:07:57.260 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:57.260 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.260 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.260 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:57.260 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.135603+0000", 2024-09-21T22:07:57.260 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:57.260 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.260 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.261 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:57.261 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.135605+0000", 2024-09-21T22:07:57.261 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9700000000000002e-06 2024-09-21T22:07:57.261 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.261 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.261 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:57.261 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.136554+0000", 2024-09-21T22:07:57.261 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00094886700000000005 2024-09-21T22:07:57.261 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.261 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.261 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:57.261 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.136582+0000", 2024-09-21T22:07:57.261 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8393000000000001e-05 2024-09-21T22:07:57.261 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.261 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.261 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:57.262 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.136596+0000", 2024-09-21T22:07:57.262 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.43e-05 2024-09-21T22:07:57.262 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.262 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.262 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:57.262 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.136691+0000", 2024-09-21T22:07:57.262 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4843e-05 2024-09-21T22:07:57.262 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.262 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.262 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:57.262 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.136873+0000", 2024-09-21T22:07:57.262 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018237099999999999 2024-09-21T22:07:57.262 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.262 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.263 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-09-21T22:07:57.263 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.137140+0000", 2024-09-21T22:07:57.263 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026642700000000002 2024-09-21T22:07:57.263 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.263 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.263 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:57.263 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.180580+0000", 2024-09-21T22:07:57.263 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.043440078 2024-09-21T22:07:57.263 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.263 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.263 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:57.263 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.192175+0000", 2024-09-21T22:07:57.263 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011594959 2024-09-21T22:07:57.263 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.264 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.264 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:57.264 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.192353+0000", 2024-09-21T22:07:57.264 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017850200000000001 2024-09-21T22:07:57.264 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.264 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.264 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:57.264 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.192441+0000", 2024-09-21T22:07:57.264 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7764999999999997e-05 2024-09-21T22:07:57.264 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:57.264 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:57.264 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:57.264 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.264 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.264 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5207.0:33 2.7 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head [write 131072~131072] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e78)", 2024-09-21T22:07:57.265 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:52.587407+0000", 2024-09-21T22:07:57.265 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.6267002179999999, 2024-09-21T22:07:57.265 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.056613749999999997, 2024-09-21T22:07:57.265 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:57.265 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:57.265 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:57.265 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:57.265 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5207", 2024-09-21T22:07:57.265 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/357731919", 2024-09-21T22:07:57.265 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 33 2024-09-21T22:07:57.265 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.265 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:57.265 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.265 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:57.265 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.587407+0000", 2024-09-21T22:07:57.266 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:57.266 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.266 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.266 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:57.266 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.587407+0000", 2024-09-21T22:07:57.266 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:57.266 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.266 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.266 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:57.266 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.587408+0000", 2024-09-21T22:07:57.266 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.59e-06 2024-09-21T22:07:57.266 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.266 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.266 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:57.266 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.587625+0000", 2024-09-21T22:07:57.267 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000216209 2024-09-21T22:07:57.267 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.267 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.267 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:57.267 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.587653+0000", 2024-09-21T22:07:57.267 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8863000000000001e-05 2024-09-21T22:07:57.267 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.267 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.267 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:57.267 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.587671+0000", 2024-09-21T22:07:57.267 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7710000000000002e-05 2024-09-21T22:07:57.267 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.267 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.267 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:57.267 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.604027+0000", 2024-09-21T22:07:57.268 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016356287000000001 2024-09-21T22:07:57.268 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.268 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.268 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:57.268 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.604170+0000", 2024-09-21T22:07:57.268 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000142769 2024-09-21T22:07:57.268 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.268 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.268 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-09-21T22:07:57.268 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.604388+0000", 2024-09-21T22:07:57.268 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000217476 2024-09-21T22:07:57.268 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.268 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.268 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:57.268 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.643757+0000", 2024-09-21T22:07:57.268 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.039369336999999997 2024-09-21T22:07:57.269 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.269 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.269 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:57.269 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.643792+0000", 2024-09-21T22:07:57.269 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4520000000000002e-05 2024-09-21T22:07:57.269 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.269 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.269 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:57.269 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.643906+0000", 2024-09-21T22:07:57.269 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000114153 2024-09-21T22:07:57.269 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.269 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.269 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:57.269 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.644021+0000", 2024-09-21T22:07:57.269 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000114836 2024-09-21T22:07:57.270 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:57.270 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:57.270 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:57.270 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.270 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.270 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5207.0:34 2.7 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head [write 262144~131072] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e78)", 2024-09-21T22:07:57.270 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:52.587708+0000", 2024-09-21T22:07:57.270 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.62639931, 2024-09-21T22:07:57.270 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.056690213000000003, 2024-09-21T22:07:57.270 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:57.270 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:57.270 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:57.270 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:57.270 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5207", 2024-09-21T22:07:57.271 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/357731919", 2024-09-21T22:07:57.271 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 34 2024-09-21T22:07:57.271 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.271 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:57.271 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.271 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:57.271 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.587708+0000", 2024-09-21T22:07:57.271 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:57.271 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.271 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.271 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:57.271 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.587708+0000", 2024-09-21T22:07:57.271 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:57.271 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.271 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.272 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:57.272 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.587709+0000", 2024-09-21T22:07:57.272 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.72e-06 2024-09-21T22:07:57.272 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.272 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.272 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:57.272 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588017+0000", 2024-09-21T22:07:57.272 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00030767499999999998 2024-09-21T22:07:57.272 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.272 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.272 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:57.272 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588037+0000", 2024-09-21T22:07:57.272 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9865999999999999e-05 2024-09-21T22:07:57.272 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.272 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.273 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:57.273 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588048+0000", 2024-09-21T22:07:57.273 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1384e-05 2024-09-21T22:07:57.273 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.273 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.273 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:57.273 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.614263+0000", 2024-09-21T22:07:57.273 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026214232000000001 2024-09-21T22:07:57.273 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.273 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.273 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:57.273 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.614397+0000", 2024-09-21T22:07:57.273 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013396299999999999 2024-09-21T22:07:57.273 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.273 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.274 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-09-21T22:07:57.274 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.614670+0000", 2024-09-21T22:07:57.274 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027295500000000002 2024-09-21T22:07:57.274 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.274 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.274 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:57.274 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.644046+0000", 2024-09-21T22:07:57.274 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029376973000000001 2024-09-21T22:07:57.274 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.274 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.274 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:57.274 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.644208+0000", 2024-09-21T22:07:57.274 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016169900000000001 2024-09-21T22:07:57.274 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.274 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.275 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:57.275 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.644335+0000", 2024-09-21T22:07:57.275 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012656000000000001 2024-09-21T22:07:57.275 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.275 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.275 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:57.275 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.644398+0000", 2024-09-21T22:07:57.275 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3186000000000001e-05 2024-09-21T22:07:57.275 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:57.275 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:57.275 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:57.275 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.275 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.275 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5207.0:35 2.7 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head [write 393216~131072] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e78)", 2024-09-21T22:07:57.275 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:52.588070+0000", 2024-09-21T22:07:57.276 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.6260374119999996, 2024-09-21T22:07:57.276 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.076411232999999995, 2024-09-21T22:07:57.276 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:57.276 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:57.276 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:57.276 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:57.276 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5207", 2024-09-21T22:07:57.276 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/357731919", 2024-09-21T22:07:57.276 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 35 2024-09-21T22:07:57.276 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.276 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:57.276 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.276 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:57.276 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588070+0000", 2024-09-21T22:07:57.276 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:57.277 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.277 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.277 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:57.277 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588070+0000", 2024-09-21T22:07:57.277 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:57.277 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.277 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.277 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:57.277 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588071+0000", 2024-09-21T22:07:57.277 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1299999999999998e-07 2024-09-21T22:07:57.277 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.277 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.277 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:57.277 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588327+0000", 2024-09-21T22:07:57.277 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025630199999999997 2024-09-21T22:07:57.278 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.278 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.278 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:57.278 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588343+0000", 2024-09-21T22:07:57.278 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6140000000000001e-05 2024-09-21T22:07:57.278 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.278 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.278 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:57.278 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588353+0000", 2024-09-21T22:07:57.278 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0043e-05 2024-09-21T22:07:57.278 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.278 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.278 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:57.278 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.623978+0000", 2024-09-21T22:07:57.279 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.035625281000000002 2024-09-21T22:07:57.279 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.279 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.279 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:57.279 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.624104+0000", 2024-09-21T22:07:57.279 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012591300000000001 2024-09-21T22:07:57.279 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.279 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.279 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-09-21T22:07:57.279 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.624296+0000", 2024-09-21T22:07:57.279 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019226200000000001 2024-09-21T22:07:57.279 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.279 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.279 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:57.279 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.654488+0000", 2024-09-21T22:07:57.279 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.030191551 2024-09-21T22:07:57.280 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.280 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.280 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:57.280 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.664329+0000", 2024-09-21T22:07:57.280 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0098405119999999992 2024-09-21T22:07:57.280 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.280 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.280 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:57.280 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.664414+0000", 2024-09-21T22:07:57.280 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5385999999999999e-05 2024-09-21T22:07:57.280 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.280 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.280 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:57.280 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.664481+0000", 2024-09-21T22:07:57.280 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6929999999999998e-05 2024-09-21T22:07:57.281 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:57.281 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:57.281 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:57.281 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.281 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.281 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5207.0:36 2.7 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head [write 524288~131072] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e78)", 2024-09-21T22:07:57.281 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:52.588370+0000", 2024-09-21T22:07:57.281 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.6257366339999999, 2024-09-21T22:07:57.281 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.076552001999999994, 2024-09-21T22:07:57.281 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:57.281 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:57.281 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:57.281 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:57.281 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5207", 2024-09-21T22:07:57.281 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/357731919", 2024-09-21T22:07:57.282 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 36 2024-09-21T22:07:57.282 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.282 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:57.282 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.282 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:57.282 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588370+0000", 2024-09-21T22:07:57.282 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:57.282 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.282 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.282 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:57.282 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588370+0000", 2024-09-21T22:07:57.282 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:57.282 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.282 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.282 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:57.283 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588372+0000", 2024-09-21T22:07:57.283 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.153e-06 2024-09-21T22:07:57.283 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.283 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.283 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:57.283 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588716+0000", 2024-09-21T22:07:57.283 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00034479200000000002 2024-09-21T22:07:57.283 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.283 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.283 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:57.283 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588752+0000", 2024-09-21T22:07:57.283 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5865999999999998e-05 2024-09-21T22:07:57.283 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.283 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.283 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:57.284 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588777+0000", 2024-09-21T22:07:57.284 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4583000000000002e-05 2024-09-21T22:07:57.284 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.284 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.284 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:57.284 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.633886+0000", 2024-09-21T22:07:57.284 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.045108941999999999 2024-09-21T22:07:57.284 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.284 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.284 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:57.284 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.634059+0000", 2024-09-21T22:07:57.284 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000173046 2024-09-21T22:07:57.284 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.284 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.284 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-09-21T22:07:57.285 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.634314+0000", 2024-09-21T22:07:57.285 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025482400000000001 2024-09-21T22:07:57.285 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.285 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.285 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:57.285 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.664541+0000", 2024-09-21T22:07:57.285 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.030227222000000002 2024-09-21T22:07:57.285 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.285 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.285 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:57.285 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.664653+0000", 2024-09-21T22:07:57.285 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011192900000000001 2024-09-21T22:07:57.285 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.285 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.285 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:57.286 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.664798+0000", 2024-09-21T22:07:57.286 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014499900000000001 2024-09-21T22:07:57.286 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.286 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.286 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:57.286 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.664922+0000", 2024-09-21T22:07:57.286 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012464600000000001 2024-09-21T22:07:57.286 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:57.286 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:57.286 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:57.286 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.286 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.286 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5207.0:37 2.7 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head [write 655360~131072] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e78)", 2024-09-21T22:07:57.286 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:52.588813+0000", 2024-09-21T22:07:57.286 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.6252936199999999, 2024-09-21T22:07:57.287 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.087074796999999995, 2024-09-21T22:07:57.287 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:57.287 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:57.287 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:57.287 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:57.287 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5207", 2024-09-21T22:07:57.287 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/357731919", 2024-09-21T22:07:57.287 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 37 2024-09-21T22:07:57.287 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.287 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:57.287 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.287 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:57.287 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588813+0000", 2024-09-21T22:07:57.287 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:57.287 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.288 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.288 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:57.288 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588813+0000", 2024-09-21T22:07:57.288 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:57.288 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.288 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.288 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:57.288 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588815+0000", 2024-09-21T22:07:57.288 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.643e-06 2024-09-21T22:07:57.288 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.288 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.288 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:57.288 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.589071+0000", 2024-09-21T22:07:57.288 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025580200000000002 2024-09-21T22:07:57.288 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.289 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.289 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:57.289 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.589098+0000", 2024-09-21T22:07:57.289 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7336000000000001e-05 2024-09-21T22:07:57.289 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.289 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.289 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:57.289 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.589121+0000", 2024-09-21T22:07:57.289 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2373000000000001e-05 2024-09-21T22:07:57.289 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.289 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.289 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:57.289 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.644491+0000", 2024-09-21T22:07:57.289 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.055369994999999998 2024-09-21T22:07:57.289 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.290 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.290 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:57.290 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.644707+0000", 2024-09-21T22:07:57.290 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000216488 2024-09-21T22:07:57.290 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.290 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.290 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-09-21T22:07:57.290 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.644908+0000", 2024-09-21T22:07:57.290 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000201196 2024-09-21T22:07:57.290 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.290 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.290 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:57.290 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.675419+0000", 2024-09-21T22:07:57.290 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.03051071 2024-09-21T22:07:57.290 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.291 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.291 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:57.291 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.675591+0000", 2024-09-21T22:07:57.291 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000172545 2024-09-21T22:07:57.291 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.291 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.291 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:57.291 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.675761+0000", 2024-09-21T22:07:57.291 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016942300000000001 2024-09-21T22:07:57.291 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.291 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.291 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:57.291 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.675888+0000", 2024-09-21T22:07:57.291 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000127286 2024-09-21T22:07:57.291 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:57.292 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:57.292 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:57.292 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.292 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.292 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5207.0:38 2.7 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head [write 786432~131072] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e78)", 2024-09-21T22:07:57.292 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:52.589156+0000", 2024-09-21T22:07:57.292 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.6249507159999999, 2024-09-21T22:07:57.292 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.087126880000000004, 2024-09-21T22:07:57.292 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:57.292 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:57.292 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:57.292 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:57.292 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5207", 2024-09-21T22:07:57.292 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/357731919", 2024-09-21T22:07:57.292 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 38 2024-09-21T22:07:57.293 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.293 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:57.293 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.293 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:57.293 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.589156+0000", 2024-09-21T22:07:57.293 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:57.293 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.293 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.293 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:57.293 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.589156+0000", 2024-09-21T22:07:57.293 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:57.293 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.293 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.293 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:57.293 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.589158+0000", 2024-09-21T22:07:57.294 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.717e-06 2024-09-21T22:07:57.294 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.294 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.294 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:57.294 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.589417+0000", 2024-09-21T22:07:57.294 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025860500000000002 2024-09-21T22:07:57.294 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.294 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.294 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:57.294 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.589447+0000", 2024-09-21T22:07:57.294 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9916e-05 2024-09-21T22:07:57.294 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.294 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.294 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:57.294 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.589467+0000", 2024-09-21T22:07:57.295 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0894e-05 2024-09-21T22:07:57.295 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.295 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.295 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:57.295 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.654576+0000", 2024-09-21T22:07:57.295 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.065108579999999999 2024-09-21T22:07:57.295 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.295 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.295 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:57.295 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.654694+0000", 2024-09-21T22:07:57.295 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000118083 2024-09-21T22:07:57.295 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.295 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.295 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-09-21T22:07:57.295 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.654880+0000", 2024-09-21T22:07:57.295 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000186069 2024-09-21T22:07:57.296 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.296 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.296 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:57.296 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.675453+0000", 2024-09-21T22:07:57.296 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020572498000000002 2024-09-21T22:07:57.296 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.296 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.296 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:57.296 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.676055+0000", 2024-09-21T22:07:57.296 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00060247000000000002 2024-09-21T22:07:57.296 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.296 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.296 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:57.296 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.676178+0000", 2024-09-21T22:07:57.297 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000122395 2024-09-21T22:07:57.297 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.297 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.297 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:57.297 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.676283+0000", 2024-09-21T22:07:57.297 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000105653 2024-09-21T22:07:57.297 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:57.297 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:57.297 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:57.297 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.297 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.297 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5207.0:39 2.7 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head [write 917504~131072] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e78)", 2024-09-21T22:07:57.297 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:52.589499+0000", 2024-09-21T22:07:57.297 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.6246081080000003, 2024-09-21T22:07:57.298 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.087365371999999997, 2024-09-21T22:07:57.298 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:57.298 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:57.298 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:57.298 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:57.298 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5207", 2024-09-21T22:07:57.298 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/357731919", 2024-09-21T22:07:57.298 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 39 2024-09-21T22:07:57.298 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.298 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:57.298 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.298 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:57.298 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.589499+0000", 2024-09-21T22:07:57.298 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:57.298 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.299 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.299 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:57.299 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.589499+0000", 2024-09-21T22:07:57.299 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:57.299 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.299 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.299 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:57.299 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.589500+0000", 2024-09-21T22:07:57.299 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.547e-06 2024-09-21T22:07:57.299 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.299 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.299 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:57.299 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.589752+0000", 2024-09-21T22:07:57.299 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025112799999999998 2024-09-21T22:07:57.299 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.300 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.300 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:57.300 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.589783+0000", 2024-09-21T22:07:57.300 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1557000000000003e-05 2024-09-21T22:07:57.300 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.300 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.300 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:57.300 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.589832+0000", 2024-09-21T22:07:57.300 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9175999999999998e-05 2024-09-21T22:07:57.300 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.300 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.300 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:57.300 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.665028+0000", 2024-09-21T22:07:57.300 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.075195293999999996 2024-09-21T22:07:57.300 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.301 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.301 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:57.301 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.665169+0000", 2024-09-21T22:07:57.301 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000141299 2024-09-21T22:07:57.301 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.301 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.301 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-09-21T22:07:57.301 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.665464+0000", 2024-09-21T22:07:57.301 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00029459899999999999 2024-09-21T22:07:57.301 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.301 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.301 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:57.301 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.676551+0000", 2024-09-21T22:07:57.301 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011087107000000001 2024-09-21T22:07:57.301 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.302 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.302 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:57.302 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.676700+0000", 2024-09-21T22:07:57.302 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000149853 2024-09-21T22:07:57.302 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.302 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.302 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:57.302 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.676802+0000", 2024-09-21T22:07:57.302 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010194199999999999 2024-09-21T22:07:57.302 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.302 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.302 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:57.302 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.676864+0000", 2024-09-21T22:07:57.302 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1870000000000002e-05 2024-09-21T22:07:57.302 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:57.303 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:57.303 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:57.303 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.303 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.303 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5207.0:74 2.7 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head [write 524288~131072] snapc 5=[5,3] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-21T22:07:57.303 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:52.938124+0000", 2024-09-21T22:07:57.303 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.2759828960000004, 2024-09-21T22:07:57.303 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.071435100000000001, 2024-09-21T22:07:57.303 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:57.303 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:57.303 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:57.303 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:57.303 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5207", 2024-09-21T22:07:57.303 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/357731919", 2024-09-21T22:07:57.303 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 74 2024-09-21T22:07:57.304 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.304 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:57.304 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.304 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:57.304 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938124+0000", 2024-09-21T22:07:57.304 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:57.304 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.304 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.304 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:57.304 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938124+0000", 2024-09-21T22:07:57.304 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:57.304 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.304 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.304 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:57.304 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938125+0000", 2024-09-21T22:07:57.305 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5600000000000005e-07 2024-09-21T22:07:57.305 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.305 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.305 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:57.305 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938218+0000", 2024-09-21T22:07:57.305 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3239999999999995e-05 2024-09-21T22:07:57.305 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.305 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.305 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:57.305 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938231+0000", 2024-09-21T22:07:57.305 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.272e-05 2024-09-21T22:07:57.305 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.305 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.305 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:57.305 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938239+0000", 2024-09-21T22:07:57.306 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5059999999999995e-06 2024-09-21T22:07:57.306 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.306 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.306 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:57.306 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.983312+0000", 2024-09-21T22:07:57.306 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.045072252 2024-09-21T22:07:57.306 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.306 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.306 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:57.306 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.983458+0000", 2024-09-21T22:07:57.306 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014628899999999999 2024-09-21T22:07:57.306 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.306 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.306 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-09-21T22:07:57.306 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.983682+0000", 2024-09-21T22:07:57.307 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022402899999999999 2024-09-21T22:07:57.307 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.307 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.307 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:57.307 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.993679+0000", 2024-09-21T22:07:57.307 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0099972740000000004 2024-09-21T22:07:57.307 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.307 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.307 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:57.307 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.009436+0000", 2024-09-21T22:07:57.307 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015756867000000001 2024-09-21T22:07:57.307 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.307 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.307 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:57.307 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.009506+0000", 2024-09-21T22:07:57.308 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0277999999999994e-05 2024-09-21T22:07:57.308 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.308 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.308 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:57.308 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.009559+0000", 2024-09-21T22:07:57.308 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2889000000000001e-05 2024-09-21T22:07:57.308 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:57.308 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:57.308 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:57.308 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.308 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.308 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5207.0:75 2.7 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head [write 655360~131072] snapc 5=[5,3] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-21T22:07:57.308 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:52.938253+0000", 2024-09-21T22:07:57.308 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.2758538399999999, 2024-09-21T22:07:57.308 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082064041000000004, 2024-09-21T22:07:57.308 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:57.309 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:57.309 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:57.309 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:57.309 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5207", 2024-09-21T22:07:57.309 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/357731919", 2024-09-21T22:07:57.309 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 75 2024-09-21T22:07:57.309 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.309 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:57.309 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.309 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:57.309 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938253+0000", 2024-09-21T22:07:57.309 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:57.309 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.309 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.310 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:57.310 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938253+0000", 2024-09-21T22:07:57.310 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:57.310 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.310 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.310 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:57.310 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938254+0000", 2024-09-21T22:07:57.310 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8599999999999998e-07 2024-09-21T22:07:57.310 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.310 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.310 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:57.310 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938354+0000", 2024-09-21T22:07:57.310 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9696000000000002e-05 2024-09-21T22:07:57.310 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.310 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.311 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:57.311 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938381+0000", 2024-09-21T22:07:57.311 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.705e-05 2024-09-21T22:07:57.311 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.311 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.311 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:57.311 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938397+0000", 2024-09-21T22:07:57.311 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5996999999999999e-05 2024-09-21T22:07:57.311 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.311 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.311 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:57.311 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.993739+0000", 2024-09-21T22:07:57.311 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.055342823999999999 2024-09-21T22:07:57.311 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.311 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.312 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:57.312 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.993835+0000", 2024-09-21T22:07:57.312 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5919000000000001e-05 2024-09-21T22:07:57.312 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.312 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.312 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-09-21T22:07:57.312 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.993993+0000", 2024-09-21T22:07:57.312 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015729900000000001 2024-09-21T22:07:57.312 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.312 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.312 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:57.312 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.009579+0000", 2024-09-21T22:07:57.312 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015586723 2024-09-21T22:07:57.312 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.312 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.313 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:57.313 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.020134+0000", 2024-09-21T22:07:57.313 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010554908 2024-09-21T22:07:57.313 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.313 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.313 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:57.313 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.020241+0000", 2024-09-21T22:07:57.313 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000106246 2024-09-21T22:07:57.313 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.313 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.313 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:57.313 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.020317+0000", 2024-09-21T22:07:57.313 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6693000000000006e-05 2024-09-21T22:07:57.313 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:57.313 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:57.314 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:57.314 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.314 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.314 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5207.0:76 2.7 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head [write 786432~131072] snapc 5=[5,3] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-21T22:07:57.314 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:52.938410+0000", 2024-09-21T22:07:57.314 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.2756966710000004, 2024-09-21T22:07:57.314 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.086926370000000003, 2024-09-21T22:07:57.314 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:57.314 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:57.314 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:57.314 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:57.314 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5207", 2024-09-21T22:07:57.314 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/357731919", 2024-09-21T22:07:57.314 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 76 2024-09-21T22:07:57.314 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.315 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:57.315 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.315 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:57.315 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938410+0000", 2024-09-21T22:07:57.315 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:57.315 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.315 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.315 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:57.315 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938410+0000", 2024-09-21T22:07:57.315 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:57.315 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.315 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.315 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:57.315 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938411+0000", 2024-09-21T22:07:57.315 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0299999999999998e-07 2024-09-21T22:07:57.316 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.316 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.316 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:57.316 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938502+0000", 2024-09-21T22:07:57.316 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0562000000000005e-05 2024-09-21T22:07:57.316 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.316 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.316 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:57.316 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938514+0000", 2024-09-21T22:07:57.316 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2327000000000001e-05 2024-09-21T22:07:57.316 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.316 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.316 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:57.316 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938522+0000", 2024-09-21T22:07:57.316 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1170000000000004e-06 2024-09-21T22:07:57.317 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.317 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.317 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:57.317 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.001427+0000", 2024-09-21T22:07:57.317 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.062904871000000001 2024-09-21T22:07:57.317 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.317 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.317 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:57.317 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.001519+0000", 2024-09-21T22:07:57.317 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1798000000000004e-05 2024-09-21T22:07:57.317 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.317 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.317 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-09-21T22:07:57.317 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.001671+0000", 2024-09-21T22:07:57.317 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015195900000000001 2024-09-21T22:07:57.317 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.318 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.318 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:57.318 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.020388+0000", 2024-09-21T22:07:57.318 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018717428000000001 2024-09-21T22:07:57.318 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.318 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.318 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:57.318 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.025041+0000", 2024-09-21T22:07:57.318 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0046524039999999997 2024-09-21T22:07:57.318 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.318 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.318 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:57.318 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.025213+0000", 2024-09-21T22:07:57.318 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017286499999999999 2024-09-21T22:07:57.318 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.319 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.319 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:57.319 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.025337+0000", 2024-09-21T22:07:57.319 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012333600000000001 2024-09-21T22:07:57.319 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:57.319 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:57.319 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:57.319 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.319 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.319 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5207.0:77 2.7 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head [write 917504~131072] snapc 5=[5,3] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-21T22:07:57.319 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:52.938534+0000", 2024-09-21T22:07:57.319 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.2755728560000001, 2024-09-21T22:07:57.319 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.087709436000000002, 2024-09-21T22:07:57.319 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:57.319 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:57.320 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:57.320 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:57.320 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5207", 2024-09-21T22:07:57.320 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/357731919", 2024-09-21T22:07:57.320 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 77 2024-09-21T22:07:57.320 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.320 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:57.320 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.320 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:57.320 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938534+0000", 2024-09-21T22:07:57.320 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:57.320 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.320 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.320 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:57.320 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938534+0000", 2024-09-21T22:07:57.320 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:57.321 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.321 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.321 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:57.321 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938535+0000", 2024-09-21T22:07:57.321 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1399999999999996e-07 2024-09-21T22:07:57.321 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.321 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.321 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:57.321 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938623+0000", 2024-09-21T22:07:57.321 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8226000000000005e-05 2024-09-21T22:07:57.321 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.321 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.321 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:57.321 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938645+0000", 2024-09-21T22:07:57.321 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1948999999999999e-05 2024-09-21T22:07:57.322 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.322 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.322 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:57.322 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938660+0000", 2024-09-21T22:07:57.322 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4776999999999999e-05 2024-09-21T22:07:57.322 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.322 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.322 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:57.322 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.009669+0000", 2024-09-21T22:07:57.322 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.071009116999999997 2024-09-21T22:07:57.322 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.322 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.322 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:57.322 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.009949+0000", 2024-09-21T22:07:57.322 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027962899999999999 2024-09-21T22:07:57.323 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.323 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.323 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-09-21T22:07:57.323 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.010355+0000", 2024-09-21T22:07:57.323 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000406334 2024-09-21T22:07:57.323 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.323 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.323 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:57.323 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.020875+0000", 2024-09-21T22:07:57.323 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010520144 2024-09-21T22:07:57.323 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.323 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.323 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:57.323 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.026086+0000", 2024-09-21T22:07:57.323 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0052107569999999999 2024-09-21T22:07:57.324 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.324 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.324 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:57.324 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.026178+0000", 2024-09-21T22:07:57.324 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2113000000000004e-05 2024-09-21T22:07:57.324 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.324 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.324 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:57.324 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.026244+0000", 2024-09-21T22:07:57.324 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5675999999999999e-05 2024-09-21T22:07:57.324 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:57.324 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:57.324 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:57.324 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.324 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.325 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5207.0:93 2.7 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head [write 131072~917504] snapc 5=[5,3] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-21T22:07:57.325 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:53.147757+0000", 2024-09-21T22:07:57.325 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.0663496779999999, 2024-09-21T22:07:57.325 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.062493043999999998, 2024-09-21T22:07:57.325 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:57.325 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:57.325 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:57.325 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:07:57.325 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5207", 2024-09-21T22:07:57.325 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/357731919", 2024-09-21T22:07:57.325 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 93 2024-09-21T22:07:57.325 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.325 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:57.325 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.325 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:57.326 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.147757+0000", 2024-09-21T22:07:57.326 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:57.326 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.326 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.326 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:57.326 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.147757+0000", 2024-09-21T22:07:57.326 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:57.326 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.326 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.326 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:57.326 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.147759+0000", 2024-09-21T22:07:57.326 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0159999999999998e-06 2024-09-21T22:07:57.326 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.326 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.326 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:57.327 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.149009+0000", 2024-09-21T22:07:57.327 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001250082 2024-09-21T22:07:57.327 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.327 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.327 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:57.327 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.149042+0000", 2024-09-21T22:07:57.327 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2617000000000003e-05 2024-09-21T22:07:57.327 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.327 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.327 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:57.327 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.149058+0000", 2024-09-21T22:07:57.327 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6113e-05 2024-09-21T22:07:57.327 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.327 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.327 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:57.328 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.149237+0000", 2024-09-21T22:07:57.328 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017834900000000001 2024-09-21T22:07:57.328 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.328 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.328 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:57.328 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.149378+0000", 2024-09-21T22:07:57.328 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014099199999999999 2024-09-21T22:07:57.328 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.328 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.328 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-09-21T22:07:57.328 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.149582+0000", 2024-09-21T22:07:57.328 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020414900000000001 2024-09-21T22:07:57.328 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.328 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.328 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:07:57.328 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.208648+0000", 2024-09-21T22:07:57.329 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.059066397999999999 2024-09-21T22:07:57.329 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.329 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.329 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:07:57.329 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.210125+0000", 2024-09-21T22:07:57.329 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001477119 2024-09-21T22:07:57.329 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.329 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.329 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:57.329 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.210200+0000", 2024-09-21T22:07:57.329 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4973000000000002e-05 2024-09-21T22:07:57.329 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.329 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.329 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:57.329 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.210250+0000", 2024-09-21T22:07:57.330 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0235999999999998e-05 2024-09-21T22:07:57.330 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:57.330 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:57.330 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:57.330 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:57.330 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:57.330 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:07:57.331 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T22:07:57.332 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:57.328+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:57.432 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:57.430+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:57.508 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:07:57.508 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:07:57.508 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-09-21T22:07:57.508 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:07:57.520 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T22:07:57.566 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:07:57.564+0000 7f2075a06640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31720) UID: 0 2024-09-21T22:07:57.635 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:57.630+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:57.646 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:07:57.647 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:07:57.647 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-21T22:07:57.647 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-21T22:07:57.647 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:07:57.661 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T22:07:57.734 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 5, 0] out_osds: [2] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-09-21T22:07:57.734 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T22:07:57.734 INFO:tasks.thrashosds.thrasher:Killing osd 2, live_osds are [0, 1, 2, 3, 4, 5] 2024-09-21T22:07:57.735 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-21T22:07:57.735 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T22:07:57.746 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:57.744+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:57.814 INFO:tasks.ceph.osd.2:Stopped 2024-09-21T22:07:57.814 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.2 2024-09-21T22:07:57.814 INFO:tasks.thrashosds.thrasher:remote for osd 2 is ubuntu@smithi039.front.sepia.ceph.com 2024-09-21T22:07:57.814 DEBUG:teuthology.orchestra.run.smithi039:> type ceph-objectstore-tool 2024-09-21T22:07:57.818 INFO:teuthology.orchestra.run.smithi039.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-21T22:07:57.819 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal --op list-pgs 2024-09-21T22:07:57.835 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:07:57.835 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-21T22:07:57.835 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-21T22:07:57.835 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-21T22:07:57.835 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.835 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4840.0:676 2.7 e63/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 63'57, mlcod=63'55)", 2024-09-21T22:07:57.836 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.627276+0000", 2024-09-21T22:07:57.836 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 33.206145104999997, 2024-09-21T22:07:57.836 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.037500787000000001, 2024-09-21T22:07:57.836 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:57.836 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:57.836 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:57.836 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:57.836 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.836 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:57.836 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.627276+0000", 2024-09-21T22:07:57.836 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:57.836 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.836 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.836 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:57.837 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.627276+0000", 2024-09-21T22:07:57.837 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:57.837 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.837 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.837 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:57.837 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.627277+0000", 2024-09-21T22:07:57.837 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.068e-06 2024-09-21T22:07:57.837 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.837 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.837 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:57.837 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.628148+0000", 2024-09-21T22:07:57.837 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00087069999999999997 2024-09-21T22:07:57.837 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.837 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.838 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:57.838 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.628170+0000", 2024-09-21T22:07:57.838 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2147999999999999e-05 2024-09-21T22:07:57.838 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.838 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.838 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:57.838 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.628179+0000", 2024-09-21T22:07:57.838 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2450000000000004e-06 2024-09-21T22:07:57.838 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.838 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.838 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:57.838 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.628279+0000", 2024-09-21T22:07:57.839 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9298999999999999e-05 2024-09-21T22:07:57.839 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.839 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.839 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:57.839 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.628305+0000", 2024-09-21T22:07:57.839 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6316999999999999e-05 2024-09-21T22:07:57.839 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.839 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.839 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:57.839 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.664714+0000", 2024-09-21T22:07:57.839 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.036408923000000003 2024-09-21T22:07:57.839 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.839 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.839 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:57.840 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.664777+0000", 2024-09-21T22:07:57.840 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3087000000000001e-05 2024-09-21T22:07:57.840 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:57.840 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:57.840 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:57.840 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.840 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.840 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4840.0:678 2.7 e63/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 63'59, mlcod=63'57)", 2024-09-21T22:07:57.840 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.679689+0000", 2024-09-21T22:07:57.840 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 33.153732605000002, 2024-09-21T22:07:57.840 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053308406000000003, 2024-09-21T22:07:57.840 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:57.840 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:57.840 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:57.841 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:57.841 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.841 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:57.841 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.679689+0000", 2024-09-21T22:07:57.841 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:57.841 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.841 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.841 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:57.841 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.679689+0000", 2024-09-21T22:07:57.841 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:57.841 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.841 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.841 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:57.842 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.679691+0000", 2024-09-21T22:07:57.842 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9400000000000001e-06 2024-09-21T22:07:57.842 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.842 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.842 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:57.842 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.680245+0000", 2024-09-21T22:07:57.842 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00055382999999999995 2024-09-21T22:07:57.842 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.842 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.842 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:57.842 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.680276+0000", 2024-09-21T22:07:57.842 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1123e-05 2024-09-21T22:07:57.842 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.842 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.843 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:57.843 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.680290+0000", 2024-09-21T22:07:57.843 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4056999999999999e-05 2024-09-21T22:07:57.843 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.843 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.843 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:57.843 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.680418+0000", 2024-09-21T22:07:57.843 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000127792 2024-09-21T22:07:57.843 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.843 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.843 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:57.843 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.680465+0000", 2024-09-21T22:07:57.843 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7360000000000001e-05 2024-09-21T22:07:57.844 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.844 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.844 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:57.844 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.732923+0000", 2024-09-21T22:07:57.844 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.052458312999999999 2024-09-21T22:07:57.844 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.844 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.844 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:57.844 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.732997+0000", 2024-09-21T22:07:57.844 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3991000000000004e-05 2024-09-21T22:07:57.844 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:57.844 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:57.844 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:57.844 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.845 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.845 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4840.0:680 2.7 e63/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 63'61, mlcod=63'59)", 2024-09-21T22:07:57.845 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.745996+0000", 2024-09-21T22:07:57.845 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 33.087425091999997, 2024-09-21T22:07:57.845 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.061804132999999997, 2024-09-21T22:07:57.845 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:57.845 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:57.845 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:57.845 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:57.845 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.845 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:57.845 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.745996+0000", 2024-09-21T22:07:57.845 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:57.845 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.845 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.846 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:57.846 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.745996+0000", 2024-09-21T22:07:57.846 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:57.846 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.846 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.846 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:57.846 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.745998+0000", 2024-09-21T22:07:57.846 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.717e-06 2024-09-21T22:07:57.846 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.846 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.846 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:57.846 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.746523+0000", 2024-09-21T22:07:57.846 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00052525399999999996 2024-09-21T22:07:57.846 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.847 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.847 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:57.847 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.746554+0000", 2024-09-21T22:07:57.847 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0586000000000002e-05 2024-09-21T22:07:57.847 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.847 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.847 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:57.847 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.746577+0000", 2024-09-21T22:07:57.847 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3079e-05 2024-09-21T22:07:57.847 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.847 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.847 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:57.847 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.746689+0000", 2024-09-21T22:07:57.847 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011177 2024-09-21T22:07:57.848 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.848 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.848 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:57.848 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.746722+0000", 2024-09-21T22:07:57.848 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2870000000000002e-05 2024-09-21T22:07:57.848 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.848 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.848 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:57.848 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.807740+0000", 2024-09-21T22:07:57.848 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.061018612 2024-09-21T22:07:57.848 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.848 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.848 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:57.848 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.807800+0000", 2024-09-21T22:07:57.849 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0244999999999997e-05 2024-09-21T22:07:57.849 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:57.849 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:57.849 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:57.849 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.849 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.849 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4840.0:682 2.7 e63/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 63'63, mlcod=63'61)", 2024-09-21T22:07:57.849 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.824126+0000", 2024-09-21T22:07:57.849 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 33.009295522999999, 2024-09-21T22:07:57.849 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.038159327, 2024-09-21T22:07:57.849 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:57.849 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:57.849 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:57.849 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:57.850 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.850 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:57.850 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.824126+0000", 2024-09-21T22:07:57.850 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:57.850 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.850 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.850 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:57.850 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.824126+0000", 2024-09-21T22:07:57.850 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:57.850 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.850 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.850 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:57.850 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.824128+0000", 2024-09-21T22:07:57.850 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9659999999999999e-06 2024-09-21T22:07:57.850 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.851 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.851 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:57.851 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.824842+0000", 2024-09-21T22:07:57.851 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00071423999999999999 2024-09-21T22:07:57.851 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.851 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.851 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:57.851 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.824883+0000", 2024-09-21T22:07:57.851 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1131999999999999e-05 2024-09-21T22:07:57.852 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.852 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.852 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:57.852 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.824931+0000", 2024-09-21T22:07:57.852 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7327000000000003e-05 2024-09-21T22:07:57.852 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.852 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.852 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:57.852 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.825058+0000", 2024-09-21T22:07:57.852 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012790499999999999 2024-09-21T22:07:57.852 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.852 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.852 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:57.852 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.825087+0000", 2024-09-21T22:07:57.853 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8192e-05 2024-09-21T22:07:57.853 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.853 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.853 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:57.853 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.862225+0000", 2024-09-21T22:07:57.853 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.037138309000000001 2024-09-21T22:07:57.853 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.853 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.853 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:57.853 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.862285+0000", 2024-09-21T22:07:57.853 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0256e-05 2024-09-21T22:07:57.853 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:57.853 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:57.853 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:57.854 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.854 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.854 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5008.0:116 2.7 e71/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 71'148, mlcod=71'146)", 2024-09-21T22:07:57.854 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:37.867612+0000", 2024-09-21T22:07:57.854 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 19.965809027999999, 2024-09-21T22:07:57.854 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.045123155999999998, 2024-09-21T22:07:57.854 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:57.854 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:57.854 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:57.854 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:57.854 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.854 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:57.854 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.867612+0000", 2024-09-21T22:07:57.854 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:57.855 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.855 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.855 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:57.855 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.867612+0000", 2024-09-21T22:07:57.855 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:57.855 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.855 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.855 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:57.855 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.867613+0000", 2024-09-21T22:07:57.855 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1200000000000002e-07 2024-09-21T22:07:57.855 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.855 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.855 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:57.855 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.868135+0000", 2024-09-21T22:07:57.856 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00052132700000000003 2024-09-21T22:07:57.856 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.856 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.856 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:57.856 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.868150+0000", 2024-09-21T22:07:57.856 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5271e-05 2024-09-21T22:07:57.856 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.856 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.856 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:57.856 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.868156+0000", 2024-09-21T22:07:57.856 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4969999999999998e-06 2024-09-21T22:07:57.856 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.856 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.856 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:57.857 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.868213+0000", 2024-09-21T22:07:57.857 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7114999999999998e-05 2024-09-21T22:07:57.857 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.857 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.857 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:57.857 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.868238+0000", 2024-09-21T22:07:57.857 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4987999999999999e-05 2024-09-21T22:07:57.857 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.857 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.857 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:57.857 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.912677+0000", 2024-09-21T22:07:57.857 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.044439019000000003 2024-09-21T22:07:57.857 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.857 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.858 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:57.858 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.912736+0000", 2024-09-21T22:07:57.858 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8127000000000001e-05 2024-09-21T22:07:57.858 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:57.858 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:57.858 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:57.858 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.858 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.858 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5008.0:118 2.7 e71/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 71'150, mlcod=71'148)", 2024-09-21T22:07:57.858 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:37.924700+0000", 2024-09-21T22:07:57.858 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 19.908721837000002, 2024-09-21T22:07:57.858 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.034650561000000003, 2024-09-21T22:07:57.858 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:57.858 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:57.859 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:57.859 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:57.859 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.859 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:57.859 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.924700+0000", 2024-09-21T22:07:57.859 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:57.859 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.859 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.859 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:57.859 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.924700+0000", 2024-09-21T22:07:57.859 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:57.859 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.860 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.860 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:57.860 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.924702+0000", 2024-09-21T22:07:57.860 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9460000000000001e-06 2024-09-21T22:07:57.860 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.860 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.860 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:57.860 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.925132+0000", 2024-09-21T22:07:57.860 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00043021200000000002 2024-09-21T22:07:57.860 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.860 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.860 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:57.860 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.925156+0000", 2024-09-21T22:07:57.860 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4451999999999999e-05 2024-09-21T22:07:57.861 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.861 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.861 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:57.861 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.925168+0000", 2024-09-21T22:07:57.861 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2249000000000001e-05 2024-09-21T22:07:57.861 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.861 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.861 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:57.861 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.925241+0000", 2024-09-21T22:07:57.861 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2502000000000005e-05 2024-09-21T22:07:57.861 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.861 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.861 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:57.861 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.925267+0000", 2024-09-21T22:07:57.861 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6471000000000001e-05 2024-09-21T22:07:57.862 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.862 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.862 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:57.862 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.959217+0000", 2024-09-21T22:07:57.862 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.033949569999999998 2024-09-21T22:07:57.862 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.862 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.862 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:57.862 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.959350+0000", 2024-09-21T22:07:57.862 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013315899999999999 2024-09-21T22:07:57.862 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:57.862 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:57.862 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:57.862 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.863 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.863 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5008.0:120 2.7 e71/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 71'152, mlcod=71'150)", 2024-09-21T22:07:57.863 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:37.970837+0000", 2024-09-21T22:07:57.863 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 19.862584586000001, 2024-09-21T22:07:57.863 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.03044943, 2024-09-21T22:07:57.863 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:57.863 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:57.863 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:57.863 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:57.863 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.863 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:57.863 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.970837+0000", 2024-09-21T22:07:57.863 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:57.863 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.863 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.864 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:57.864 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.970837+0000", 2024-09-21T22:07:57.864 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:57.864 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.864 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.864 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:57.864 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.970838+0000", 2024-09-21T22:07:57.864 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4649999999999999e-06 2024-09-21T22:07:57.864 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.864 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.864 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:57.864 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.971265+0000", 2024-09-21T22:07:57.864 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00042665800000000001 2024-09-21T22:07:57.864 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.865 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.865 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:57.865 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.971285+0000", 2024-09-21T22:07:57.865 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9639999999999998e-05 2024-09-21T22:07:57.865 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.865 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.865 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:57.865 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.971294+0000", 2024-09-21T22:07:57.865 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7210000000000006e-06 2024-09-21T22:07:57.865 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.865 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.865 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:57.865 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.971354+0000", 2024-09-21T22:07:57.865 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9613e-05 2024-09-21T22:07:57.866 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.866 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.866 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:57.866 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.971381+0000", 2024-09-21T22:07:57.866 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7538999999999998e-05 2024-09-21T22:07:57.866 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.866 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.866 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:57.866 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.001199+0000", 2024-09-21T22:07:57.866 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029817537000000002 2024-09-21T22:07:57.866 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.866 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.866 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:57.866 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.001286+0000", 2024-09-21T22:07:57.867 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7256999999999995e-05 2024-09-21T22:07:57.867 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:57.867 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:57.867 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:57.867 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.867 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.867 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5008.0:122 2.7 e71/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 71'154, mlcod=71'152)", 2024-09-21T22:07:57.867 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:38.014709+0000", 2024-09-21T22:07:57.867 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 19.818712711, 2024-09-21T22:07:57.867 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029567408999999999, 2024-09-21T22:07:57.867 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:57.867 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:57.867 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:57.867 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:57.868 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.868 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:57.868 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.014709+0000", 2024-09-21T22:07:57.868 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:57.868 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.868 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.868 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:57.868 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.014709+0000", 2024-09-21T22:07:57.868 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:57.868 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.868 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.868 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:57.868 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.014709+0000", 2024-09-21T22:07:57.868 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9599999999999999e-07 2024-09-21T22:07:57.868 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.869 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.869 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:57.869 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.015201+0000", 2024-09-21T22:07:57.869 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00049175599999999999 2024-09-21T22:07:57.869 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.869 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.869 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:57.869 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.015215+0000", 2024-09-21T22:07:57.869 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3692999999999999e-05 2024-09-21T22:07:57.869 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.869 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.869 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:57.869 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.015225+0000", 2024-09-21T22:07:57.869 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.815e-06 2024-09-21T22:07:57.870 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.870 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.870 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:57.870 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.015287+0000", 2024-09-21T22:07:57.870 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1858000000000003e-05 2024-09-21T22:07:57.870 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.870 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.870 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:57.870 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.015315+0000", 2024-09-21T22:07:57.870 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8490999999999999e-05 2024-09-21T22:07:57.870 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.870 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.870 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:57.870 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.044182+0000", 2024-09-21T22:07:57.871 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.028866936999999999 2024-09-21T22:07:57.871 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.871 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.871 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:57.871 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.044276+0000", 2024-09-21T22:07:57.871 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4163e-05 2024-09-21T22:07:57.871 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:57.871 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:57.871 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:57.871 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.871 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.871 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5207.0:33 2.7 e78/16 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head v 78'174, mlcod=78'170)", 2024-09-21T22:07:57.871 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:52.604638+0000", 2024-09-21T22:07:57.871 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 5.2287837489999998, 2024-09-21T22:07:57.871 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020363816, 2024-09-21T22:07:57.872 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:57.872 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:57.872 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:57.872 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:57.872 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.872 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:57.872 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.604638+0000", 2024-09-21T22:07:57.872 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:57.872 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.872 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.872 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:57.872 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.604638+0000", 2024-09-21T22:07:57.872 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:57.872 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.872 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.873 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:57.873 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.604640+0000", 2024-09-21T22:07:57.873 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8929999999999999e-06 2024-09-21T22:07:57.873 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.873 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.873 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:57.873 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.604719+0000", 2024-09-21T22:07:57.873 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9116999999999995e-05 2024-09-21T22:07:57.873 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.873 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.873 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:57.873 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.604751+0000", 2024-09-21T22:07:57.873 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2316e-05 2024-09-21T22:07:57.873 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.873 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.874 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:57.874 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.604770+0000", 2024-09-21T22:07:57.874 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9406e-05 2024-09-21T22:07:57.874 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.874 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.874 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:57.874 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.604930+0000", 2024-09-21T22:07:57.874 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015912599999999999 2024-09-21T22:07:57.874 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.874 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.874 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:57.874 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.604981+0000", 2024-09-21T22:07:57.874 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1010000000000001e-05 2024-09-21T22:07:57.874 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.874 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.875 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:57.875 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.624865+0000", 2024-09-21T22:07:57.875 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019884545999999999 2024-09-21T22:07:57.875 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.875 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.875 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:57.875 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.625001+0000", 2024-09-21T22:07:57.875 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013640200000000001 2024-09-21T22:07:57.875 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:57.875 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:57.875 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:57.875 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.875 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.875 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5207.0:34 2.7 e78/16 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head v 78'175, mlcod=78'170)", 2024-09-21T22:07:57.875 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:52.614890+0000", 2024-09-21T22:07:57.876 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 5.2185311429999999, 2024-09-21T22:07:57.876 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019973422000000001, 2024-09-21T22:07:57.876 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:57.876 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:57.876 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:57.876 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:57.876 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.876 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:57.876 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.614890+0000", 2024-09-21T22:07:57.876 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:57.876 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.876 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.876 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:57.876 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.614890+0000", 2024-09-21T22:07:57.877 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:57.877 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.877 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.877 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:57.877 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.614892+0000", 2024-09-21T22:07:57.877 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8899999999999999e-06 2024-09-21T22:07:57.877 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.877 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.877 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:57.877 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.615033+0000", 2024-09-21T22:07:57.877 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014115199999999999 2024-09-21T22:07:57.877 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.877 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.877 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:57.878 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.615069+0000", 2024-09-21T22:07:57.878 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5892999999999997e-05 2024-09-21T22:07:57.878 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.878 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.878 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:57.878 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.615090+0000", 2024-09-21T22:07:57.878 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.037e-05 2024-09-21T22:07:57.878 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.878 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.878 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:57.878 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.615217+0000", 2024-09-21T22:07:57.878 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012768299999999999 2024-09-21T22:07:57.878 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.879 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.879 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:57.879 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.615270+0000", 2024-09-21T22:07:57.879 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3050000000000002e-05 2024-09-21T22:07:57.879 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.879 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.879 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:57.879 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.634730+0000", 2024-09-21T22:07:57.879 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019459860999999998 2024-09-21T22:07:57.879 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.879 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.879 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:57.879 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.634864+0000", 2024-09-21T22:07:57.879 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000133523 2024-09-21T22:07:57.880 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:57.880 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:57.880 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:57.880 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.880 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.880 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5207.0:35 2.7 e78/16 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head v 78'176, mlcod=78'170)", 2024-09-21T22:07:57.880 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:52.624606+0000", 2024-09-21T22:07:57.880 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 5.2088152670000003, 2024-09-21T22:07:57.880 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020065888, 2024-09-21T22:07:57.880 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:57.880 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:57.880 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:57.880 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:57.880 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.880 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:57.881 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.624606+0000", 2024-09-21T22:07:57.881 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:57.881 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.881 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.881 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:57.881 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.624606+0000", 2024-09-21T22:07:57.881 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:57.881 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.881 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.881 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:57.881 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.624608+0000", 2024-09-21T22:07:57.881 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7600000000000001e-06 2024-09-21T22:07:57.881 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.881 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.881 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:57.881 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.624802+0000", 2024-09-21T22:07:57.882 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019425900000000001 2024-09-21T22:07:57.882 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.882 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.882 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:57.882 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.624860+0000", 2024-09-21T22:07:57.882 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7886000000000001e-05 2024-09-21T22:07:57.882 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.882 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.882 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:57.882 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.624882+0000", 2024-09-21T22:07:57.882 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2277e-05 2024-09-21T22:07:57.882 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.882 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.882 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:57.883 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.625081+0000", 2024-09-21T22:07:57.883 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019864500000000001 2024-09-21T22:07:57.883 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.883 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.883 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:57.883 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.625131+0000", 2024-09-21T22:07:57.883 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0120000000000001e-05 2024-09-21T22:07:57.883 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.883 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.883 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:57.883 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.644538+0000", 2024-09-21T22:07:57.883 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019406667999999998 2024-09-21T22:07:57.883 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.883 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.884 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:57.884 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.644672+0000", 2024-09-21T22:07:57.884 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013427299999999999 2024-09-21T22:07:57.884 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:57.884 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:57.884 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:57.884 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.884 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.884 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5207.0:36 2.7 e78/16 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head v 78'177, mlcod=78'171)", 2024-09-21T22:07:57.884 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:52.634570+0000", 2024-09-21T22:07:57.884 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 5.1988514029999999, 2024-09-21T22:07:57.884 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020665352000000001, 2024-09-21T22:07:57.884 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:57.884 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:57.885 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:57.885 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:57.885 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.885 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:57.885 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.634570+0000", 2024-09-21T22:07:57.885 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:57.885 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.885 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.885 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:57.885 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.634570+0000", 2024-09-21T22:07:57.885 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:57.885 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.885 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.885 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:57.885 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.634572+0000", 2024-09-21T22:07:57.886 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7439999999999999e-06 2024-09-21T22:07:57.886 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.886 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.886 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:57.886 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.634715+0000", 2024-09-21T22:07:57.886 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014291600000000001 2024-09-21T22:07:57.886 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.886 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.886 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:57.886 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.634739+0000", 2024-09-21T22:07:57.886 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4615999999999999e-05 2024-09-21T22:07:57.886 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.886 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.886 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:57.887 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.634752+0000", 2024-09-21T22:07:57.887 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.27e-05 2024-09-21T22:07:57.887 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.887 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.887 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:57.887 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.634953+0000", 2024-09-21T22:07:57.887 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020067500000000001 2024-09-21T22:07:57.887 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.887 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.887 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:57.887 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.635008+0000", 2024-09-21T22:07:57.887 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5504000000000003e-05 2024-09-21T22:07:57.887 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.887 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.888 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:57.888 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.655111+0000", 2024-09-21T22:07:57.888 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.02010323 2024-09-21T22:07:57.888 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.888 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.888 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:57.888 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.655235+0000", 2024-09-21T22:07:57.888 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000123967 2024-09-21T22:07:57.888 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:57.888 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:57.888 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:57.888 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.888 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.888 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5207.0:37 2.7 e78/16 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head v 78'178, mlcod=78'174)", 2024-09-21T22:07:57.889 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:52.645132+0000", 2024-09-21T22:07:57.889 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 5.188289385, 2024-09-21T22:07:57.889 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020185980999999999, 2024-09-21T22:07:57.889 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:57.889 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:57.889 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:57.889 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:57.889 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.889 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:57.889 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.645132+0000", 2024-09-21T22:07:57.889 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:57.889 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.889 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.889 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:57.890 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.645132+0000", 2024-09-21T22:07:57.890 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:57.890 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.890 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.890 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:57.890 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.645134+0000", 2024-09-21T22:07:57.890 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1770000000000001e-06 2024-09-21T22:07:57.890 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.890 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.890 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:57.890 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.645259+0000", 2024-09-21T22:07:57.890 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012473900000000001 2024-09-21T22:07:57.890 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.890 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.891 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:57.891 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.645296+0000", 2024-09-21T22:07:57.891 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6572999999999997e-05 2024-09-21T22:07:57.891 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.891 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.891 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:57.891 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.645315+0000", 2024-09-21T22:07:57.891 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9083e-05 2024-09-21T22:07:57.891 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.891 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.891 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:57.891 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.645460+0000", 2024-09-21T22:07:57.891 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014552599999999999 2024-09-21T22:07:57.891 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.892 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.892 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:57.892 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.645516+0000", 2024-09-21T22:07:57.892 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6369000000000002e-05 2024-09-21T22:07:57.892 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.892 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.892 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:57.892 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.665226+0000", 2024-09-21T22:07:57.892 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019709340999999998 2024-09-21T22:07:57.892 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.892 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.892 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:57.892 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.665318+0000", 2024-09-21T22:07:57.892 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2173e-05 2024-09-21T22:07:57.892 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:57.893 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:57.893 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:57.893 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.893 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.893 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5207.0:38 2.7 e78/16 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head v 78'179, mlcod=78'174)", 2024-09-21T22:07:57.893 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:52.655070+0000", 2024-09-21T22:07:57.893 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 5.1783509240000001, 2024-09-21T22:07:57.893 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020312226999999999, 2024-09-21T22:07:57.893 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:57.893 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:57.893 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:57.893 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:57.893 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.894 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:57.894 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.655070+0000", 2024-09-21T22:07:57.894 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:57.894 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.894 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.894 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:57.894 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.655070+0000", 2024-09-21T22:07:57.894 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:57.894 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.894 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.894 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:57.894 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.655073+0000", 2024-09-21T22:07:57.894 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.26e-06 2024-09-21T22:07:57.894 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.895 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.895 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:57.895 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.655200+0000", 2024-09-21T22:07:57.895 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012738300000000001 2024-09-21T22:07:57.895 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.895 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.895 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:57.895 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.655231+0000", 2024-09-21T22:07:57.895 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0865999999999999e-05 2024-09-21T22:07:57.895 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.895 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.895 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:57.895 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.655253+0000", 2024-09-21T22:07:57.895 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1523e-05 2024-09-21T22:07:57.896 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.896 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.896 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:57.896 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.655390+0000", 2024-09-21T22:07:57.896 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013705299999999999 2024-09-21T22:07:57.896 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.896 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.896 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:57.896 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.655448+0000", 2024-09-21T22:07:57.896 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8462999999999999e-05 2024-09-21T22:07:57.896 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.896 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.896 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:57.897 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.675275+0000", 2024-09-21T22:07:57.897 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019827000000000001 2024-09-21T22:07:57.897 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.897 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.897 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:57.897 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.675383+0000", 2024-09-21T22:07:57.897 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000107679 2024-09-21T22:07:57.897 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:57.897 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:57.897 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:57.897 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.897 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.897 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5207.0:70 2.7 e79/16 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head v 79'184, mlcod=79'181)", 2024-09-21T22:07:57.898 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:52.938385+0000", 2024-09-21T22:07:57.898 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.8950368050000002, 2024-09-21T22:07:57.898 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023928324000000001, 2024-09-21T22:07:57.898 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:57.898 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:57.898 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:57.898 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:57.898 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.898 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:57.898 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938385+0000", 2024-09-21T22:07:57.898 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:57.898 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.898 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.899 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:57.899 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938385+0000", 2024-09-21T22:07:57.899 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:57.899 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.899 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.899 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:57.899 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938387+0000", 2024-09-21T22:07:57.899 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1639999999999999e-06 2024-09-21T22:07:57.899 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.899 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.899 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:57.899 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938511+0000", 2024-09-21T22:07:57.899 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000124485 2024-09-21T22:07:57.899 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.900 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.900 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:57.900 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938549+0000", 2024-09-21T22:07:57.900 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7347e-05 2024-09-21T22:07:57.900 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.900 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.900 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:57.900 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938570+0000", 2024-09-21T22:07:57.900 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1013e-05 2024-09-21T22:07:57.900 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.900 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.900 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:57.900 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938718+0000", 2024-09-21T22:07:57.900 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000148789 2024-09-21T22:07:57.901 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.901 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.901 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:57.901 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938769+0000", 2024-09-21T22:07:57.901 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1002999999999999e-05 2024-09-21T22:07:57.901 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.901 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.901 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:57.901 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.962199+0000", 2024-09-21T22:07:57.901 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023429783999999999 2024-09-21T22:07:57.901 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.901 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.901 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:57.901 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.962313+0000", 2024-09-21T22:07:57.901 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000113739 2024-09-21T22:07:57.902 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:57.902 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:57.902 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:57.902 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.902 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.902 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5207.0:71 2.7 e79/16 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head v 79'185, mlcod=79'181)", 2024-09-21T22:07:57.902 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:52.951522+0000", 2024-09-21T22:07:57.902 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.8818989129999997, 2024-09-21T22:07:57.902 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021108028000000001, 2024-09-21T22:07:57.902 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:57.902 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:57.902 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:57.902 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:57.903 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.903 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:57.903 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.951522+0000", 2024-09-21T22:07:57.903 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:57.903 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.903 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.903 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:57.903 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.951522+0000", 2024-09-21T22:07:57.903 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:57.903 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.903 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.903 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:57.903 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.951524+0000", 2024-09-21T22:07:57.903 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9199999999999998e-06 2024-09-21T22:07:57.904 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.904 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.904 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:57.904 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.951686+0000", 2024-09-21T22:07:57.904 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016202499999999999 2024-09-21T22:07:57.904 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.904 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.904 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:57.904 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.951713+0000", 2024-09-21T22:07:57.904 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7007000000000001e-05 2024-09-21T22:07:57.904 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.904 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.904 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:57.904 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.951734+0000", 2024-09-21T22:07:57.905 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0228999999999999e-05 2024-09-21T22:07:57.905 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.905 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.905 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:57.905 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.952740+0000", 2024-09-21T22:07:57.905 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001006631 2024-09-21T22:07:57.905 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.905 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.905 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:57.905 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.952778+0000", 2024-09-21T22:07:57.905 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7316e-05 2024-09-21T22:07:57.905 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.905 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.905 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:57.906 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.972505+0000", 2024-09-21T22:07:57.906 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019727776999999998 2024-09-21T22:07:57.906 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.906 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.906 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:57.906 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.972631+0000", 2024-09-21T22:07:57.906 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012512299999999999 2024-09-21T22:07:57.906 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:57.906 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:57.906 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:57.906 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.906 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.906 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5207.0:72 2.7 e79/16 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head v 79'186, mlcod=79'181)", 2024-09-21T22:07:57.907 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:52.962684+0000", 2024-09-21T22:07:57.907 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.8707378370000001, 2024-09-21T22:07:57.907 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020226731000000001, 2024-09-21T22:07:57.907 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:57.907 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:57.907 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:57.907 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:57.907 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.907 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:57.907 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.962684+0000", 2024-09-21T22:07:57.907 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:57.907 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.907 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.907 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:57.908 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.962684+0000", 2024-09-21T22:07:57.908 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:57.908 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.908 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.908 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:57.908 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.962685+0000", 2024-09-21T22:07:57.908 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5269999999999999e-06 2024-09-21T22:07:57.908 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.908 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.908 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:57.908 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.962806+0000", 2024-09-21T22:07:57.908 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000121192 2024-09-21T22:07:57.908 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.908 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.909 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:57.909 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.962848+0000", 2024-09-21T22:07:57.909 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1387000000000002e-05 2024-09-21T22:07:57.909 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.909 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.909 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:57.909 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.962873+0000", 2024-09-21T22:07:57.909 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5069999999999999e-05 2024-09-21T22:07:57.909 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.909 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.909 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:57.909 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.963011+0000", 2024-09-21T22:07:57.909 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013869899999999999 2024-09-21T22:07:57.909 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.909 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.910 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:57.910 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.963057+0000", 2024-09-21T22:07:57.911 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5793e-05 2024-09-21T22:07:57.911 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.911 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.911 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:57.911 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.982810+0000", 2024-09-21T22:07:57.911 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019752914 2024-09-21T22:07:57.911 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.911 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.911 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:57.911 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.982910+0000", 2024-09-21T22:07:57.911 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010014900000000001 2024-09-21T22:07:57.911 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:57.911 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:57.911 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:57.912 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.912 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.912 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5207.0:75 2.7 e79/16 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head v 79'189, mlcod=79'186)", 2024-09-21T22:07:57.912 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:52.994175+0000", 2024-09-21T22:07:57.912 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.8392459460000001, 2024-09-21T22:07:57.912 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019827546000000001, 2024-09-21T22:07:57.912 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:57.912 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:57.912 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:57.912 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:57.912 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.912 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:57.913 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.994175+0000", 2024-09-21T22:07:57.913 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:57.913 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.913 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.913 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:57.913 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.994175+0000", 2024-09-21T22:07:57.913 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:57.913 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.913 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.913 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:57.913 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.994177+0000", 2024-09-21T22:07:57.913 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5829999999999999e-06 2024-09-21T22:07:57.913 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.913 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.913 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:57.914 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.994264+0000", 2024-09-21T22:07:57.914 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7182999999999995e-05 2024-09-21T22:07:57.914 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.914 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.914 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:57.914 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.994294+0000", 2024-09-21T22:07:57.914 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0093000000000001e-05 2024-09-21T22:07:57.914 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.914 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.914 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:57.914 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.994318+0000", 2024-09-21T22:07:57.914 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3969999999999999e-05 2024-09-21T22:07:57.914 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.915 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.915 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:57.915 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.994443+0000", 2024-09-21T22:07:57.915 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000124459 2024-09-21T22:07:57.915 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.915 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.915 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:57.915 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.994475+0000", 2024-09-21T22:07:57.915 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2432999999999999e-05 2024-09-21T22:07:57.915 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.915 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.915 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:57.915 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.013934+0000", 2024-09-21T22:07:57.915 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019459309000000001 2024-09-21T22:07:57.915 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.916 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.916 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:57.916 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.014003+0000", 2024-09-21T22:07:57.916 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8516000000000006e-05 2024-09-21T22:07:57.916 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:57.916 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:57.916 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:57.916 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.916 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.916 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5207.0:76 2.7 e79/16 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head v 79'190, mlcod=79'186)", 2024-09-21T22:07:57.916 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:53.001917+0000", 2024-09-21T22:07:57.916 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.831504464, 2024-09-21T22:07:57.916 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022783040000000001, 2024-09-21T22:07:57.916 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:57.917 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:57.917 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:57.917 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:57.917 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.917 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:57.917 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.001917+0000", 2024-09-21T22:07:57.917 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:57.917 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.917 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.917 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:57.917 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.001917+0000", 2024-09-21T22:07:57.917 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:57.917 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.917 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.917 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:57.918 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.001919+0000", 2024-09-21T22:07:57.918 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.703e-06 2024-09-21T22:07:57.918 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.918 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.918 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:57.918 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.002006+0000", 2024-09-21T22:07:57.918 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6978999999999995e-05 2024-09-21T22:07:57.918 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.918 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.918 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:57.918 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.002042+0000", 2024-09-21T22:07:57.918 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6847000000000001e-05 2024-09-21T22:07:57.918 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.918 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.919 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:57.919 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.002057+0000", 2024-09-21T22:07:57.919 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.419e-05 2024-09-21T22:07:57.919 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.919 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.919 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:57.919 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.004387+0000", 2024-09-21T22:07:57.919 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002330365 2024-09-21T22:07:57.919 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.919 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.919 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:57.919 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.004420+0000", 2024-09-21T22:07:57.919 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3170000000000003e-05 2024-09-21T22:07:57.919 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.920 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.920 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:57.920 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.024610+0000", 2024-09-21T22:07:57.920 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020190084000000001 2024-09-21T22:07:57.920 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.920 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.920 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:57.920 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.024700+0000", 2024-09-21T22:07:57.920 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9702000000000003e-05 2024-09-21T22:07:57.920 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:57.920 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:57.920 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:57.920 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.920 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.921 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5207.0:93 2.7 e79/16 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head v 79'193, mlcod=79'191)", 2024-09-21T22:07:57.921 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:53.149813+0000", 2024-09-21T22:07:57.921 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.6836084050000002, 2024-09-21T22:07:57.921 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.060053738000000002, 2024-09-21T22:07:57.921 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:07:57.921 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:07:57.921 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:07:57.921 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:07:57.921 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.921 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:07:57.921 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.149813+0000", 2024-09-21T22:07:57.921 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:57.921 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.921 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.921 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:07:57.922 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.149813+0000", 2024-09-21T22:07:57.922 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:07:57.922 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.922 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.922 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:07:57.922 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.149815+0000", 2024-09-21T22:07:57.922 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9099999999999999e-06 2024-09-21T22:07:57.922 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.922 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.922 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:07:57.922 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.150828+0000", 2024-09-21T22:07:57.922 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0010129099999999999 2024-09-21T22:07:57.922 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.922 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.922 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:07:57.923 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.150860+0000", 2024-09-21T22:07:57.923 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2443000000000001e-05 2024-09-21T22:07:57.923 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.923 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.923 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:07:57.923 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.150874+0000", 2024-09-21T22:07:57.923 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3627e-05 2024-09-21T22:07:57.923 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.923 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.923 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:07:57.923 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.151010+0000", 2024-09-21T22:07:57.923 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013656599999999999 2024-09-21T22:07:57.923 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.923 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.923 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:07:57.924 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.151047+0000", 2024-09-21T22:07:57.924 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6495999999999998e-05 2024-09-21T22:07:57.924 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.924 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.924 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:07:57.924 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.209791+0000", 2024-09-21T22:07:57.924 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058744004000000002 2024-09-21T22:07:57.924 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:07:57.924 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:07:57.924 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:07:57.924 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.209867+0000", 2024-09-21T22:07:57.924 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5782000000000001e-05 2024-09-21T22:07:57.924 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:57.924 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:57.924 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:57.925 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:07:57.925 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:07:57.925 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:07:57.925 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-21T22:07:57.927 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:07:57.924+0000 7f2075a06640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31720) UID: 0 2024-09-21T22:07:57.927 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T22:07:58.026 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:58.022+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:58.058 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:07:58.058 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:07:58.058 INFO:teuthology.orchestra.run.smithi039.stdout: "num_ops": 0 2024-09-21T22:07:58.058 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:07:58.072 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T22:07:58.164 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:58.162+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:58.171 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T22:07:58.228 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:58.226+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:58.234 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:07:58.234 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:07:58.234 INFO:teuthology.orchestra.run.smithi039.stdout: "complaint_time": 30, 2024-09-21T22:07:58.234 INFO:teuthology.orchestra.run.smithi039.stdout: "num_blocked_ops": 0 2024-09-21T22:07:58.234 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:07:58.250 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T22:07:58.313 INFO:teuthology.orchestra.run.smithi039.stdout:osd.0: {} 2024-09-21T22:07:58.314 INFO:teuthology.orchestra.run.smithi039.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T22:07:58.329 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:58.324+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:58.330 INFO:teuthology.orchestra.run.smithi039.stdout:osd.1: {} 2024-09-21T22:07:58.331 INFO:teuthology.orchestra.run.smithi039.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T22:07:58.331 INFO:teuthology.orchestra.run.smithi039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-21T22:07:58.352 INFO:teuthology.orchestra.run.smithi039.stdout:osd.3: {} 2024-09-21T22:07:58.352 INFO:teuthology.orchestra.run.smithi039.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T22:07:58.384 INFO:teuthology.orchestra.run.smithi039.stdout:osd.4: {} 2024-09-21T22:07:58.384 INFO:teuthology.orchestra.run.smithi039.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T22:07:58.405 INFO:teuthology.orchestra.run.smithi039.stdout:osd.5: {} 2024-09-21T22:07:58.405 INFO:teuthology.orchestra.run.smithi039.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T22:07:58.408 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T22:07:58.409 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T22:07:58.416 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:58.416 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:58.430 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:58.428+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:58.437 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T22:07:58.531 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:07:58.528+0000 7f2075a06640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31720) UID: 0 2024-09-21T22:07:58.598 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:58.598 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:58.612 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T22:07:58.633 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:07:58.628+0000 7f2075a06640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31720) UID: 0 2024-09-21T22:07:58.731 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:58.731 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:58.734 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:58.728+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:58.748 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T22:07:58.835 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:07:58.832+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:07:58.872 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:58.872 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:58.886 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T22:07:58.936 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:58.934+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:58.947 INFO:teuthology.orchestra.run.smithi039.stderr:nodeep-scrub is unset 2024-09-21T22:07:59.024 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:59.024 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:59.037 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:07:59.032+0000 7f2075a06640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31720) UID: 0 2024-09-21T22:07:59.038 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T22:07:59.137 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:07:59.132+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:07:59.209 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:59.210 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:59.224 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T22:07:59.238 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:59.234+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:59.339 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:07:59.336+0000 7f2075a06640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31720) UID: 0 2024-09-21T22:07:59.400 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:59.400 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:59.414 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T22:07:59.458 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:07:59.456+0000 7f2075a06640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31720) UID: 0 2024-09-21T22:07:59.541 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:59.538+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:59.584 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:59.584 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:59.600 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T22:07:59.642 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:07:59.638+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:07:59.727 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:59.727 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:59.741 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T22:07:59.743 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:59.742+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:59.844 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:59.842+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:07:59.861 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:07:59.862 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:07:59.876 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T22:07:59.944 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:07:59.942+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:08:00.004 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:00.005 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:00.020 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T22:08:00.046 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:08:00.042+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:08:00.108 INFO:tasks.thrashosds.thrasher:No PGs found for osd.2 2024-09-21T22:08:00.140 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:00.140 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:00.152 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:08:00.149+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:08:00.154 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T22:08:00.247 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:08:00.244+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:08:00.277 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:00.277 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:00.292 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T22:08:00.348 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:08:00.344+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:08:00.410 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T22:08:00.429 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:00.429 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:00.444 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T22:08:00.449 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:08:00.448+0000 7f2075a06640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31720) UID: 0 2024-09-21T22:08:00.549 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:08:00.545+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:08:00.556 INFO:teuthology.orchestra.run.smithi039.stdout:osd.0: {} 2024-09-21T22:08:00.556 INFO:teuthology.orchestra.run.smithi039.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T22:08:00.574 INFO:teuthology.orchestra.run.smithi039.stdout:osd.1: {} 2024-09-21T22:08:00.574 INFO:teuthology.orchestra.run.smithi039.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T22:08:00.574 INFO:teuthology.orchestra.run.smithi039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-21T22:08:00.595 INFO:teuthology.orchestra.run.smithi039.stdout:osd.3: {} 2024-09-21T22:08:00.596 INFO:teuthology.orchestra.run.smithi039.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T22:08:00.616 INFO:teuthology.orchestra.run.smithi039.stdout:osd.4: {} 2024-09-21T22:08:00.616 INFO:teuthology.orchestra.run.smithi039.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T22:08:00.617 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:08:00.617 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:08:00.617 INFO:teuthology.orchestra.run.smithi039.stdout: "complaint_time": 30, 2024-09-21T22:08:00.617 INFO:teuthology.orchestra.run.smithi039.stdout: "num_blocked_ops": 0 2024-09-21T22:08:00.617 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:08:00.632 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T22:08:00.637 INFO:teuthology.orchestra.run.smithi039.stdout:osd.5: {} 2024-09-21T22:08:00.638 INFO:teuthology.orchestra.run.smithi039.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T22:08:00.643 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T22:08:00.643 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T22:08:00.651 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:08:00.648+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:08:00.752 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:08:00.752+0000 7f2075a06640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31720) UID: 0 2024-09-21T22:08:00.765 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:08:00.765 INFO:teuthology.orchestra.run.smithi039.stdout: "size": 20, 2024-09-21T22:08:00.765 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 600, 2024-09-21T22:08:00.765 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [ 2024-09-21T22:08:00.765 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.765 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:694 2.5 e63/37 2:b1d55633:::rbd_data.126483d75f2d.000000000000000b:head v 63'26, mlcod=63'24)", 2024-09-21T22:08:00.765 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.163199+0000", 2024-09-21T22:08:00.766 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 35.598666178000002, 2024-09-21T22:08:00.766 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.034676672999999998, 2024-09-21T22:08:00.766 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:00.766 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:00.766 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:00.766 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:00.766 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.766 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:00.766 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.163199+0000", 2024-09-21T22:08:00.766 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:00.766 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.766 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.766 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:00.767 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.163199+0000", 2024-09-21T22:08:00.767 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:00.767 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.767 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.767 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:00.767 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.163200+0000", 2024-09-21T22:08:00.767 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.703e-06 2024-09-21T22:08:00.767 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.767 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.767 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:00.767 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.164013+0000", 2024-09-21T22:08:00.767 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000812447 2024-09-21T22:08:00.767 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.767 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.768 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:00.768 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.164042+0000", 2024-09-21T22:08:00.768 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9006e-05 2024-09-21T22:08:00.768 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.768 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.768 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:00.768 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.164064+0000", 2024-09-21T22:08:00.768 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1719999999999999e-05 2024-09-21T22:08:00.768 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.768 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.768 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:00.768 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.164143+0000", 2024-09-21T22:08:00.768 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.9283999999999995e-05 2024-09-21T22:08:00.769 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.769 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.769 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:00.769 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.164165+0000", 2024-09-21T22:08:00.769 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.2379e-05 2024-09-21T22:08:00.769 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.769 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.769 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:00.769 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.197827+0000", 2024-09-21T22:08:00.769 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.033661782000000001 2024-09-21T22:08:00.769 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.769 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.769 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:00.769 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.197875+0000", 2024-09-21T22:08:00.770 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8352000000000001e-05 2024-09-21T22:08:00.770 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:00.770 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:00.770 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:00.770 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.770 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.770 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:696 2.5 e63/37 2:b1d55633:::rbd_data.126483d75f2d.000000000000000b:head v 63'28, mlcod=63'26)", 2024-09-21T22:08:00.770 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.218913+0000", 2024-09-21T22:08:00.770 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 35.542951731999999, 2024-09-21T22:08:00.770 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.056187599999999997, 2024-09-21T22:08:00.770 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:00.770 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:00.770 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:00.771 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:00.771 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.771 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:00.771 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.218913+0000", 2024-09-21T22:08:00.771 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:00.771 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.771 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.771 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:00.771 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.218913+0000", 2024-09-21T22:08:00.771 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:00.771 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.771 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.771 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:00.772 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.218914+0000", 2024-09-21T22:08:00.772 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.1930000000000001e-06 2024-09-21T22:08:00.772 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.772 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.772 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:00.772 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.219721+0000", 2024-09-21T22:08:00.772 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00080701500000000001 2024-09-21T22:08:00.772 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.772 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.772 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:00.772 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.219743+0000", 2024-09-21T22:08:00.772 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1364000000000001e-05 2024-09-21T22:08:00.772 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.772 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.773 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:00.773 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.219759+0000", 2024-09-21T22:08:00.773 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.611e-05 2024-09-21T22:08:00.773 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.773 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.773 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:00.773 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.219908+0000", 2024-09-21T22:08:00.773 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000149252 2024-09-21T22:08:00.773 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.773 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.773 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:00.773 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.219951+0000", 2024-09-21T22:08:00.773 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.2518999999999997e-05 2024-09-21T22:08:00.773 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.774 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.774 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:00.774 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.275031+0000", 2024-09-21T22:08:00.774 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.055080241000000002 2024-09-21T22:08:00.774 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.774 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.774 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:00.774 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.275101+0000", 2024-09-21T22:08:00.774 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.9906000000000005e-05 2024-09-21T22:08:00.774 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:00.774 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:00.774 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:00.774 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.774 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.774 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:698 2.5 e63/37 2:b1d55633:::rbd_data.126483d75f2d.000000000000000b:head v 63'30, mlcod=63'28)", 2024-09-21T22:08:00.775 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.289319+0000", 2024-09-21T22:08:00.775 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 35.472545523000001, 2024-09-21T22:08:00.775 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.053930944000000001, 2024-09-21T22:08:00.775 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:00.775 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:00.775 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:00.775 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:00.775 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.775 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:00.775 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.289319+0000", 2024-09-21T22:08:00.775 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:00.775 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.775 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.775 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:00.776 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.289319+0000", 2024-09-21T22:08:00.776 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:00.776 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.776 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.776 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:00.776 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.289321+0000", 2024-09-21T22:08:00.776 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.221e-06 2024-09-21T22:08:00.776 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.776 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.776 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:00.776 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.290091+0000", 2024-09-21T22:08:00.776 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00077069300000000003 2024-09-21T22:08:00.776 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.777 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.777 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:00.777 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.290109+0000", 2024-09-21T22:08:00.777 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7218999999999999e-05 2024-09-21T22:08:00.777 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.777 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.777 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:00.777 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.290122+0000", 2024-09-21T22:08:00.777 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3678e-05 2024-09-21T22:08:00.777 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.777 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.777 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:00.777 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.290274+0000", 2024-09-21T22:08:00.777 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00015205899999999999 2024-09-21T22:08:00.777 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.778 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.778 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:00.778 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.290313+0000", 2024-09-21T22:08:00.778 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.8636000000000001e-05 2024-09-21T22:08:00.778 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.778 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.778 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:00.778 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.343179+0000", 2024-09-21T22:08:00.778 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.052866199000000003 2024-09-21T22:08:00.778 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.778 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.778 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:00.778 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.343250+0000", 2024-09-21T22:08:00.778 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.1239e-05 2024-09-21T22:08:00.779 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:00.779 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:00.779 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:00.779 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.779 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.779 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:702 2.5 e63/37 2:a70ae99f:::rbd_data.126483d75f2d.000000000000000c:head v 63'34, mlcod=63'32)", 2024-09-21T22:08:00.779 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.392532+0000", 2024-09-21T22:08:00.779 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 35.369332935000003, 2024-09-21T22:08:00.779 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.035703990999999997, 2024-09-21T22:08:00.779 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:00.779 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:00.779 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:00.779 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:00.779 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.779 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:00.780 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.392532+0000", 2024-09-21T22:08:00.780 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:00.780 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.780 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.780 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:00.780 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.392532+0000", 2024-09-21T22:08:00.780 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:00.780 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.780 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.780 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:00.780 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.392534+0000", 2024-09-21T22:08:00.780 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.64e-06 2024-09-21T22:08:00.780 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.780 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.781 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:00.781 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.393321+0000", 2024-09-21T22:08:00.781 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00078705699999999997 2024-09-21T22:08:00.781 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.781 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.781 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:00.781 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.393347+0000", 2024-09-21T22:08:00.781 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.6735999999999999e-05 2024-09-21T22:08:00.781 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.781 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.781 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:00.781 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.393361+0000", 2024-09-21T22:08:00.781 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3346000000000001e-05 2024-09-21T22:08:00.781 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.781 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.781 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:00.782 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.393469+0000", 2024-09-21T22:08:00.782 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000108039 2024-09-21T22:08:00.782 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.782 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.782 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:00.782 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.393512+0000", 2024-09-21T22:08:00.782 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.2862999999999999e-05 2024-09-21T22:08:00.782 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.782 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.782 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:00.782 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.428170+0000", 2024-09-21T22:08:00.782 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.034658858000000001 2024-09-21T22:08:00.782 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.783 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.783 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:00.783 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.428236+0000", 2024-09-21T22:08:00.783 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.5451999999999996e-05 2024-09-21T22:08:00.783 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:00.783 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:00.783 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:00.783 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.783 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.783 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:708 2.1 e63/53 2:837ddd83:::rbd_data.126483d75f2d.000000000000000d:head v 63'6, mlcod=57'4)", 2024-09-21T22:08:00.783 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.517007+0000", 2024-09-21T22:08:00.783 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 35.244858008999998, 2024-09-21T22:08:00.783 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.032495584000000001, 2024-09-21T22:08:00.783 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:00.783 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:00.784 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:00.784 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:00.784 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.784 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:00.784 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.517007+0000", 2024-09-21T22:08:00.784 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:00.784 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.784 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.784 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:00.784 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.517007+0000", 2024-09-21T22:08:00.784 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:00.784 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.784 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.784 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:00.784 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.517008+0000", 2024-09-21T22:08:00.785 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.604e-06 2024-09-21T22:08:00.785 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.785 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.785 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:00.785 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.517603+0000", 2024-09-21T22:08:00.785 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00059461900000000001 2024-09-21T22:08:00.785 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.785 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.785 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:00.785 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.517632+0000", 2024-09-21T22:08:00.785 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8663e-05 2024-09-21T22:08:00.785 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.785 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.785 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:00.786 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.517647+0000", 2024-09-21T22:08:00.786 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4833e-05 2024-09-21T22:08:00.786 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.786 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.786 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:00.786 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.517714+0000", 2024-09-21T22:08:00.786 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.7669000000000006e-05 2024-09-21T22:08:00.786 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.786 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.786 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:00.786 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.517737+0000", 2024-09-21T22:08:00.786 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.2469000000000001e-05 2024-09-21T22:08:00.786 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.786 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.787 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:00.787 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.549370+0000", 2024-09-21T22:08:00.787 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.031633221000000003 2024-09-21T22:08:00.787 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.787 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.787 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:00.787 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.549502+0000", 2024-09-21T22:08:00.787 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013250600000000001 2024-09-21T22:08:00.787 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:00.787 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:00.787 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:00.787 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.787 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.788 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:712 2.1 e63/53 2:837ddd83:::rbd_data.126483d75f2d.000000000000000d:head v 63'10, mlcod=63'8)", 2024-09-21T22:08:00.788 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.604029+0000", 2024-09-21T22:08:00.788 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 35.157836070000002, 2024-09-21T22:08:00.788 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.037982371000000001, 2024-09-21T22:08:00.788 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:00.788 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:00.788 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:00.788 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:00.788 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.788 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:00.788 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.604029+0000", 2024-09-21T22:08:00.788 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:00.788 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.788 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.789 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:00.789 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.604029+0000", 2024-09-21T22:08:00.789 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:00.789 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.789 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.789 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:00.789 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.604031+0000", 2024-09-21T22:08:00.789 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8870000000000001e-06 2024-09-21T22:08:00.789 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.789 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.789 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:00.789 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.604395+0000", 2024-09-21T22:08:00.789 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00036422000000000001 2024-09-21T22:08:00.789 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.790 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.790 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:00.790 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.604424+0000", 2024-09-21T22:08:00.790 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8609e-05 2024-09-21T22:08:00.790 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.790 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.790 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:00.790 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.604437+0000", 2024-09-21T22:08:00.790 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3086e-05 2024-09-21T22:08:00.790 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.790 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.790 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:00.790 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.604509+0000", 2024-09-21T22:08:00.790 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.2448999999999997e-05 2024-09-21T22:08:00.791 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.791 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.791 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:00.791 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.604531+0000", 2024-09-21T22:08:00.791 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1602000000000001e-05 2024-09-21T22:08:00.791 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.791 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.791 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:00.791 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.641885+0000", 2024-09-21T22:08:00.791 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.037354545000000003 2024-09-21T22:08:00.791 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.791 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.791 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:00.791 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.642011+0000", 2024-09-21T22:08:00.792 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00012597300000000001 2024-09-21T22:08:00.792 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:00.792 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:00.792 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:00.792 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.792 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.792 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:720 2.1 e63/53 2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head v 63'18, mlcod=63'16)", 2024-09-21T22:08:00.792 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.805111+0000", 2024-09-21T22:08:00.792 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 34.956754009999997, 2024-09-21T22:08:00.792 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.046082864000000001, 2024-09-21T22:08:00.792 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:00.792 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:00.792 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:00.792 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:00.793 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.793 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:00.793 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.805111+0000", 2024-09-21T22:08:00.793 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:00.793 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.793 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.793 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:00.793 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.805111+0000", 2024-09-21T22:08:00.793 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:00.793 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.793 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.793 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:00.793 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.805112+0000", 2024-09-21T22:08:00.793 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.02e-06 2024-09-21T22:08:00.793 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.794 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.794 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:00.794 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.805429+0000", 2024-09-21T22:08:00.794 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00031740799999999998 2024-09-21T22:08:00.794 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.794 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.794 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:00.794 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.805459+0000", 2024-09-21T22:08:00.794 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.955e-05 2024-09-21T22:08:00.794 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.794 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.794 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:00.794 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.805472+0000", 2024-09-21T22:08:00.794 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3533000000000001e-05 2024-09-21T22:08:00.795 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.795 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.795 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:00.795 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.805581+0000", 2024-09-21T22:08:00.795 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000108194 2024-09-21T22:08:00.795 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.795 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.795 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:00.795 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.805615+0000", 2024-09-21T22:08:00.795 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.4310000000000002e-05 2024-09-21T22:08:00.795 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.795 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.795 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:00.795 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.851146+0000", 2024-09-21T22:08:00.796 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.045531092000000002 2024-09-21T22:08:00.796 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.796 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.796 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:00.796 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.851194+0000", 2024-09-21T22:08:00.796 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.7756999999999997e-05 2024-09-21T22:08:00.796 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:00.796 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:00.796 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:00.796 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.796 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.796 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:723 2.1 e63/53 2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head v 63'20, mlcod=63'18)", 2024-09-21T22:08:00.796 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.862341+0000", 2024-09-21T22:08:00.796 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 34.899524075999999, 2024-09-21T22:08:00.796 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.036454671000000001, 2024-09-21T22:08:00.797 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:00.797 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:00.797 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:00.797 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:00.797 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.797 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:00.797 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.862341+0000", 2024-09-21T22:08:00.797 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:00.797 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.797 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.797 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:00.797 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.862341+0000", 2024-09-21T22:08:00.797 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:00.797 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.797 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.798 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:00.798 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.862342+0000", 2024-09-21T22:08:00.798 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.0299999999999997e-07 2024-09-21T22:08:00.798 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.798 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.798 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:00.798 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.862721+0000", 2024-09-21T22:08:00.798 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00037889800000000002 2024-09-21T22:08:00.798 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.798 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.798 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:00.798 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.862738+0000", 2024-09-21T22:08:00.798 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6957999999999999e-05 2024-09-21T22:08:00.798 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.799 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.799 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:00.799 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.862747+0000", 2024-09-21T22:08:00.799 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.0729999999999994e-06 2024-09-21T22:08:00.799 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.799 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.799 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:00.799 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.862830+0000", 2024-09-21T22:08:00.799 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.3113000000000002e-05 2024-09-21T22:08:00.799 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.799 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.799 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:00.799 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.862860+0000", 2024-09-21T22:08:00.799 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9980000000000001e-05 2024-09-21T22:08:00.799 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.800 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.800 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:00.800 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.898705+0000", 2024-09-21T22:08:00.800 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.035845063000000003 2024-09-21T22:08:00.800 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.800 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.800 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:00.800 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.898795+0000", 2024-09-21T22:08:00.800 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.0682999999999999e-05 2024-09-21T22:08:00.800 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:00.800 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:00.800 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:00.800 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.800 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.800 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:136 2.5 e72/37 2:b1d55633:::rbd_data.126483d75f2d.000000000000000b:head v 72'44, mlcod=72'42)", 2024-09-21T22:08:00.801 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.386273+0000", 2024-09-21T22:08:00.801 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 22.375591961000001, 2024-09-21T22:08:00.801 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.033806744999999999, 2024-09-21T22:08:00.801 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:00.801 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:00.801 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:00.801 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:00.801 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.801 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:00.801 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.386273+0000", 2024-09-21T22:08:00.801 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:00.801 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.801 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.801 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:00.801 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.386273+0000", 2024-09-21T22:08:00.802 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:00.802 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.802 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.802 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:00.802 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.386274+0000", 2024-09-21T22:08:00.802 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.8299999999999996e-07 2024-09-21T22:08:00.802 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.802 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.802 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:00.802 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.387108+0000", 2024-09-21T22:08:00.802 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00083467800000000005 2024-09-21T22:08:00.802 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.802 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.803 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:00.803 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.387121+0000", 2024-09-21T22:08:00.803 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2823000000000001e-05 2024-09-21T22:08:00.803 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.803 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.803 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:00.803 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.387128+0000", 2024-09-21T22:08:00.803 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.404e-06 2024-09-21T22:08:00.803 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.803 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.803 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:00.803 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.387179+0000", 2024-09-21T22:08:00.803 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.1888000000000002e-05 2024-09-21T22:08:00.803 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.804 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.804 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:00.804 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.387201+0000", 2024-09-21T22:08:00.804 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.2019999999999999e-05 2024-09-21T22:08:00.804 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.804 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.804 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:00.804 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.419993+0000", 2024-09-21T22:08:00.804 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.032791182000000002 2024-09-21T22:08:00.804 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.804 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.804 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:00.804 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.420080+0000", 2024-09-21T22:08:00.804 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.7066999999999998e-05 2024-09-21T22:08:00.805 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:00.805 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:00.805 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:00.805 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.805 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.805 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:138 2.5 e72/37 2:b1d55633:::rbd_data.126483d75f2d.000000000000000b:head v 72'46, mlcod=72'44)", 2024-09-21T22:08:00.805 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.430104+0000", 2024-09-21T22:08:00.805 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 22.331760613, 2024-09-21T22:08:00.805 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.034087458000000001, 2024-09-21T22:08:00.805 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:00.805 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:00.805 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:00.805 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:00.805 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.806 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:00.806 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.430104+0000", 2024-09-21T22:08:00.806 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:00.806 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.806 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.806 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:00.806 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.430104+0000", 2024-09-21T22:08:00.806 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:00.806 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.806 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.806 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:00.806 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.430105+0000", 2024-09-21T22:08:00.806 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.1680000000000001e-06 2024-09-21T22:08:00.806 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.806 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.807 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:00.807 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.430864+0000", 2024-09-21T22:08:00.807 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00075819400000000001 2024-09-21T22:08:00.807 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.807 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.807 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:00.807 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.430881+0000", 2024-09-21T22:08:00.807 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7000999999999999e-05 2024-09-21T22:08:00.807 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.807 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.807 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:00.807 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.430889+0000", 2024-09-21T22:08:00.807 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.9540000000000004e-06 2024-09-21T22:08:00.807 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.808 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.808 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:00.808 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.430959+0000", 2024-09-21T22:08:00.808 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.0858999999999994e-05 2024-09-21T22:08:00.808 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.808 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.808 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:00.808 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.430996+0000", 2024-09-21T22:08:00.808 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.6443999999999999e-05 2024-09-21T22:08:00.808 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.808 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.808 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:00.808 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.464087+0000", 2024-09-21T22:08:00.808 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.033090799999999997 2024-09-21T22:08:00.808 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.809 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.809 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:00.809 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.464192+0000", 2024-09-21T22:08:00.809 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00010503800000000001 2024-09-21T22:08:00.809 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:00.809 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:00.809 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:00.809 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.809 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.809 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:140 2.5 e72/37 2:b1d55633:::rbd_data.126483d75f2d.000000000000000b:head v 72'48, mlcod=72'46)", 2024-09-21T22:08:00.809 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.475737+0000", 2024-09-21T22:08:00.809 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 22.286127664999999, 2024-09-21T22:08:00.809 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.037632025, 2024-09-21T22:08:00.809 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:00.810 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:00.810 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:00.810 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:00.810 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.810 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:00.810 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.475737+0000", 2024-09-21T22:08:00.810 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:00.810 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.810 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.810 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:00.810 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.475737+0000", 2024-09-21T22:08:00.810 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:00.811 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.811 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.811 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:00.811 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.475739+0000", 2024-09-21T22:08:00.811 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.666e-06 2024-09-21T22:08:00.811 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.811 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.811 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:00.811 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.476538+0000", 2024-09-21T22:08:00.811 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00079927399999999999 2024-09-21T22:08:00.811 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.811 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.811 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:00.811 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.476556+0000", 2024-09-21T22:08:00.811 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7666999999999999e-05 2024-09-21T22:08:00.812 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.812 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.812 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:00.812 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.476565+0000", 2024-09-21T22:08:00.812 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.0650000000000005e-06 2024-09-21T22:08:00.812 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.812 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.812 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:00.812 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.476651+0000", 2024-09-21T22:08:00.812 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.5916999999999997e-05 2024-09-21T22:08:00.812 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.812 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.812 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:00.813 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.476696+0000", 2024-09-21T22:08:00.813 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.5537000000000001e-05 2024-09-21T22:08:00.813 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.813 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.813 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:00.813 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.513300+0000", 2024-09-21T22:08:00.813 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.036603498999999998 2024-09-21T22:08:00.813 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.813 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.813 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:00.813 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.513369+0000", 2024-09-21T22:08:00.813 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.9400000000000006e-05 2024-09-21T22:08:00.813 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:00.814 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:00.814 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:00.814 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.814 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.814 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:145 2.5 e72/37 2:a70ae99f:::rbd_data.126483d75f2d.000000000000000c:head v 72'52, mlcod=72'50)", 2024-09-21T22:08:00.814 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.578171+0000", 2024-09-21T22:08:00.814 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 22.183693815000002, 2024-09-21T22:08:00.814 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.037181537000000001, 2024-09-21T22:08:00.814 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:00.814 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:00.814 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:00.814 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:00.814 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.814 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:00.815 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.578171+0000", 2024-09-21T22:08:00.815 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:00.815 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.815 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.815 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:00.815 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.578171+0000", 2024-09-21T22:08:00.815 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:00.815 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.815 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.815 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:00.815 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.578172+0000", 2024-09-21T22:08:00.815 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.017e-06 2024-09-21T22:08:00.815 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.815 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.815 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:00.816 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.579000+0000", 2024-09-21T22:08:00.816 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00082799499999999999 2024-09-21T22:08:00.816 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.816 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.816 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:00.816 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.579020+0000", 2024-09-21T22:08:00.816 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.0165000000000001e-05 2024-09-21T22:08:00.816 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.816 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.816 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:00.816 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.579036+0000", 2024-09-21T22:08:00.816 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5767e-05 2024-09-21T22:08:00.816 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.816 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.816 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:00.817 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.579106+0000", 2024-09-21T22:08:00.817 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.0091000000000004e-05 2024-09-21T22:08:00.817 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.817 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.817 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:00.817 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.579131+0000", 2024-09-21T22:08:00.817 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4978e-05 2024-09-21T22:08:00.817 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.817 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.817 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:00.817 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.615249+0000", 2024-09-21T22:08:00.817 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.036118355999999997 2024-09-21T22:08:00.817 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.817 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.818 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:00.818 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.615353+0000", 2024-09-21T22:08:00.818 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000103168 2024-09-21T22:08:00.818 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:00.818 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:00.818 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:00.818 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.818 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.818 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:147 2.5 e72/37 2:a70ae99f:::rbd_data.126483d75f2d.000000000000000c:head v 72'54, mlcod=72'52)", 2024-09-21T22:08:00.818 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.627203+0000", 2024-09-21T22:08:00.818 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 22.134661702999999, 2024-09-21T22:08:00.818 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.038203972000000003, 2024-09-21T22:08:00.818 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:00.819 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:00.819 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:00.819 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:00.819 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.819 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:00.819 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.627203+0000", 2024-09-21T22:08:00.819 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:00.819 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.819 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.819 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:00.819 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.627203+0000", 2024-09-21T22:08:00.819 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:00.819 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.819 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.820 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:00.820 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.627204+0000", 2024-09-21T22:08:00.820 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3069999999999999e-06 2024-09-21T22:08:00.820 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.820 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.820 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:00.820 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.628114+0000", 2024-09-21T22:08:00.820 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00090942099999999999 2024-09-21T22:08:00.820 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.820 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.820 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:00.820 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.628140+0000", 2024-09-21T22:08:00.820 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5882000000000001e-05 2024-09-21T22:08:00.820 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.821 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.821 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:00.821 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.628162+0000", 2024-09-21T22:08:00.821 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.2293000000000002e-05 2024-09-21T22:08:00.821 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.821 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.821 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:00.821 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.628250+0000", 2024-09-21T22:08:00.821 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.7973000000000006e-05 2024-09-21T22:08:00.821 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.821 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.821 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:00.821 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.628272+0000", 2024-09-21T22:08:00.821 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1682999999999999e-05 2024-09-21T22:08:00.822 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.822 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.822 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:00.822 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.665322+0000", 2024-09-21T22:08:00.822 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.037050650999999997 2024-09-21T22:08:00.822 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.822 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.822 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:00.822 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.665407+0000", 2024-09-21T22:08:00.822 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.4761999999999999e-05 2024-09-21T22:08:00.822 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:00.822 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:00.822 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:00.823 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.823 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.823 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:149 2.5 e72/37 2:a70ae99f:::rbd_data.126483d75f2d.000000000000000c:head v 72'56, mlcod=72'54)", 2024-09-21T22:08:00.823 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.690377+0000", 2024-09-21T22:08:00.823 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 22.071487959999999, 2024-09-21T22:08:00.823 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.049254799000000002, 2024-09-21T22:08:00.823 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:00.823 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:00.823 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:00.823 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:00.823 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.823 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:00.823 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.690377+0000", 2024-09-21T22:08:00.823 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:00.823 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.824 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.824 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:00.824 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.690377+0000", 2024-09-21T22:08:00.824 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:00.824 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.824 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.824 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:00.824 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.690378+0000", 2024-09-21T22:08:00.824 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.344e-06 2024-09-21T22:08:00.824 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.824 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.824 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:00.824 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.691302+0000", 2024-09-21T22:08:00.824 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00092322400000000005 2024-09-21T22:08:00.825 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.825 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.825 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:00.825 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.691321+0000", 2024-09-21T22:08:00.825 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9928999999999998e-05 2024-09-21T22:08:00.825 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.825 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.825 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:00.825 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.691345+0000", 2024-09-21T22:08:00.825 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.372e-05 2024-09-21T22:08:00.825 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.825 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.825 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:00.825 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.691442+0000", 2024-09-21T22:08:00.826 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.7270999999999998e-05 2024-09-21T22:08:00.826 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.826 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.826 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:00.826 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.691512+0000", 2024-09-21T22:08:00.826 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.9183000000000005e-05 2024-09-21T22:08:00.826 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.826 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.826 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:00.826 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.739545+0000", 2024-09-21T22:08:00.826 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.048033657 2024-09-21T22:08:00.826 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.826 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.826 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:00.827 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.739632+0000", 2024-09-21T22:08:00.827 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.6471000000000006e-05 2024-09-21T22:08:00.827 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:00.827 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:00.827 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:00.827 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.827 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.827 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:156 2.1 e72/53 2:837ddd83:::rbd_data.126483d75f2d.000000000000000d:head v 72'28, mlcod=72'26)", 2024-09-21T22:08:00.827 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.824856+0000", 2024-09-21T22:08:00.827 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 21.937008521999999, 2024-09-21T22:08:00.827 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.034320985999999998, 2024-09-21T22:08:00.827 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:00.827 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:00.828 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:00.828 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:00.828 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.828 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:00.828 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.824856+0000", 2024-09-21T22:08:00.828 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:00.828 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.828 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.828 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:00.828 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.824856+0000", 2024-09-21T22:08:00.828 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:00.828 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.828 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.828 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:00.829 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.824858+0000", 2024-09-21T22:08:00.829 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2979999999999999e-06 2024-09-21T22:08:00.829 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.829 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.829 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:00.829 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.825174+0000", 2024-09-21T22:08:00.829 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000316335 2024-09-21T22:08:00.829 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.829 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.829 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:00.829 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.825193+0000", 2024-09-21T22:08:00.829 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9436000000000002e-05 2024-09-21T22:08:00.829 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.829 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.829 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:00.830 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.825202+0000", 2024-09-21T22:08:00.830 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.8359999999999998e-06 2024-09-21T22:08:00.830 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.830 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.830 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:00.830 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.825286+0000", 2024-09-21T22:08:00.830 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.3683999999999994e-05 2024-09-21T22:08:00.830 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.830 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.830 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:00.830 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.825314+0000", 2024-09-21T22:08:00.830 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.7653000000000001e-05 2024-09-21T22:08:00.830 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.830 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.830 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:00.831 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.859122+0000", 2024-09-21T22:08:00.831 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.033808624000000002 2024-09-21T22:08:00.831 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.831 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.831 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:00.831 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.859177+0000", 2024-09-21T22:08:00.831 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.5120000000000001e-05 2024-09-21T22:08:00.831 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:00.831 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:00.831 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:00.831 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.831 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.831 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:158 2.1 e72/53 2:837ddd83:::rbd_data.126483d75f2d.000000000000000d:head v 72'30, mlcod=72'28)", 2024-09-21T22:08:00.831 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.869277+0000", 2024-09-21T22:08:00.832 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 21.892588324999998, 2024-09-21T22:08:00.832 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.040508390999999998, 2024-09-21T22:08:00.832 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:00.832 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:00.832 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:00.832 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:00.832 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.832 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:00.832 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.869277+0000", 2024-09-21T22:08:00.832 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:00.832 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.832 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.832 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:00.832 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.869277+0000", 2024-09-21T22:08:00.832 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:00.833 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.833 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.833 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:00.833 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.869278+0000", 2024-09-21T22:08:00.833 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5269999999999999e-06 2024-09-21T22:08:00.833 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.833 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.833 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:00.833 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.869902+0000", 2024-09-21T22:08:00.833 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00062399000000000005 2024-09-21T22:08:00.833 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.833 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.833 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:00.833 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.869924+0000", 2024-09-21T22:08:00.834 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1875999999999999e-05 2024-09-21T22:08:00.834 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.834 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.834 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:00.834 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.869937+0000", 2024-09-21T22:08:00.834 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2685e-05 2024-09-21T22:08:00.834 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.834 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.834 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:00.834 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.870058+0000", 2024-09-21T22:08:00.834 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000120862 2024-09-21T22:08:00.834 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.834 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.834 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:00.834 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.870093+0000", 2024-09-21T22:08:00.835 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.5896000000000003e-05 2024-09-21T22:08:00.835 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.835 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.835 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:00.835 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.909728+0000", 2024-09-21T22:08:00.835 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.039634461000000003 2024-09-21T22:08:00.835 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.835 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.835 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:00.835 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.909785+0000", 2024-09-21T22:08:00.835 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.7093999999999999e-05 2024-09-21T22:08:00.835 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:00.835 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:00.836 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:00.836 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.836 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.836 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:161 2.1 e72/53 2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head v 72'32, mlcod=72'30)", 2024-09-21T22:08:00.836 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.924455+0000", 2024-09-21T22:08:00.836 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 21.837409962999999, 2024-09-21T22:08:00.836 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.038366863000000001, 2024-09-21T22:08:00.836 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:00.836 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:00.836 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:00.836 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:00.836 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.836 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:00.836 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.924455+0000", 2024-09-21T22:08:00.836 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:00.837 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.837 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.837 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:00.837 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.924455+0000", 2024-09-21T22:08:00.837 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:00.837 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.837 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.837 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:00.837 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.924456+0000", 2024-09-21T22:08:00.837 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.003e-06 2024-09-21T22:08:00.837 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.837 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.837 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:00.837 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.924811+0000", 2024-09-21T22:08:00.837 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00035519499999999999 2024-09-21T22:08:00.838 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.838 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.838 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:00.838 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.924826+0000", 2024-09-21T22:08:00.838 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5296999999999999e-05 2024-09-21T22:08:00.838 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.838 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.838 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:00.838 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.924834+0000", 2024-09-21T22:08:00.838 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.5379999999999997e-06 2024-09-21T22:08:00.838 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.838 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.838 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:00.838 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.924910+0000", 2024-09-21T22:08:00.839 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.6488000000000004e-05 2024-09-21T22:08:00.839 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.839 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.839 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:00.839 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.924934+0000", 2024-09-21T22:08:00.839 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3598999999999999e-05 2024-09-21T22:08:00.839 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.839 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.839 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:00.839 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.962739+0000", 2024-09-21T22:08:00.839 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.037805370999999997 2024-09-21T22:08:00.839 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.839 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.839 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:00.839 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.962822+0000", 2024-09-21T22:08:00.840 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.2372000000000003e-05 2024-09-21T22:08:00.840 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:00.840 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:00.840 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:00.840 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.840 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.840 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:163 2.1 e72/53 2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head v 72'34, mlcod=72'32)", 2024-09-21T22:08:00.840 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.971683+0000", 2024-09-21T22:08:00.840 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 21.790181423, 2024-09-21T22:08:00.840 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.043171320999999999, 2024-09-21T22:08:00.840 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:00.840 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:00.840 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:00.840 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:00.840 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.841 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:00.841 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.971683+0000", 2024-09-21T22:08:00.841 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:00.841 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.841 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.841 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:00.841 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.971683+0000", 2024-09-21T22:08:00.841 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:00.841 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.841 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.841 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:00.841 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.971684+0000", 2024-09-21T22:08:00.841 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.6600000000000005e-07 2024-09-21T22:08:00.841 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.841 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.842 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:00.842 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.971945+0000", 2024-09-21T22:08:00.842 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00026101899999999998 2024-09-21T22:08:00.842 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.842 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.842 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:00.842 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.971959+0000", 2024-09-21T22:08:00.842 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4054e-05 2024-09-21T22:08:00.842 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.842 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.842 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:00.842 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.971969+0000", 2024-09-21T22:08:00.842 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.499e-06 2024-09-21T22:08:00.842 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.843 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.843 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:00.843 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.972037+0000", 2024-09-21T22:08:00.843 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.8448000000000006e-05 2024-09-21T22:08:00.843 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.843 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.843 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:00.843 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.972057+0000", 2024-09-21T22:08:00.843 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9817999999999999e-05 2024-09-21T22:08:00.843 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.843 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.843 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:00.843 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.014770+0000", 2024-09-21T22:08:00.843 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.042712464999999998 2024-09-21T22:08:00.843 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.844 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.844 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:00.844 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.014855+0000", 2024-09-21T22:08:00.844 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.5152000000000001e-05 2024-09-21T22:08:00.844 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:00.844 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:00.844 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:00.844 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.844 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.844 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:165 2.1 e72/53 2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head v 72'36, mlcod=72'34)", 2024-09-21T22:08:00.844 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:39.028631+0000", 2024-09-21T22:08:00.844 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 21.733233945999999, 2024-09-21T22:08:00.844 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.033430561999999997, 2024-09-21T22:08:00.844 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:00.845 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:00.845 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:00.845 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:00.845 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.845 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:00.845 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.028631+0000", 2024-09-21T22:08:00.845 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:00.845 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.845 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.845 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:00.845 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.028631+0000", 2024-09-21T22:08:00.845 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:00.845 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.845 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.845 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:00.845 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.028633+0000", 2024-09-21T22:08:00.846 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5799999999999999e-06 2024-09-21T22:08:00.846 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.846 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.846 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:00.846 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.029209+0000", 2024-09-21T22:08:00.846 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00057670799999999999 2024-09-21T22:08:00.846 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.846 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.846 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:00.846 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.029238+0000", 2024-09-21T22:08:00.846 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8943000000000001e-05 2024-09-21T22:08:00.846 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.846 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.846 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:00.846 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.029252+0000", 2024-09-21T22:08:00.847 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3672999999999999e-05 2024-09-21T22:08:00.847 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.847 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.847 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:00.847 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.029328+0000", 2024-09-21T22:08:00.847 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.6497999999999998e-05 2024-09-21T22:08:00.847 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.847 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.847 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:00.847 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.029353+0000", 2024-09-21T22:08:00.847 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5060999999999999e-05 2024-09-21T22:08:00.847 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.847 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.847 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:00.847 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.061993+0000", 2024-09-21T22:08:00.848 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.032639286000000003 2024-09-21T22:08:00.848 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.848 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.848 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:00.848 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.062062+0000", 2024-09-21T22:08:00.848 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.8813000000000007e-05 2024-09-21T22:08:00.848 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:00.848 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:00.848 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:00.848 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.848 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.848 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:167 2.1 e72/53 2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head v 72'38, mlcod=72'36)", 2024-09-21T22:08:00.848 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:39.077324+0000", 2024-09-21T22:08:00.848 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 21.684541132, 2024-09-21T22:08:00.848 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.036031961000000001, 2024-09-21T22:08:00.849 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:00.849 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:00.849 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:00.849 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:00.849 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.849 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:00.849 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.077324+0000", 2024-09-21T22:08:00.849 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:00.849 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.849 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.849 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:00.849 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.077324+0000", 2024-09-21T22:08:00.849 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:00.849 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.850 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.850 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:00.850 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.077326+0000", 2024-09-21T22:08:00.850 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.75e-06 2024-09-21T22:08:00.850 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.850 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.850 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:00.850 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.077719+0000", 2024-09-21T22:08:00.850 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00039322599999999999 2024-09-21T22:08:00.850 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.850 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.850 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:00.851 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.077749+0000", 2024-09-21T22:08:00.851 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9895000000000001e-05 2024-09-21T22:08:00.851 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.851 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.851 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:00.851 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.077763+0000", 2024-09-21T22:08:00.851 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4517e-05 2024-09-21T22:08:00.851 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.851 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.851 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:00.851 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.077843+0000", 2024-09-21T22:08:00.851 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.9994999999999996e-05 2024-09-21T22:08:00.851 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.851 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.851 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:00.852 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.077867+0000", 2024-09-21T22:08:00.852 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3640000000000001e-05 2024-09-21T22:08:00.852 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.852 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.852 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:00.852 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.113216+0000", 2024-09-21T22:08:00.852 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.035349168 2024-09-21T22:08:00.852 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:00.852 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:00.852 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:00.852 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.113356+0000", 2024-09-21T22:08:00.852 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013977 2024-09-21T22:08:00.852 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:00.852 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:00.852 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:00.853 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:00.853 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:00.853 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:08:00.855 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T22:08:00.855 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:08:00.849+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:08:00.955 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:08:00.952+0000 7f2075a06640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31720) UID: 0 2024-09-21T22:08:00.970 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-21T22:08:01.011 INFO:tasks.workunit.client.0.smithi157.stderr: Creating snap: 10% complete... Creating snap: 20% complete... Creating snap: 100% complete...done. 2024-09-21T22:08:01.022 INFO:tasks.workunit.client.0.smithi157.stderr:+ _sudo dd if=/tmp/tmp.GqKuv7rsZi/data of=/dev/nbd1 bs=1M count=1 oflag=direct 2024-09-21T22:08:01.022 INFO:tasks.workunit.client.0.smithi157.stderr:+ local cmd 2024-09-21T22:08:01.022 INFO:tasks.workunit.client.0.smithi157.stderr:++ id -u 2024-09-21T22:08:01.024 INFO:tasks.workunit.client.0.smithi157.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T22:08:01.025 INFO:tasks.workunit.client.0.smithi157.stderr:++ which dd 2024-09-21T22:08:01.027 INFO:tasks.workunit.client.0.smithi157.stderr:+ cmd=/usr/bin/dd 2024-09-21T22:08:01.027 INFO:tasks.workunit.client.0.smithi157.stderr:+ shift 2024-09-21T22:08:01.027 INFO:tasks.workunit.client.0.smithi157.stderr:+ sudo -nE /usr/bin/dd if=/tmp/tmp.GqKuv7rsZi/data of=/dev/nbd1 bs=1M count=1 oflag=direct 2024-09-21T22:08:01.030 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:08:01.030 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:08:01.030 INFO:teuthology.orchestra.run.smithi039.stdout: "num_ops": 0 2024-09-21T22:08:01.030 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:08:01.046 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T22:08:01.096 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:08:01.093+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:08:01.118 INFO:tasks.workunit.client.0.smithi157.stderr:1+0 records in 2024-09-21T22:08:01.118 INFO:tasks.workunit.client.0.smithi157.stderr:1+0 records out 2024-09-21T22:08:01.118 INFO:tasks.workunit.client.0.smithi157.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.083681 s, 12.5 MB/s 2024-09-21T22:08:01.120 INFO:tasks.workunit.client.0.smithi157.stderr:+ unmap_device /dev/nbd1 23672 2024-09-21T22:08:01.120 INFO:tasks.workunit.client.0.smithi157.stderr:+ local args=/dev/nbd1 2024-09-21T22:08:01.120 INFO:tasks.workunit.client.0.smithi157.stderr:+ local pid=23672 2024-09-21T22:08:01.120 INFO:tasks.workunit.client.0.smithi157.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd1 2024-09-21T22:08:01.120 INFO:tasks.workunit.client.0.smithi157.stderr:+ local cmd 2024-09-21T22:08:01.120 INFO:tasks.workunit.client.0.smithi157.stderr:++ id -u 2024-09-21T22:08:01.122 INFO:tasks.workunit.client.0.smithi157.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T22:08:01.122 INFO:tasks.workunit.client.0.smithi157.stderr:++ which rbd 2024-09-21T22:08:01.125 INFO:tasks.workunit.client.0.smithi157.stderr:+ cmd=/usr/bin/rbd 2024-09-21T22:08:01.125 INFO:tasks.workunit.client.0.smithi157.stderr:+ shift 2024-09-21T22:08:01.125 INFO:tasks.workunit.client.0.smithi157.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd1 2024-09-21T22:08:01.157 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:08:01.152+0000 7f2075a06640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31720) UID: 0 2024-09-21T22:08:01.165 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:08:01.165 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:08:01.165 INFO:teuthology.orchestra.run.smithi039.stdout: "complaint_time": 30, 2024-09-21T22:08:01.165 INFO:teuthology.orchestra.run.smithi039.stdout: "num_blocked_ops": 0 2024-09-21T22:08:01.165 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:08:01.180 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T22:08:01.257 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:08:01.252+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:08:01.351 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:08:01.351 INFO:teuthology.orchestra.run.smithi039.stdout: "size": 20, 2024-09-21T22:08:01.351 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 600, 2024-09-21T22:08:01.351 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [ 2024-09-21T22:08:01.351 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.351 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.3 55/54 [PushOp(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head, version: 52'2, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head@52'2, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T22:08:01.352 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.084379+0000", 2024-09-21T22:08:01.352 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 49.263036460000002, 2024-09-21T22:08:01.352 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.051020674000000002, 2024-09-21T22:08:01.352 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:01.352 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:01.352 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:08:01.352 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:01.352 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.352 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:01.352 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.084379+0000", 2024-09-21T22:08:01.352 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:01.352 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.352 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.353 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:01.353 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.084379+0000", 2024-09-21T22:08:01.353 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:01.353 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.353 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.353 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:01.353 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.084381+0000", 2024-09-21T22:08:01.353 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.73e-06 2024-09-21T22:08:01.353 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.353 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.353 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:01.353 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.085494+0000", 2024-09-21T22:08:01.353 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011131559999999999 2024-09-21T22:08:01.353 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.354 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.354 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:01.354 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.085542+0000", 2024-09-21T22:08:01.354 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8334999999999997e-05 2024-09-21T22:08:01.354 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.354 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.354 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:01.354 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.085591+0000", 2024-09-21T22:08:01.354 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.9165000000000001e-05 2024-09-21T22:08:01.354 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.354 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.354 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:01.354 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.085777+0000", 2024-09-21T22:08:01.354 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000186152 2024-09-21T22:08:01.355 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.355 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.355 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:01.355 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.085796+0000", 2024-09-21T22:08:01.355 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8612999999999999e-05 2024-09-21T22:08:01.355 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.355 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.355 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:01.355 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.135399+0000", 2024-09-21T22:08:01.355 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.049603522999999997 2024-09-21T22:08:01.355 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:01.355 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:01.355 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:01.356 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.356 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.356 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.3 55/54 [PushOp(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head, version: 52'2, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head@52'2, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:08:01.356 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.139554+0000", 2024-09-21T22:08:01.356 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 49.207861356999999, 2024-09-21T22:08:01.356 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.056215389999999997, 2024-09-21T22:08:01.356 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:01.356 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:01.356 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:08:01.356 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:01.356 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.356 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:01.357 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.139554+0000", 2024-09-21T22:08:01.357 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:01.357 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.357 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.357 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:01.357 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.139554+0000", 2024-09-21T22:08:01.357 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:01.357 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.357 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.357 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:01.357 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.139556+0000", 2024-09-21T22:08:01.357 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7999999999999999e-06 2024-09-21T22:08:01.357 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.357 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.358 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:01.358 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.140671+0000", 2024-09-21T22:08:01.358 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001115612 2024-09-21T22:08:01.358 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.358 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.358 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:01.358 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.140720+0000", 2024-09-21T22:08:01.358 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8631999999999998e-05 2024-09-21T22:08:01.358 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.358 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.358 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:01.358 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.140752+0000", 2024-09-21T22:08:01.358 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.2199000000000001e-05 2024-09-21T22:08:01.359 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.359 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.359 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:01.359 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.140900+0000", 2024-09-21T22:08:01.359 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00014786000000000001 2024-09-21T22:08:01.359 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.359 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.359 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:01.359 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.140917+0000", 2024-09-21T22:08:01.359 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6753999999999999e-05 2024-09-21T22:08:01.359 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.359 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.359 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:01.359 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.195769+0000", 2024-09-21T22:08:01.360 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.054852533000000002 2024-09-21T22:08:01.360 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:01.360 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:01.360 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:01.360 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.360 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.360 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.3 55/54 [PushOp(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head, version: 52'2, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head@52'2, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:08:01.360 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.199187+0000", 2024-09-21T22:08:01.360 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 49.148227812999998, 2024-09-21T22:08:01.360 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.051543975999999998, 2024-09-21T22:08:01.360 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:01.360 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:01.360 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:08:01.360 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:01.361 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.361 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:01.361 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.199187+0000", 2024-09-21T22:08:01.361 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:01.361 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.361 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.361 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:01.361 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.199187+0000", 2024-09-21T22:08:01.361 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:01.361 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.361 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.361 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:01.362 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.199189+0000", 2024-09-21T22:08:01.362 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7969999999999999e-06 2024-09-21T22:08:01.362 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.362 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.362 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:01.362 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.200355+0000", 2024-09-21T22:08:01.362 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011660139999999999 2024-09-21T22:08:01.362 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.363 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.363 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:01.363 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.200405+0000", 2024-09-21T22:08:01.363 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.0124000000000003e-05 2024-09-21T22:08:01.363 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.363 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.363 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:01.363 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.200457+0000", 2024-09-21T22:08:01.363 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.1177999999999997e-05 2024-09-21T22:08:01.363 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.363 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.363 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:01.363 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.200635+0000", 2024-09-21T22:08:01.363 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00017841299999999999 2024-09-21T22:08:01.364 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.364 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.364 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:01.364 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.200653+0000", 2024-09-21T22:08:01.364 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7944999999999998e-05 2024-09-21T22:08:01.364 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.364 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.364 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:01.364 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.250731+0000", 2024-09-21T22:08:01.364 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.050078505000000002 2024-09-21T22:08:01.364 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:01.364 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:01.364 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:01.364 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.365 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.365 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.3 55/54 [PushOp(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head, version: 52'3, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head@52'3, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:08:01.365 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.343754+0000", 2024-09-21T22:08:01.365 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 49.003661102000002, 2024-09-21T22:08:01.365 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.063560607000000005, 2024-09-21T22:08:01.365 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:01.365 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:01.365 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:08:01.365 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:01.365 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.365 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:01.365 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.343754+0000", 2024-09-21T22:08:01.365 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:01.365 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.365 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.366 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:01.366 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.343754+0000", 2024-09-21T22:08:01.366 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:01.366 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.366 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.366 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:01.366 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.343756+0000", 2024-09-21T22:08:01.366 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7069999999999999e-06 2024-09-21T22:08:01.366 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.366 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.366 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:01.366 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.344864+0000", 2024-09-21T22:08:01.366 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011085159999999999 2024-09-21T22:08:01.366 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.367 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.367 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:01.367 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.344914+0000", 2024-09-21T22:08:01.367 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.9462000000000002e-05 2024-09-21T22:08:01.367 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.367 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.367 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:01.367 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.344961+0000", 2024-09-21T22:08:01.367 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.7633999999999998e-05 2024-09-21T22:08:01.367 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.367 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.368 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:01.368 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.345171+0000", 2024-09-21T22:08:01.368 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000209652 2024-09-21T22:08:01.368 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.368 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.368 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:01.368 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.345193+0000", 2024-09-21T22:08:01.368 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1985999999999999e-05 2024-09-21T22:08:01.368 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.368 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.368 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:01.368 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.407315+0000", 2024-09-21T22:08:01.368 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.06212165 2024-09-21T22:08:01.368 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:01.368 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:01.369 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:01.369 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.369 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.369 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.3 55/54 [PushOp(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head, version: 52'3, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head@52'3, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:08:01.369 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.410551+0000", 2024-09-21T22:08:01.369 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 48.936863971000001, 2024-09-21T22:08:01.369 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.054517522999999998, 2024-09-21T22:08:01.369 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:01.369 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:01.369 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:08:01.369 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:01.369 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.369 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:01.369 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.410551+0000", 2024-09-21T22:08:01.370 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:01.370 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.370 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.370 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:01.370 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.410551+0000", 2024-09-21T22:08:01.370 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:01.370 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.370 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.370 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:01.370 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.410553+0000", 2024-09-21T22:08:01.370 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5200000000000001e-06 2024-09-21T22:08:01.370 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.370 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.371 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:01.371 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.411747+0000", 2024-09-21T22:08:01.371 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011946089999999999 2024-09-21T22:08:01.371 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.371 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.371 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:01.371 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.411796+0000", 2024-09-21T22:08:01.371 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8717999999999997e-05 2024-09-21T22:08:01.371 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.371 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.371 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:01.371 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.411844+0000", 2024-09-21T22:08:01.371 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8291999999999998e-05 2024-09-21T22:08:01.371 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.372 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.372 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:01.372 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.412066+0000", 2024-09-21T22:08:01.372 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00022115899999999999 2024-09-21T22:08:01.372 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.372 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.372 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:01.372 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.412081+0000", 2024-09-21T22:08:01.372 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5654e-05 2024-09-21T22:08:01.372 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.372 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.372 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:01.372 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.465069+0000", 2024-09-21T22:08:01.372 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.052987570999999997 2024-09-21T22:08:01.373 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:01.373 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:01.373 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:01.373 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.373 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.373 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.3 55/54 [PushOp(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head, version: 52'3, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head@52'3, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 4194304, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:08:01.373 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.468339+0000", 2024-09-21T22:08:01.373 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 48.879076306999998, 2024-09-21T22:08:01.373 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.064010024999999998, 2024-09-21T22:08:01.373 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:01.373 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:01.373 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:08:01.373 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:01.373 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.374 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:01.374 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.468339+0000", 2024-09-21T22:08:01.374 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:01.374 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.374 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.374 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:01.374 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.468339+0000", 2024-09-21T22:08:01.374 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:01.374 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.374 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.374 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:01.375 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.468340+0000", 2024-09-21T22:08:01.375 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4330000000000001e-06 2024-09-21T22:08:01.375 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.375 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.375 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:01.375 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.469484+0000", 2024-09-21T22:08:01.375 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001143535 2024-09-21T22:08:01.375 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.375 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.375 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:01.375 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.469533+0000", 2024-09-21T22:08:01.375 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8980999999999997e-05 2024-09-21T22:08:01.375 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.376 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.376 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:01.376 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.469581+0000", 2024-09-21T22:08:01.376 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8551999999999999e-05 2024-09-21T22:08:01.376 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.376 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.376 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:01.376 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.469804+0000", 2024-09-21T22:08:01.376 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00022287699999999999 2024-09-21T22:08:01.376 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.376 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.376 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:01.376 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.469829+0000", 2024-09-21T22:08:01.377 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4766e-05 2024-09-21T22:08:01.377 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.377 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.377 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:01.377 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.532349+0000", 2024-09-21T22:08:01.377 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.062519880999999999 2024-09-21T22:08:01.377 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:01.377 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:01.377 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:01.377 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.377 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.377 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4840.0:668 2.3 2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:08:01.378 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.409114+0000", 2024-09-21T22:08:01.378 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 36.938301101, 2024-09-21T22:08:01.378 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.048703758999999999, 2024-09-21T22:08:01.378 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:01.378 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:01.378 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:01.378 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-21T22:08:01.378 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4840", 2024-09-21T22:08:01.378 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:08:01.378 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 668 2024-09-21T22:08:01.378 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.379 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:01.379 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.379 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:01.379 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.409114+0000", 2024-09-21T22:08:01.379 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:01.379 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.379 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.379 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:01.379 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.409114+0000", 2024-09-21T22:08:01.379 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:01.379 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.380 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.380 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:01.380 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.409115+0000", 2024-09-21T22:08:01.380 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.114e-06 2024-09-21T22:08:01.380 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.380 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.380 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:01.380 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.410308+0000", 2024-09-21T22:08:01.380 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011930949999999999 2024-09-21T22:08:01.380 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.380 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.380 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:01.380 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.410325+0000", 2024-09-21T22:08:01.381 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7088e-05 2024-09-21T22:08:01.381 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.381 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.381 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:01.381 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.410334+0000", 2024-09-21T22:08:01.381 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.2549999999999997e-06 2024-09-21T22:08:01.381 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.381 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.381 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:01.381 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.410393+0000", 2024-09-21T22:08:01.381 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.9327999999999998e-05 2024-09-21T22:08:01.381 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.381 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.381 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:01.382 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.410482+0000", 2024-09-21T22:08:01.382 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.9019999999999998e-05 2024-09-21T22:08:01.382 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.382 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.382 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-21T22:08:01.382 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.410617+0000", 2024-09-21T22:08:01.382 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000135232 2024-09-21T22:08:01.382 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.382 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.382 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:01.382 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.457051+0000", 2024-09-21T22:08:01.383 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.04643415 2024-09-21T22:08:01.383 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.383 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.383 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-21T22:08:01.383 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.457675+0000", 2024-09-21T22:08:01.383 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00062332700000000002 2024-09-21T22:08:01.383 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.383 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.383 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:01.383 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.457735+0000", 2024-09-21T22:08:01.383 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.029e-05 2024-09-21T22:08:01.383 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.383 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.383 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:01.383 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.457818+0000", 2024-09-21T22:08:01.384 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.2860000000000003e-05 2024-09-21T22:08:01.384 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:01.384 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:01.384 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:01.384 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.384 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.384 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4840.0:670 2.3 2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:08:01.384 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.466980+0000", 2024-09-21T22:08:01.384 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 36.880435032999998, 2024-09-21T22:08:01.384 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.050913930000000003, 2024-09-21T22:08:01.384 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:01.384 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:01.384 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:01.384 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-21T22:08:01.384 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4840", 2024-09-21T22:08:01.385 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:08:01.385 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 670 2024-09-21T22:08:01.385 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.385 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:01.385 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.385 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:01.385 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.466980+0000", 2024-09-21T22:08:01.385 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:01.385 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.385 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.385 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:01.385 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.466980+0000", 2024-09-21T22:08:01.385 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:01.385 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.385 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.386 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:01.386 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.466981+0000", 2024-09-21T22:08:01.386 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0839999999999999e-06 2024-09-21T22:08:01.386 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.386 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.386 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:01.386 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.467815+0000", 2024-09-21T22:08:01.386 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00083330399999999999 2024-09-21T22:08:01.386 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.386 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.386 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:01.386 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.467832+0000", 2024-09-21T22:08:01.387 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7601e-05 2024-09-21T22:08:01.387 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.387 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.387 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:01.387 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.467840+0000", 2024-09-21T22:08:01.387 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.2220000000000002e-06 2024-09-21T22:08:01.387 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.387 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.387 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:01.387 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.467974+0000", 2024-09-21T22:08:01.387 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000133411 2024-09-21T22:08:01.387 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.387 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.387 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:01.388 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.468223+0000", 2024-09-21T22:08:01.388 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00024906400000000001 2024-09-21T22:08:01.388 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.388 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.388 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-21T22:08:01.388 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.468616+0000", 2024-09-21T22:08:01.388 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000393433 2024-09-21T22:08:01.388 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.388 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.388 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:01.388 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.517016+0000", 2024-09-21T22:08:01.388 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.048399708 2024-09-21T22:08:01.388 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.388 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.388 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-21T22:08:01.389 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.517693+0000", 2024-09-21T22:08:01.389 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00067681000000000002 2024-09-21T22:08:01.389 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.389 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.389 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:01.389 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.517789+0000", 2024-09-21T22:08:01.389 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.5778000000000004e-05 2024-09-21T22:08:01.389 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.389 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.389 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:01.389 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.517894+0000", 2024-09-21T22:08:01.389 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000105515 2024-09-21T22:08:01.389 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:01.389 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:01.389 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:01.390 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.390 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.390 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4840.0:672 2.3 2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:08:01.390 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.527127+0000", 2024-09-21T22:08:01.390 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 36.820288570999999, 2024-09-21T22:08:01.390 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.047345602000000001, 2024-09-21T22:08:01.390 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:01.390 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:01.390 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:01.390 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-21T22:08:01.390 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4840", 2024-09-21T22:08:01.390 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:08:01.390 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 672 2024-09-21T22:08:01.391 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.391 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:01.391 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.391 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:01.391 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.527127+0000", 2024-09-21T22:08:01.391 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:01.391 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.391 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.391 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:01.391 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.527127+0000", 2024-09-21T22:08:01.391 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:01.391 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.391 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.391 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:01.392 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.527128+0000", 2024-09-21T22:08:01.392 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2810000000000001e-06 2024-09-21T22:08:01.392 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.392 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.392 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:01.392 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.527962+0000", 2024-09-21T22:08:01.392 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00083445199999999996 2024-09-21T22:08:01.392 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.392 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.392 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:01.392 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.527981+0000", 2024-09-21T22:08:01.392 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9009e-05 2024-09-21T22:08:01.392 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.392 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.392 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:01.393 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.527990+0000", 2024-09-21T22:08:01.393 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.8619999999999992e-06 2024-09-21T22:08:01.393 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.393 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.393 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:01.393 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.528061+0000", 2024-09-21T22:08:01.393 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.0226000000000002e-05 2024-09-21T22:08:01.393 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.393 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.393 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:01.393 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.528156+0000", 2024-09-21T22:08:01.393 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.5724999999999996e-05 2024-09-21T22:08:01.393 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.393 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.394 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-21T22:08:01.394 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.528303+0000", 2024-09-21T22:08:01.394 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000146374 2024-09-21T22:08:01.394 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.394 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.394 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:01.394 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.573772+0000", 2024-09-21T22:08:01.394 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.045469745999999998 2024-09-21T22:08:01.394 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.394 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.394 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-21T22:08:01.394 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.574299+0000", 2024-09-21T22:08:01.394 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00052649200000000004 2024-09-21T22:08:01.394 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.395 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.395 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:01.395 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.574366+0000", 2024-09-21T22:08:01.395 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.6872000000000006e-05 2024-09-21T22:08:01.395 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.395 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.395 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:01.395 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.574472+0000", 2024-09-21T22:08:01.395 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000106563 2024-09-21T22:08:01.395 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:01.395 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:01.395 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:01.395 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.395 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.396 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4840.0:686 2.3 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:08:01.396 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.943561+0000", 2024-09-21T22:08:01.396 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 36.403854295000002, 2024-09-21T22:08:01.396 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.049021493999999999, 2024-09-21T22:08:01.396 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:01.396 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:01.396 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:01.396 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-21T22:08:01.396 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4840", 2024-09-21T22:08:01.396 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:08:01.396 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 686 2024-09-21T22:08:01.396 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.396 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:01.396 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.396 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:01.397 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.943561+0000", 2024-09-21T22:08:01.397 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:01.397 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.397 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.397 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:01.397 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.943561+0000", 2024-09-21T22:08:01.397 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:01.397 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.397 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.397 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:01.397 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.943563+0000", 2024-09-21T22:08:01.397 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9420000000000002e-06 2024-09-21T22:08:01.398 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.398 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.398 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:01.398 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.944546+0000", 2024-09-21T22:08:01.398 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00098357099999999992 2024-09-21T22:08:01.398 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.398 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.398 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:01.398 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.944565+0000", 2024-09-21T22:08:01.398 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8621e-05 2024-09-21T22:08:01.398 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.398 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.398 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:01.398 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.944575+0000", 2024-09-21T22:08:01.399 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.4639999999999995e-06 2024-09-21T22:08:01.399 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.399 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.399 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:01.399 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.944647+0000", 2024-09-21T22:08:01.399 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.2786000000000004e-05 2024-09-21T22:08:01.399 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.399 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.399 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:01.399 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.944746+0000", 2024-09-21T22:08:01.399 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.8196999999999995e-05 2024-09-21T22:08:01.399 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.399 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.399 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-21T22:08:01.399 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.944890+0000", 2024-09-21T22:08:01.400 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000144804 2024-09-21T22:08:01.400 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.400 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.400 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:01.400 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.992404+0000", 2024-09-21T22:08:01.400 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.047513211999999999 2024-09-21T22:08:01.400 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.400 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.400 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-21T22:08:01.400 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.992480+0000", 2024-09-21T22:08:01.400 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.6491999999999999e-05 2024-09-21T22:08:01.400 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.400 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.400 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:01.400 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.992521+0000", 2024-09-21T22:08:01.401 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.0608000000000002e-05 2024-09-21T22:08:01.401 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.401 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.401 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:01.401 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.992582+0000", 2024-09-21T22:08:01.401 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.1797000000000005e-05 2024-09-21T22:08:01.401 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:01.401 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:01.401 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:01.401 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.401 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.401 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4840.0:720 2.1 2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:08:01.401 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.803827+0000", 2024-09-21T22:08:01.401 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 35.543588018999998, 2024-09-21T22:08:01.401 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.049890510999999998, 2024-09-21T22:08:01.402 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:01.402 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:01.402 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:01.402 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-21T22:08:01.402 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4840", 2024-09-21T22:08:01.402 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:08:01.402 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 720 2024-09-21T22:08:01.402 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.402 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:01.402 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.402 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:01.402 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.803827+0000", 2024-09-21T22:08:01.402 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:01.402 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.403 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.403 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:01.403 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.803827+0000", 2024-09-21T22:08:01.403 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:01.403 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.403 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.403 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:01.403 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.803828+0000", 2024-09-21T22:08:01.403 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.569999999999999e-07 2024-09-21T22:08:01.403 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.403 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.403 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:01.403 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.804689+0000", 2024-09-21T22:08:01.403 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00086121799999999997 2024-09-21T22:08:01.403 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.404 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.404 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:01.404 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.804706+0000", 2024-09-21T22:08:01.404 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6266999999999999e-05 2024-09-21T22:08:01.404 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.404 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.404 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:01.404 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.804713+0000", 2024-09-21T22:08:01.404 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.1010000000000003e-06 2024-09-21T22:08:01.404 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.404 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.404 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:01.404 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.804781+0000", 2024-09-21T22:08:01.404 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.7871000000000001e-05 2024-09-21T22:08:01.404 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.405 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.405 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:01.405 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.804859+0000", 2024-09-21T22:08:01.405 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.7891000000000004e-05 2024-09-21T22:08:01.405 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.405 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.405 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 0", 2024-09-21T22:08:01.405 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.804971+0000", 2024-09-21T22:08:01.405 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00011275100000000001 2024-09-21T22:08:01.405 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.405 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.405 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:01.405 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.852704+0000", 2024-09-21T22:08:01.405 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.047732361000000001 2024-09-21T22:08:01.406 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.406 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.406 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-21T22:08:01.406 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.853446+0000", 2024-09-21T22:08:01.406 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00074258799999999997 2024-09-21T22:08:01.406 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.406 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.406 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:01.406 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.853608+0000", 2024-09-21T22:08:01.406 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00016190900000000001 2024-09-21T22:08:01.406 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.406 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.406 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:01.407 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.853718+0000", 2024-09-21T22:08:01.407 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000109597 2024-09-21T22:08:01.407 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:01.407 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:01.407 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:01.407 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.407 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.407 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4840.0:723 2.1 2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:08:01.407 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.860543+0000", 2024-09-21T22:08:01.407 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 35.486872585, 2024-09-21T22:08:01.407 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.062354957000000003, 2024-09-21T22:08:01.407 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:01.407 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:01.407 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:01.407 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-21T22:08:01.408 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4840", 2024-09-21T22:08:01.408 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:08:01.408 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 723 2024-09-21T22:08:01.408 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.408 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:01.408 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.408 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:01.408 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.860543+0000", 2024-09-21T22:08:01.408 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:01.408 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.408 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.408 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:01.408 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.860543+0000", 2024-09-21T22:08:01.408 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:01.409 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.409 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.409 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:01.409 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.860544+0000", 2024-09-21T22:08:01.409 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.1119999999999999e-06 2024-09-21T22:08:01.409 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.409 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.409 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:01.409 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.861400+0000", 2024-09-21T22:08:01.409 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000856393 2024-09-21T22:08:01.410 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.410 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.410 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:01.410 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.861419+0000", 2024-09-21T22:08:01.410 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8992e-05 2024-09-21T22:08:01.410 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.410 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.410 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:01.410 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.861428+0000", 2024-09-21T22:08:01.410 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.551e-06 2024-09-21T22:08:01.410 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.410 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.410 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:01.410 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.861555+0000", 2024-09-21T22:08:01.411 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000127286 2024-09-21T22:08:01.411 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.411 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.411 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:01.411 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.861816+0000", 2024-09-21T22:08:01.411 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00026100899999999998 2024-09-21T22:08:01.411 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.411 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.411 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 0", 2024-09-21T22:08:01.411 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.862189+0000", 2024-09-21T22:08:01.411 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00037261899999999998 2024-09-21T22:08:01.411 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.411 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.411 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:01.411 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.921514+0000", 2024-09-21T22:08:01.411 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.059325210000000003 2024-09-21T22:08:01.412 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.412 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.412 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-21T22:08:01.412 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.922721+0000", 2024-09-21T22:08:01.412 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0012066819999999999 2024-09-21T22:08:01.412 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.412 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.412 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:01.412 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.922797+0000", 2024-09-21T22:08:01.412 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.6162999999999996e-05 2024-09-21T22:08:01.412 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.412 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.412 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:01.412 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.922898+0000", 2024-09-21T22:08:01.412 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00010094000000000001 2024-09-21T22:08:01.412 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:01.413 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:01.413 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:01.413 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.413 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.413 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:725 2.0 e63/16 2:08b4f96a:::rbd_data.126483d75f2d.000000000000000f:head v 63'33, mlcod=63'31)", 2024-09-21T22:08:01.413 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.936778+0000", 2024-09-21T22:08:01.413 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 35.410636842999999, 2024-09-21T22:08:01.413 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.047671937999999997, 2024-09-21T22:08:01.413 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:01.413 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:01.413 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:01.413 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:01.413 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.413 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:01.413 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.936778+0000", 2024-09-21T22:08:01.413 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:01.414 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.414 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.414 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:01.414 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.936778+0000", 2024-09-21T22:08:01.414 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:01.414 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.414 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.414 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:01.414 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.936780+0000", 2024-09-21T22:08:01.414 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7799999999999999e-06 2024-09-21T22:08:01.414 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.414 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.414 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:01.414 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.938475+0000", 2024-09-21T22:08:01.414 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0016951749999999999 2024-09-21T22:08:01.415 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.415 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.415 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:01.415 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.938506+0000", 2024-09-21T22:08:01.415 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.0865999999999999e-05 2024-09-21T22:08:01.415 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.415 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.415 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:01.415 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.938520+0000", 2024-09-21T22:08:01.415 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3825999999999999e-05 2024-09-21T22:08:01.415 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.415 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.415 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:01.415 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.938624+0000", 2024-09-21T22:08:01.415 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00010391900000000001 2024-09-21T22:08:01.415 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.416 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.416 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:01.416 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.938651+0000", 2024-09-21T22:08:01.416 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.7481999999999999e-05 2024-09-21T22:08:01.416 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.416 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.416 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:01.416 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.984369+0000", 2024-09-21T22:08:01.416 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.045717595 2024-09-21T22:08:01.416 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.416 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.416 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:01.416 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.984450+0000", 2024-09-21T22:08:01.416 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.1295e-05 2024-09-21T22:08:01.416 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:01.417 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:01.417 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:01.417 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.417 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.417 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:727 2.0 e63/16 2:08b4f96a:::rbd_data.126483d75f2d.000000000000000f:head v 63'35, mlcod=63'33)", 2024-09-21T22:08:01.417 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.998090+0000", 2024-09-21T22:08:01.417 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 35.349325667000002, 2024-09-21T22:08:01.417 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.052205704999999998, 2024-09-21T22:08:01.417 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:01.417 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:01.417 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:01.417 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:01.417 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.417 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:01.417 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.998090+0000", 2024-09-21T22:08:01.417 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:01.418 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.418 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.418 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:01.418 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.998090+0000", 2024-09-21T22:08:01.418 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:01.418 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.418 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.418 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:01.418 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.998091+0000", 2024-09-21T22:08:01.418 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6929999999999999e-06 2024-09-21T22:08:01.418 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.418 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.418 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:01.418 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.998940+0000", 2024-09-21T22:08:01.419 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00084875200000000004 2024-09-21T22:08:01.419 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.419 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.419 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:01.419 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.998970+0000", 2024-09-21T22:08:01.419 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9709999999999998e-05 2024-09-21T22:08:01.419 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.419 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.419 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:01.419 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.998984+0000", 2024-09-21T22:08:01.419 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3739e-05 2024-09-21T22:08:01.419 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.419 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.419 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:01.419 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.999097+0000", 2024-09-21T22:08:01.420 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000113484 2024-09-21T22:08:01.420 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.420 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.420 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:01.420 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.999135+0000", 2024-09-21T22:08:01.420 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.8405999999999998e-05 2024-09-21T22:08:01.420 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.420 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.420 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:01.420 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.050245+0000", 2024-09-21T22:08:01.420 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.051109935000000002 2024-09-21T22:08:01.420 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.420 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.420 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:01.421 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.050295+0000", 2024-09-21T22:08:01.421 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.9985999999999999e-05 2024-09-21T22:08:01.421 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:01.421 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:01.421 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:01.421 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.421 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.421 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:729 2.0 e63/16 2:08b4f96a:::rbd_data.126483d75f2d.000000000000000f:head v 63'37, mlcod=63'35)", 2024-09-21T22:08:01.421 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:26.062424+0000", 2024-09-21T22:08:01.421 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 35.284991044999998, 2024-09-21T22:08:01.421 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.047312186999999999, 2024-09-21T22:08:01.421 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:01.421 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:01.421 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:01.421 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:01.422 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.422 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:01.422 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.062424+0000", 2024-09-21T22:08:01.422 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:01.422 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.422 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.422 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:01.422 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.062424+0000", 2024-09-21T22:08:01.422 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:01.422 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.422 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.422 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:01.422 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.062426+0000", 2024-09-21T22:08:01.422 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7129999999999999e-06 2024-09-21T22:08:01.423 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.423 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.423 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:01.423 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.063211+0000", 2024-09-21T22:08:01.423 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00078488500000000003 2024-09-21T22:08:01.423 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.423 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.423 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:01.423 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.063241+0000", 2024-09-21T22:08:01.423 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.0056000000000002e-05 2024-09-21T22:08:01.423 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.423 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.423 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:01.423 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.063255+0000", 2024-09-21T22:08:01.423 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4022999999999999e-05 2024-09-21T22:08:01.424 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.424 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.424 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:01.424 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.063384+0000", 2024-09-21T22:08:01.424 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00012920600000000001 2024-09-21T22:08:01.424 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.424 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.424 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:01.424 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.063421+0000", 2024-09-21T22:08:01.424 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.7242e-05 2024-09-21T22:08:01.424 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.424 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.424 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:01.424 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.109666+0000", 2024-09-21T22:08:01.425 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.046244446000000002 2024-09-21T22:08:01.425 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.425 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.425 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:01.425 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.109736+0000", 2024-09-21T22:08:01.425 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.0616000000000003e-05 2024-09-21T22:08:01.425 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:01.425 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:01.425 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:01.425 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.425 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.425 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:731 2.0 e63/16 2:08b4f96a:::rbd_data.126483d75f2d.000000000000000f:head v 63'39, mlcod=63'37)", 2024-09-21T22:08:01.425 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:26.135946+0000", 2024-09-21T22:08:01.425 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 35.211469538000003, 2024-09-21T22:08:01.425 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.054290359000000003, 2024-09-21T22:08:01.426 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:01.426 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:01.426 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:01.426 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:01.426 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.426 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:01.426 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.135946+0000", 2024-09-21T22:08:01.426 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:01.426 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.426 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.426 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:01.426 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.135946+0000", 2024-09-21T22:08:01.426 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:01.426 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.427 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.427 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:01.427 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.135948+0000", 2024-09-21T22:08:01.427 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7999999999999999e-06 2024-09-21T22:08:01.427 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.427 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.427 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:01.427 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.136730+0000", 2024-09-21T22:08:01.427 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00078271899999999997 2024-09-21T22:08:01.427 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.427 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.427 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:01.427 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.136760+0000", 2024-09-21T22:08:01.428 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9876e-05 2024-09-21T22:08:01.428 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.428 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.428 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:01.428 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.136774+0000", 2024-09-21T22:08:01.428 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3478999999999999e-05 2024-09-21T22:08:01.428 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.428 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.428 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:01.428 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.136944+0000", 2024-09-21T22:08:01.428 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000170626 2024-09-21T22:08:01.428 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.428 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.428 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:01.428 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.136985+0000", 2024-09-21T22:08:01.429 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.0377999999999999e-05 2024-09-21T22:08:01.429 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.429 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.429 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:01.429 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.190137+0000", 2024-09-21T22:08:01.429 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.053152438000000003 2024-09-21T22:08:01.429 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.429 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.429 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:01.429 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.190236+0000", 2024-09-21T22:08:01.429 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.9042999999999993e-05 2024-09-21T22:08:01.429 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:01.429 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:01.429 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:01.430 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.430 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.430 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:125 2.3 e71/66 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head v 71'31, mlcod=71'29)", 2024-09-21T22:08:01.430 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.066855+0000", 2024-09-21T22:08:01.430 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 23.280560102999999, 2024-09-21T22:08:01.430 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.059856059000000003, 2024-09-21T22:08:01.430 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:01.430 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:01.430 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:01.430 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:01.430 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.430 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:01.430 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.066855+0000", 2024-09-21T22:08:01.430 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:01.431 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.431 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.431 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:01.431 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.066855+0000", 2024-09-21T22:08:01.431 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:01.431 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.431 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.431 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:01.431 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.066857+0000", 2024-09-21T22:08:01.431 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.807e-06 2024-09-21T22:08:01.431 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.431 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.431 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:01.431 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.068365+0000", 2024-09-21T22:08:01.432 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0015077910000000001 2024-09-21T22:08:01.432 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.432 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.432 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:01.432 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.068395+0000", 2024-09-21T22:08:01.432 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.0326000000000001e-05 2024-09-21T22:08:01.432 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.432 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.432 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:01.432 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.068415+0000", 2024-09-21T22:08:01.432 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9871999999999999e-05 2024-09-21T22:08:01.432 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.432 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.432 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:01.432 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.068580+0000", 2024-09-21T22:08:01.433 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000165317 2024-09-21T22:08:01.433 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.433 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.433 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:01.433 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.068618+0000", 2024-09-21T22:08:01.433 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.7753000000000003e-05 2024-09-21T22:08:01.433 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.433 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.433 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:01.433 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.126623+0000", 2024-09-21T22:08:01.433 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.058004642000000002 2024-09-21T22:08:01.433 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.433 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.433 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:01.433 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.126711+0000", 2024-09-21T22:08:01.434 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.8551e-05 2024-09-21T22:08:01.434 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:01.434 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:01.434 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:01.434 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.434 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.434 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:127 2.3 e71/66 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head v 71'33, mlcod=71'31)", 2024-09-21T22:08:01.434 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.140864+0000", 2024-09-21T22:08:01.434 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 23.206551075, 2024-09-21T22:08:01.434 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.052512380999999997, 2024-09-21T22:08:01.434 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:01.434 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:01.434 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:01.434 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:01.435 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.435 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:01.435 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.140864+0000", 2024-09-21T22:08:01.435 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:01.435 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.435 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.435 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:01.435 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.140864+0000", 2024-09-21T22:08:01.435 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:01.435 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.435 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.435 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:01.436 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.140866+0000", 2024-09-21T22:08:01.436 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.694e-06 2024-09-21T22:08:01.436 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.436 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.436 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:01.436 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.141712+0000", 2024-09-21T22:08:01.436 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00084581499999999998 2024-09-21T22:08:01.436 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.436 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.436 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:01.436 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.141742+0000", 2024-09-21T22:08:01.436 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9865999999999998e-05 2024-09-21T22:08:01.436 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.437 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.437 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:01.437 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.141761+0000", 2024-09-21T22:08:01.437 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9473000000000001e-05 2024-09-21T22:08:01.437 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.437 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.437 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:01.437 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.141919+0000", 2024-09-21T22:08:01.437 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00015758300000000001 2024-09-21T22:08:01.437 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.437 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.437 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:01.437 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.141955+0000", 2024-09-21T22:08:01.438 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.6613000000000003e-05 2024-09-21T22:08:01.438 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.438 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.438 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:01.438 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.193333+0000", 2024-09-21T22:08:01.438 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.051377260000000001 2024-09-21T22:08:01.438 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.438 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.438 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:01.438 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.193377+0000", 2024-09-21T22:08:01.438 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.4076999999999998e-05 2024-09-21T22:08:01.438 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:01.439 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:01.439 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:01.439 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.439 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.439 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:129 2.3 e71/66 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head v 71'35, mlcod=71'33)", 2024-09-21T22:08:01.439 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.209458+0000", 2024-09-21T22:08:01.439 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 23.137957325999999, 2024-09-21T22:08:01.439 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.056595578000000001, 2024-09-21T22:08:01.439 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:01.439 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:01.439 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:01.439 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:01.439 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.439 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:01.439 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.209458+0000", 2024-09-21T22:08:01.440 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:01.440 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.440 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.440 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:01.440 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.209458+0000", 2024-09-21T22:08:01.440 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:01.440 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.440 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.440 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:01.440 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.209460+0000", 2024-09-21T22:08:01.440 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7600000000000001e-06 2024-09-21T22:08:01.440 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.440 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.440 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:01.441 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.210317+0000", 2024-09-21T22:08:01.441 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00085771700000000003 2024-09-21T22:08:01.441 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.441 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.441 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:01.441 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.210346+0000", 2024-09-21T22:08:01.441 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8334e-05 2024-09-21T22:08:01.441 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.441 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.441 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:01.441 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.210365+0000", 2024-09-21T22:08:01.441 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9439000000000001e-05 2024-09-21T22:08:01.441 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.441 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.442 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:01.442 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.210544+0000", 2024-09-21T22:08:01.442 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00017845599999999999 2024-09-21T22:08:01.442 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.442 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.442 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:01.442 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.210584+0000", 2024-09-21T22:08:01.442 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.0423000000000002e-05 2024-09-21T22:08:01.442 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.442 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.442 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:01.442 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.265970+0000", 2024-09-21T22:08:01.442 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.055385765000000003 2024-09-21T22:08:01.442 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.443 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.443 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:01.443 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.266054+0000", 2024-09-21T22:08:01.443 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.3683999999999994e-05 2024-09-21T22:08:01.443 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:01.443 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:01.443 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:01.443 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.443 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.443 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:131 2.3 e71/66 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head v 71'37, mlcod=71'35)", 2024-09-21T22:08:01.443 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.279550+0000", 2024-09-21T22:08:01.443 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 23.067865487999999, 2024-09-21T22:08:01.443 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.052290419999999997, 2024-09-21T22:08:01.443 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:01.444 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:01.444 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:01.444 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:01.444 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.444 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:01.444 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.279550+0000", 2024-09-21T22:08:01.444 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:01.444 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.444 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.444 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:01.444 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.279550+0000", 2024-09-21T22:08:01.444 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:01.444 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.444 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.444 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:01.445 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.279552+0000", 2024-09-21T22:08:01.445 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9439999999999999e-06 2024-09-21T22:08:01.445 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.445 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.445 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:01.445 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.280296+0000", 2024-09-21T22:08:01.445 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00074417000000000005 2024-09-21T22:08:01.445 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.445 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.445 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:01.445 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.280324+0000", 2024-09-21T22:08:01.445 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8045999999999999e-05 2024-09-21T22:08:01.445 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.445 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.445 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:01.446 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.280345+0000", 2024-09-21T22:08:01.446 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.0849999999999999e-05 2024-09-21T22:08:01.446 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.446 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.446 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:01.446 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.280497+0000", 2024-09-21T22:08:01.446 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00015252500000000001 2024-09-21T22:08:01.446 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.446 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.446 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:01.446 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.280531+0000", 2024-09-21T22:08:01.446 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.4034e-05 2024-09-21T22:08:01.446 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.446 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.446 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:01.446 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.331769+0000", 2024-09-21T22:08:01.447 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.051237999999999999 2024-09-21T22:08:01.447 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:01.447 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:01.447 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:01.447 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.331840+0000", 2024-09-21T22:08:01.447 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.0851000000000003e-05 2024-09-21T22:08:01.447 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:01.447 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:01.447 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:01.447 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:01.447 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:01.447 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:08:01.449 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T22:08:01.449 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:08:01.445+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:08:01.550 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:08:01.545+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:08:01.585 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:08:01.585 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:08:01.585 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-09-21T22:08:01.585 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:08:01.600 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T22:08:01.650 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:08:01.648+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:08:01.719 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:08:01.719 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:08:01.719 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-21T22:08:01.720 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-21T22:08:01.720 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:08:01.734 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T22:08:01.778 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:08:01.776+0000 7f2075a06640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31720) UID: 0 2024-09-21T22:08:01.852 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:08:01.848+0000 7f2075a06640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31720) UID: 0 2024-09-21T22:08:01.910 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:08:01.910 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-21T22:08:01.910 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-21T22:08:01.910 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-21T22:08:01.910 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.910 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:08:01.910 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.036355+0000", 2024-09-21T22:08:01.910 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 119.87056977, 2024-09-21T22:08:01.910 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.062441934999999997, 2024-09-21T22:08:01.910 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:01.911 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:01.911 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:01.911 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:01.911 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:08:01.911 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:08:01.911 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1 2024-09-21T22:08:01.911 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.911 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:01.911 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.911 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:01.911 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036355+0000", 2024-09-21T22:08:01.911 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:01.911 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.911 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.911 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:01.912 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036355+0000", 2024-09-21T22:08:01.912 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:01.912 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.912 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.912 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:01.912 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036357+0000", 2024-09-21T22:08:01.912 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.446e-06 2024-09-21T22:08:01.912 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.912 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.912 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:01.912 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036373+0000", 2024-09-21T22:08:01.912 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5447e-05 2024-09-21T22:08:01.912 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.912 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.912 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:01.913 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036399+0000", 2024-09-21T22:08:01.913 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6423000000000001e-05 2024-09-21T22:08:01.913 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.913 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.913 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:01.913 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036420+0000", 2024-09-21T22:08:01.913 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1129999999999999e-05 2024-09-21T22:08:01.913 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.913 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.913 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:01.913 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036557+0000", 2024-09-21T22:08:01.913 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013686899999999999 2024-09-21T22:08:01.913 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.913 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.914 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-21T22:08:01.914 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036585+0000", 2024-09-21T22:08:01.914 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8362999999999999e-05 2024-09-21T22:08:01.914 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.914 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.914 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:01.914 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.095068+0000", 2024-09-21T22:08:01.914 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058482972000000001 2024-09-21T22:08:01.914 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.914 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.914 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:01.914 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.095516+0000", 2024-09-21T22:08:01.914 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00044802999999999998 2024-09-21T22:08:01.915 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.915 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.915 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:08:01.915 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.095869+0000", 2024-09-21T22:08:01.915 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000352104 2024-09-21T22:08:01.915 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.915 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.915 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:01.915 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.097055+0000", 2024-09-21T22:08:01.915 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011865599999999999 2024-09-21T22:08:01.915 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.915 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.915 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:01.915 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.098637+0000", 2024-09-21T22:08:01.915 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001581458 2024-09-21T22:08:01.916 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.916 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.916 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:01.916 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.098728+0000", 2024-09-21T22:08:01.916 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1043000000000002e-05 2024-09-21T22:08:01.916 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.916 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.916 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:01.916 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.098797+0000", 2024-09-21T22:08:01.916 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9090000000000004e-05 2024-09-21T22:08:01.916 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:01.916 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:01.916 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:01.916 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.917 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.917 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4840.0:680 2.7 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:08:01.917 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.744506+0000", 2024-09-21T22:08:01.917 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 37.162418916, 2024-09-21T22:08:01.917 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.063865055000000004, 2024-09-21T22:08:01.917 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:01.917 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:01.917 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:01.917 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:01.917 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4840", 2024-09-21T22:08:01.917 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:08:01.917 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 680 2024-09-21T22:08:01.917 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.917 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:01.918 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.918 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:01.918 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.744506+0000", 2024-09-21T22:08:01.918 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:01.918 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.918 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.918 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:01.918 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.744506+0000", 2024-09-21T22:08:01.918 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:01.918 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.918 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.918 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:01.918 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.744508+0000", 2024-09-21T22:08:01.918 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.103e-06 2024-09-21T22:08:01.918 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.919 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.919 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:01.919 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.745371+0000", 2024-09-21T22:08:01.919 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00086322199999999997 2024-09-21T22:08:01.919 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.919 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.919 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:01.919 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.745401+0000", 2024-09-21T22:08:01.919 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9955999999999999e-05 2024-09-21T22:08:01.919 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.919 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.919 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:01.919 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.745416+0000", 2024-09-21T22:08:01.919 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4873e-05 2024-09-21T22:08:01.919 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.920 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.920 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:01.920 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.745509+0000", 2024-09-21T22:08:01.920 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3171000000000006e-05 2024-09-21T22:08:01.920 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.920 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.920 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:01.920 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.745632+0000", 2024-09-21T22:08:01.920 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012272699999999999 2024-09-21T22:08:01.920 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.920 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.920 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-09-21T22:08:01.920 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.745789+0000", 2024-09-21T22:08:01.920 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015781399999999999 2024-09-21T22:08:01.920 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.921 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.921 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:01.921 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.785494+0000", 2024-09-21T22:08:01.921 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.039704401 2024-09-21T22:08:01.921 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.921 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.921 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:01.921 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.808090+0000", 2024-09-21T22:08:01.921 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022596207 2024-09-21T22:08:01.921 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.921 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.921 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:01.921 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.808246+0000", 2024-09-21T22:08:01.921 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015567200000000001 2024-09-21T22:08:01.921 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.922 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.922 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:01.922 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.808371+0000", 2024-09-21T22:08:01.922 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012490900000000001 2024-09-21T22:08:01.922 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:01.922 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:01.922 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:01.922 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.922 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.922 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4840.0:682 2.7 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:08:01.922 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.822415+0000", 2024-09-21T22:08:01.922 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 37.084509353000001, 2024-09-21T22:08:01.922 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.065044633000000004, 2024-09-21T22:08:01.922 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:01.923 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:01.923 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:01.923 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:01.923 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4840", 2024-09-21T22:08:01.923 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:08:01.923 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 682 2024-09-21T22:08:01.923 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.923 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:01.923 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.923 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:01.923 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.822415+0000", 2024-09-21T22:08:01.923 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:01.923 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.923 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.924 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:01.924 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.822415+0000", 2024-09-21T22:08:01.924 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:01.924 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.924 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.924 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:01.924 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.822417+0000", 2024-09-21T22:08:01.924 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8029999999999999e-06 2024-09-21T22:08:01.924 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.924 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.924 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:01.924 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.823308+0000", 2024-09-21T22:08:01.924 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00089146500000000003 2024-09-21T22:08:01.924 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.925 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.925 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:01.925 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.823339+0000", 2024-09-21T22:08:01.925 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0493000000000001e-05 2024-09-21T22:08:01.925 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.925 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.925 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:01.925 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.823354+0000", 2024-09-21T22:08:01.925 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5472999999999999e-05 2024-09-21T22:08:01.925 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.925 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.925 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:01.925 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.823476+0000", 2024-09-21T22:08:01.925 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000121209 2024-09-21T22:08:01.926 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.926 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.926 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:01.926 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.823650+0000", 2024-09-21T22:08:01.926 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017394599999999999 2024-09-21T22:08:01.926 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.926 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.926 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-09-21T22:08:01.926 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.823873+0000", 2024-09-21T22:08:01.926 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022314899999999999 2024-09-21T22:08:01.926 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.926 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.927 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:01.927 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.884396+0000", 2024-09-21T22:08:01.927 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.060523674999999999 2024-09-21T22:08:01.927 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.927 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.927 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:01.927 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.887238+0000", 2024-09-21T22:08:01.927 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028420450000000001 2024-09-21T22:08:01.927 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.927 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.927 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:01.927 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.887369+0000", 2024-09-21T22:08:01.927 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013010699999999999 2024-09-21T22:08:01.927 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.928 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.928 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:01.928 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.887460+0000", 2024-09-21T22:08:01.928 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1267999999999994e-05 2024-09-21T22:08:01.928 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:01.928 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:01.928 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:01.928 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.928 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.928 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4840.0:692 2.5 2:b1d55633:::rbd_data.126483d75f2d.000000000000000b:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:08:01.928 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:25.091048+0000", 2024-09-21T22:08:01.928 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 36.815876819000003, 2024-09-21T22:08:01.928 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.060917092999999999, 2024-09-21T22:08:01.929 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:01.929 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:01.929 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:01.929 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:01.929 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4840", 2024-09-21T22:08:01.929 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:08:01.929 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 692 2024-09-21T22:08:01.929 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.929 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:01.929 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.929 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:01.929 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.091048+0000", 2024-09-21T22:08:01.929 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:01.929 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.930 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.930 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:01.930 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.091048+0000", 2024-09-21T22:08:01.930 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:01.930 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.930 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.930 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:01.930 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.091050+0000", 2024-09-21T22:08:01.930 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.03e-06 2024-09-21T22:08:01.930 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.930 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.930 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:01.931 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.091989+0000", 2024-09-21T22:08:01.931 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00093932100000000001 2024-09-21T22:08:01.931 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.931 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.931 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:01.931 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.092023+0000", 2024-09-21T22:08:01.931 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4310000000000002e-05 2024-09-21T22:08:01.931 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.931 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.931 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:01.931 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.092040+0000", 2024-09-21T22:08:01.931 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6426000000000002e-05 2024-09-21T22:08:01.932 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.932 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.932 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:01.932 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.092175+0000", 2024-09-21T22:08:01.932 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013500299999999999 2024-09-21T22:08:01.932 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.932 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.932 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:01.932 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.092307+0000", 2024-09-21T22:08:01.932 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000132562 2024-09-21T22:08:01.932 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.932 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.932 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:08:01.932 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.092499+0000", 2024-09-21T22:08:01.932 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000191972 2024-09-21T22:08:01.933 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.933 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.933 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:01.933 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.151065+0000", 2024-09-21T22:08:01.933 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058565514999999999 2024-09-21T22:08:01.933 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.933 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.933 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:01.933 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.151844+0000", 2024-09-21T22:08:01.933 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00077930800000000004 2024-09-21T22:08:01.933 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.933 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.933 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:01.933 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.151916+0000", 2024-09-21T22:08:01.934 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1921999999999993e-05 2024-09-21T22:08:01.934 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.934 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.934 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:01.934 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.151965+0000", 2024-09-21T22:08:01.934 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8724000000000003e-05 2024-09-21T22:08:01.934 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:01.934 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:01.934 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:01.934 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.934 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.934 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4840.0:696 2.5 2:b1d55633:::rbd_data.126483d75f2d.000000000000000b:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:08:01.935 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:25.218641+0000", 2024-09-21T22:08:01.935 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 36.688283550999998, 2024-09-21T22:08:01.935 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058363669999999999, 2024-09-21T22:08:01.935 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:01.935 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:01.935 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:01.935 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:01.935 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4840", 2024-09-21T22:08:01.935 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:08:01.935 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 696 2024-09-21T22:08:01.935 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.935 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:01.935 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.935 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:01.936 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.218641+0000", 2024-09-21T22:08:01.936 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:01.936 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.936 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.936 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:01.936 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.218641+0000", 2024-09-21T22:08:01.936 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:01.936 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.936 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.936 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:01.936 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.218642+0000", 2024-09-21T22:08:01.936 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.266e-06 2024-09-21T22:08:01.936 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.936 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.937 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:01.937 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.219432+0000", 2024-09-21T22:08:01.937 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00078986300000000002 2024-09-21T22:08:01.937 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.937 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.937 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:01.937 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.219450+0000", 2024-09-21T22:08:01.937 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7526000000000001e-05 2024-09-21T22:08:01.937 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.937 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.937 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:01.937 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.219459+0000", 2024-09-21T22:08:01.937 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3000000000000007e-06 2024-09-21T22:08:01.937 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.938 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.938 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:01.938 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.219655+0000", 2024-09-21T22:08:01.938 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019599099999999999 2024-09-21T22:08:01.938 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.938 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.938 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:01.938 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.219917+0000", 2024-09-21T22:08:01.938 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026198200000000001 2024-09-21T22:08:01.938 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.938 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.938 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:08:01.938 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.220198+0000", 2024-09-21T22:08:01.939 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00028154500000000002 2024-09-21T22:08:01.939 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.939 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.939 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:01.939 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.268856+0000", 2024-09-21T22:08:01.939 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.048657687999999998 2024-09-21T22:08:01.939 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.939 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.939 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:01.939 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.276903+0000", 2024-09-21T22:08:01.939 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0080472249999999999 2024-09-21T22:08:01.939 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.939 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.939 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:01.939 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.276958+0000", 2024-09-21T22:08:01.940 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4184999999999997e-05 2024-09-21T22:08:01.940 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.940 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.940 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:01.940 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.277005+0000", 2024-09-21T22:08:01.940 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7098999999999997e-05 2024-09-21T22:08:01.940 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:01.940 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:01.940 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:01.940 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.940 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.940 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4840.0:729 2.0 2:08b4f96a:::rbd_data.126483d75f2d.000000000000000f:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:08:01.940 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:26.062012+0000", 2024-09-21T22:08:01.940 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 35.844912317999999, 2024-09-21T22:08:01.940 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.060656823999999998, 2024-09-21T22:08:01.941 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:01.941 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:01.941 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:01.941 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:01.941 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4840", 2024-09-21T22:08:01.941 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:08:01.941 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 729 2024-09-21T22:08:01.941 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.941 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:01.941 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.941 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:01.941 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.062012+0000", 2024-09-21T22:08:01.942 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:01.942 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.942 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.942 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:01.942 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.062012+0000", 2024-09-21T22:08:01.942 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:01.942 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.942 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.942 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:01.942 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.062013+0000", 2024-09-21T22:08:01.942 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.243e-06 2024-09-21T22:08:01.942 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.942 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.943 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:01.943 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.062827+0000", 2024-09-21T22:08:01.943 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00081374799999999999 2024-09-21T22:08:01.943 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.943 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.943 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:01.943 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.062845+0000", 2024-09-21T22:08:01.943 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.808e-05 2024-09-21T22:08:01.943 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.943 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.943 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:01.943 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.062854+0000", 2024-09-21T22:08:01.943 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1039999999999996e-06 2024-09-21T22:08:01.943 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.944 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.944 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:01.944 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.062970+0000", 2024-09-21T22:08:01.944 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011601 2024-09-21T22:08:01.944 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.944 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.944 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:01.944 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.063230+0000", 2024-09-21T22:08:01.944 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025918799999999999 2024-09-21T22:08:01.944 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.944 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.944 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-09-21T22:08:01.944 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.063641+0000", 2024-09-21T22:08:01.944 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00041152799999999998 2024-09-21T22:08:01.945 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.945 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.945 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:01.945 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.122015+0000", 2024-09-21T22:08:01.945 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058374246999999997 2024-09-21T22:08:01.945 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.945 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.945 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:01.945 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.122451+0000", 2024-09-21T22:08:01.945 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000435771 2024-09-21T22:08:01.945 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.945 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.945 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:01.946 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.122576+0000", 2024-09-21T22:08:01.946 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012469099999999999 2024-09-21T22:08:01.946 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.946 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.946 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:01.946 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.122669+0000", 2024-09-21T22:08:01.946 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3214000000000006e-05 2024-09-21T22:08:01.946 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:01.946 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:01.946 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:01.946 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.946 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.946 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4840.0:731 2.0 2:08b4f96a:::rbd_data.126483d75f2d.000000000000000f:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:08:01.946 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:26.135603+0000", 2024-09-21T22:08:01.947 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 35.771321817999997, 2024-09-21T22:08:01.947 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.056838474999999999, 2024-09-21T22:08:01.947 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:01.947 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:01.947 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:01.947 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:01.947 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4840", 2024-09-21T22:08:01.947 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:08:01.947 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 731 2024-09-21T22:08:01.947 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.947 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:01.947 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.947 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:01.947 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.135603+0000", 2024-09-21T22:08:01.948 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:01.948 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.948 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.948 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:01.948 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.135603+0000", 2024-09-21T22:08:01.948 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:01.948 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.948 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.948 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:01.948 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.135605+0000", 2024-09-21T22:08:01.948 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9700000000000002e-06 2024-09-21T22:08:01.948 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.948 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.948 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:01.949 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.136554+0000", 2024-09-21T22:08:01.949 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00094886700000000005 2024-09-21T22:08:01.949 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.949 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.949 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:01.949 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.136582+0000", 2024-09-21T22:08:01.949 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8393000000000001e-05 2024-09-21T22:08:01.949 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.949 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.949 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:01.949 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.136596+0000", 2024-09-21T22:08:01.949 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.43e-05 2024-09-21T22:08:01.949 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.949 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.950 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:01.950 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.136691+0000", 2024-09-21T22:08:01.950 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4843e-05 2024-09-21T22:08:01.950 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.950 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.950 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:01.950 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.136873+0000", 2024-09-21T22:08:01.950 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018237099999999999 2024-09-21T22:08:01.950 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.950 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.950 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-09-21T22:08:01.950 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.137140+0000", 2024-09-21T22:08:01.950 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026642700000000002 2024-09-21T22:08:01.950 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.951 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.951 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:01.951 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.180580+0000", 2024-09-21T22:08:01.951 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.043440078 2024-09-21T22:08:01.951 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.951 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.951 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:01.951 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.192175+0000", 2024-09-21T22:08:01.951 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011594959 2024-09-21T22:08:01.951 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.951 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.951 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:01.951 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.192353+0000", 2024-09-21T22:08:01.951 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017850200000000001 2024-09-21T22:08:01.952 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.952 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.952 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:01.952 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.192441+0000", 2024-09-21T22:08:01.952 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7764999999999997e-05 2024-09-21T22:08:01.952 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:01.952 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:01.952 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:01.952 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.952 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.952 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5207.0:33 2.7 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head [write 131072~131072] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e78)", 2024-09-21T22:08:01.952 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:52.587407+0000", 2024-09-21T22:08:01.952 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.3195177049999991, 2024-09-21T22:08:01.952 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.056613749999999997, 2024-09-21T22:08:01.952 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:01.953 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:01.953 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:01.953 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:01.953 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5207", 2024-09-21T22:08:01.953 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/357731919", 2024-09-21T22:08:01.953 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 33 2024-09-21T22:08:01.953 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.953 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:01.953 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.953 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:01.953 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.587407+0000", 2024-09-21T22:08:01.953 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:01.953 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.953 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.954 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:01.954 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.587407+0000", 2024-09-21T22:08:01.954 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:01.954 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.954 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.954 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:01.954 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.587408+0000", 2024-09-21T22:08:01.954 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.59e-06 2024-09-21T22:08:01.954 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.954 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.954 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:01.954 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.587625+0000", 2024-09-21T22:08:01.954 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000216209 2024-09-21T22:08:01.954 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.955 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.955 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:01.955 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.587653+0000", 2024-09-21T22:08:01.955 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8863000000000001e-05 2024-09-21T22:08:01.955 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.955 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.955 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:01.955 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.587671+0000", 2024-09-21T22:08:01.955 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7710000000000002e-05 2024-09-21T22:08:01.955 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.955 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.955 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:01.955 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.604027+0000", 2024-09-21T22:08:01.955 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.016356287000000001 2024-09-21T22:08:01.955 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.956 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.956 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:01.956 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.604170+0000", 2024-09-21T22:08:01.956 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000142769 2024-09-21T22:08:01.956 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.956 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.956 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-09-21T22:08:01.956 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.604388+0000", 2024-09-21T22:08:01.956 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000217476 2024-09-21T22:08:01.956 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.956 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.956 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:01.956 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.643757+0000", 2024-09-21T22:08:01.957 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.039369336999999997 2024-09-21T22:08:01.957 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.957 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.957 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:01.957 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.643792+0000", 2024-09-21T22:08:01.957 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4520000000000002e-05 2024-09-21T22:08:01.957 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.957 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.957 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:01.957 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.643906+0000", 2024-09-21T22:08:01.957 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000114153 2024-09-21T22:08:01.957 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.957 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.957 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:01.958 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.644021+0000", 2024-09-21T22:08:01.958 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000114836 2024-09-21T22:08:01.958 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:01.958 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:01.958 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:01.958 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.958 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.958 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5207.0:34 2.7 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head [write 262144~131072] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e78)", 2024-09-21T22:08:01.958 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:52.587708+0000", 2024-09-21T22:08:01.958 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.3192167969999993, 2024-09-21T22:08:01.958 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.056690213000000003, 2024-09-21T22:08:01.958 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:01.958 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:01.959 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:01.959 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:01.959 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5207", 2024-09-21T22:08:01.959 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/357731919", 2024-09-21T22:08:01.959 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 34 2024-09-21T22:08:01.959 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.959 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:01.959 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.959 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:01.959 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.587708+0000", 2024-09-21T22:08:01.959 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:01.959 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.959 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.959 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:01.960 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.587708+0000", 2024-09-21T22:08:01.960 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:01.960 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.960 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.960 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:01.960 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.587709+0000", 2024-09-21T22:08:01.960 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.72e-06 2024-09-21T22:08:01.960 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.960 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.960 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:01.960 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588017+0000", 2024-09-21T22:08:01.960 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00030767499999999998 2024-09-21T22:08:01.960 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.960 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.960 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:01.961 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588037+0000", 2024-09-21T22:08:01.961 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9865999999999999e-05 2024-09-21T22:08:01.961 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.961 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.961 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:01.961 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588048+0000", 2024-09-21T22:08:01.961 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1384e-05 2024-09-21T22:08:01.961 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.961 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.961 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:01.961 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.614263+0000", 2024-09-21T22:08:01.961 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026214232000000001 2024-09-21T22:08:01.961 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.961 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.962 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:01.962 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.614397+0000", 2024-09-21T22:08:01.962 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013396299999999999 2024-09-21T22:08:01.962 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.962 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.962 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-09-21T22:08:01.962 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.614670+0000", 2024-09-21T22:08:01.962 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027295500000000002 2024-09-21T22:08:01.962 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.962 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.962 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:01.962 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.644046+0000", 2024-09-21T22:08:01.962 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029376973000000001 2024-09-21T22:08:01.962 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.963 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.963 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:01.963 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.644208+0000", 2024-09-21T22:08:01.963 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016169900000000001 2024-09-21T22:08:01.963 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.963 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.963 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:01.963 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.644335+0000", 2024-09-21T22:08:01.963 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012656000000000001 2024-09-21T22:08:01.963 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.963 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.963 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:01.963 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.644398+0000", 2024-09-21T22:08:01.963 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3186000000000001e-05 2024-09-21T22:08:01.963 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:01.963 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:01.964 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:01.964 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.964 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.964 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5207.0:35 2.7 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head [write 393216~131072] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e78)", 2024-09-21T22:08:01.964 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:52.588070+0000", 2024-09-21T22:08:01.964 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.3188548989999997, 2024-09-21T22:08:01.964 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.076411232999999995, 2024-09-21T22:08:01.964 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:01.964 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:01.964 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:01.964 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:01.964 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5207", 2024-09-21T22:08:01.964 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/357731919", 2024-09-21T22:08:01.964 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 35 2024-09-21T22:08:01.965 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.965 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:01.965 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.965 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:01.965 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588070+0000", 2024-09-21T22:08:01.965 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:01.965 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.965 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.965 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:01.965 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588070+0000", 2024-09-21T22:08:01.965 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:01.965 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.965 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.965 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:01.966 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588071+0000", 2024-09-21T22:08:01.966 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1299999999999998e-07 2024-09-21T22:08:01.966 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.966 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.966 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:01.966 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588327+0000", 2024-09-21T22:08:01.966 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025630199999999997 2024-09-21T22:08:01.966 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.966 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.966 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:01.966 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588343+0000", 2024-09-21T22:08:01.966 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6140000000000001e-05 2024-09-21T22:08:01.966 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.966 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.967 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:01.967 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588353+0000", 2024-09-21T22:08:01.967 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0043e-05 2024-09-21T22:08:01.967 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.967 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.967 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:01.967 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.623978+0000", 2024-09-21T22:08:01.967 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.035625281000000002 2024-09-21T22:08:01.967 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.967 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.967 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:01.967 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.624104+0000", 2024-09-21T22:08:01.967 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012591300000000001 2024-09-21T22:08:01.967 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.967 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.968 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-09-21T22:08:01.968 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.624296+0000", 2024-09-21T22:08:01.968 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019226200000000001 2024-09-21T22:08:01.968 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.968 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.968 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:01.968 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.654488+0000", 2024-09-21T22:08:01.968 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.030191551 2024-09-21T22:08:01.968 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.968 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.968 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:01.968 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.664329+0000", 2024-09-21T22:08:01.968 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0098405119999999992 2024-09-21T22:08:01.968 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.969 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.969 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:01.969 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.664414+0000", 2024-09-21T22:08:01.969 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5385999999999999e-05 2024-09-21T22:08:01.969 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.969 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.969 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:01.969 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.664481+0000", 2024-09-21T22:08:01.969 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6929999999999998e-05 2024-09-21T22:08:01.969 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:01.969 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:01.969 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:01.969 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.969 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.970 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5207.0:36 2.7 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head [write 524288~131072] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e78)", 2024-09-21T22:08:01.970 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:52.588370+0000", 2024-09-21T22:08:01.970 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.318554121, 2024-09-21T22:08:01.970 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.076552001999999994, 2024-09-21T22:08:01.970 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:01.970 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:01.970 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:01.970 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:01.970 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5207", 2024-09-21T22:08:01.970 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/357731919", 2024-09-21T22:08:01.970 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 36 2024-09-21T22:08:01.970 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.970 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:01.970 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.971 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:01.971 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588370+0000", 2024-09-21T22:08:01.971 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:01.971 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.971 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.971 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:01.971 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588370+0000", 2024-09-21T22:08:01.971 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:01.971 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.971 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.971 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:01.971 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588372+0000", 2024-09-21T22:08:01.971 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.153e-06 2024-09-21T22:08:01.971 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.972 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.972 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:01.972 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588716+0000", 2024-09-21T22:08:01.972 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00034479200000000002 2024-09-21T22:08:01.972 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.972 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.972 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:01.972 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588752+0000", 2024-09-21T22:08:01.972 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5865999999999998e-05 2024-09-21T22:08:01.972 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.972 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.972 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:01.972 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588777+0000", 2024-09-21T22:08:01.973 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4583000000000002e-05 2024-09-21T22:08:01.973 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.973 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.973 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:01.973 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.633886+0000", 2024-09-21T22:08:01.973 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.045108941999999999 2024-09-21T22:08:01.973 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.973 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.973 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:01.973 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.634059+0000", 2024-09-21T22:08:01.973 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000173046 2024-09-21T22:08:01.973 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.973 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.973 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-09-21T22:08:01.973 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.634314+0000", 2024-09-21T22:08:01.974 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025482400000000001 2024-09-21T22:08:01.974 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.974 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.974 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:01.974 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.664541+0000", 2024-09-21T22:08:01.974 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.030227222000000002 2024-09-21T22:08:01.974 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.974 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.974 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:01.974 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.664653+0000", 2024-09-21T22:08:01.974 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011192900000000001 2024-09-21T22:08:01.974 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.974 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.975 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:01.975 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.664798+0000", 2024-09-21T22:08:01.975 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014499900000000001 2024-09-21T22:08:01.975 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.975 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.975 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:01.975 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.664922+0000", 2024-09-21T22:08:01.975 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012464600000000001 2024-09-21T22:08:01.975 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:01.975 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:01.975 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:01.975 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.975 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.975 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5207.0:37 2.7 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head [write 655360~131072] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e78)", 2024-09-21T22:08:01.975 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:52.588813+0000", 2024-09-21T22:08:01.976 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.318111107, 2024-09-21T22:08:01.976 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.087074796999999995, 2024-09-21T22:08:01.976 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:01.976 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:01.976 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:01.976 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:01.976 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5207", 2024-09-21T22:08:01.976 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/357731919", 2024-09-21T22:08:01.976 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 37 2024-09-21T22:08:01.976 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.976 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:01.976 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.976 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:01.976 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588813+0000", 2024-09-21T22:08:01.977 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:01.977 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.977 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.977 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:01.977 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588813+0000", 2024-09-21T22:08:01.977 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:01.977 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.977 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.977 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:01.977 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588815+0000", 2024-09-21T22:08:01.977 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.643e-06 2024-09-21T22:08:01.977 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.977 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.977 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:01.978 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.589071+0000", 2024-09-21T22:08:01.978 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025580200000000002 2024-09-21T22:08:01.978 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.978 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.978 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:01.978 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.589098+0000", 2024-09-21T22:08:01.978 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7336000000000001e-05 2024-09-21T22:08:01.978 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.978 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.978 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:01.978 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.589121+0000", 2024-09-21T22:08:01.978 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2373000000000001e-05 2024-09-21T22:08:01.978 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.978 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.979 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:01.979 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.644491+0000", 2024-09-21T22:08:01.979 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.055369994999999998 2024-09-21T22:08:01.979 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.979 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.979 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:01.979 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.644707+0000", 2024-09-21T22:08:01.979 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000216488 2024-09-21T22:08:01.979 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.979 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.979 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-09-21T22:08:01.979 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.644908+0000", 2024-09-21T22:08:01.979 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000201196 2024-09-21T22:08:01.980 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.980 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.980 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:01.980 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.675419+0000", 2024-09-21T22:08:01.980 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.03051071 2024-09-21T22:08:01.980 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.980 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.980 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:01.980 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.675591+0000", 2024-09-21T22:08:01.980 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000172545 2024-09-21T22:08:01.980 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.980 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.980 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:01.980 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.675761+0000", 2024-09-21T22:08:01.981 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016942300000000001 2024-09-21T22:08:01.981 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.981 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.981 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:01.981 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.675888+0000", 2024-09-21T22:08:01.981 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000127286 2024-09-21T22:08:01.981 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:01.981 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:01.981 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:01.981 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.981 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.981 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5207.0:38 2.7 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head [write 786432~131072] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e78)", 2024-09-21T22:08:01.981 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:52.589156+0000", 2024-09-21T22:08:01.981 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.317768203, 2024-09-21T22:08:01.982 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.087126880000000004, 2024-09-21T22:08:01.982 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:01.982 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:01.982 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:01.982 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:01.982 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5207", 2024-09-21T22:08:01.982 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/357731919", 2024-09-21T22:08:01.982 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 38 2024-09-21T22:08:01.982 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.982 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:01.982 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.982 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:01.982 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.589156+0000", 2024-09-21T22:08:01.983 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:01.983 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.983 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.983 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:01.983 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.589156+0000", 2024-09-21T22:08:01.983 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:01.983 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.983 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.983 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:01.983 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.589158+0000", 2024-09-21T22:08:01.983 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.717e-06 2024-09-21T22:08:01.983 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.983 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.983 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:01.984 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.589417+0000", 2024-09-21T22:08:01.984 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025860500000000002 2024-09-21T22:08:01.984 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.984 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.984 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:01.984 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.589447+0000", 2024-09-21T22:08:01.984 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9916e-05 2024-09-21T22:08:01.984 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.984 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.984 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:01.984 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.589467+0000", 2024-09-21T22:08:01.984 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0894e-05 2024-09-21T22:08:01.984 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.984 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.985 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:01.985 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.654576+0000", 2024-09-21T22:08:01.985 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.065108579999999999 2024-09-21T22:08:01.985 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.985 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.985 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:01.985 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.654694+0000", 2024-09-21T22:08:01.985 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000118083 2024-09-21T22:08:01.985 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.985 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.985 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-09-21T22:08:01.985 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.654880+0000", 2024-09-21T22:08:01.985 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000186069 2024-09-21T22:08:01.986 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.986 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.986 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:01.986 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.675453+0000", 2024-09-21T22:08:01.986 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020572498000000002 2024-09-21T22:08:01.986 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.986 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.986 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:01.986 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.676055+0000", 2024-09-21T22:08:01.986 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00060247000000000002 2024-09-21T22:08:01.986 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.986 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.986 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:01.986 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.676178+0000", 2024-09-21T22:08:01.986 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000122395 2024-09-21T22:08:01.987 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.987 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.987 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:01.987 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.676283+0000", 2024-09-21T22:08:01.987 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000105653 2024-09-21T22:08:01.987 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:01.987 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:01.987 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:01.987 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.987 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.987 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5207.0:39 2.7 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head [write 917504~131072] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e78)", 2024-09-21T22:08:01.987 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:52.589499+0000", 2024-09-21T22:08:01.987 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.3174255949999996, 2024-09-21T22:08:01.987 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.087365371999999997, 2024-09-21T22:08:01.988 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:01.988 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:01.988 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:01.988 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:01.988 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5207", 2024-09-21T22:08:01.988 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/357731919", 2024-09-21T22:08:01.988 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 39 2024-09-21T22:08:01.988 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.988 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:01.988 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.988 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:01.988 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.589499+0000", 2024-09-21T22:08:01.988 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:01.988 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.988 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.989 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:01.989 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.589499+0000", 2024-09-21T22:08:01.989 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:01.989 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.989 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.989 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:01.989 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.589500+0000", 2024-09-21T22:08:01.989 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.547e-06 2024-09-21T22:08:01.989 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.989 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.989 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:01.989 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.589752+0000", 2024-09-21T22:08:01.989 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025112799999999998 2024-09-21T22:08:01.989 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.990 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.990 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:01.990 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.589783+0000", 2024-09-21T22:08:01.990 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1557000000000003e-05 2024-09-21T22:08:01.990 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.990 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.990 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:01.990 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.589832+0000", 2024-09-21T22:08:01.990 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9175999999999998e-05 2024-09-21T22:08:01.990 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.990 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.990 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:01.990 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.665028+0000", 2024-09-21T22:08:01.990 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.075195293999999996 2024-09-21T22:08:01.991 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.991 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.991 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:01.991 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.665169+0000", 2024-09-21T22:08:01.991 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000141299 2024-09-21T22:08:01.991 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.991 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.991 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-09-21T22:08:01.991 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.665464+0000", 2024-09-21T22:08:01.991 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00029459899999999999 2024-09-21T22:08:01.991 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.991 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.991 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:01.991 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.676551+0000", 2024-09-21T22:08:01.992 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011087107000000001 2024-09-21T22:08:01.992 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.992 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.992 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:01.992 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.676700+0000", 2024-09-21T22:08:01.992 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000149853 2024-09-21T22:08:01.992 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.992 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.992 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:01.992 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.676802+0000", 2024-09-21T22:08:01.992 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010194199999999999 2024-09-21T22:08:01.992 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.992 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.992 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:01.993 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.676864+0000", 2024-09-21T22:08:01.993 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1870000000000002e-05 2024-09-21T22:08:01.993 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:01.993 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:01.993 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:01.993 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.993 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.993 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5207.0:74 2.7 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head [write 524288~131072] snapc 5=[5,3] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-21T22:08:01.993 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:52.938124+0000", 2024-09-21T22:08:01.993 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.9688003829999996, 2024-09-21T22:08:01.993 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.071435100000000001, 2024-09-21T22:08:01.993 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:01.993 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:01.993 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:01.994 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:01.994 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5207", 2024-09-21T22:08:01.994 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/357731919", 2024-09-21T22:08:01.994 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 74 2024-09-21T22:08:01.994 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.994 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:01.994 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.994 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:01.994 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938124+0000", 2024-09-21T22:08:01.994 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:01.994 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.994 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.994 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:01.994 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938124+0000", 2024-09-21T22:08:01.995 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:01.995 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.995 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.995 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:01.995 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938125+0000", 2024-09-21T22:08:01.995 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5600000000000005e-07 2024-09-21T22:08:01.995 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.995 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.995 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:01.995 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938218+0000", 2024-09-21T22:08:01.995 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3239999999999995e-05 2024-09-21T22:08:01.995 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.995 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.995 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:01.996 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938231+0000", 2024-09-21T22:08:01.996 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.272e-05 2024-09-21T22:08:01.996 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.996 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.996 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:01.996 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938239+0000", 2024-09-21T22:08:01.996 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5059999999999995e-06 2024-09-21T22:08:01.996 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.996 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.996 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:01.996 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.983312+0000", 2024-09-21T22:08:01.996 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.045072252 2024-09-21T22:08:01.996 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.996 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.997 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:01.997 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.983458+0000", 2024-09-21T22:08:01.997 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014628899999999999 2024-09-21T22:08:01.997 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.997 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.997 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-09-21T22:08:01.997 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.983682+0000", 2024-09-21T22:08:01.997 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022402899999999999 2024-09-21T22:08:01.997 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.997 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.997 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:01.997 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.993679+0000", 2024-09-21T22:08:01.997 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0099972740000000004 2024-09-21T22:08:01.998 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.998 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.998 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:01.998 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.009436+0000", 2024-09-21T22:08:01.998 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015756867000000001 2024-09-21T22:08:01.998 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.998 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.998 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:01.998 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.009506+0000", 2024-09-21T22:08:01.998 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0277999999999994e-05 2024-09-21T22:08:01.998 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.998 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.998 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:01.998 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.009559+0000", 2024-09-21T22:08:01.999 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2889000000000001e-05 2024-09-21T22:08:01.999 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:01.999 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:01.999 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:01.999 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:01.999 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:01.999 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5207.0:75 2.7 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head [write 655360~131072] snapc 5=[5,3] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-21T22:08:01.999 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:52.938253+0000", 2024-09-21T22:08:01.999 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.9686713269999991, 2024-09-21T22:08:01.999 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082064041000000004, 2024-09-21T22:08:01.999 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:01.999 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:01.999 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:01.999 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:01.999 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5207", 2024-09-21T22:08:02.000 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/357731919", 2024-09-21T22:08:02.000 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 75 2024-09-21T22:08:02.000 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.000 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:02.000 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.000 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:02.000 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938253+0000", 2024-09-21T22:08:02.000 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:02.000 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.000 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.000 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:02.000 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938253+0000", 2024-09-21T22:08:02.000 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:02.000 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.000 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.001 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:02.001 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938254+0000", 2024-09-21T22:08:02.001 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8599999999999998e-07 2024-09-21T22:08:02.001 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.001 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.001 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:02.001 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938354+0000", 2024-09-21T22:08:02.001 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9696000000000002e-05 2024-09-21T22:08:02.001 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.001 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.001 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:02.001 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938381+0000", 2024-09-21T22:08:02.001 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.705e-05 2024-09-21T22:08:02.001 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.002 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.002 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:02.002 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938397+0000", 2024-09-21T22:08:02.002 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5996999999999999e-05 2024-09-21T22:08:02.002 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.002 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.002 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:02.002 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.993739+0000", 2024-09-21T22:08:02.002 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.055342823999999999 2024-09-21T22:08:02.002 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.002 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.002 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:02.002 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.993835+0000", 2024-09-21T22:08:02.002 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5919000000000001e-05 2024-09-21T22:08:02.002 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.003 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.003 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-09-21T22:08:02.003 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.993993+0000", 2024-09-21T22:08:02.003 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015729900000000001 2024-09-21T22:08:02.003 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.003 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.003 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:02.003 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.009579+0000", 2024-09-21T22:08:02.003 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015586723 2024-09-21T22:08:02.003 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.003 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.003 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:02.003 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.020134+0000", 2024-09-21T22:08:02.003 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010554908 2024-09-21T22:08:02.003 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.004 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.004 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:02.004 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.020241+0000", 2024-09-21T22:08:02.004 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000106246 2024-09-21T22:08:02.004 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.004 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.004 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:02.004 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.020317+0000", 2024-09-21T22:08:02.004 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6693000000000006e-05 2024-09-21T22:08:02.004 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:02.004 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:02.004 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:02.004 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.004 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.004 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5207.0:76 2.7 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head [write 786432~131072] snapc 5=[5,3] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-21T22:08:02.005 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:52.938410+0000", 2024-09-21T22:08:02.005 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.9685141579999996, 2024-09-21T22:08:02.005 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.086926370000000003, 2024-09-21T22:08:02.005 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:02.005 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:02.005 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:02.005 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:02.005 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5207", 2024-09-21T22:08:02.005 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/357731919", 2024-09-21T22:08:02.005 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 76 2024-09-21T22:08:02.005 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.005 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:02.005 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.005 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:02.006 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938410+0000", 2024-09-21T22:08:02.006 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:02.006 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.006 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.006 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:02.006 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938410+0000", 2024-09-21T22:08:02.006 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:02.006 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.006 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.006 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:02.006 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938411+0000", 2024-09-21T22:08:02.006 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0299999999999998e-07 2024-09-21T22:08:02.006 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.006 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.007 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:02.007 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938502+0000", 2024-09-21T22:08:02.007 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0562000000000005e-05 2024-09-21T22:08:02.007 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.007 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.007 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:02.007 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938514+0000", 2024-09-21T22:08:02.007 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2327000000000001e-05 2024-09-21T22:08:02.007 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.007 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.007 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:02.007 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938522+0000", 2024-09-21T22:08:02.007 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1170000000000004e-06 2024-09-21T22:08:02.007 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.008 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:02.008 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.001427+0000", 2024-09-21T22:08:02.008 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.062904871000000001 2024-09-21T22:08:02.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.008 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:02.008 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.001519+0000", 2024-09-21T22:08:02.008 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1798000000000004e-05 2024-09-21T22:08:02.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.008 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-09-21T22:08:02.008 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.001671+0000", 2024-09-21T22:08:02.008 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015195900000000001 2024-09-21T22:08:02.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.009 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.009 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:02.009 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.020388+0000", 2024-09-21T22:08:02.009 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018717428000000001 2024-09-21T22:08:02.009 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.009 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.009 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:02.009 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.025041+0000", 2024-09-21T22:08:02.009 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0046524039999999997 2024-09-21T22:08:02.009 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.009 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.009 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:02.009 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.025213+0000", 2024-09-21T22:08:02.009 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017286499999999999 2024-09-21T22:08:02.009 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.010 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.010 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:02.010 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.025337+0000", 2024-09-21T22:08:02.010 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012333600000000001 2024-09-21T22:08:02.010 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:02.010 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:02.010 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:02.010 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.010 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.010 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5207.0:77 2.7 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head [write 917504~131072] snapc 5=[5,3] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-21T22:08:02.010 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:52.938534+0000", 2024-09-21T22:08:02.010 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.9683903429999994, 2024-09-21T22:08:02.010 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.087709436000000002, 2024-09-21T22:08:02.010 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:02.011 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:02.011 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:02.011 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:02.011 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5207", 2024-09-21T22:08:02.011 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/357731919", 2024-09-21T22:08:02.011 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 77 2024-09-21T22:08:02.011 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.011 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:02.011 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.011 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:02.011 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938534+0000", 2024-09-21T22:08:02.011 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:02.011 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.011 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.011 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:02.012 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938534+0000", 2024-09-21T22:08:02.012 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:02.012 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.012 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:02.012 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938535+0000", 2024-09-21T22:08:02.012 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1399999999999996e-07 2024-09-21T22:08:02.012 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.012 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:02.012 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938623+0000", 2024-09-21T22:08:02.012 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8226000000000005e-05 2024-09-21T22:08:02.012 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.012 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:02.013 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938645+0000", 2024-09-21T22:08:02.013 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1948999999999999e-05 2024-09-21T22:08:02.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.013 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.013 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:02.013 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938660+0000", 2024-09-21T22:08:02.013 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4776999999999999e-05 2024-09-21T22:08:02.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.013 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.013 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:02.013 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.009669+0000", 2024-09-21T22:08:02.013 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.071009116999999997 2024-09-21T22:08:02.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.014 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.014 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:02.014 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.009949+0000", 2024-09-21T22:08:02.014 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027962899999999999 2024-09-21T22:08:02.014 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.014 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.014 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-09-21T22:08:02.014 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.010355+0000", 2024-09-21T22:08:02.014 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000406334 2024-09-21T22:08:02.014 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.014 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.014 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:02.014 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.020875+0000", 2024-09-21T22:08:02.014 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010520144 2024-09-21T22:08:02.015 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.015 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.015 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:02.015 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.026086+0000", 2024-09-21T22:08:02.015 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0052107569999999999 2024-09-21T22:08:02.015 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.015 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.015 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:02.015 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.026178+0000", 2024-09-21T22:08:02.015 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2113000000000004e-05 2024-09-21T22:08:02.015 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.015 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.015 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:02.015 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.026244+0000", 2024-09-21T22:08:02.015 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5675999999999999e-05 2024-09-21T22:08:02.016 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:02.016 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:02.016 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:02.016 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.016 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.016 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5207.0:93 2.7 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head [write 131072~917504] snapc 5=[5,3] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-21T22:08:02.016 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:53.147757+0000", 2024-09-21T22:08:02.016 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.7591671650000009, 2024-09-21T22:08:02.016 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.062493043999999998, 2024-09-21T22:08:02.016 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:02.016 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:02.016 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:02.016 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:02.016 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5207", 2024-09-21T22:08:02.016 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/357731919", 2024-09-21T22:08:02.017 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 93 2024-09-21T22:08:02.017 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.017 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:02.017 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.017 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:02.017 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.147757+0000", 2024-09-21T22:08:02.017 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:02.017 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.017 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.017 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:02.017 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.147757+0000", 2024-09-21T22:08:02.017 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:02.017 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.017 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.018 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:02.018 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.147759+0000", 2024-09-21T22:08:02.018 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0159999999999998e-06 2024-09-21T22:08:02.018 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.018 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.018 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:02.018 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.149009+0000", 2024-09-21T22:08:02.018 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001250082 2024-09-21T22:08:02.018 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.018 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.018 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:02.018 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.149042+0000", 2024-09-21T22:08:02.018 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2617000000000003e-05 2024-09-21T22:08:02.018 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.019 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.019 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:02.019 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.149058+0000", 2024-09-21T22:08:02.019 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6113e-05 2024-09-21T22:08:02.019 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.019 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.019 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:02.019 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.149237+0000", 2024-09-21T22:08:02.019 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017834900000000001 2024-09-21T22:08:02.019 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.019 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.019 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:02.019 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.149378+0000", 2024-09-21T22:08:02.019 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014099199999999999 2024-09-21T22:08:02.020 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.020 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.020 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-09-21T22:08:02.020 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.149582+0000", 2024-09-21T22:08:02.020 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020414900000000001 2024-09-21T22:08:02.020 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.020 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.020 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:02.020 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.208648+0000", 2024-09-21T22:08:02.020 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.059066397999999999 2024-09-21T22:08:02.020 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.020 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.020 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:02.020 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.210125+0000", 2024-09-21T22:08:02.021 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001477119 2024-09-21T22:08:02.021 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.021 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.021 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:02.021 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.210200+0000", 2024-09-21T22:08:02.021 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4973000000000002e-05 2024-09-21T22:08:02.021 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.021 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.021 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:02.021 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.210250+0000", 2024-09-21T22:08:02.021 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0235999999999998e-05 2024-09-21T22:08:02.021 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:02.021 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:02.022 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:02.022 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.022 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.022 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5207.0:128 2.7 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head [write 131072~917504] snapc 6=[6,5,3] ondisk+write+known_if_redirected+supports_pool_eio e83)", 2024-09-21T22:08:02.022 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:08:01.056777+0000", 2024-09-21T22:08:02.022 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.85014769499999998, 2024-09-21T22:08:02.022 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.062294929999999998, 2024-09-21T22:08:02.022 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:02.022 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:02.022 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:02.022 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:02.022 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5207", 2024-09-21T22:08:02.022 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/357731919", 2024-09-21T22:08:02.022 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 128 2024-09-21T22:08:02.023 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.023 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:02.023 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.023 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:02.023 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.056777+0000", 2024-09-21T22:08:02.023 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:02.023 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.023 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.023 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:02.023 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.056777+0000", 2024-09-21T22:08:02.023 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:02.023 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.023 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.023 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:02.023 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.056779+0000", 2024-09-21T22:08:02.024 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4669999999999998e-06 2024-09-21T22:08:02.024 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.024 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.024 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:02.024 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.058094+0000", 2024-09-21T22:08:02.024 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001314602 2024-09-21T22:08:02.024 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.024 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.024 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:02.024 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.058126+0000", 2024-09-21T22:08:02.024 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1625999999999998e-05 2024-09-21T22:08:02.024 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.024 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.024 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:02.025 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.058141+0000", 2024-09-21T22:08:02.025 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5274e-05 2024-09-21T22:08:02.025 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.025 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.025 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:02.025 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.058267+0000", 2024-09-21T22:08:02.025 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000126682 2024-09-21T22:08:02.025 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.025 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.025 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:02.025 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.058420+0000", 2024-09-21T22:08:02.025 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015212599999999999 2024-09-21T22:08:02.025 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.025 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.026 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-09-21T22:08:02.026 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.058644+0000", 2024-09-21T22:08:02.026 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000224749 2024-09-21T22:08:02.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.026 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.026 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:02.026 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.117883+0000", 2024-09-21T22:08:02.026 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.059238617 2024-09-21T22:08:02.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.026 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.026 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:02.026 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.118942+0000", 2024-09-21T22:08:02.026 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0010588780000000001 2024-09-21T22:08:02.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.027 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.027 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:02.027 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.119011+0000", 2024-09-21T22:08:02.027 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8802999999999998e-05 2024-09-21T22:08:02.027 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.027 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.027 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:02.027 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.119072+0000", 2024-09-21T22:08:02.027 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1105999999999994e-05 2024-09-21T22:08:02.027 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:02.027 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:02.027 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:02.027 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:02.027 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:02.027 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:08:02.028 INFO:teuthology.orchestra.run.smithi039.stderr:noscrub is set 2024-09-21T22:08:02.029 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T22:08:02.030 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:08:02.024+0000 7f2075a06640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31720) UID: 0 2024-09-21T22:08:02.130 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:08:02.128+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:08:02.202 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:08:02.202 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:08:02.202 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-09-21T22:08:02.202 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:08:02.216 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T22:08:02.231 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:08:02.225+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:08:02.332 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:08:02.328+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:08:02.383 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:08:02.383 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:08:02.383 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-21T22:08:02.383 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-21T22:08:02.383 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:08:02.397 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T22:08:02.442 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:08:02.440+0000 7f2075a06640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31720) UID: 0 2024-09-21T22:08:02.534 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:08:02.529+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:08:02.572 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:08:02.573 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-21T22:08:02.573 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-21T22:08:02.573 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-21T22:08:02.573 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.573 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4840.0:676 2.7 e63/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 63'57, mlcod=63'55)", 2024-09-21T22:08:02.573 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.627276+0000", 2024-09-21T22:08:02.573 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 37.943564348000002, 2024-09-21T22:08:02.573 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.037500787000000001, 2024-09-21T22:08:02.573 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:02.573 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:02.573 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:02.573 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:02.574 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.574 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:02.574 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.627276+0000", 2024-09-21T22:08:02.574 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:02.574 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.574 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.574 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:02.574 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.627276+0000", 2024-09-21T22:08:02.574 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:02.574 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.574 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.574 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:02.574 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.627277+0000", 2024-09-21T22:08:02.575 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.068e-06 2024-09-21T22:08:02.575 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.575 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.575 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:02.575 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.628148+0000", 2024-09-21T22:08:02.575 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00087069999999999997 2024-09-21T22:08:02.575 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.575 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.575 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:02.575 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.628170+0000", 2024-09-21T22:08:02.575 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2147999999999999e-05 2024-09-21T22:08:02.575 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.575 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.575 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:02.575 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.628179+0000", 2024-09-21T22:08:02.576 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2450000000000004e-06 2024-09-21T22:08:02.576 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.576 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.576 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:02.576 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.628279+0000", 2024-09-21T22:08:02.576 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9298999999999999e-05 2024-09-21T22:08:02.576 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.576 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.576 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:02.576 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.628305+0000", 2024-09-21T22:08:02.576 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6316999999999999e-05 2024-09-21T22:08:02.576 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.576 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.576 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:02.577 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.664714+0000", 2024-09-21T22:08:02.577 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.036408923000000003 2024-09-21T22:08:02.577 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.577 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.577 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:02.577 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.664777+0000", 2024-09-21T22:08:02.577 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3087000000000001e-05 2024-09-21T22:08:02.577 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:02.577 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:02.577 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:02.577 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.577 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.578 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4840.0:678 2.7 e63/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 63'59, mlcod=63'57)", 2024-09-21T22:08:02.578 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.679689+0000", 2024-09-21T22:08:02.578 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 37.891151848, 2024-09-21T22:08:02.578 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053308406000000003, 2024-09-21T22:08:02.578 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:02.578 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:02.578 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:02.578 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:02.578 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.578 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:02.578 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.679689+0000", 2024-09-21T22:08:02.578 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:02.578 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.578 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.579 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:02.579 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.679689+0000", 2024-09-21T22:08:02.579 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:02.579 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.579 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.579 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:02.579 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.679691+0000", 2024-09-21T22:08:02.579 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9400000000000001e-06 2024-09-21T22:08:02.579 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.579 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.580 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:02.580 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.680245+0000", 2024-09-21T22:08:02.580 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00055382999999999995 2024-09-21T22:08:02.580 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.580 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.580 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:02.580 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.680276+0000", 2024-09-21T22:08:02.580 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1123e-05 2024-09-21T22:08:02.580 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.580 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.580 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:02.581 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.680290+0000", 2024-09-21T22:08:02.581 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4056999999999999e-05 2024-09-21T22:08:02.581 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.581 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.581 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:02.581 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.680418+0000", 2024-09-21T22:08:02.581 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000127792 2024-09-21T22:08:02.581 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.581 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.581 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:02.581 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.680465+0000", 2024-09-21T22:08:02.581 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7360000000000001e-05 2024-09-21T22:08:02.581 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.581 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.582 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:02.582 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.732923+0000", 2024-09-21T22:08:02.582 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.052458312999999999 2024-09-21T22:08:02.582 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.582 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.582 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:02.582 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.732997+0000", 2024-09-21T22:08:02.582 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3991000000000004e-05 2024-09-21T22:08:02.582 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:02.582 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:02.582 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:02.582 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.582 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.582 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4840.0:680 2.7 e63/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 63'61, mlcod=63'59)", 2024-09-21T22:08:02.583 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.745996+0000", 2024-09-21T22:08:02.583 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 37.824844335000002, 2024-09-21T22:08:02.583 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.061804132999999997, 2024-09-21T22:08:02.583 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:02.583 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:02.583 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:02.583 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:02.583 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.583 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:02.583 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.745996+0000", 2024-09-21T22:08:02.583 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:02.583 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.583 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.583 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:02.584 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.745996+0000", 2024-09-21T22:08:02.584 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:02.584 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.584 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.584 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:02.584 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.745998+0000", 2024-09-21T22:08:02.584 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.717e-06 2024-09-21T22:08:02.584 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.584 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.584 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:02.584 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.746523+0000", 2024-09-21T22:08:02.584 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00052525399999999996 2024-09-21T22:08:02.584 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.584 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.584 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:02.585 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.746554+0000", 2024-09-21T22:08:02.585 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0586000000000002e-05 2024-09-21T22:08:02.585 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.585 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.585 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:02.585 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.746577+0000", 2024-09-21T22:08:02.585 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3079e-05 2024-09-21T22:08:02.585 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.585 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.585 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:02.585 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.746689+0000", 2024-09-21T22:08:02.585 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011177 2024-09-21T22:08:02.585 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.585 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.585 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:02.586 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.746722+0000", 2024-09-21T22:08:02.586 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2870000000000002e-05 2024-09-21T22:08:02.586 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.586 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.586 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:02.586 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.807740+0000", 2024-09-21T22:08:02.586 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.061018612 2024-09-21T22:08:02.586 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.586 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.586 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:02.586 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.807800+0000", 2024-09-21T22:08:02.586 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0244999999999997e-05 2024-09-21T22:08:02.586 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:02.586 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:02.587 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:02.587 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.587 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.587 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4840.0:682 2.7 e63/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 63'63, mlcod=63'61)", 2024-09-21T22:08:02.587 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.824126+0000", 2024-09-21T22:08:02.587 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 37.746714765999997, 2024-09-21T22:08:02.587 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.038159327, 2024-09-21T22:08:02.587 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:02.587 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:02.587 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:02.587 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:02.587 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.587 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:02.587 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.824126+0000", 2024-09-21T22:08:02.587 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:02.588 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.588 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.588 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:02.588 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.824126+0000", 2024-09-21T22:08:02.588 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:02.588 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.588 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.588 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:02.588 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.824128+0000", 2024-09-21T22:08:02.588 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9659999999999999e-06 2024-09-21T22:08:02.588 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.588 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.588 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:02.588 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.824842+0000", 2024-09-21T22:08:02.588 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00071423999999999999 2024-09-21T22:08:02.589 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.589 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.589 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:02.589 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.824883+0000", 2024-09-21T22:08:02.589 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1131999999999999e-05 2024-09-21T22:08:02.589 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.589 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.589 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:02.589 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.824931+0000", 2024-09-21T22:08:02.589 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7327000000000003e-05 2024-09-21T22:08:02.590 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.590 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.590 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:02.590 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.825058+0000", 2024-09-21T22:08:02.590 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012790499999999999 2024-09-21T22:08:02.590 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.590 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.590 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:02.590 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.825087+0000", 2024-09-21T22:08:02.590 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8192e-05 2024-09-21T22:08:02.590 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.590 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.590 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:02.590 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.862225+0000", 2024-09-21T22:08:02.591 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.037138309000000001 2024-09-21T22:08:02.591 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.591 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.591 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:02.591 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.862285+0000", 2024-09-21T22:08:02.591 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0256e-05 2024-09-21T22:08:02.591 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:02.591 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:02.591 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:02.591 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.591 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.591 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5008.0:116 2.7 e71/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 71'148, mlcod=71'146)", 2024-09-21T22:08:02.591 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:37.867612+0000", 2024-09-21T22:08:02.591 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 24.703228271, 2024-09-21T22:08:02.592 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.045123155999999998, 2024-09-21T22:08:02.592 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:02.592 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:02.592 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:02.592 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:02.592 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.592 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:02.592 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.867612+0000", 2024-09-21T22:08:02.592 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:02.592 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.592 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.592 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:02.592 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.867612+0000", 2024-09-21T22:08:02.592 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:02.592 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.593 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.593 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:02.593 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.867613+0000", 2024-09-21T22:08:02.593 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1200000000000002e-07 2024-09-21T22:08:02.593 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.593 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.593 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:02.593 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.868135+0000", 2024-09-21T22:08:02.593 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00052132700000000003 2024-09-21T22:08:02.593 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.593 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.593 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:02.593 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.868150+0000", 2024-09-21T22:08:02.593 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5271e-05 2024-09-21T22:08:02.593 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.593 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.594 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:02.594 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.868156+0000", 2024-09-21T22:08:02.594 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4969999999999998e-06 2024-09-21T22:08:02.594 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.594 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.594 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:02.594 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.868213+0000", 2024-09-21T22:08:02.594 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7114999999999998e-05 2024-09-21T22:08:02.594 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.594 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.594 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:02.594 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.868238+0000", 2024-09-21T22:08:02.594 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4987999999999999e-05 2024-09-21T22:08:02.595 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.595 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.595 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:02.595 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.912677+0000", 2024-09-21T22:08:02.595 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.044439019000000003 2024-09-21T22:08:02.595 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.595 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.595 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:02.595 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.912736+0000", 2024-09-21T22:08:02.595 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8127000000000001e-05 2024-09-21T22:08:02.595 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:02.595 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:02.595 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:02.595 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.595 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.596 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5008.0:118 2.7 e71/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 71'150, mlcod=71'148)", 2024-09-21T22:08:02.596 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:37.924700+0000", 2024-09-21T22:08:02.596 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 24.64614108, 2024-09-21T22:08:02.596 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.034650561000000003, 2024-09-21T22:08:02.596 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:02.596 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:02.596 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:02.596 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:02.596 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.596 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:02.596 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.924700+0000", 2024-09-21T22:08:02.596 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:02.596 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.596 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.597 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:02.597 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.924700+0000", 2024-09-21T22:08:02.597 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:02.597 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.597 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.597 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:02.597 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.924702+0000", 2024-09-21T22:08:02.597 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9460000000000001e-06 2024-09-21T22:08:02.597 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.597 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.597 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:02.597 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.925132+0000", 2024-09-21T22:08:02.597 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00043021200000000002 2024-09-21T22:08:02.597 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.597 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.598 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:02.598 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.925156+0000", 2024-09-21T22:08:02.598 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4451999999999999e-05 2024-09-21T22:08:02.598 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.598 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.598 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:02.598 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.925168+0000", 2024-09-21T22:08:02.598 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2249000000000001e-05 2024-09-21T22:08:02.598 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.598 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.598 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:02.598 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.925241+0000", 2024-09-21T22:08:02.598 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2502000000000005e-05 2024-09-21T22:08:02.598 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.599 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.599 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:02.599 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.925267+0000", 2024-09-21T22:08:02.599 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6471000000000001e-05 2024-09-21T22:08:02.599 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.599 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.599 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:02.599 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.959217+0000", 2024-09-21T22:08:02.599 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.033949569999999998 2024-09-21T22:08:02.599 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.599 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.599 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:02.599 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.959350+0000", 2024-09-21T22:08:02.600 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013315899999999999 2024-09-21T22:08:02.600 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:02.600 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:02.600 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:02.600 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.600 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.600 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5008.0:120 2.7 e71/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 71'152, mlcod=71'150)", 2024-09-21T22:08:02.600 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:37.970837+0000", 2024-09-21T22:08:02.600 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 24.600003828999998, 2024-09-21T22:08:02.600 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.03044943, 2024-09-21T22:08:02.600 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:02.600 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:02.600 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:02.600 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:02.600 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.601 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:02.601 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.970837+0000", 2024-09-21T22:08:02.601 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:02.601 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.601 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.601 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:02.601 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.970837+0000", 2024-09-21T22:08:02.601 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:02.601 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.601 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.601 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:02.601 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.970838+0000", 2024-09-21T22:08:02.601 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4649999999999999e-06 2024-09-21T22:08:02.601 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.602 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.602 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:02.602 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.971265+0000", 2024-09-21T22:08:02.602 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00042665800000000001 2024-09-21T22:08:02.602 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.602 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.602 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:02.602 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.971285+0000", 2024-09-21T22:08:02.602 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9639999999999998e-05 2024-09-21T22:08:02.602 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.602 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.602 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:02.602 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.971294+0000", 2024-09-21T22:08:02.602 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7210000000000006e-06 2024-09-21T22:08:02.603 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.603 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.603 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:02.603 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.971354+0000", 2024-09-21T22:08:02.603 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9613e-05 2024-09-21T22:08:02.603 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.603 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.603 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:02.603 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.971381+0000", 2024-09-21T22:08:02.603 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7538999999999998e-05 2024-09-21T22:08:02.603 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.603 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.603 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:02.603 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.001199+0000", 2024-09-21T22:08:02.603 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029817537000000002 2024-09-21T22:08:02.604 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.604 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.604 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:02.604 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.001286+0000", 2024-09-21T22:08:02.604 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7256999999999995e-05 2024-09-21T22:08:02.604 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:02.604 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:02.604 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:02.604 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.604 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.604 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5008.0:122 2.7 e71/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 71'154, mlcod=71'152)", 2024-09-21T22:08:02.604 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:38.014709+0000", 2024-09-21T22:08:02.604 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 24.556131954000001, 2024-09-21T22:08:02.604 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029567408999999999, 2024-09-21T22:08:02.605 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:02.605 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:02.605 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:02.605 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:02.605 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.605 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:02.605 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.014709+0000", 2024-09-21T22:08:02.605 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:02.605 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.605 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.605 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:02.605 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.014709+0000", 2024-09-21T22:08:02.605 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:02.605 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.605 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.606 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:02.606 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.014709+0000", 2024-09-21T22:08:02.606 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9599999999999999e-07 2024-09-21T22:08:02.606 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.606 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.606 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:02.606 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.015201+0000", 2024-09-21T22:08:02.606 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00049175599999999999 2024-09-21T22:08:02.606 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.606 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.606 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:02.606 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.015215+0000", 2024-09-21T22:08:02.606 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3692999999999999e-05 2024-09-21T22:08:02.606 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.607 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.607 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:02.607 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.015225+0000", 2024-09-21T22:08:02.607 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.815e-06 2024-09-21T22:08:02.607 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.607 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.607 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:02.607 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.015287+0000", 2024-09-21T22:08:02.607 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1858000000000003e-05 2024-09-21T22:08:02.607 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.607 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.607 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:02.607 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.015315+0000", 2024-09-21T22:08:02.607 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8490999999999999e-05 2024-09-21T22:08:02.607 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.608 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.608 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:02.608 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.044182+0000", 2024-09-21T22:08:02.608 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.028866936999999999 2024-09-21T22:08:02.608 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.608 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.608 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:02.610 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.044276+0000", 2024-09-21T22:08:02.610 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4163e-05 2024-09-21T22:08:02.610 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:02.610 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:02.610 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:02.610 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.610 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.610 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5207.0:33 2.7 e78/16 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head v 78'174, mlcod=78'170)", 2024-09-21T22:08:02.611 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:52.604638+0000", 2024-09-21T22:08:02.611 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.9662029919999995, 2024-09-21T22:08:02.611 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020363816, 2024-09-21T22:08:02.611 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:02.611 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:02.611 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:02.611 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:02.611 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.611 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:02.611 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.604638+0000", 2024-09-21T22:08:02.611 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:02.611 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.612 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.612 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:02.612 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.604638+0000", 2024-09-21T22:08:02.612 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:02.612 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.612 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.612 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:02.612 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.604640+0000", 2024-09-21T22:08:02.612 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8929999999999999e-06 2024-09-21T22:08:02.612 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.612 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.613 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:02.613 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.604719+0000", 2024-09-21T22:08:02.613 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9116999999999995e-05 2024-09-21T22:08:02.613 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.613 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.613 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:02.613 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.604751+0000", 2024-09-21T22:08:02.613 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2316e-05 2024-09-21T22:08:02.613 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.613 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.613 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:02.613 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.604770+0000", 2024-09-21T22:08:02.613 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9406e-05 2024-09-21T22:08:02.614 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.614 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.614 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:02.614 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.604930+0000", 2024-09-21T22:08:02.614 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015912599999999999 2024-09-21T22:08:02.614 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.614 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.614 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:02.614 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.604981+0000", 2024-09-21T22:08:02.614 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1010000000000001e-05 2024-09-21T22:08:02.614 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.614 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.615 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:02.615 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.624865+0000", 2024-09-21T22:08:02.615 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019884545999999999 2024-09-21T22:08:02.615 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.615 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.615 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:02.615 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.625001+0000", 2024-09-21T22:08:02.615 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013640200000000001 2024-09-21T22:08:02.615 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:02.615 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:02.615 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:02.615 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.615 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.616 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5207.0:34 2.7 e78/16 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head v 78'175, mlcod=78'170)", 2024-09-21T22:08:02.616 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:52.614890+0000", 2024-09-21T22:08:02.616 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.9559503859999996, 2024-09-21T22:08:02.616 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019973422000000001, 2024-09-21T22:08:02.616 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:02.616 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:02.616 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:02.616 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:02.616 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.616 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:02.616 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.614890+0000", 2024-09-21T22:08:02.616 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:02.617 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.617 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.617 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:02.617 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.614890+0000", 2024-09-21T22:08:02.617 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:02.617 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.617 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.617 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:02.617 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.614892+0000", 2024-09-21T22:08:02.617 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8899999999999999e-06 2024-09-21T22:08:02.617 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.617 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.618 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:02.618 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.615033+0000", 2024-09-21T22:08:02.618 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014115199999999999 2024-09-21T22:08:02.618 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.618 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.618 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:02.618 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.615069+0000", 2024-09-21T22:08:02.618 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5892999999999997e-05 2024-09-21T22:08:02.618 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.618 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.618 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:02.619 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.615090+0000", 2024-09-21T22:08:02.619 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.037e-05 2024-09-21T22:08:02.619 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.619 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.619 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:02.619 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.615217+0000", 2024-09-21T22:08:02.619 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012768299999999999 2024-09-21T22:08:02.619 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.619 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.619 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:02.619 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.615270+0000", 2024-09-21T22:08:02.619 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3050000000000002e-05 2024-09-21T22:08:02.619 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.620 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.620 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:02.620 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.634730+0000", 2024-09-21T22:08:02.620 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019459860999999998 2024-09-21T22:08:02.620 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.620 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.620 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:02.620 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.634864+0000", 2024-09-21T22:08:02.620 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000133523 2024-09-21T22:08:02.620 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:02.620 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:02.620 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:02.621 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.621 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.621 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5207.0:35 2.7 e78/16 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head v 78'176, mlcod=78'170)", 2024-09-21T22:08:02.621 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:52.624606+0000", 2024-09-21T22:08:02.621 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.94623451, 2024-09-21T22:08:02.621 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020065888, 2024-09-21T22:08:02.621 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:02.621 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:02.621 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:02.621 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:02.621 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.621 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:02.621 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.624606+0000", 2024-09-21T22:08:02.622 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:02.622 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.622 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.622 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:02.622 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.624606+0000", 2024-09-21T22:08:02.622 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:02.622 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.622 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.622 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:02.622 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.624608+0000", 2024-09-21T22:08:02.622 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7600000000000001e-06 2024-09-21T22:08:02.622 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.623 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.623 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:02.623 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.624802+0000", 2024-09-21T22:08:02.623 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019425900000000001 2024-09-21T22:08:02.623 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.623 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.623 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:02.623 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.624860+0000", 2024-09-21T22:08:02.623 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7886000000000001e-05 2024-09-21T22:08:02.623 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.623 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.623 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:02.624 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.624882+0000", 2024-09-21T22:08:02.624 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2277e-05 2024-09-21T22:08:02.624 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.624 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.624 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:02.624 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.625081+0000", 2024-09-21T22:08:02.624 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019864500000000001 2024-09-21T22:08:02.624 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.624 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.624 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:02.624 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.625131+0000", 2024-09-21T22:08:02.624 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0120000000000001e-05 2024-09-21T22:08:02.624 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.625 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.625 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:02.625 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.644538+0000", 2024-09-21T22:08:02.625 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019406667999999998 2024-09-21T22:08:02.625 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.625 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.625 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:02.625 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.644672+0000", 2024-09-21T22:08:02.625 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013427299999999999 2024-09-21T22:08:02.625 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:02.625 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:02.625 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:02.626 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.626 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.626 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5207.0:36 2.7 e78/16 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head v 78'177, mlcod=78'171)", 2024-09-21T22:08:02.626 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:52.634570+0000", 2024-09-21T22:08:02.626 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.9362706460000005, 2024-09-21T22:08:02.626 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020665352000000001, 2024-09-21T22:08:02.626 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:02.626 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:02.626 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:02.626 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:02.626 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.626 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:02.626 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.634570+0000", 2024-09-21T22:08:02.627 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:02.627 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.627 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.627 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:02.627 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.634570+0000", 2024-09-21T22:08:02.627 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:02.627 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.627 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.627 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:02.627 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.634572+0000", 2024-09-21T22:08:02.627 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7439999999999999e-06 2024-09-21T22:08:02.627 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.628 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.628 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:02.628 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.634715+0000", 2024-09-21T22:08:02.628 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014291600000000001 2024-09-21T22:08:02.628 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.628 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.628 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:02.628 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.634739+0000", 2024-09-21T22:08:02.628 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4615999999999999e-05 2024-09-21T22:08:02.628 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.628 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.628 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:02.629 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.634752+0000", 2024-09-21T22:08:02.629 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.27e-05 2024-09-21T22:08:02.629 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.629 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.629 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:02.629 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.634953+0000", 2024-09-21T22:08:02.629 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020067500000000001 2024-09-21T22:08:02.629 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.629 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.629 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:02.629 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.635008+0000", 2024-09-21T22:08:02.630 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5504000000000003e-05 2024-09-21T22:08:02.630 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.630 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.630 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:02.630 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.655111+0000", 2024-09-21T22:08:02.630 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.02010323 2024-09-21T22:08:02.630 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.630 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.630 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:02.630 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.655235+0000", 2024-09-21T22:08:02.630 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000123967 2024-09-21T22:08:02.630 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:02.631 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:02.631 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:02.631 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.631 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.631 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5207.0:37 2.7 e78/16 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head v 78'178, mlcod=78'174)", 2024-09-21T22:08:02.631 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:52.645132+0000", 2024-09-21T22:08:02.631 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.9257086280000006, 2024-09-21T22:08:02.631 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020185980999999999, 2024-09-21T22:08:02.631 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:02.631 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:02.631 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:02.631 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:02.632 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.632 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:02.632 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.645132+0000", 2024-09-21T22:08:02.632 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:02.632 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.632 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.632 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:02.632 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.645132+0000", 2024-09-21T22:08:02.632 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:02.632 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.632 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.632 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:02.633 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.645134+0000", 2024-09-21T22:08:02.633 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1770000000000001e-06 2024-09-21T22:08:02.633 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.633 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.633 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:02.633 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.645259+0000", 2024-09-21T22:08:02.633 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012473900000000001 2024-09-21T22:08:02.633 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.633 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.633 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:02.633 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.645296+0000", 2024-09-21T22:08:02.633 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6572999999999997e-05 2024-09-21T22:08:02.634 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.634 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.634 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:02.634 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.645315+0000", 2024-09-21T22:08:02.634 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9083e-05 2024-09-21T22:08:02.634 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.634 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.634 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:02.634 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.645460+0000", 2024-09-21T22:08:02.634 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014552599999999999 2024-09-21T22:08:02.634 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.635 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.635 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:02.635 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.645516+0000", 2024-09-21T22:08:02.635 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6369000000000002e-05 2024-09-21T22:08:02.635 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.635 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.635 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:02.635 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.665226+0000", 2024-09-21T22:08:02.635 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019709340999999998 2024-09-21T22:08:02.635 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.635 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.636 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:02.636 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.665318+0000", 2024-09-21T22:08:02.636 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2173e-05 2024-09-21T22:08:02.636 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:02.636 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:02.636 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:02.636 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.636 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.636 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5207.0:38 2.7 e78/16 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head v 78'179, mlcod=78'174)", 2024-09-21T22:08:02.636 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:52.655070+0000", 2024-09-21T22:08:02.636 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.9157701669999998, 2024-09-21T22:08:02.637 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020312226999999999, 2024-09-21T22:08:02.637 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:02.637 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:02.637 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:02.637 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:02.637 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.637 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:02.637 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.655070+0000", 2024-09-21T22:08:02.637 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:02.637 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.637 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.637 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:02.638 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.655070+0000", 2024-09-21T22:08:02.638 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:02.638 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.638 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.638 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:02.638 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.655073+0000", 2024-09-21T22:08:02.638 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.26e-06 2024-09-21T22:08:02.638 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.638 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.638 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:02.638 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.655200+0000", 2024-09-21T22:08:02.639 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012738300000000001 2024-09-21T22:08:02.639 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.639 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.639 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:02.639 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.655231+0000", 2024-09-21T22:08:02.639 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0865999999999999e-05 2024-09-21T22:08:02.639 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.639 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.639 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:02.639 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.655253+0000", 2024-09-21T22:08:02.639 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1523e-05 2024-09-21T22:08:02.640 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.640 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.640 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:02.640 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.655390+0000", 2024-09-21T22:08:02.640 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013705299999999999 2024-09-21T22:08:02.640 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.640 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.640 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:02.640 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.655448+0000", 2024-09-21T22:08:02.640 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8462999999999999e-05 2024-09-21T22:08:02.640 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.640 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.641 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:02.641 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.675275+0000", 2024-09-21T22:08:02.641 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019827000000000001 2024-09-21T22:08:02.641 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.641 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.641 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:02.641 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.675383+0000", 2024-09-21T22:08:02.641 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000107679 2024-09-21T22:08:02.641 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:02.641 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:02.641 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:02.641 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.641 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.642 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5207.0:70 2.7 e79/16 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head v 79'184, mlcod=79'181)", 2024-09-21T22:08:02.642 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:52.938385+0000", 2024-09-21T22:08:02.642 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.6324560479999999, 2024-09-21T22:08:02.642 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023928324000000001, 2024-09-21T22:08:02.642 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:02.642 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:02.642 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:02.642 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:02.642 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.642 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:02.642 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938385+0000", 2024-09-21T22:08:02.643 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:02.643 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.643 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.643 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:02.643 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938385+0000", 2024-09-21T22:08:02.643 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:02.643 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.643 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.643 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:02.643 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938387+0000", 2024-09-21T22:08:02.643 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1639999999999999e-06 2024-09-21T22:08:02.643 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.644 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.644 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:02.644 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938511+0000", 2024-09-21T22:08:02.644 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000124485 2024-09-21T22:08:02.644 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.644 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.644 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:02.644 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938549+0000", 2024-09-21T22:08:02.644 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7347e-05 2024-09-21T22:08:02.644 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.644 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.644 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:02.645 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938570+0000", 2024-09-21T22:08:02.645 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1013e-05 2024-09-21T22:08:02.645 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.645 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.645 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:02.645 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938718+0000", 2024-09-21T22:08:02.645 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000148789 2024-09-21T22:08:02.645 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.645 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.645 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:02.645 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938769+0000", 2024-09-21T22:08:02.645 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1002999999999999e-05 2024-09-21T22:08:02.645 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.646 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.646 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:02.646 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.962199+0000", 2024-09-21T22:08:02.646 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023429783999999999 2024-09-21T22:08:02.646 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.646 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.646 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:02.646 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.962313+0000", 2024-09-21T22:08:02.646 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000113739 2024-09-21T22:08:02.646 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:02.646 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:02.646 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:02.647 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.647 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.647 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5207.0:71 2.7 e79/16 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head v 79'185, mlcod=79'181)", 2024-09-21T22:08:02.647 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:52.951522+0000", 2024-09-21T22:08:02.647 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.6193181560000003, 2024-09-21T22:08:02.647 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021108028000000001, 2024-09-21T22:08:02.647 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:02.647 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:02.647 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:02.647 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:02.647 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.648 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:02.648 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.951522+0000", 2024-09-21T22:08:02.648 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:02.648 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.648 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.648 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:02.648 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.951522+0000", 2024-09-21T22:08:02.648 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:02.648 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.648 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.648 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:02.648 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.951524+0000", 2024-09-21T22:08:02.649 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9199999999999998e-06 2024-09-21T22:08:02.649 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.649 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.649 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:02.649 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.951686+0000", 2024-09-21T22:08:02.649 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016202499999999999 2024-09-21T22:08:02.649 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.649 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.649 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:02.649 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.951713+0000", 2024-09-21T22:08:02.649 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7007000000000001e-05 2024-09-21T22:08:02.650 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.650 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.650 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:02.650 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.951734+0000", 2024-09-21T22:08:02.650 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0228999999999999e-05 2024-09-21T22:08:02.650 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.650 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.650 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:02.650 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.952740+0000", 2024-09-21T22:08:02.650 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001006631 2024-09-21T22:08:02.650 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.650 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.651 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:02.651 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.952778+0000", 2024-09-21T22:08:02.651 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7316e-05 2024-09-21T22:08:02.651 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.651 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.651 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:02.651 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.972505+0000", 2024-09-21T22:08:02.651 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019727776999999998 2024-09-21T22:08:02.651 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.651 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.651 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:02.651 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.972631+0000", 2024-09-21T22:08:02.652 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012512299999999999 2024-09-21T22:08:02.652 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:02.652 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:02.652 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:02.652 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.652 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.652 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5207.0:72 2.7 e79/16 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head v 79'186, mlcod=79'181)", 2024-09-21T22:08:02.652 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:52.962684+0000", 2024-09-21T22:08:02.652 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.6081570799999998, 2024-09-21T22:08:02.652 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020226731000000001, 2024-09-21T22:08:02.652 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:02.652 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:02.653 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:02.653 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:02.653 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.653 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:02.653 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.962684+0000", 2024-09-21T22:08:02.653 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:02.653 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.653 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.653 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:02.654 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.962684+0000", 2024-09-21T22:08:02.654 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:02.654 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.654 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.654 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:02.654 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.962685+0000", 2024-09-21T22:08:02.654 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5269999999999999e-06 2024-09-21T22:08:02.654 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.654 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.654 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:02.654 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.962806+0000", 2024-09-21T22:08:02.654 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000121192 2024-09-21T22:08:02.655 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.655 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.655 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:02.655 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.962848+0000", 2024-09-21T22:08:02.655 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1387000000000002e-05 2024-09-21T22:08:02.655 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.655 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.655 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:02.655 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.962873+0000", 2024-09-21T22:08:02.655 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5069999999999999e-05 2024-09-21T22:08:02.655 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.655 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.656 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:02.656 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.963011+0000", 2024-09-21T22:08:02.656 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013869899999999999 2024-09-21T22:08:02.656 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.656 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.656 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:02.656 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.963057+0000", 2024-09-21T22:08:02.656 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5793e-05 2024-09-21T22:08:02.656 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.656 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.656 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:02.656 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.982810+0000", 2024-09-21T22:08:02.657 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019752914 2024-09-21T22:08:02.657 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.657 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.657 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:02.657 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.982910+0000", 2024-09-21T22:08:02.657 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010014900000000001 2024-09-21T22:08:02.657 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:02.657 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:02.657 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:02.657 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.658 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.658 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5207.0:76 2.7 e79/16 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head v 79'190, mlcod=79'186)", 2024-09-21T22:08:02.658 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:53.001917+0000", 2024-09-21T22:08:02.658 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.5689237069999997, 2024-09-21T22:08:02.658 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022783040000000001, 2024-09-21T22:08:02.658 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:02.658 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:02.658 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:02.658 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:02.658 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.658 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:02.658 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.001917+0000", 2024-09-21T22:08:02.659 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:02.659 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.659 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.659 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:02.659 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.001917+0000", 2024-09-21T22:08:02.659 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:02.659 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.659 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.659 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:02.659 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.001919+0000", 2024-09-21T22:08:02.659 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.703e-06 2024-09-21T22:08:02.659 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.659 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.660 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:02.660 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.002006+0000", 2024-09-21T22:08:02.660 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6978999999999995e-05 2024-09-21T22:08:02.660 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.660 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.660 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:02.660 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.002042+0000", 2024-09-21T22:08:02.660 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6847000000000001e-05 2024-09-21T22:08:02.660 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.660 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.660 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:02.661 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.002057+0000", 2024-09-21T22:08:02.661 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.419e-05 2024-09-21T22:08:02.661 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.661 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.661 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:02.661 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.004387+0000", 2024-09-21T22:08:02.661 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002330365 2024-09-21T22:08:02.661 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.661 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.661 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:02.661 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.004420+0000", 2024-09-21T22:08:02.661 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3170000000000003e-05 2024-09-21T22:08:02.662 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.662 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.662 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:02.662 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.024610+0000", 2024-09-21T22:08:02.662 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020190084000000001 2024-09-21T22:08:02.662 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.662 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.662 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:02.662 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.024700+0000", 2024-09-21T22:08:02.662 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9702000000000003e-05 2024-09-21T22:08:02.662 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:02.662 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:02.663 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:02.663 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.663 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.663 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5207.0:93 2.7 e79/16 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head v 79'193, mlcod=79'191)", 2024-09-21T22:08:02.663 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:53.149813+0000", 2024-09-21T22:08:02.663 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.4210276480000008, 2024-09-21T22:08:02.663 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.060053738000000002, 2024-09-21T22:08:02.663 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:02.663 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:02.663 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:02.663 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:02.663 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.664 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:02.664 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.149813+0000", 2024-09-21T22:08:02.664 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:02.664 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.664 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.664 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:02.664 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.149813+0000", 2024-09-21T22:08:02.664 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:02.664 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.664 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.664 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:02.664 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.149815+0000", 2024-09-21T22:08:02.665 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9099999999999999e-06 2024-09-21T22:08:02.665 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.665 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.665 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:02.665 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.150828+0000", 2024-09-21T22:08:02.665 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0010129099999999999 2024-09-21T22:08:02.665 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.665 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.665 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:02.665 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.150860+0000", 2024-09-21T22:08:02.665 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2443000000000001e-05 2024-09-21T22:08:02.665 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.666 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.666 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:02.666 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.150874+0000", 2024-09-21T22:08:02.666 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3627e-05 2024-09-21T22:08:02.666 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.666 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.666 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:02.666 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.151010+0000", 2024-09-21T22:08:02.666 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013656599999999999 2024-09-21T22:08:02.666 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.666 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.666 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:02.666 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.151047+0000", 2024-09-21T22:08:02.667 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6495999999999998e-05 2024-09-21T22:08:02.667 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.667 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.667 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:02.667 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.209791+0000", 2024-09-21T22:08:02.667 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058744004000000002 2024-09-21T22:08:02.667 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.667 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.667 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:02.667 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.209867+0000", 2024-09-21T22:08:02.667 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5782000000000001e-05 2024-09-21T22:08:02.667 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:02.668 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:02.668 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:02.668 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.668 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.668 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5207.0:128 2.7 e83/16 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head v 83'198, mlcod=83'195)", 2024-09-21T22:08:02.668 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:08:01.058831+0000", 2024-09-21T22:08:02.668 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 1.512010013, 2024-09-21T22:08:02.668 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058096534999999998, 2024-09-21T22:08:02.668 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:02.668 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:02.668 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:02.668 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:02.668 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.669 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:02.669 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.058831+0000", 2024-09-21T22:08:02.669 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:02.669 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.669 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.669 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:02.669 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.058831+0000", 2024-09-21T22:08:02.669 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:02.669 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.669 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.669 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:02.669 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.058833+0000", 2024-09-21T22:08:02.669 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0660000000000002e-06 2024-09-21T22:08:02.670 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.670 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.670 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:02.670 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.059877+0000", 2024-09-21T22:08:02.670 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0010447 2024-09-21T22:08:02.670 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.670 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.670 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:02.670 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.059908+0000", 2024-09-21T22:08:02.670 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0800000000000003e-05 2024-09-21T22:08:02.670 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.670 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.671 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:02.671 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.059923+0000", 2024-09-21T22:08:02.671 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4497e-05 2024-09-21T22:08:02.671 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.671 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.671 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:02.671 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.060053+0000", 2024-09-21T22:08:02.671 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013002900000000001 2024-09-21T22:08:02.671 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.671 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.671 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:02.671 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.060091+0000", 2024-09-21T22:08:02.671 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8389999999999997e-05 2024-09-21T22:08:02.672 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.672 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.672 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:02.672 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.116840+0000", 2024-09-21T22:08:02.672 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.056749253999999999 2024-09-21T22:08:02.672 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:02.672 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:02.672 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:02.672 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.116927+0000", 2024-09-21T22:08:02.672 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6798999999999993e-05 2024-09-21T22:08:02.672 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:02.672 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:02.673 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:02.673 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:02.673 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:02.673 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:08:02.673 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T22:08:02.675 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:08:02.669+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:08:02.675 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T22:08:02.776 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:08:02.772+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:08:02.806 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:08:02.806 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:08:02.806 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-09-21T22:08:02.806 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:08:02.821 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T22:08:02.830 INFO:teuthology.orchestra.run.smithi039.stdout:osd.0: {} 2024-09-21T22:08:02.830 INFO:teuthology.orchestra.run.smithi039.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T22:08:02.847 INFO:teuthology.orchestra.run.smithi039.stdout:osd.1: {} 2024-09-21T22:08:02.847 INFO:teuthology.orchestra.run.smithi039.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T22:08:02.848 INFO:teuthology.orchestra.run.smithi039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-21T22:08:02.867 INFO:teuthology.orchestra.run.smithi039.stdout:osd.3: {} 2024-09-21T22:08:02.867 INFO:teuthology.orchestra.run.smithi039.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T22:08:02.876 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:08:02.873+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:08:02.885 INFO:teuthology.orchestra.run.smithi039.stdout:osd.4: {} 2024-09-21T22:08:02.885 INFO:teuthology.orchestra.run.smithi039.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T22:08:02.904 INFO:teuthology.orchestra.run.smithi039.stdout:osd.5: {} 2024-09-21T22:08:02.904 INFO:teuthology.orchestra.run.smithi039.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T22:08:02.909 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T22:08:02.910 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T22:08:02.977 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:08:02.972+0000 7f2075a06640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31720) UID: 0 2024-09-21T22:08:02.993 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:02.993 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:03.007 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T22:08:03.094 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:08:03.092+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:08:03.178 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:03.178 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:03.179 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:08:03.176+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:08:03.193 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T22:08:03.280 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:08:03.276+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:08:03.329 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:03.330 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:03.344 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T22:08:03.388 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:08:03.385+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:08:03.471 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:03.471 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:03.482 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:08:03.477+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:08:03.485 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T22:08:03.583 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:08:03.577+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:08:03.656 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:03.656 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:03.672 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T22:08:03.684 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:08:03.684+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:08:03.785 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:08:03.781+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:08:03.841 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:03.841 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:03.855 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T22:08:03.886 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:08:03.884+0000 7f2075a06640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31720) UID: 0 2024-09-21T22:08:03.987 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:08:03.981+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:08:04.029 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:04.030 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:04.045 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T22:08:04.049 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-21T22:08:04.092 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:08:04.089+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:08:04.189 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:08:04.188+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:08:04.218 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:04.219 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:04.235 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T22:08:04.290 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:08:04.288+0000 7f2075a06640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31720) UID: 0 2024-09-21T22:08:04.371 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:04.371 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:04.385 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T22:08:04.426 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:08:04.424+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:08:04.491 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:08:04.489+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:08:04.544 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:04.544 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:04.558 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T22:08:04.602 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:08:04.600+0000 7f2075a06640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31720) UID: 0 2024-09-21T22:08:04.686 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:04.686 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:04.693 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:08:04.688+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:08:04.701 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T22:08:04.794 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:08:04.792+0000 7f2075a06640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31720) UID: 0 2024-09-21T22:08:04.872 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:04.874 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:04.887 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T22:08:04.897 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:08:04.893+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:08:04.912 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T22:08:04.997 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:08:04.993+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:08:05.036 INFO:teuthology.orchestra.run.smithi039.stderr:nodeep-scrub is set 2024-09-21T22:08:05.071 INFO:teuthology.orchestra.run.smithi039.stdout:osd.0: {} 2024-09-21T22:08:05.071 INFO:teuthology.orchestra.run.smithi039.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T22:08:05.079 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:05.079 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:05.088 INFO:teuthology.orchestra.run.smithi039.stdout:osd.1: {} 2024-09-21T22:08:05.088 INFO:teuthology.orchestra.run.smithi039.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T22:08:05.088 INFO:teuthology.orchestra.run.smithi039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-21T22:08:05.094 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T22:08:05.097 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:08:05.093+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:08:05.108 INFO:tasks.thrashosds.thrasher:in_osds: [1, 3, 4, 5, 0] out_osds: [2] dead_osds: [2] live_osds: [0, 1, 3, 4, 5] 2024-09-21T22:08:05.108 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T22:08:05.109 INFO:tasks.thrashosds.thrasher:Removing osd 1, in_osds are: [1, 3, 4, 5, 0] 2024-09-21T22:08:05.109 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 1 2024-09-21T22:08:05.109 INFO:teuthology.orchestra.run.smithi039.stdout:osd.3: {} 2024-09-21T22:08:05.110 INFO:teuthology.orchestra.run.smithi039.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T22:08:05.128 INFO:teuthology.orchestra.run.smithi039.stdout:osd.4: {} 2024-09-21T22:08:05.128 INFO:teuthology.orchestra.run.smithi039.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T22:08:05.146 INFO:teuthology.orchestra.run.smithi039.stdout:osd.5: {} 2024-09-21T22:08:05.146 INFO:teuthology.orchestra.run.smithi039.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T22:08:05.149 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T22:08:05.149 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T22:08:05.199 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:08:05.196+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:08:05.230 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:08:05.230 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-21T22:08:05.230 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-21T22:08:05.230 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-21T22:08:05.230 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.230 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4840.0:676 2.7 e63/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 63'57, mlcod=63'55)", 2024-09-21T22:08:05.230 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.627276+0000", 2024-09-21T22:08:05.230 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 40.600999274000003, 2024-09-21T22:08:05.231 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.037500787000000001, 2024-09-21T22:08:05.231 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:05.231 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:05.231 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:05.231 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:05.231 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.231 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:05.231 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.627276+0000", 2024-09-21T22:08:05.231 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:05.231 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.231 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.231 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:05.232 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.627276+0000", 2024-09-21T22:08:05.232 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:05.232 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.232 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.232 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:05.232 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.627277+0000", 2024-09-21T22:08:05.232 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.068e-06 2024-09-21T22:08:05.232 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.232 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.232 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:05.232 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.628148+0000", 2024-09-21T22:08:05.232 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00087069999999999997 2024-09-21T22:08:05.232 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.232 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.232 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:05.233 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.628170+0000", 2024-09-21T22:08:05.233 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2147999999999999e-05 2024-09-21T22:08:05.233 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.233 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.233 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:05.233 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.628179+0000", 2024-09-21T22:08:05.233 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2450000000000004e-06 2024-09-21T22:08:05.233 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.233 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.233 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:05.233 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.628279+0000", 2024-09-21T22:08:05.233 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9298999999999999e-05 2024-09-21T22:08:05.233 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.233 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.233 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:05.234 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.628305+0000", 2024-09-21T22:08:05.234 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6316999999999999e-05 2024-09-21T22:08:05.234 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.234 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.234 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:05.234 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.664714+0000", 2024-09-21T22:08:05.234 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.036408923000000003 2024-09-21T22:08:05.234 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.234 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.234 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:05.234 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.664777+0000", 2024-09-21T22:08:05.234 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3087000000000001e-05 2024-09-21T22:08:05.234 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:05.234 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:05.235 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:05.235 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.235 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.235 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4840.0:678 2.7 e63/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 63'59, mlcod=63'57)", 2024-09-21T22:08:05.235 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.679689+0000", 2024-09-21T22:08:05.235 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 40.548586774, 2024-09-21T22:08:05.235 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053308406000000003, 2024-09-21T22:08:05.235 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:05.235 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:05.235 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:05.235 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:05.235 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.235 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:05.235 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.679689+0000", 2024-09-21T22:08:05.236 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:05.236 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.236 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.236 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:05.236 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.679689+0000", 2024-09-21T22:08:05.236 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:05.236 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.236 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.236 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:05.236 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.679691+0000", 2024-09-21T22:08:05.236 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9400000000000001e-06 2024-09-21T22:08:05.236 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.236 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.236 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:05.236 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.680245+0000", 2024-09-21T22:08:05.237 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00055382999999999995 2024-09-21T22:08:05.237 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.237 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.237 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:05.237 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.680276+0000", 2024-09-21T22:08:05.237 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1123e-05 2024-09-21T22:08:05.237 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.237 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.237 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:05.237 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.680290+0000", 2024-09-21T22:08:05.237 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4056999999999999e-05 2024-09-21T22:08:05.237 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.237 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.237 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:05.237 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.680418+0000", 2024-09-21T22:08:05.238 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000127792 2024-09-21T22:08:05.238 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.238 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.238 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:05.238 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.680465+0000", 2024-09-21T22:08:05.238 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7360000000000001e-05 2024-09-21T22:08:05.238 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.238 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.238 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:05.238 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.732923+0000", 2024-09-21T22:08:05.238 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.052458312999999999 2024-09-21T22:08:05.238 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.238 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.239 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:05.239 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.732997+0000", 2024-09-21T22:08:05.239 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3991000000000004e-05 2024-09-21T22:08:05.239 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:05.239 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:05.239 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:05.239 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.239 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.239 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4840.0:680 2.7 e63/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 63'61, mlcod=63'59)", 2024-09-21T22:08:05.239 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.745996+0000", 2024-09-21T22:08:05.239 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 40.482279261000002, 2024-09-21T22:08:05.239 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.061804132999999997, 2024-09-21T22:08:05.239 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:05.239 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:05.239 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:05.240 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:05.240 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.240 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:05.240 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.745996+0000", 2024-09-21T22:08:05.240 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:05.240 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.240 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.240 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:05.240 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.745996+0000", 2024-09-21T22:08:05.240 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:05.240 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.240 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.240 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:05.240 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.745998+0000", 2024-09-21T22:08:05.240 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.717e-06 2024-09-21T22:08:05.241 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.241 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.241 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:05.241 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.746523+0000", 2024-09-21T22:08:05.241 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00052525399999999996 2024-09-21T22:08:05.241 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.241 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.241 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:05.241 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.746554+0000", 2024-09-21T22:08:05.242 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0586000000000002e-05 2024-09-21T22:08:05.242 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.242 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.242 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:05.242 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.746577+0000", 2024-09-21T22:08:05.242 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3079e-05 2024-09-21T22:08:05.242 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.242 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.242 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:05.242 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.746689+0000", 2024-09-21T22:08:05.242 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011177 2024-09-21T22:08:05.242 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.242 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.242 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:05.243 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.746722+0000", 2024-09-21T22:08:05.243 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2870000000000002e-05 2024-09-21T22:08:05.243 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.243 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.243 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:05.243 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.807740+0000", 2024-09-21T22:08:05.243 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.061018612 2024-09-21T22:08:05.243 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.243 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.243 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:05.243 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.807800+0000", 2024-09-21T22:08:05.243 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0244999999999997e-05 2024-09-21T22:08:05.243 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:05.244 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:05.244 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:05.244 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.244 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.244 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4840.0:682 2.7 e63/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 63'63, mlcod=63'61)", 2024-09-21T22:08:05.244 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.824126+0000", 2024-09-21T22:08:05.244 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 40.404149691999997, 2024-09-21T22:08:05.244 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.038159327, 2024-09-21T22:08:05.244 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:05.244 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:05.244 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:05.244 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:05.244 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.244 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:05.245 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.824126+0000", 2024-09-21T22:08:05.245 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:05.245 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.245 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.245 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:05.245 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.824126+0000", 2024-09-21T22:08:05.245 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:05.245 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.245 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.245 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:05.245 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.824128+0000", 2024-09-21T22:08:05.245 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9659999999999999e-06 2024-09-21T22:08:05.245 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.245 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.246 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:05.246 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.824842+0000", 2024-09-21T22:08:05.246 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00071423999999999999 2024-09-21T22:08:05.246 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.246 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.246 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:05.246 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.824883+0000", 2024-09-21T22:08:05.246 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1131999999999999e-05 2024-09-21T22:08:05.246 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.246 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.246 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:05.246 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.824931+0000", 2024-09-21T22:08:05.247 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7327000000000003e-05 2024-09-21T22:08:05.247 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.247 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.247 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:05.247 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.825058+0000", 2024-09-21T22:08:05.247 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012790499999999999 2024-09-21T22:08:05.247 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.247 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.247 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:05.247 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.825087+0000", 2024-09-21T22:08:05.248 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8192e-05 2024-09-21T22:08:05.248 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.248 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.248 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:05.248 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.862225+0000", 2024-09-21T22:08:05.248 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.037138309000000001 2024-09-21T22:08:05.248 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.248 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.248 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:05.248 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.862285+0000", 2024-09-21T22:08:05.248 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0256e-05 2024-09-21T22:08:05.248 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:05.248 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:05.249 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:05.249 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.249 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.249 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5008.0:116 2.7 e71/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 71'148, mlcod=71'146)", 2024-09-21T22:08:05.249 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:37.867612+0000", 2024-09-21T22:08:05.249 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 27.360663197000001, 2024-09-21T22:08:05.249 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.045123155999999998, 2024-09-21T22:08:05.249 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:05.249 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:05.249 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:05.249 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:05.249 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.249 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:05.249 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.867612+0000", 2024-09-21T22:08:05.249 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:05.250 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.250 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.250 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:05.250 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.867612+0000", 2024-09-21T22:08:05.250 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:05.250 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.250 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.250 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:05.250 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.867613+0000", 2024-09-21T22:08:05.250 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1200000000000002e-07 2024-09-21T22:08:05.250 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.250 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.251 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:05.251 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.868135+0000", 2024-09-21T22:08:05.251 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00052132700000000003 2024-09-21T22:08:05.251 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.251 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.251 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:05.251 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.868150+0000", 2024-09-21T22:08:05.251 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5271e-05 2024-09-21T22:08:05.251 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.251 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.252 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:05.252 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.868156+0000", 2024-09-21T22:08:05.252 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4969999999999998e-06 2024-09-21T22:08:05.252 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.252 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.252 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:05.252 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.868213+0000", 2024-09-21T22:08:05.252 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7114999999999998e-05 2024-09-21T22:08:05.252 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.252 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.252 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:05.252 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.868238+0000", 2024-09-21T22:08:05.252 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4987999999999999e-05 2024-09-21T22:08:05.253 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.253 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.253 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:05.253 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.912677+0000", 2024-09-21T22:08:05.253 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.044439019000000003 2024-09-21T22:08:05.253 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.253 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.253 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:05.253 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.912736+0000", 2024-09-21T22:08:05.253 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8127000000000001e-05 2024-09-21T22:08:05.253 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:05.253 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:05.253 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:05.253 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.253 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.254 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5008.0:118 2.7 e71/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 71'150, mlcod=71'148)", 2024-09-21T22:08:05.254 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:37.924700+0000", 2024-09-21T22:08:05.254 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 27.303576006, 2024-09-21T22:08:05.254 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.034650561000000003, 2024-09-21T22:08:05.254 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:05.254 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:05.254 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:05.254 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:05.254 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.254 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:05.254 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.924700+0000", 2024-09-21T22:08:05.254 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:05.254 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.254 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.255 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:05.255 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.924700+0000", 2024-09-21T22:08:05.255 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:05.255 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.255 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.255 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:05.255 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.924702+0000", 2024-09-21T22:08:05.255 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9460000000000001e-06 2024-09-21T22:08:05.255 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.255 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.255 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:05.255 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.925132+0000", 2024-09-21T22:08:05.255 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00043021200000000002 2024-09-21T22:08:05.255 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.256 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.256 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:05.256 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.925156+0000", 2024-09-21T22:08:05.256 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4451999999999999e-05 2024-09-21T22:08:05.256 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.256 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.256 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:05.256 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.925168+0000", 2024-09-21T22:08:05.256 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2249000000000001e-05 2024-09-21T22:08:05.256 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.256 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.256 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:05.256 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.925241+0000", 2024-09-21T22:08:05.256 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2502000000000005e-05 2024-09-21T22:08:05.257 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.257 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.257 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:05.257 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.925267+0000", 2024-09-21T22:08:05.257 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6471000000000001e-05 2024-09-21T22:08:05.257 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.257 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.257 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:05.257 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.959217+0000", 2024-09-21T22:08:05.257 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.033949569999999998 2024-09-21T22:08:05.257 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.257 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.257 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:05.257 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.959350+0000", 2024-09-21T22:08:05.257 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013315899999999999 2024-09-21T22:08:05.258 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:05.258 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:05.258 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:05.258 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.258 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.258 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5008.0:120 2.7 e71/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 71'152, mlcod=71'150)", 2024-09-21T22:08:05.258 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:37.970837+0000", 2024-09-21T22:08:05.258 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 27.257438754999999, 2024-09-21T22:08:05.258 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.03044943, 2024-09-21T22:08:05.258 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:05.258 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:05.258 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:05.258 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:05.259 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.259 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:05.259 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.970837+0000", 2024-09-21T22:08:05.259 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:05.259 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.259 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.259 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:05.259 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.970837+0000", 2024-09-21T22:08:05.259 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:05.259 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.259 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.259 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:05.259 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.970838+0000", 2024-09-21T22:08:05.259 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4649999999999999e-06 2024-09-21T22:08:05.259 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.260 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.260 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:05.260 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.971265+0000", 2024-09-21T22:08:05.260 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00042665800000000001 2024-09-21T22:08:05.260 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.260 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.260 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:05.260 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.971285+0000", 2024-09-21T22:08:05.260 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9639999999999998e-05 2024-09-21T22:08:05.260 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.260 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.260 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:05.260 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.971294+0000", 2024-09-21T22:08:05.260 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7210000000000006e-06 2024-09-21T22:08:05.261 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.261 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.261 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:05.261 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.971354+0000", 2024-09-21T22:08:05.261 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9613e-05 2024-09-21T22:08:05.261 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.261 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.261 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:05.261 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.971381+0000", 2024-09-21T22:08:05.261 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7538999999999998e-05 2024-09-21T22:08:05.261 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.261 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.261 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:05.262 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.001199+0000", 2024-09-21T22:08:05.262 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029817537000000002 2024-09-21T22:08:05.262 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.262 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.262 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:05.262 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.001286+0000", 2024-09-21T22:08:05.262 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7256999999999995e-05 2024-09-21T22:08:05.262 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:05.262 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:05.263 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:05.263 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.263 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.263 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5008.0:122 2.7 e71/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 71'154, mlcod=71'152)", 2024-09-21T22:08:05.263 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:38.014709+0000", 2024-09-21T22:08:05.263 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 27.213566879999998, 2024-09-21T22:08:05.263 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029567408999999999, 2024-09-21T22:08:05.263 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:05.263 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:05.263 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:05.263 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:05.263 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.264 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:05.264 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.014709+0000", 2024-09-21T22:08:05.264 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:05.264 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.264 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.264 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:05.264 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.014709+0000", 2024-09-21T22:08:05.264 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:05.264 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.264 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.264 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:05.264 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.014709+0000", 2024-09-21T22:08:05.264 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9599999999999999e-07 2024-09-21T22:08:05.264 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.264 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.264 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:05.265 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.015201+0000", 2024-09-21T22:08:05.265 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00049175599999999999 2024-09-21T22:08:05.265 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.265 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.265 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:05.265 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.015215+0000", 2024-09-21T22:08:05.265 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3692999999999999e-05 2024-09-21T22:08:05.265 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.265 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.265 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:05.265 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.015225+0000", 2024-09-21T22:08:05.265 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.815e-06 2024-09-21T22:08:05.265 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.265 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.265 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:05.266 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.015287+0000", 2024-09-21T22:08:05.266 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1858000000000003e-05 2024-09-21T22:08:05.266 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.266 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.266 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:05.266 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.015315+0000", 2024-09-21T22:08:05.266 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8490999999999999e-05 2024-09-21T22:08:05.266 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.266 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.266 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:05.266 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.044182+0000", 2024-09-21T22:08:05.266 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.028866936999999999 2024-09-21T22:08:05.266 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.266 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.267 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:05.267 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.044276+0000", 2024-09-21T22:08:05.267 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4163e-05 2024-09-21T22:08:05.267 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:05.267 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:05.267 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:05.267 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.267 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.267 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5207.0:33 2.7 e78/16 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head v 78'174, mlcod=78'170)", 2024-09-21T22:08:05.267 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:52.604638+0000", 2024-09-21T22:08:05.267 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 12.623637918, 2024-09-21T22:08:05.267 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020363816, 2024-09-21T22:08:05.267 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:05.267 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:05.268 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:05.268 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:05.268 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.268 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:05.268 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.604638+0000", 2024-09-21T22:08:05.268 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:05.268 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.268 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.268 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:05.268 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.604638+0000", 2024-09-21T22:08:05.268 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:05.268 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.268 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.268 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:05.269 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.604640+0000", 2024-09-21T22:08:05.269 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8929999999999999e-06 2024-09-21T22:08:05.269 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.269 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.269 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:05.269 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.604719+0000", 2024-09-21T22:08:05.269 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.9116999999999995e-05 2024-09-21T22:08:05.269 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.269 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.269 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:05.269 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.604751+0000", 2024-09-21T22:08:05.269 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2316e-05 2024-09-21T22:08:05.269 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.269 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.269 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:05.269 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.604770+0000", 2024-09-21T22:08:05.270 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9406e-05 2024-09-21T22:08:05.270 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.270 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.270 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:05.270 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.604930+0000", 2024-09-21T22:08:05.270 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015912599999999999 2024-09-21T22:08:05.270 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.270 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.270 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:05.270 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.604981+0000", 2024-09-21T22:08:05.270 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1010000000000001e-05 2024-09-21T22:08:05.270 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.270 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.270 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:05.271 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.624865+0000", 2024-09-21T22:08:05.271 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019884545999999999 2024-09-21T22:08:05.271 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.271 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.271 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:05.271 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.625001+0000", 2024-09-21T22:08:05.271 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013640200000000001 2024-09-21T22:08:05.271 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:05.271 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:05.271 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:05.271 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.271 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.271 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5207.0:34 2.7 e78/16 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head v 78'175, mlcod=78'170)", 2024-09-21T22:08:05.272 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:52.614890+0000", 2024-09-21T22:08:05.272 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 12.613385312, 2024-09-21T22:08:05.272 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019973422000000001, 2024-09-21T22:08:05.272 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:05.272 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:05.272 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:05.272 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:05.272 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.272 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:05.272 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.614890+0000", 2024-09-21T22:08:05.272 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:05.273 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.273 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.273 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:05.273 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.614890+0000", 2024-09-21T22:08:05.273 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:05.273 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.273 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.273 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:05.273 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.614892+0000", 2024-09-21T22:08:05.273 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8899999999999999e-06 2024-09-21T22:08:05.273 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.273 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.273 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:05.273 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.615033+0000", 2024-09-21T22:08:05.273 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014115199999999999 2024-09-21T22:08:05.274 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.274 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.274 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:05.274 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.615069+0000", 2024-09-21T22:08:05.274 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5892999999999997e-05 2024-09-21T22:08:05.274 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.274 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.274 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:05.274 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.615090+0000", 2024-09-21T22:08:05.274 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.037e-05 2024-09-21T22:08:05.274 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.274 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.274 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:05.274 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.615217+0000", 2024-09-21T22:08:05.275 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012768299999999999 2024-09-21T22:08:05.275 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.275 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.275 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:05.275 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.615270+0000", 2024-09-21T22:08:05.275 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3050000000000002e-05 2024-09-21T22:08:05.275 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.275 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.275 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:05.275 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.634730+0000", 2024-09-21T22:08:05.275 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019459860999999998 2024-09-21T22:08:05.275 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.275 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.275 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:05.275 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.634864+0000", 2024-09-21T22:08:05.276 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000133523 2024-09-21T22:08:05.276 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:05.276 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:05.276 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:05.276 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.276 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.276 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5207.0:35 2.7 e78/16 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head v 78'176, mlcod=78'170)", 2024-09-21T22:08:05.276 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:52.624606+0000", 2024-09-21T22:08:05.276 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 12.603669436000001, 2024-09-21T22:08:05.276 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020065888, 2024-09-21T22:08:05.276 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:05.276 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:05.276 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:05.276 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:05.277 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.277 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:05.277 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.624606+0000", 2024-09-21T22:08:05.277 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:05.277 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.277 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.277 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:05.277 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.624606+0000", 2024-09-21T22:08:05.277 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:05.277 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.277 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.277 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:05.277 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.624608+0000", 2024-09-21T22:08:05.278 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7600000000000001e-06 2024-09-21T22:08:05.278 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.278 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.278 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:05.278 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.624802+0000", 2024-09-21T22:08:05.278 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019425900000000001 2024-09-21T22:08:05.278 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.278 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.278 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:05.278 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.624860+0000", 2024-09-21T22:08:05.278 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7886000000000001e-05 2024-09-21T22:08:05.278 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.278 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.278 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:05.278 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.624882+0000", 2024-09-21T22:08:05.279 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2277e-05 2024-09-21T22:08:05.279 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.279 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.279 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:05.279 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.625081+0000", 2024-09-21T22:08:05.279 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019864500000000001 2024-09-21T22:08:05.279 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.279 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.279 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:05.279 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.625131+0000", 2024-09-21T22:08:05.279 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0120000000000001e-05 2024-09-21T22:08:05.279 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.279 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.279 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:05.279 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.644538+0000", 2024-09-21T22:08:05.280 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019406667999999998 2024-09-21T22:08:05.280 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.280 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.280 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:05.280 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.644672+0000", 2024-09-21T22:08:05.280 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013427299999999999 2024-09-21T22:08:05.280 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:05.280 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:05.280 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:05.280 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.280 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.280 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5207.0:36 2.7 e78/16 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head v 78'177, mlcod=78'171)", 2024-09-21T22:08:05.280 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:52.634570+0000", 2024-09-21T22:08:05.280 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 12.593705571999999, 2024-09-21T22:08:05.280 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020665352000000001, 2024-09-21T22:08:05.280 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:05.281 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:05.281 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:05.281 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:05.281 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.281 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:05.281 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.634570+0000", 2024-09-21T22:08:05.281 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:05.281 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.281 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.281 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:05.281 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.634570+0000", 2024-09-21T22:08:05.282 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:05.282 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.282 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.282 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:05.282 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.634572+0000", 2024-09-21T22:08:05.282 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7439999999999999e-06 2024-09-21T22:08:05.282 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.282 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.282 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:05.282 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.634715+0000", 2024-09-21T22:08:05.282 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014291600000000001 2024-09-21T22:08:05.283 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.283 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.283 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:05.283 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.634739+0000", 2024-09-21T22:08:05.283 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4615999999999999e-05 2024-09-21T22:08:05.283 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.283 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.283 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:05.283 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.634752+0000", 2024-09-21T22:08:05.283 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.27e-05 2024-09-21T22:08:05.283 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.283 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.283 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:05.283 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.634953+0000", 2024-09-21T22:08:05.283 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020067500000000001 2024-09-21T22:08:05.284 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.284 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.284 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:05.284 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.635008+0000", 2024-09-21T22:08:05.284 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5504000000000003e-05 2024-09-21T22:08:05.284 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.284 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.284 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:05.284 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.655111+0000", 2024-09-21T22:08:05.284 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.02010323 2024-09-21T22:08:05.284 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.284 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.284 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:05.284 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.655235+0000", 2024-09-21T22:08:05.284 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000123967 2024-09-21T22:08:05.285 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:05.285 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:05.285 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:05.285 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.285 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.285 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5207.0:37 2.7 e78/16 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head v 78'178, mlcod=78'174)", 2024-09-21T22:08:05.285 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:52.645132+0000", 2024-09-21T22:08:05.285 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 12.583143553999999, 2024-09-21T22:08:05.285 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020185980999999999, 2024-09-21T22:08:05.285 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:05.285 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:05.285 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:05.285 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:05.285 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.285 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:05.286 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.645132+0000", 2024-09-21T22:08:05.286 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:05.286 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.286 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.286 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:05.286 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.645132+0000", 2024-09-21T22:08:05.286 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:05.286 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.286 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.286 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:05.286 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.645134+0000", 2024-09-21T22:08:05.286 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1770000000000001e-06 2024-09-21T22:08:05.286 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.287 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.287 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:05.287 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.645259+0000", 2024-09-21T22:08:05.287 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012473900000000001 2024-09-21T22:08:05.287 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.287 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.287 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:05.287 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.645296+0000", 2024-09-21T22:08:05.287 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6572999999999997e-05 2024-09-21T22:08:05.287 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.287 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.287 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:05.287 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.645315+0000", 2024-09-21T22:08:05.287 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9083e-05 2024-09-21T22:08:05.287 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.288 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.288 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:05.288 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.645460+0000", 2024-09-21T22:08:05.288 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014552599999999999 2024-09-21T22:08:05.288 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.288 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.288 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:05.288 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.645516+0000", 2024-09-21T22:08:05.288 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6369000000000002e-05 2024-09-21T22:08:05.288 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.288 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.288 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:05.288 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.665226+0000", 2024-09-21T22:08:05.288 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019709340999999998 2024-09-21T22:08:05.288 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.289 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.289 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:05.289 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.665318+0000", 2024-09-21T22:08:05.289 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2173e-05 2024-09-21T22:08:05.289 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:05.289 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:05.289 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:05.289 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.289 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.289 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5207.0:38 2.7 e78/16 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head v 78'179, mlcod=78'174)", 2024-09-21T22:08:05.289 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:52.655070+0000", 2024-09-21T22:08:05.289 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 12.573205093, 2024-09-21T22:08:05.289 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020312226999999999, 2024-09-21T22:08:05.289 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:05.289 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:05.290 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:05.290 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:05.290 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.290 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:05.290 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.655070+0000", 2024-09-21T22:08:05.290 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:05.290 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.290 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.290 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:05.290 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.655070+0000", 2024-09-21T22:08:05.290 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:05.290 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.290 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.291 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:05.291 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.655073+0000", 2024-09-21T22:08:05.291 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.26e-06 2024-09-21T22:08:05.291 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.291 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.291 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:05.291 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.655200+0000", 2024-09-21T22:08:05.291 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012738300000000001 2024-09-21T22:08:05.291 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.291 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.291 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:05.291 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.655231+0000", 2024-09-21T22:08:05.291 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0865999999999999e-05 2024-09-21T22:08:05.291 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.291 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.292 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:05.292 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.655253+0000", 2024-09-21T22:08:05.292 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1523e-05 2024-09-21T22:08:05.292 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.292 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.292 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:05.292 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.655390+0000", 2024-09-21T22:08:05.292 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013705299999999999 2024-09-21T22:08:05.292 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.292 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.292 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:05.292 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.655448+0000", 2024-09-21T22:08:05.293 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8462999999999999e-05 2024-09-21T22:08:05.293 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.293 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.293 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:05.293 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.675275+0000", 2024-09-21T22:08:05.293 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019827000000000001 2024-09-21T22:08:05.293 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.293 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.293 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:05.293 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.675383+0000", 2024-09-21T22:08:05.293 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000107679 2024-09-21T22:08:05.293 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:05.293 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:05.293 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:05.294 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.294 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.294 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5207.0:70 2.7 e79/16 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head v 79'184, mlcod=79'181)", 2024-09-21T22:08:05.294 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:52.938385+0000", 2024-09-21T22:08:05.294 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 12.289890974, 2024-09-21T22:08:05.294 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023928324000000001, 2024-09-21T22:08:05.294 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:05.294 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:05.294 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:05.294 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:05.294 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.294 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:05.294 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938385+0000", 2024-09-21T22:08:05.294 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:05.295 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.295 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.295 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:05.295 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938385+0000", 2024-09-21T22:08:05.295 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:05.295 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.295 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.295 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:05.295 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938387+0000", 2024-09-21T22:08:05.295 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1639999999999999e-06 2024-09-21T22:08:05.295 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.295 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.295 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:05.295 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938511+0000", 2024-09-21T22:08:05.295 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000124485 2024-09-21T22:08:05.296 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.296 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.296 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:05.296 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938549+0000", 2024-09-21T22:08:05.296 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7347e-05 2024-09-21T22:08:05.296 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.296 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.296 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:05.296 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938570+0000", 2024-09-21T22:08:05.296 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1013e-05 2024-09-21T22:08:05.296 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.296 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.296 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:05.296 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938718+0000", 2024-09-21T22:08:05.297 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000148789 2024-09-21T22:08:05.297 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.297 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.297 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:05.297 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938769+0000", 2024-09-21T22:08:05.297 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1002999999999999e-05 2024-09-21T22:08:05.297 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.297 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.297 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:05.297 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.962199+0000", 2024-09-21T22:08:05.297 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023429783999999999 2024-09-21T22:08:05.297 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.297 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.297 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:05.297 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.962313+0000", 2024-09-21T22:08:05.297 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000113739 2024-09-21T22:08:05.298 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:05.298 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:05.298 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:05.298 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.298 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.298 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5207.0:71 2.7 e79/16 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head v 79'185, mlcod=79'181)", 2024-09-21T22:08:05.298 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:52.951522+0000", 2024-09-21T22:08:05.298 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 12.276753082000001, 2024-09-21T22:08:05.298 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021108028000000001, 2024-09-21T22:08:05.298 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:05.298 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:05.298 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:05.298 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:05.299 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.299 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:05.299 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.951522+0000", 2024-09-21T22:08:05.299 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:05.299 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.299 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.299 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:05.299 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.951522+0000", 2024-09-21T22:08:05.299 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:05.299 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.299 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.299 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:05.299 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.951524+0000", 2024-09-21T22:08:05.299 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9199999999999998e-06 2024-09-21T22:08:05.299 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.300 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.300 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:05.300 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.951686+0000", 2024-09-21T22:08:05.300 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016202499999999999 2024-09-21T22:08:05.300 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.300 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.300 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:05.300 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.951713+0000", 2024-09-21T22:08:05.300 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7007000000000001e-05 2024-09-21T22:08:05.300 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.300 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.300 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:05.300 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.951734+0000", 2024-09-21T22:08:05.300 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0228999999999999e-05 2024-09-21T22:08:05.300 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.300 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.300 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:05.301 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.952740+0000", 2024-09-21T22:08:05.301 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001006631 2024-09-21T22:08:05.301 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.301 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.301 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:05.301 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.952778+0000", 2024-09-21T22:08:05.301 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7316e-05 2024-09-21T22:08:05.301 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.301 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.301 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:05.301 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.972505+0000", 2024-09-21T22:08:05.301 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019727776999999998 2024-09-21T22:08:05.301 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.301 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.302 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:05.302 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.972631+0000", 2024-09-21T22:08:05.302 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012512299999999999 2024-09-21T22:08:05.302 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:05.302 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:05.302 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:05.302 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.302 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.302 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5207.0:72 2.7 e79/16 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head v 79'186, mlcod=79'181)", 2024-09-21T22:08:05.302 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:52.962684+0000", 2024-09-21T22:08:05.302 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 12.265592006, 2024-09-21T22:08:05.302 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020226731000000001, 2024-09-21T22:08:05.302 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:05.303 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:05.303 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:05.303 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:05.303 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.303 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:05.303 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.962684+0000", 2024-09-21T22:08:05.303 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:05.303 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.303 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.303 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:05.303 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.962684+0000", 2024-09-21T22:08:05.303 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:05.303 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.303 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.303 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:05.303 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.962685+0000", 2024-09-21T22:08:05.304 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5269999999999999e-06 2024-09-21T22:08:05.304 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.304 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.304 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:05.304 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.962806+0000", 2024-09-21T22:08:05.304 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000121192 2024-09-21T22:08:05.304 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.304 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.304 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:05.304 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.962848+0000", 2024-09-21T22:08:05.304 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1387000000000002e-05 2024-09-21T22:08:05.304 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.304 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.304 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:05.304 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.962873+0000", 2024-09-21T22:08:05.304 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5069999999999999e-05 2024-09-21T22:08:05.305 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.305 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.305 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:05.305 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.963011+0000", 2024-09-21T22:08:05.305 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013869899999999999 2024-09-21T22:08:05.305 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.305 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.305 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:05.305 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.963057+0000", 2024-09-21T22:08:05.305 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5793e-05 2024-09-21T22:08:05.305 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.305 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.305 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:05.305 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.982810+0000", 2024-09-21T22:08:05.305 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019752914 2024-09-21T22:08:05.306 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.306 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.306 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:05.306 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.982910+0000", 2024-09-21T22:08:05.306 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010014900000000001 2024-09-21T22:08:05.306 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:05.306 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:05.306 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:05.306 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.306 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.306 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5207.0:76 2.7 e79/16 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head v 79'190, mlcod=79'186)", 2024-09-21T22:08:05.306 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:53.001917+0000", 2024-09-21T22:08:05.307 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 12.226358633, 2024-09-21T22:08:05.307 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022783040000000001, 2024-09-21T22:08:05.307 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:05.307 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:05.307 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:05.307 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:05.307 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.307 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:05.307 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.001917+0000", 2024-09-21T22:08:05.307 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:05.307 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.307 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.307 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:05.307 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.001917+0000", 2024-09-21T22:08:05.307 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:05.308 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.308 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.308 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:05.308 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.001919+0000", 2024-09-21T22:08:05.308 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.703e-06 2024-09-21T22:08:05.308 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.308 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.308 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:05.308 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.002006+0000", 2024-09-21T22:08:05.308 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6978999999999995e-05 2024-09-21T22:08:05.308 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.308 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.308 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:05.308 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.002042+0000", 2024-09-21T22:08:05.308 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6847000000000001e-05 2024-09-21T22:08:05.309 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.309 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.309 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:05.309 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.002057+0000", 2024-09-21T22:08:05.309 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.419e-05 2024-09-21T22:08:05.309 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.309 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.309 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:05.309 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.004387+0000", 2024-09-21T22:08:05.309 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002330365 2024-09-21T22:08:05.309 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.309 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.309 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:05.310 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.004420+0000", 2024-09-21T22:08:05.310 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3170000000000003e-05 2024-09-21T22:08:05.310 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.310 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.310 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:05.310 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.024610+0000", 2024-09-21T22:08:05.310 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020190084000000001 2024-09-21T22:08:05.310 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.310 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.310 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:05.310 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.024700+0000", 2024-09-21T22:08:05.310 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9702000000000003e-05 2024-09-21T22:08:05.310 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:05.310 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:05.310 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:05.311 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.311 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.311 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5207.0:93 2.7 e79/16 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head v 79'193, mlcod=79'191)", 2024-09-21T22:08:05.311 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:53.149813+0000", 2024-09-21T22:08:05.311 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 12.078462574, 2024-09-21T22:08:05.311 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.060053738000000002, 2024-09-21T22:08:05.311 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:05.311 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:05.311 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:05.311 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:05.311 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.311 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:05.311 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.149813+0000", 2024-09-21T22:08:05.311 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:05.312 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.312 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.312 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:05.312 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.149813+0000", 2024-09-21T22:08:05.312 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:05.312 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.312 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.312 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:05.312 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.149815+0000", 2024-09-21T22:08:05.312 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9099999999999999e-06 2024-09-21T22:08:05.312 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.312 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.312 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:05.312 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.150828+0000", 2024-09-21T22:08:05.312 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0010129099999999999 2024-09-21T22:08:05.313 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.313 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.313 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:05.313 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.150860+0000", 2024-09-21T22:08:05.313 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2443000000000001e-05 2024-09-21T22:08:05.313 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.313 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.313 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:05.313 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.150874+0000", 2024-09-21T22:08:05.313 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3627e-05 2024-09-21T22:08:05.313 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.313 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.313 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:05.314 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.151010+0000", 2024-09-21T22:08:05.314 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013656599999999999 2024-09-21T22:08:05.314 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.314 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.314 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:05.314 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.151047+0000", 2024-09-21T22:08:05.314 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6495999999999998e-05 2024-09-21T22:08:05.314 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.314 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.314 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:05.314 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.209791+0000", 2024-09-21T22:08:05.314 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058744004000000002 2024-09-21T22:08:05.314 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.315 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.315 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:05.315 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.209867+0000", 2024-09-21T22:08:05.315 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5782000000000001e-05 2024-09-21T22:08:05.315 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:05.315 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:05.315 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:05.315 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.315 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.315 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5207.0:128 2.7 e83/16 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head v 83'198, mlcod=83'195)", 2024-09-21T22:08:05.315 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:08:01.058831+0000", 2024-09-21T22:08:05.315 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.1694449389999999, 2024-09-21T22:08:05.315 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058096534999999998, 2024-09-21T22:08:05.315 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:05.315 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:05.316 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:05.316 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:05.316 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.316 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:05.316 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.058831+0000", 2024-09-21T22:08:05.316 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:05.316 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.316 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.316 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:05.316 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.058831+0000", 2024-09-21T22:08:05.316 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:05.316 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.316 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.316 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:05.317 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.058833+0000", 2024-09-21T22:08:05.317 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0660000000000002e-06 2024-09-21T22:08:05.317 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.317 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.317 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:05.317 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.059877+0000", 2024-09-21T22:08:05.317 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0010447 2024-09-21T22:08:05.317 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.317 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.317 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:05.317 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.059908+0000", 2024-09-21T22:08:05.317 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0800000000000003e-05 2024-09-21T22:08:05.317 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.317 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.317 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:05.318 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.059923+0000", 2024-09-21T22:08:05.318 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4497e-05 2024-09-21T22:08:05.318 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.318 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.318 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:05.318 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.060053+0000", 2024-09-21T22:08:05.318 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013002900000000001 2024-09-21T22:08:05.318 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.318 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.318 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:05.319 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.060091+0000", 2024-09-21T22:08:05.319 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8389999999999997e-05 2024-09-21T22:08:05.319 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.319 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.319 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:05.319 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.116840+0000", 2024-09-21T22:08:05.319 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.056749253999999999 2024-09-21T22:08:05.319 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.319 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.319 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:05.319 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.116927+0000", 2024-09-21T22:08:05.319 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6798999999999993e-05 2024-09-21T22:08:05.319 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:05.319 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:05.319 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:05.320 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:05.320 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:05.320 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:08:05.322 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T22:08:05.322 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:08:05.316+0000 7f2075a06640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31720) UID: 0 2024-09-21T22:08:05.421 INFO:tasks.ceph.mon.a.smithi039.stderr:2024-09-21T22:08:05.417+0000 7f9967196640 -1 mon.a@0(leader).osd e85 definitely_dead 0 2024-09-21T22:08:05.421 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:08:05.417+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:08:05.494 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:08:05.494 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:08:05.495 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-09-21T22:08:05.495 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:08:05.509 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T22:08:05.558 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:08:05.556+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:08:05.623 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:08:05.620+0000 7f2075a06640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31720) UID: 0 2024-09-21T22:08:05.635 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:08:05.636 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:08:05.636 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-21T22:08:05.636 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-21T22:08:05.636 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:08:05.650 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T22:08:05.738 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:08:05.736+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:08:05.824 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:08:05.824 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-21T22:08:05.824 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-21T22:08:05.824 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-21T22:08:05.824 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.824 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5031.0:3 2.2 2:53d5e6d0:::rbd_header.126483d75f2d:head [call rbd.get_size in=11b,call,call rbd.get_features in=15b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e65)", 2024-09-21T22:08:05.824 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:27.373436+0000", 2024-09-21T22:08:05.824 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 38.448255678000002, 2024-09-21T22:08:05.824 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.86524013700000002, 2024-09-21T22:08:05.824 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:05.825 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:05.825 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:08:05.825 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:05.825 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5031", 2024-09-21T22:08:05.825 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/3232705659", 2024-09-21T22:08:05.825 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3 2024-09-21T22:08:05.825 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.825 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:05.825 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.825 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:05.825 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:27.373436+0000", 2024-09-21T22:08:05.825 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:05.825 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.825 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.825 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:05.826 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:27.373436+0000", 2024-09-21T22:08:05.826 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:05.826 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.826 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.826 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:05.826 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:27.373439+0000", 2024-09-21T22:08:05.826 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5969999999999999e-06 2024-09-21T22:08:05.826 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.826 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.826 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:05.826 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:27.373450+0000", 2024-09-21T22:08:05.826 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1326e-05 2024-09-21T22:08:05.826 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.826 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.827 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:05.827 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:27.373471+0000", 2024-09-21T22:08:05.827 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0169999999999998e-05 2024-09-21T22:08:05.827 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.827 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.827 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:05.827 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:27.373486+0000", 2024-09-21T22:08:05.827 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5483000000000001e-05 2024-09-21T22:08:05.827 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.827 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.827 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:05.827 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:27.373602+0000", 2024-09-21T22:08:05.827 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000115603 2024-09-21T22:08:05.827 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.827 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.828 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-21T22:08:05.828 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:27.373617+0000", 2024-09-21T22:08:05.828 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4956999999999999e-05 2024-09-21T22:08:05.828 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.828 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.828 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:05.828 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.234578+0000", 2024-09-21T22:08:05.828 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.86096097900000002 2024-09-21T22:08:05.828 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.828 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.828 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for missing object", 2024-09-21T22:08:05.828 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.234771+0000", 2024-09-21T22:08:05.828 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019394700000000001 2024-09-21T22:08:05.828 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.828 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.829 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:05.829 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.237470+0000", 2024-09-21T22:08:05.829 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026981319999999998 2024-09-21T22:08:05.829 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.829 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.829 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:05.829 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.237791+0000", 2024-09-21T22:08:05.829 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00032138499999999999 2024-09-21T22:08:05.829 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.829 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.829 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:05.829 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.238677+0000", 2024-09-21T22:08:05.829 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00088555800000000005 2024-09-21T22:08:05.829 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:05.829 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:05.830 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:05.830 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.830 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.830 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.2 66/65 [PushOp(2:4481fad7:ns::rbd_data.127f79bc213d.0000000000000006:head, version: 57'9, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:4481fad7:ns::rbd_data.127f79bc213d.0000000000000006:head@57'9, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T22:08:05.830 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:28.247056+0000", 2024-09-21T22:08:05.830 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 37.574636015000003, 2024-09-21T22:08:05.830 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.045025012000000003, 2024-09-21T22:08:05.830 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:05.830 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:05.830 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:08:05.830 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:05.830 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.830 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:05.830 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.247056+0000", 2024-09-21T22:08:05.831 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:05.831 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.831 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.831 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:05.831 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.247056+0000", 2024-09-21T22:08:05.831 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:05.831 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.831 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.831 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:05.831 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.247057+0000", 2024-09-21T22:08:05.831 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2500000000000001e-06 2024-09-21T22:08:05.831 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.831 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.831 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:05.831 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.249384+0000", 2024-09-21T22:08:05.832 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0023265619999999999 2024-09-21T22:08:05.832 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.832 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.832 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:05.832 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.249429+0000", 2024-09-21T22:08:05.832 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5392999999999997e-05 2024-09-21T22:08:05.832 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.832 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.832 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:05.832 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.249476+0000", 2024-09-21T22:08:05.832 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6570000000000003e-05 2024-09-21T22:08:05.832 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.832 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.832 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:05.832 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.249611+0000", 2024-09-21T22:08:05.833 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000135143 2024-09-21T22:08:05.833 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.833 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.833 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:05.833 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.249626+0000", 2024-09-21T22:08:05.833 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.469e-05 2024-09-21T22:08:05.833 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.833 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.833 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:05.833 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.292081+0000", 2024-09-21T22:08:05.833 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.042455404000000002 2024-09-21T22:08:05.833 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:05.833 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:05.833 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:05.833 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.834 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.834 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5031.0:9 2.2 2:53d5e6d0:::rbd_header.126483d75f2d:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-21T22:08:05.834 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:28.250163+0000", 2024-09-21T22:08:05.834 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 37.571528620999999, 2024-09-21T22:08:05.834 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.042986430999999999, 2024-09-21T22:08:05.834 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:05.834 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:05.834 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:08:05.834 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:05.834 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5031", 2024-09-21T22:08:05.834 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/3232705659", 2024-09-21T22:08:05.834 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 9 2024-09-21T22:08:05.834 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.834 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:05.834 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.835 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:05.835 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.250163+0000", 2024-09-21T22:08:05.835 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:05.835 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.835 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.835 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:05.835 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.250163+0000", 2024-09-21T22:08:05.835 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:05.835 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.835 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.835 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:05.835 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.250166+0000", 2024-09-21T22:08:05.835 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7039999999999999e-06 2024-09-21T22:08:05.835 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.835 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.836 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:05.836 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.250175+0000", 2024-09-21T22:08:05.836 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7560000000000005e-06 2024-09-21T22:08:05.836 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.836 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.836 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:05.836 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.250199+0000", 2024-09-21T22:08:05.836 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4049999999999998e-05 2024-09-21T22:08:05.836 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.836 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.836 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:05.836 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.250218+0000", 2024-09-21T22:08:05.836 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9236e-05 2024-09-21T22:08:05.836 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.836 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.837 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:05.837 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.292193+0000", 2024-09-21T22:08:05.837 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.041974694 2024-09-21T22:08:05.837 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.837 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.837 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:05.837 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.292492+0000", 2024-09-21T22:08:05.837 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00029900100000000002 2024-09-21T22:08:05.837 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.837 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.837 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:05.837 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.293150+0000", 2024-09-21T22:08:05.837 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00065799000000000001 2024-09-21T22:08:05.837 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:05.837 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:05.838 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:05.838 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.838 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.838 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5031.0:12 2.2 2:53d5e6d0:::rbd_header.126483d75f2d:head [watch watch cookie 140495695137040] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-09-21T22:08:05.838 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:28.295580+0000", 2024-09-21T22:08:05.838 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 37.526111733999997, 2024-09-21T22:08:05.838 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.033090870000000001, 2024-09-21T22:08:05.838 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:05.838 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:05.838 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:05.838 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:05.838 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5031", 2024-09-21T22:08:05.838 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/3232705659", 2024-09-21T22:08:05.838 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 12 2024-09-21T22:08:05.839 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.839 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:05.839 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.839 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:05.839 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.295580+0000", 2024-09-21T22:08:05.839 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:05.839 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.839 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.839 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:05.839 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.295580+0000", 2024-09-21T22:08:05.839 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:05.839 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.839 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.839 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:05.839 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.295581+0000", 2024-09-21T22:08:05.840 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6799999999999999e-07 2024-09-21T22:08:05.840 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.840 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.840 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:05.840 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.295586+0000", 2024-09-21T22:08:05.840 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8659999999999997e-06 2024-09-21T22:08:05.840 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.840 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.840 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:05.840 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.295599+0000", 2024-09-21T22:08:05.840 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2897999999999999e-05 2024-09-21T22:08:05.840 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.840 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.840 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:05.840 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.295614+0000", 2024-09-21T22:08:05.841 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5406000000000001e-05 2024-09-21T22:08:05.841 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.841 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.841 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:05.841 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.295665+0000", 2024-09-21T22:08:05.841 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0501000000000003e-05 2024-09-21T22:08:05.841 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.841 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.841 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:05.841 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.295727+0000", 2024-09-21T22:08:05.841 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2073e-05 2024-09-21T22:08:05.841 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.841 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.841 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-09-21T22:08:05.841 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.295849+0000", 2024-09-21T22:08:05.842 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012190500000000001 2024-09-21T22:08:05.842 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.842 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.842 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:05.842 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.296429+0000", 2024-09-21T22:08:05.842 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00058019699999999996 2024-09-21T22:08:05.842 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.842 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.842 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:05.842 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.328308+0000", 2024-09-21T22:08:05.842 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.031878849000000001 2024-09-21T22:08:05.842 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.842 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.842 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:05.842 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.328421+0000", 2024-09-21T22:08:05.843 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000113322 2024-09-21T22:08:05.843 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.843 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.843 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:05.843 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.328671+0000", 2024-09-21T22:08:05.843 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024998499999999999 2024-09-21T22:08:05.843 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:05.843 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:05.843 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:05.843 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.843 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.843 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.2 66/65 [PushOp(2:4481fad7:ns::rbd_data.127f79bc213d.0000000000000006:head, version: 57'9, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:4481fad7:ns::rbd_data.127f79bc213d.0000000000000006:head@57'9, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:08:05.843 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:28.296301+0000", 2024-09-21T22:08:05.843 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 37.525390813000001, 2024-09-21T22:08:05.843 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.031905291000000002, 2024-09-21T22:08:05.844 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:05.844 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:05.844 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:08:05.844 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:05.844 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.844 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:05.844 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.296301+0000", 2024-09-21T22:08:05.844 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:05.844 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.844 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.844 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:05.844 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.296301+0000", 2024-09-21T22:08:05.844 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:05.844 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.844 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.845 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:05.845 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.296315+0000", 2024-09-21T22:08:05.845 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3867e-05 2024-09-21T22:08:05.845 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.845 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.845 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:05.845 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.297432+0000", 2024-09-21T22:08:05.845 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011164759999999999 2024-09-21T22:08:05.845 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.845 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.845 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:05.845 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.297454+0000", 2024-09-21T22:08:05.845 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2813e-05 2024-09-21T22:08:05.845 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.845 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.846 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:05.846 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.297479+0000", 2024-09-21T22:08:05.846 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4817e-05 2024-09-21T22:08:05.846 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.846 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.846 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:05.846 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.297582+0000", 2024-09-21T22:08:05.846 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010261500000000001 2024-09-21T22:08:05.846 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.846 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.846 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:05.846 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.297600+0000", 2024-09-21T22:08:05.846 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8323000000000001e-05 2024-09-21T22:08:05.846 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.847 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.847 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:05.847 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.328207+0000", 2024-09-21T22:08:05.847 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.030606379999999999 2024-09-21T22:08:05.847 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:05.847 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:05.847 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:05.847 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.847 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.847 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop_reply(client.5031.0:12 2.2 e66/65 ondisk, result = 0)", 2024-09-21T22:08:05.847 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:28.297512+0000", 2024-09-21T22:08:05.847 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 37.52418024, 2024-09-21T22:08:05.847 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.031170693999999999, 2024-09-21T22:08:05.847 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:05.847 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:05.847 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:08:05.848 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:05.848 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.848 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:05.848 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.297512+0000", 2024-09-21T22:08:05.848 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:05.848 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.848 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.848 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:05.848 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.297512+0000", 2024-09-21T22:08:05.848 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:05.848 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.848 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.848 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:05.848 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.297512+0000", 2024-09-21T22:08:05.849 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7700000000000005e-07 2024-09-21T22:08:05.849 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.849 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.849 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:05.849 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.297517+0000", 2024-09-21T22:08:05.849 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1039999999999998e-06 2024-09-21T22:08:05.849 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.849 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.849 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:05.849 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.297528+0000", 2024-09-21T22:08:05.849 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0633e-05 2024-09-21T22:08:05.849 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.849 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.849 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:05.849 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.297537+0000", 2024-09-21T22:08:05.850 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7199999999999995e-06 2024-09-21T22:08:05.850 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.850 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.850 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:05.850 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.328273+0000", 2024-09-21T22:08:05.850 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.030736574999999999 2024-09-21T22:08:05.850 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.850 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.850 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:05.850 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.328293+0000", 2024-09-21T22:08:05.850 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9304e-05 2024-09-21T22:08:05.850 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.850 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.850 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:05.850 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.328683+0000", 2024-09-21T22:08:05.851 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00038988100000000002 2024-09-21T22:08:05.851 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:05.851 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:05.851 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:05.851 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.851 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.851 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.2 66/65 [PushOp(2:4481fad7:ns::rbd_data.127f79bc213d.0000000000000006:head, version: 57'9, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:4481fad7:ns::rbd_data.127f79bc213d.0000000000000006:head@57'9, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:08:05.851 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:28.331166+0000", 2024-09-21T22:08:05.851 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 37.490525871000003, 2024-09-21T22:08:05.851 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.039995378999999998, 2024-09-21T22:08:05.851 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:05.851 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:05.851 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:08:05.851 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:05.852 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.852 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:05.852 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.331166+0000", 2024-09-21T22:08:05.852 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:05.852 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.852 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.852 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:05.852 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.331166+0000", 2024-09-21T22:08:05.852 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:05.852 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.852 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.852 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:05.852 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.331167+0000", 2024-09-21T22:08:05.852 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9199999999999999e-07 2024-09-21T22:08:05.852 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.853 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.853 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:05.853 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.332127+0000", 2024-09-21T22:08:05.853 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00095969800000000004 2024-09-21T22:08:05.853 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.853 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.853 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:05.853 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.332149+0000", 2024-09-21T22:08:05.853 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1974999999999999e-05 2024-09-21T22:08:05.853 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.853 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.853 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:05.853 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.332161+0000", 2024-09-21T22:08:05.853 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2276000000000001e-05 2024-09-21T22:08:05.853 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.854 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.854 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:05.854 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.332290+0000", 2024-09-21T22:08:05.854 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000129214 2024-09-21T22:08:05.854 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.854 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.854 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:05.854 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.332312+0000", 2024-09-21T22:08:05.854 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1308999999999999e-05 2024-09-21T22:08:05.854 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.854 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.854 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:05.854 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.371162+0000", 2024-09-21T22:08:05.854 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.038850015000000002 2024-09-21T22:08:05.854 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:05.855 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:05.855 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:05.855 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.855 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.855 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4811.0:3 2.3 2:c4c92e5a:::rbd_trash:head [call rbd.trash_list in=13b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-21T22:08:05.855 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:28.475306+0000", 2024-09-21T22:08:05.855 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 37.346386326999998, 2024-09-21T22:08:05.855 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.77996670099999998, 2024-09-21T22:08:05.855 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:05.855 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:05.855 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:08:05.855 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:05.855 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4811", 2024-09-21T22:08:05.855 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/2028736390", 2024-09-21T22:08:05.856 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3 2024-09-21T22:08:05.856 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.856 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:05.856 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.856 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:05.856 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.475306+0000", 2024-09-21T22:08:05.856 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:05.856 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.856 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.856 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:05.856 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.475306+0000", 2024-09-21T22:08:05.856 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:05.856 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.856 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.856 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:05.856 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.475308+0000", 2024-09-21T22:08:05.857 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8959999999999999e-06 2024-09-21T22:08:05.857 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.857 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.857 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:05.857 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.475314+0000", 2024-09-21T22:08:05.857 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2280000000000003e-06 2024-09-21T22:08:05.857 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.857 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.857 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:05.857 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.475329+0000", 2024-09-21T22:08:05.857 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5054e-05 2024-09-21T22:08:05.857 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.857 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.857 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:05.857 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.475338+0000", 2024-09-21T22:08:05.858 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3770000000000002e-06 2024-09-21T22:08:05.858 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.858 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.858 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:05.858 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.475484+0000", 2024-09-21T22:08:05.858 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014612300000000001 2024-09-21T22:08:05.858 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.858 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.858 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-21T22:08:05.858 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.475506+0000", 2024-09-21T22:08:05.858 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1319999999999999e-05 2024-09-21T22:08:05.858 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.858 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.858 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:05.858 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.254433+0000", 2024-09-21T22:08:05.859 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.778927708 2024-09-21T22:08:05.859 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.859 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.859 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:05.859 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.254979+0000", 2024-09-21T22:08:05.859 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00054573100000000004 2024-09-21T22:08:05.859 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.859 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.859 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:05.859 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.255272+0000", 2024-09-21T22:08:05.859 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000293264 2024-09-21T22:08:05.859 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:05.859 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:05.859 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:05.859 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.860 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.860 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.3 67/66 [PushOp(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head, version: 63'11, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head@63'11, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T22:08:05.860 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:29.262595+0000", 2024-09-21T22:08:05.860 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 36.559097143999999, 2024-09-21T22:08:05.860 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.036027272999999999, 2024-09-21T22:08:05.860 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:05.860 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:05.860 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:08:05.860 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:05.860 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.860 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:05.860 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.262595+0000", 2024-09-21T22:08:05.860 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:05.860 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.861 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.861 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:05.861 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.262595+0000", 2024-09-21T22:08:05.861 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:05.861 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.861 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.861 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:05.861 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.262596+0000", 2024-09-21T22:08:05.861 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4300000000000001e-06 2024-09-21T22:08:05.861 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.861 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.861 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:05.861 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.264505+0000", 2024-09-21T22:08:05.861 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019085650000000001 2024-09-21T22:08:05.861 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.862 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.862 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:05.862 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.264557+0000", 2024-09-21T22:08:05.862 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2040000000000003e-05 2024-09-21T22:08:05.862 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.862 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.862 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:05.862 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.264589+0000", 2024-09-21T22:08:05.862 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1636e-05 2024-09-21T22:08:05.862 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.862 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.862 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:05.862 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.264778+0000", 2024-09-21T22:08:05.862 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018935600000000001 2024-09-21T22:08:05.863 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.863 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.863 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:05.863 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.264803+0000", 2024-09-21T22:08:05.863 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4692999999999999e-05 2024-09-21T22:08:05.863 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.863 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.863 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:05.863 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.298622+0000", 2024-09-21T22:08:05.863 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.033819553000000002 2024-09-21T22:08:05.863 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:05.863 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:05.863 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:05.863 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.863 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.864 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.3 67/66 [PushOp(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head, version: 63'11, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head@63'11, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:08:05.864 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:29.301490+0000", 2024-09-21T22:08:05.864 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 36.520202357999999, 2024-09-21T22:08:05.864 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.041178293999999997, 2024-09-21T22:08:05.864 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:05.864 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:05.864 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:08:05.864 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:05.864 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.864 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:05.864 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.301490+0000", 2024-09-21T22:08:05.864 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:05.865 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.865 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.865 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:05.865 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.301490+0000", 2024-09-21T22:08:05.865 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:05.865 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.865 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.865 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:05.865 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.301491+0000", 2024-09-21T22:08:05.865 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.513e-06 2024-09-21T22:08:05.865 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.865 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.865 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:05.865 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.302815+0000", 2024-09-21T22:08:05.866 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001323722 2024-09-21T22:08:05.866 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.866 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.866 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:05.866 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.302863+0000", 2024-09-21T22:08:05.866 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8449999999999999e-05 2024-09-21T22:08:05.866 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.866 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.866 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:05.866 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.302895+0000", 2024-09-21T22:08:05.866 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1072999999999999e-05 2024-09-21T22:08:05.866 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.866 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.866 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:05.866 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.303014+0000", 2024-09-21T22:08:05.867 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011907400000000001 2024-09-21T22:08:05.867 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.867 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.867 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:05.867 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.303029+0000", 2024-09-21T22:08:05.867 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5322e-05 2024-09-21T22:08:05.867 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.867 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.867 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:05.867 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.342668+0000", 2024-09-21T22:08:05.867 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.039639140000000003 2024-09-21T22:08:05.867 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:05.867 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:05.867 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:05.867 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.868 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.868 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.3 67/66 [PushOp(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head, version: 63'11, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head@63'11, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:08:05.868 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:29.345626+0000", 2024-09-21T22:08:05.868 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 36.476065624, 2024-09-21T22:08:05.868 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.034820216000000001, 2024-09-21T22:08:05.868 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:05.868 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:05.868 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:08:05.868 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:05.868 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.868 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:05.868 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.345626+0000", 2024-09-21T22:08:05.868 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:05.868 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.868 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.869 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:05.869 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.345626+0000", 2024-09-21T22:08:05.869 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:05.869 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.869 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.869 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:05.869 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.345628+0000", 2024-09-21T22:08:05.869 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3799999999999999e-06 2024-09-21T22:08:05.869 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.869 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.869 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:05.869 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.347215+0000", 2024-09-21T22:08:05.869 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015868760000000001 2024-09-21T22:08:05.869 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.869 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.870 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:05.870 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.347283+0000", 2024-09-21T22:08:05.870 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8267000000000002e-05 2024-09-21T22:08:05.870 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.870 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.870 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:05.870 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.347336+0000", 2024-09-21T22:08:05.870 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3455999999999998e-05 2024-09-21T22:08:05.870 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.870 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.870 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:05.870 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.347463+0000", 2024-09-21T22:08:05.870 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012614399999999999 2024-09-21T22:08:05.870 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.871 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.871 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:05.871 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.347473+0000", 2024-09-21T22:08:05.871 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0462000000000001e-05 2024-09-21T22:08:05.871 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.871 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.871 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:05.871 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.380447+0000", 2024-09-21T22:08:05.871 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.032973631000000003 2024-09-21T22:08:05.871 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:05.871 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:05.871 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:05.871 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.871 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.872 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.3 67/66 [PushOp(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head, version: 63'11, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head@63'11, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 4194304, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:08:05.872 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:29.383677+0000", 2024-09-21T22:08:05.872 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 36.438014746999997, 2024-09-21T22:08:05.872 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.031566611000000001, 2024-09-21T22:08:05.872 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:05.872 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:05.872 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:08:05.872 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:05.872 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.872 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:05.872 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.383677+0000", 2024-09-21T22:08:05.872 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:05.872 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.872 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.872 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:05.873 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.383677+0000", 2024-09-21T22:08:05.873 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:05.873 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.873 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.873 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:05.873 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.383679+0000", 2024-09-21T22:08:05.873 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4500000000000001e-06 2024-09-21T22:08:05.873 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.873 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.873 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:05.873 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.384982+0000", 2024-09-21T22:08:05.873 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001303069 2024-09-21T22:08:05.873 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.873 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.874 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:05.874 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.385028+0000", 2024-09-21T22:08:05.874 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5655999999999998e-05 2024-09-21T22:08:05.874 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.874 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.874 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:05.874 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.385056+0000", 2024-09-21T22:08:05.874 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8877000000000002e-05 2024-09-21T22:08:05.874 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.874 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.874 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:05.875 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.385161+0000", 2024-09-21T22:08:05.875 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000104325 2024-09-21T22:08:05.875 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.875 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.875 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:05.875 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.385173+0000", 2024-09-21T22:08:05.875 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2493e-05 2024-09-21T22:08:05.875 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.875 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.875 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:05.875 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.415244+0000", 2024-09-21T22:08:05.875 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.030070741000000002 2024-09-21T22:08:05.875 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:05.875 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:05.876 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:05.876 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.876 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.876 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.3 67/66 [PushOp(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head, version: 63'19, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head@63'19, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T22:08:05.876 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:29.420098+0000", 2024-09-21T22:08:05.876 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 36.401593789000003, 2024-09-21T22:08:05.876 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.067283846999999994, 2024-09-21T22:08:05.876 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:05.876 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:05.876 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:08:05.876 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:05.876 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.876 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:05.876 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.420098+0000", 2024-09-21T22:08:05.876 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:05.877 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.877 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.877 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:05.877 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.420098+0000", 2024-09-21T22:08:05.877 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:05.877 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.877 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.877 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:05.877 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.420100+0000", 2024-09-21T22:08:05.877 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.716e-06 2024-09-21T22:08:05.877 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.877 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.877 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:05.877 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.421949+0000", 2024-09-21T22:08:05.877 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0018485190000000001 2024-09-21T22:08:05.878 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.878 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.878 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:05.878 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.421996+0000", 2024-09-21T22:08:05.878 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7089000000000003e-05 2024-09-21T22:08:05.878 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.878 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.878 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:05.878 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.422042+0000", 2024-09-21T22:08:05.878 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6010000000000002e-05 2024-09-21T22:08:05.878 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.878 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.878 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:05.878 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.422231+0000", 2024-09-21T22:08:05.878 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018960900000000001 2024-09-21T22:08:05.879 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.879 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.879 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:05.879 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.422254+0000", 2024-09-21T22:08:05.879 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2864e-05 2024-09-21T22:08:05.879 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.879 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.879 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:05.879 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.487382+0000", 2024-09-21T22:08:05.879 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.065128039999999998 2024-09-21T22:08:05.879 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:05.879 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:05.879 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:05.879 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.879 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.880 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.3 67/66 [PushOp(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head, version: 63'19, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head@63'19, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:08:05.880 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:29.491783+0000", 2024-09-21T22:08:05.880 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 36.329909444000002, 2024-09-21T22:08:05.880 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.04476596, 2024-09-21T22:08:05.880 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:05.880 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:05.880 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:08:05.880 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:05.880 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.880 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:05.880 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.491783+0000", 2024-09-21T22:08:05.880 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:05.880 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.880 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.881 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:05.881 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.491783+0000", 2024-09-21T22:08:05.881 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:05.881 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.881 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.881 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:05.881 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.491784+0000", 2024-09-21T22:08:05.881 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.17e-06 2024-09-21T22:08:05.881 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.881 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.881 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:05.881 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.493126+0000", 2024-09-21T22:08:05.881 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0013424489999999999 2024-09-21T22:08:05.881 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.881 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.882 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:05.882 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.493170+0000", 2024-09-21T22:08:05.882 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4153000000000002e-05 2024-09-21T22:08:05.882 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.882 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.882 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:05.882 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.493218+0000", 2024-09-21T22:08:05.882 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7392999999999998e-05 2024-09-21T22:08:05.882 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.882 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.882 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:05.882 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.493392+0000", 2024-09-21T22:08:05.882 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017392699999999999 2024-09-21T22:08:05.882 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.882 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.883 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:05.883 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.493416+0000", 2024-09-21T22:08:05.883 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4117e-05 2024-09-21T22:08:05.883 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.883 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.883 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:05.883 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.536549+0000", 2024-09-21T22:08:05.883 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.043132750999999997 2024-09-21T22:08:05.883 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:05.883 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:05.883 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:05.883 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.883 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.883 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.3 67/66 [PushOp(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head, version: 63'19, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head@63'19, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:08:05.884 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:29.539831+0000", 2024-09-21T22:08:05.884 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 36.281860684000002, 2024-09-21T22:08:05.884 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.036233222000000002, 2024-09-21T22:08:05.884 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:05.884 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:05.884 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:08:05.884 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:05.884 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.884 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:05.884 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.539831+0000", 2024-09-21T22:08:05.884 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:05.884 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.884 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.884 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:05.884 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.539831+0000", 2024-09-21T22:08:05.885 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:05.885 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.885 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.885 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:05.885 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.539833+0000", 2024-09-21T22:08:05.885 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5200000000000001e-06 2024-09-21T22:08:05.885 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.885 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.885 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:05.885 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.541130+0000", 2024-09-21T22:08:05.885 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0012972719999999999 2024-09-21T22:08:05.885 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.885 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.885 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:05.885 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.541179+0000", 2024-09-21T22:08:05.886 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8507000000000002e-05 2024-09-21T22:08:05.886 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.886 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.886 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:05.886 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.541226+0000", 2024-09-21T22:08:05.886 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7663000000000001e-05 2024-09-21T22:08:05.886 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.886 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.886 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:05.886 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.541361+0000", 2024-09-21T22:08:05.886 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000135065 2024-09-21T22:08:05.886 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.886 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.886 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:05.887 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.541374+0000", 2024-09-21T22:08:05.887 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2295e-05 2024-09-21T22:08:05.887 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.887 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.887 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:05.887 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.576065+0000", 2024-09-21T22:08:05.887 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.034690899999999997 2024-09-21T22:08:05.887 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:05.887 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:05.887 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:05.887 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.887 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.887 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.3 67/66 [PushOp(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head, version: 63'19, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head@63'19, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 4194304, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:08:05.887 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:29.579455+0000", 2024-09-21T22:08:05.888 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 36.242237375999999, 2024-09-21T22:08:05.888 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.034295594999999998, 2024-09-21T22:08:05.888 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:05.888 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:05.888 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:08:05.888 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:05.888 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.888 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:05.888 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.579455+0000", 2024-09-21T22:08:05.888 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:05.888 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.888 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.888 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:05.888 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.579455+0000", 2024-09-21T22:08:05.888 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:05.889 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.889 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.889 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:05.889 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.579456+0000", 2024-09-21T22:08:05.889 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.404e-06 2024-09-21T22:08:05.889 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.889 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.889 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:05.889 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.580763+0000", 2024-09-21T22:08:05.889 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001307325 2024-09-21T22:08:05.889 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.889 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.889 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:05.889 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.580811+0000", 2024-09-21T22:08:05.889 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7843000000000003e-05 2024-09-21T22:08:05.890 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.890 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.890 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:05.890 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.580860+0000", 2024-09-21T22:08:05.890 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9036999999999998e-05 2024-09-21T22:08:05.890 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.890 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.890 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:05.890 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.580988+0000", 2024-09-21T22:08:05.890 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000127529 2024-09-21T22:08:05.890 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.890 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.890 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:05.890 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.580998+0000", 2024-09-21T22:08:05.891 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0617e-05 2024-09-21T22:08:05.891 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.891 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.891 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:05.891 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.613750+0000", 2024-09-21T22:08:05.891 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.032751839999999997 2024-09-21T22:08:05.891 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:05.891 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:05.891 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:05.891 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.891 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.891 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5008.0:125 2.3 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-21T22:08:05.891 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:38.066298+0000", 2024-09-21T22:08:05.891 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 27.755393712, 2024-09-21T22:08:05.891 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.062817136999999995, 2024-09-21T22:08:05.892 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:05.892 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:05.892 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:05.892 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:05.892 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5008", 2024-09-21T22:08:05.892 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/1400114666", 2024-09-21T22:08:05.892 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 125 2024-09-21T22:08:05.892 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.892 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:05.892 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.892 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:05.892 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.066298+0000", 2024-09-21T22:08:05.892 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:05.892 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.892 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.893 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:05.893 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.066298+0000", 2024-09-21T22:08:05.893 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:05.893 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.893 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.893 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:05.893 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.066300+0000", 2024-09-21T22:08:05.893 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.474e-06 2024-09-21T22:08:05.893 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.893 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.893 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:05.893 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.067443+0000", 2024-09-21T22:08:05.894 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011428009999999999 2024-09-21T22:08:05.894 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.894 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.894 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:05.894 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.067477+0000", 2024-09-21T22:08:05.894 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4059e-05 2024-09-21T22:08:05.894 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.894 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.894 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:05.894 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.067492+0000", 2024-09-21T22:08:05.894 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5054e-05 2024-09-21T22:08:05.894 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.894 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.894 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:05.895 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.067638+0000", 2024-09-21T22:08:05.895 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014583900000000001 2024-09-21T22:08:05.895 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.895 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.895 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:05.895 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.067882+0000", 2024-09-21T22:08:05.895 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024415500000000002 2024-09-21T22:08:05.895 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.895 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.895 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-09-21T22:08:05.895 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.068247+0000", 2024-09-21T22:08:05.895 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00036570800000000002 2024-09-21T22:08:05.895 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.895 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.895 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:05.896 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.119971+0000", 2024-09-21T22:08:05.896 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.051723866 2024-09-21T22:08:05.896 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.896 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.896 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:05.896 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.128849+0000", 2024-09-21T22:08:05.896 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0088779489999999996 2024-09-21T22:08:05.896 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.896 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.896 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:05.896 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.129000+0000", 2024-09-21T22:08:05.896 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015062499999999999 2024-09-21T22:08:05.896 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.896 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.896 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:05.897 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.129116+0000", 2024-09-21T22:08:05.897 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011560700000000001 2024-09-21T22:08:05.897 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:05.897 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:05.897 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:05.897 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.897 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.897 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5008.0:127 2.3 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-21T22:08:05.897 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:38.140308+0000", 2024-09-21T22:08:05.897 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 27.681383610000001, 2024-09-21T22:08:05.897 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.055290512999999999, 2024-09-21T22:08:05.897 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:05.897 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:05.897 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:05.898 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:05.898 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5008", 2024-09-21T22:08:05.898 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/1400114666", 2024-09-21T22:08:05.898 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 127 2024-09-21T22:08:05.898 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.898 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:05.898 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.898 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:05.898 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.140308+0000", 2024-09-21T22:08:05.898 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:05.898 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.898 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.898 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:05.898 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.140308+0000", 2024-09-21T22:08:05.898 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:05.899 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.899 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.899 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:05.899 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.140310+0000", 2024-09-21T22:08:05.899 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6509999999999999e-06 2024-09-21T22:08:05.899 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.899 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.899 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:05.899 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.141504+0000", 2024-09-21T22:08:05.899 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011935750000000001 2024-09-21T22:08:05.899 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.899 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.899 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:05.899 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.141522+0000", 2024-09-21T22:08:05.899 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8349e-05 2024-09-21T22:08:05.900 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.900 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.900 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:05.900 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.141531+0000", 2024-09-21T22:08:05.900 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3239999999999992e-06 2024-09-21T22:08:05.900 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.900 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.900 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:05.900 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.141661+0000", 2024-09-21T22:08:05.900 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013001000000000001 2024-09-21T22:08:05.900 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.900 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.900 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:05.900 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.141907+0000", 2024-09-21T22:08:05.900 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024547499999999999 2024-09-21T22:08:05.901 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.901 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.901 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-09-21T22:08:05.901 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.142274+0000", 2024-09-21T22:08:05.901 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00036716400000000001 2024-09-21T22:08:05.901 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.901 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.901 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:05.901 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.189864+0000", 2024-09-21T22:08:05.901 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047589617000000001 2024-09-21T22:08:05.901 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.901 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.901 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:05.901 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.195406+0000", 2024-09-21T22:08:05.902 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0055425860000000004 2024-09-21T22:08:05.902 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.902 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.902 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:05.902 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.195507+0000", 2024-09-21T22:08:05.902 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000100862 2024-09-21T22:08:05.902 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.902 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.902 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:05.902 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.195599+0000", 2024-09-21T22:08:05.902 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1899999999999998e-05 2024-09-21T22:08:05.902 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:05.902 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:05.902 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:05.902 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.903 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.903 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5008.0:129 2.3 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head [write 2228224~917504 in=917504b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-21T22:08:05.903 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:38.209297+0000", 2024-09-21T22:08:05.903 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 27.612394729000002, 2024-09-21T22:08:05.903 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.060441054000000001, 2024-09-21T22:08:05.903 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:05.903 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:05.903 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:05.903 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:05.903 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5008", 2024-09-21T22:08:05.903 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/1400114666", 2024-09-21T22:08:05.903 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 129 2024-09-21T22:08:05.903 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.903 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:05.903 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.904 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:05.904 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.209297+0000", 2024-09-21T22:08:05.904 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:05.904 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.904 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.904 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:05.904 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.209297+0000", 2024-09-21T22:08:05.904 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:05.904 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.904 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.904 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:05.904 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.209299+0000", 2024-09-21T22:08:05.904 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.274e-06 2024-09-21T22:08:05.904 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.904 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.905 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:05.905 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.210077+0000", 2024-09-21T22:08:05.905 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00077849799999999997 2024-09-21T22:08:05.905 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.905 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.905 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:05.905 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.210094+0000", 2024-09-21T22:08:05.905 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6772000000000001e-05 2024-09-21T22:08:05.905 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.905 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.905 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:05.905 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.210103+0000", 2024-09-21T22:08:05.905 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6780000000000006e-06 2024-09-21T22:08:05.905 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.905 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.906 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:05.906 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.210254+0000", 2024-09-21T22:08:05.906 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015141699999999999 2024-09-21T22:08:05.906 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.906 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.906 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:05.906 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.210501+0000", 2024-09-21T22:08:05.906 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000246498 2024-09-21T22:08:05.906 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.906 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.906 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-09-21T22:08:05.906 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.210872+0000", 2024-09-21T22:08:05.906 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00037147500000000002 2024-09-21T22:08:05.906 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.906 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.907 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:05.907 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.269019+0000", 2024-09-21T22:08:05.907 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058147114 2024-09-21T22:08:05.907 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.907 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.907 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:05.907 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.269638+0000", 2024-09-21T22:08:05.907 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00061896499999999997 2024-09-21T22:08:05.907 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.907 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.907 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:05.907 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.269693+0000", 2024-09-21T22:08:05.907 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4786e-05 2024-09-21T22:08:05.908 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.908 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.908 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:05.908 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.269738+0000", 2024-09-21T22:08:05.908 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5577000000000001e-05 2024-09-21T22:08:05.908 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:05.908 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:05.908 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:05.908 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.908 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.908 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5008.0:131 2.3 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-21T22:08:05.908 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:38.279879+0000", 2024-09-21T22:08:05.908 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 27.541813039000001, 2024-09-21T22:08:05.908 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.054184296999999999, 2024-09-21T22:08:05.908 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:05.909 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:05.909 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:05.909 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:05.909 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5008", 2024-09-21T22:08:05.909 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/1400114666", 2024-09-21T22:08:05.909 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 131 2024-09-21T22:08:05.909 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.909 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:05.909 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.909 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:05.909 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.279879+0000", 2024-09-21T22:08:05.909 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:05.909 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.909 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.910 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:05.910 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.279879+0000", 2024-09-21T22:08:05.910 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:05.910 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.910 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.910 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:05.910 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.279880+0000", 2024-09-21T22:08:05.910 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.136e-06 2024-09-21T22:08:05.910 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.910 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.910 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:05.910 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.280636+0000", 2024-09-21T22:08:05.910 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00075625900000000001 2024-09-21T22:08:05.910 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.910 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.911 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:05.911 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.280648+0000", 2024-09-21T22:08:05.911 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2026e-05 2024-09-21T22:08:05.911 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.911 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.911 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:05.911 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.280654+0000", 2024-09-21T22:08:05.911 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8839999999999999e-06 2024-09-21T22:08:05.911 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.911 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.911 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:05.911 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.280739+0000", 2024-09-21T22:08:05.911 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4196000000000005e-05 2024-09-21T22:08:05.911 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.911 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.912 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:05.912 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.280808+0000", 2024-09-21T22:08:05.912 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9685999999999997e-05 2024-09-21T22:08:05.912 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.912 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.912 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-09-21T22:08:05.912 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.280900+0000", 2024-09-21T22:08:05.912 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1407000000000001e-05 2024-09-21T22:08:05.912 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.912 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.912 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:05.912 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.307730+0000", 2024-09-21T22:08:05.912 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026830568999999999 2024-09-21T22:08:05.912 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.912 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.913 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:05.913 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.333901+0000", 2024-09-21T22:08:05.913 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026170787000000001 2024-09-21T22:08:05.913 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.913 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.913 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:05.913 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.333996+0000", 2024-09-21T22:08:05.913 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4723999999999996e-05 2024-09-21T22:08:05.913 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:05.913 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:05.913 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:05.913 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.334063+0000", 2024-09-21T22:08:05.913 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7623e-05 2024-09-21T22:08:05.913 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:05.913 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:05.914 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:05.914 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:05.914 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:05.914 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:08:05.915 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T22:08:05.916 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:08:05.913+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:08:06.016 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:08:06.013+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:08:06.055 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:08:06.055 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:08:06.055 INFO:teuthology.orchestra.run.smithi039.stdout: "num_ops": 0 2024-09-21T22:08:06.055 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:08:06.055 INFO:teuthology.orchestra.run.smithi039.stderr:marked out osd.1. 2024-09-21T22:08:06.079 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T22:08:06.117 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:08:06.112+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:08:06.218 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:08:06.213+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:08:06.251 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:08:06.251 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:08:06.251 INFO:teuthology.orchestra.run.smithi039.stdout: "complaint_time": 30, 2024-09-21T22:08:06.251 INFO:teuthology.orchestra.run.smithi039.stdout: "num_blocked_ops": 0 2024-09-21T22:08:06.251 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:08:06.265 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T22:08:06.356 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:08:06.353+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:08:06.419 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:08:06.413+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:08:06.441 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:08:06.441 INFO:teuthology.orchestra.run.smithi039.stdout: "size": 20, 2024-09-21T22:08:06.441 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 600, 2024-09-21T22:08:06.441 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [ 2024-09-21T22:08:06.441 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.441 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:694 2.5 e63/37 2:b1d55633:::rbd_data.126483d75f2d.000000000000000b:head v 63'26, mlcod=63'24)", 2024-09-21T22:08:06.441 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.163199+0000", 2024-09-21T22:08:06.442 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 41.275694778000002, 2024-09-21T22:08:06.442 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.034676672999999998, 2024-09-21T22:08:06.442 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:06.442 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:06.442 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:06.442 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:06.442 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.442 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:06.442 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.163199+0000", 2024-09-21T22:08:06.442 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:06.442 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.442 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.442 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:06.443 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.163199+0000", 2024-09-21T22:08:06.443 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:06.443 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.443 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.443 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:06.443 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.163200+0000", 2024-09-21T22:08:06.443 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.703e-06 2024-09-21T22:08:06.443 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.443 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.443 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:06.443 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.164013+0000", 2024-09-21T22:08:06.443 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000812447 2024-09-21T22:08:06.443 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.443 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.443 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:06.443 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.164042+0000", 2024-09-21T22:08:06.444 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9006e-05 2024-09-21T22:08:06.444 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.444 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.444 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:06.444 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.164064+0000", 2024-09-21T22:08:06.444 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1719999999999999e-05 2024-09-21T22:08:06.444 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.444 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.444 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:06.444 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.164143+0000", 2024-09-21T22:08:06.444 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.9283999999999995e-05 2024-09-21T22:08:06.444 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.444 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.444 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:06.444 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.164165+0000", 2024-09-21T22:08:06.445 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.2379e-05 2024-09-21T22:08:06.445 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.445 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.445 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:06.445 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.197827+0000", 2024-09-21T22:08:06.445 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.033661782000000001 2024-09-21T22:08:06.445 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.445 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.445 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:06.445 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.197875+0000", 2024-09-21T22:08:06.445 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8352000000000001e-05 2024-09-21T22:08:06.445 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:06.445 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:06.445 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:06.446 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.446 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.446 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:696 2.5 e63/37 2:b1d55633:::rbd_data.126483d75f2d.000000000000000b:head v 63'28, mlcod=63'26)", 2024-09-21T22:08:06.446 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.218913+0000", 2024-09-21T22:08:06.446 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 41.219980331999999, 2024-09-21T22:08:06.446 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.056187599999999997, 2024-09-21T22:08:06.446 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:06.446 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:06.446 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:06.446 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:06.446 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.446 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:06.446 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.218913+0000", 2024-09-21T22:08:06.446 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:06.447 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.447 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.447 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:06.447 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.218913+0000", 2024-09-21T22:08:06.447 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:06.447 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.447 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.447 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:06.447 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.218914+0000", 2024-09-21T22:08:06.447 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.1930000000000001e-06 2024-09-21T22:08:06.447 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.447 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.447 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:06.447 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.219721+0000", 2024-09-21T22:08:06.448 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00080701500000000001 2024-09-21T22:08:06.448 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.448 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.448 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:06.448 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.219743+0000", 2024-09-21T22:08:06.448 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1364000000000001e-05 2024-09-21T22:08:06.448 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.448 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.448 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:06.448 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.219759+0000", 2024-09-21T22:08:06.448 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.611e-05 2024-09-21T22:08:06.448 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.448 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.448 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:06.449 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.219908+0000", 2024-09-21T22:08:06.449 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000149252 2024-09-21T22:08:06.449 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.449 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.449 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:06.449 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.219951+0000", 2024-09-21T22:08:06.449 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.2518999999999997e-05 2024-09-21T22:08:06.449 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.449 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.449 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:06.449 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.275031+0000", 2024-09-21T22:08:06.449 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.055080241000000002 2024-09-21T22:08:06.449 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.449 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.449 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:06.450 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.275101+0000", 2024-09-21T22:08:06.450 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.9906000000000005e-05 2024-09-21T22:08:06.450 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:06.450 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:06.450 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:06.450 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.450 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.450 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:698 2.5 e63/37 2:b1d55633:::rbd_data.126483d75f2d.000000000000000b:head v 63'30, mlcod=63'28)", 2024-09-21T22:08:06.450 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.289319+0000", 2024-09-21T22:08:06.450 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 41.149574123000001, 2024-09-21T22:08:06.450 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.053930944000000001, 2024-09-21T22:08:06.450 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:06.451 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:06.451 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:06.451 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:06.451 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.451 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:06.451 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.289319+0000", 2024-09-21T22:08:06.451 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:06.451 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.451 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.451 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:06.451 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.289319+0000", 2024-09-21T22:08:06.451 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:06.451 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.451 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.451 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:06.452 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.289321+0000", 2024-09-21T22:08:06.452 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.221e-06 2024-09-21T22:08:06.452 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.452 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.452 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:06.452 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.290091+0000", 2024-09-21T22:08:06.452 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00077069300000000003 2024-09-21T22:08:06.452 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.452 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.452 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:06.452 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.290109+0000", 2024-09-21T22:08:06.452 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7218999999999999e-05 2024-09-21T22:08:06.452 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.452 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.453 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:06.453 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.290122+0000", 2024-09-21T22:08:06.453 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3678e-05 2024-09-21T22:08:06.453 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.453 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.453 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:06.453 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.290274+0000", 2024-09-21T22:08:06.453 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00015205899999999999 2024-09-21T22:08:06.453 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.453 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.453 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:06.453 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.290313+0000", 2024-09-21T22:08:06.453 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.8636000000000001e-05 2024-09-21T22:08:06.453 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.454 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.454 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:06.454 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.343179+0000", 2024-09-21T22:08:06.454 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.052866199000000003 2024-09-21T22:08:06.454 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.454 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.454 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:06.454 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.343250+0000", 2024-09-21T22:08:06.454 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.1239e-05 2024-09-21T22:08:06.454 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:06.454 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:06.454 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:06.454 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.455 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.455 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:702 2.5 e63/37 2:a70ae99f:::rbd_data.126483d75f2d.000000000000000c:head v 63'34, mlcod=63'32)", 2024-09-21T22:08:06.455 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.392532+0000", 2024-09-21T22:08:06.455 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 41.046361535000003, 2024-09-21T22:08:06.455 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.035703990999999997, 2024-09-21T22:08:06.455 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:06.455 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:06.455 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:06.455 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:06.455 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.455 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:06.455 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.392532+0000", 2024-09-21T22:08:06.455 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:06.455 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.456 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.456 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:06.456 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.392532+0000", 2024-09-21T22:08:06.456 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:06.456 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.456 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.456 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:06.456 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.392534+0000", 2024-09-21T22:08:06.456 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.64e-06 2024-09-21T22:08:06.456 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.456 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.456 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:06.456 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.393321+0000", 2024-09-21T22:08:06.456 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00078705699999999997 2024-09-21T22:08:06.457 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.457 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.457 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:06.457 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.393347+0000", 2024-09-21T22:08:06.457 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.6735999999999999e-05 2024-09-21T22:08:06.457 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.457 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.457 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:06.457 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.393361+0000", 2024-09-21T22:08:06.457 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3346000000000001e-05 2024-09-21T22:08:06.457 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.457 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.457 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:06.457 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.393469+0000", 2024-09-21T22:08:06.458 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000108039 2024-09-21T22:08:06.458 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.458 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.458 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:06.458 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.393512+0000", 2024-09-21T22:08:06.458 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.2862999999999999e-05 2024-09-21T22:08:06.458 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.458 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.458 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:06.458 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.428170+0000", 2024-09-21T22:08:06.458 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.034658858000000001 2024-09-21T22:08:06.458 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.458 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.458 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:06.459 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.428236+0000", 2024-09-21T22:08:06.459 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.5451999999999996e-05 2024-09-21T22:08:06.459 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:06.459 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:06.459 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:06.459 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.459 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.459 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:708 2.1 e63/53 2:837ddd83:::rbd_data.126483d75f2d.000000000000000d:head v 63'6, mlcod=57'4)", 2024-09-21T22:08:06.459 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.517007+0000", 2024-09-21T22:08:06.459 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 40.921886608999998, 2024-09-21T22:08:06.459 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.032495584000000001, 2024-09-21T22:08:06.459 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:06.459 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:06.459 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:06.459 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:06.460 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.460 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:06.460 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.517007+0000", 2024-09-21T22:08:06.460 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:06.460 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.460 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.460 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:06.460 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.517007+0000", 2024-09-21T22:08:06.460 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:06.460 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.460 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.460 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:06.460 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.517008+0000", 2024-09-21T22:08:06.460 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.604e-06 2024-09-21T22:08:06.461 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.461 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.461 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:06.461 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.517603+0000", 2024-09-21T22:08:06.461 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00059461900000000001 2024-09-21T22:08:06.461 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.461 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.461 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:06.461 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.517632+0000", 2024-09-21T22:08:06.461 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8663e-05 2024-09-21T22:08:06.461 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.461 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.461 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:06.461 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.517647+0000", 2024-09-21T22:08:06.461 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4833e-05 2024-09-21T22:08:06.462 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.462 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.462 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:06.462 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.517714+0000", 2024-09-21T22:08:06.462 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.7669000000000006e-05 2024-09-21T22:08:06.462 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.462 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.462 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:06.462 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.517737+0000", 2024-09-21T22:08:06.462 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.2469000000000001e-05 2024-09-21T22:08:06.462 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.462 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.462 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:06.463 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.549370+0000", 2024-09-21T22:08:06.463 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.031633221000000003 2024-09-21T22:08:06.463 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.463 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.463 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:06.463 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.549502+0000", 2024-09-21T22:08:06.463 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013250600000000001 2024-09-21T22:08:06.463 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:06.463 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:06.463 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:06.463 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.463 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.463 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:712 2.1 e63/53 2:837ddd83:::rbd_data.126483d75f2d.000000000000000d:head v 63'10, mlcod=63'8)", 2024-09-21T22:08:06.463 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.604029+0000", 2024-09-21T22:08:06.463 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 40.834864670000002, 2024-09-21T22:08:06.464 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.037982371000000001, 2024-09-21T22:08:06.464 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:06.464 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:06.464 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:06.464 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:06.464 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.464 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:06.464 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.604029+0000", 2024-09-21T22:08:06.464 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:06.464 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.464 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.464 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:06.464 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.604029+0000", 2024-09-21T22:08:06.464 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:06.464 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.465 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.465 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:06.465 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.604031+0000", 2024-09-21T22:08:06.465 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8870000000000001e-06 2024-09-21T22:08:06.465 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.465 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.465 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:06.465 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.604395+0000", 2024-09-21T22:08:06.465 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00036422000000000001 2024-09-21T22:08:06.465 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.465 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.465 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:06.465 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.604424+0000", 2024-09-21T22:08:06.465 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8609e-05 2024-09-21T22:08:06.465 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.466 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.466 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:06.466 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.604437+0000", 2024-09-21T22:08:06.466 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3086e-05 2024-09-21T22:08:06.466 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.466 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.466 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:06.466 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.604509+0000", 2024-09-21T22:08:06.466 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.2448999999999997e-05 2024-09-21T22:08:06.466 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.466 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.466 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:06.466 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.604531+0000", 2024-09-21T22:08:06.466 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1602000000000001e-05 2024-09-21T22:08:06.467 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.467 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.467 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:06.467 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.641885+0000", 2024-09-21T22:08:06.467 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.037354545000000003 2024-09-21T22:08:06.467 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.467 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.467 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:06.467 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.642011+0000", 2024-09-21T22:08:06.467 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00012597300000000001 2024-09-21T22:08:06.467 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:06.467 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:06.467 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:06.467 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.467 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.468 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:720 2.1 e63/53 2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head v 63'18, mlcod=63'16)", 2024-09-21T22:08:06.468 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.805111+0000", 2024-09-21T22:08:06.468 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 40.633782609999997, 2024-09-21T22:08:06.468 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.046082864000000001, 2024-09-21T22:08:06.468 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:06.468 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:06.468 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:06.468 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:06.468 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.468 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:06.468 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.805111+0000", 2024-09-21T22:08:06.468 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:06.468 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.468 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.468 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:06.469 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.805111+0000", 2024-09-21T22:08:06.469 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:06.469 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.469 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.469 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:06.469 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.805112+0000", 2024-09-21T22:08:06.469 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.02e-06 2024-09-21T22:08:06.469 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.469 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.469 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:06.469 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.805429+0000", 2024-09-21T22:08:06.469 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00031740799999999998 2024-09-21T22:08:06.469 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.469 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.470 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:06.470 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.805459+0000", 2024-09-21T22:08:06.470 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.955e-05 2024-09-21T22:08:06.470 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.470 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.470 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:06.470 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.805472+0000", 2024-09-21T22:08:06.470 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3533000000000001e-05 2024-09-21T22:08:06.470 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.470 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.470 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:06.470 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.805581+0000", 2024-09-21T22:08:06.470 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000108194 2024-09-21T22:08:06.470 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.471 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.471 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:06.471 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.805615+0000", 2024-09-21T22:08:06.471 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.4310000000000002e-05 2024-09-21T22:08:06.471 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.471 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.471 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:06.471 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.851146+0000", 2024-09-21T22:08:06.471 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.045531092000000002 2024-09-21T22:08:06.471 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.471 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.471 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:06.471 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.851194+0000", 2024-09-21T22:08:06.471 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.7756999999999997e-05 2024-09-21T22:08:06.471 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:06.472 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:06.472 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:06.472 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.472 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.472 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:723 2.1 e63/53 2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head v 63'20, mlcod=63'18)", 2024-09-21T22:08:06.472 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.862341+0000", 2024-09-21T22:08:06.472 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 40.576552675999999, 2024-09-21T22:08:06.472 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.036454671000000001, 2024-09-21T22:08:06.472 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:06.472 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:06.472 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:06.472 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:06.472 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.472 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:06.473 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.862341+0000", 2024-09-21T22:08:06.473 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:06.473 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.473 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.473 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:06.473 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.862341+0000", 2024-09-21T22:08:06.473 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:06.473 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.473 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.473 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:06.473 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.862342+0000", 2024-09-21T22:08:06.473 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.0299999999999997e-07 2024-09-21T22:08:06.473 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.474 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.474 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:06.474 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.862721+0000", 2024-09-21T22:08:06.474 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00037889800000000002 2024-09-21T22:08:06.474 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.474 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.474 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:06.474 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.862738+0000", 2024-09-21T22:08:06.474 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6957999999999999e-05 2024-09-21T22:08:06.474 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.474 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.474 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:06.474 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.862747+0000", 2024-09-21T22:08:06.475 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.0729999999999994e-06 2024-09-21T22:08:06.475 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.475 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.475 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:06.475 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.862830+0000", 2024-09-21T22:08:06.475 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.3113000000000002e-05 2024-09-21T22:08:06.475 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.475 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.475 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:06.475 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.862860+0000", 2024-09-21T22:08:06.475 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9980000000000001e-05 2024-09-21T22:08:06.475 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.475 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.476 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:06.476 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.898705+0000", 2024-09-21T22:08:06.476 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.035845063000000003 2024-09-21T22:08:06.476 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.476 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.476 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:06.476 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.898795+0000", 2024-09-21T22:08:06.476 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.0682999999999999e-05 2024-09-21T22:08:06.476 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:06.476 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:06.476 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:06.476 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.476 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.476 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:136 2.5 e72/37 2:b1d55633:::rbd_data.126483d75f2d.000000000000000b:head v 72'44, mlcod=72'42)", 2024-09-21T22:08:06.477 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.386273+0000", 2024-09-21T22:08:06.477 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 28.052620561000001, 2024-09-21T22:08:06.477 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.033806744999999999, 2024-09-21T22:08:06.477 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:06.477 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:06.477 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:06.477 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:06.477 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.477 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:06.477 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.386273+0000", 2024-09-21T22:08:06.478 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:06.478 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.478 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.478 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:06.478 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.386273+0000", 2024-09-21T22:08:06.478 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:06.478 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.478 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.478 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:06.478 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.386274+0000", 2024-09-21T22:08:06.478 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.8299999999999996e-07 2024-09-21T22:08:06.478 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.478 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.478 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:06.479 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.387108+0000", 2024-09-21T22:08:06.479 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00083467800000000005 2024-09-21T22:08:06.479 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.479 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.479 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:06.479 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.387121+0000", 2024-09-21T22:08:06.479 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2823000000000001e-05 2024-09-21T22:08:06.479 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.479 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.479 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:06.479 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.387128+0000", 2024-09-21T22:08:06.479 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.404e-06 2024-09-21T22:08:06.479 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.479 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.480 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:06.480 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.387179+0000", 2024-09-21T22:08:06.480 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.1888000000000002e-05 2024-09-21T22:08:06.480 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.480 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.480 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:06.480 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.387201+0000", 2024-09-21T22:08:06.480 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.2019999999999999e-05 2024-09-21T22:08:06.480 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.480 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.480 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:06.480 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.419993+0000", 2024-09-21T22:08:06.480 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.032791182000000002 2024-09-21T22:08:06.480 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.481 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.481 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:06.481 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.420080+0000", 2024-09-21T22:08:06.481 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.7066999999999998e-05 2024-09-21T22:08:06.481 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:06.481 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:06.481 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:06.481 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.481 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.481 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:138 2.5 e72/37 2:b1d55633:::rbd_data.126483d75f2d.000000000000000b:head v 72'46, mlcod=72'44)", 2024-09-21T22:08:06.481 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.430104+0000", 2024-09-21T22:08:06.481 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 28.008789213, 2024-09-21T22:08:06.481 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.034087458000000001, 2024-09-21T22:08:06.481 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:06.482 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:06.482 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:06.482 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:06.482 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.482 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:06.482 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.430104+0000", 2024-09-21T22:08:06.482 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:06.482 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.482 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.482 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:06.482 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.430104+0000", 2024-09-21T22:08:06.482 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:06.482 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.482 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.482 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:06.483 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.430105+0000", 2024-09-21T22:08:06.483 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.1680000000000001e-06 2024-09-21T22:08:06.483 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.483 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.483 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:06.483 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.430864+0000", 2024-09-21T22:08:06.483 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00075819400000000001 2024-09-21T22:08:06.483 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.483 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.483 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:06.483 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.430881+0000", 2024-09-21T22:08:06.483 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7000999999999999e-05 2024-09-21T22:08:06.483 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.483 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.484 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:06.484 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.430889+0000", 2024-09-21T22:08:06.484 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.9540000000000004e-06 2024-09-21T22:08:06.484 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.484 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.484 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:06.484 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.430959+0000", 2024-09-21T22:08:06.484 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.0858999999999994e-05 2024-09-21T22:08:06.484 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.484 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.484 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:06.484 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.430996+0000", 2024-09-21T22:08:06.484 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.6443999999999999e-05 2024-09-21T22:08:06.484 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.484 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.485 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:06.485 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.464087+0000", 2024-09-21T22:08:06.485 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.033090799999999997 2024-09-21T22:08:06.485 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.485 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.485 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:06.485 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.464192+0000", 2024-09-21T22:08:06.485 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00010503800000000001 2024-09-21T22:08:06.485 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:06.485 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:06.485 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:06.485 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.485 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.485 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:140 2.5 e72/37 2:b1d55633:::rbd_data.126483d75f2d.000000000000000b:head v 72'48, mlcod=72'46)", 2024-09-21T22:08:06.485 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.475737+0000", 2024-09-21T22:08:06.486 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 27.963156264999999, 2024-09-21T22:08:06.486 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.037632025, 2024-09-21T22:08:06.486 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:06.486 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:06.486 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:06.486 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:06.486 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.486 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:06.486 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.475737+0000", 2024-09-21T22:08:06.486 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:06.486 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.486 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.486 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:06.486 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.475737+0000", 2024-09-21T22:08:06.487 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:06.487 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.487 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.487 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:06.487 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.475739+0000", 2024-09-21T22:08:06.487 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.666e-06 2024-09-21T22:08:06.487 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.487 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.487 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:06.487 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.476538+0000", 2024-09-21T22:08:06.487 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00079927399999999999 2024-09-21T22:08:06.487 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.487 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.487 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:06.487 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.476556+0000", 2024-09-21T22:08:06.488 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7666999999999999e-05 2024-09-21T22:08:06.488 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.488 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.488 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:06.488 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.476565+0000", 2024-09-21T22:08:06.488 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.0650000000000005e-06 2024-09-21T22:08:06.488 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.488 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.488 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:06.488 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.476651+0000", 2024-09-21T22:08:06.488 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.5916999999999997e-05 2024-09-21T22:08:06.488 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.488 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.488 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:06.488 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.476696+0000", 2024-09-21T22:08:06.489 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.5537000000000001e-05 2024-09-21T22:08:06.489 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.489 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.489 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:06.489 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.513300+0000", 2024-09-21T22:08:06.489 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.036603498999999998 2024-09-21T22:08:06.489 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.489 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.489 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:06.489 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.513369+0000", 2024-09-21T22:08:06.490 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.9400000000000006e-05 2024-09-21T22:08:06.490 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:06.490 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:06.490 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:06.490 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.490 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.490 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:145 2.5 e72/37 2:a70ae99f:::rbd_data.126483d75f2d.000000000000000c:head v 72'52, mlcod=72'50)", 2024-09-21T22:08:06.490 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.578171+0000", 2024-09-21T22:08:06.490 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 27.860722415000001, 2024-09-21T22:08:06.490 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.037181537000000001, 2024-09-21T22:08:06.490 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:06.490 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:06.490 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:06.490 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:06.491 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.491 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:06.491 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.578171+0000", 2024-09-21T22:08:06.491 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:06.491 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.491 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.491 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:06.491 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.578171+0000", 2024-09-21T22:08:06.491 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:06.491 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.491 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.491 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:06.491 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.578172+0000", 2024-09-21T22:08:06.491 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.017e-06 2024-09-21T22:08:06.492 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.492 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.492 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:06.492 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.579000+0000", 2024-09-21T22:08:06.492 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00082799499999999999 2024-09-21T22:08:06.492 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.492 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.492 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:06.492 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.579020+0000", 2024-09-21T22:08:06.492 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.0165000000000001e-05 2024-09-21T22:08:06.492 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.492 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.492 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:06.492 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.579036+0000", 2024-09-21T22:08:06.492 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5767e-05 2024-09-21T22:08:06.493 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.493 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.493 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:06.493 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.579106+0000", 2024-09-21T22:08:06.493 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.0091000000000004e-05 2024-09-21T22:08:06.493 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.493 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.493 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:06.493 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.579131+0000", 2024-09-21T22:08:06.493 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4978e-05 2024-09-21T22:08:06.493 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.493 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.493 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:06.493 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.615249+0000", 2024-09-21T22:08:06.494 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.036118355999999997 2024-09-21T22:08:06.494 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.494 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.494 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:06.494 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.615353+0000", 2024-09-21T22:08:06.494 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000103168 2024-09-21T22:08:06.494 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:06.494 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:06.494 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:06.494 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.494 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.494 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:147 2.5 e72/37 2:a70ae99f:::rbd_data.126483d75f2d.000000000000000c:head v 72'54, mlcod=72'52)", 2024-09-21T22:08:06.494 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.627203+0000", 2024-09-21T22:08:06.494 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 27.811690302999999, 2024-09-21T22:08:06.494 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.038203972000000003, 2024-09-21T22:08:06.495 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:06.495 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:06.495 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:06.495 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:06.495 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.495 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:06.495 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.627203+0000", 2024-09-21T22:08:06.495 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:06.495 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.495 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.495 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:06.495 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.627203+0000", 2024-09-21T22:08:06.495 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:06.495 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.496 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.496 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:06.496 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.627204+0000", 2024-09-21T22:08:06.496 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3069999999999999e-06 2024-09-21T22:08:06.496 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.496 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.496 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:06.496 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.628114+0000", 2024-09-21T22:08:06.496 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00090942099999999999 2024-09-21T22:08:06.496 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.496 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.496 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:06.497 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.628140+0000", 2024-09-21T22:08:06.497 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5882000000000001e-05 2024-09-21T22:08:06.497 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.497 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.497 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:06.497 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.628162+0000", 2024-09-21T22:08:06.497 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.2293000000000002e-05 2024-09-21T22:08:06.497 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.497 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.497 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:06.497 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.628250+0000", 2024-09-21T22:08:06.497 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.7973000000000006e-05 2024-09-21T22:08:06.497 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.497 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.498 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:06.498 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.628272+0000", 2024-09-21T22:08:06.498 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1682999999999999e-05 2024-09-21T22:08:06.498 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.498 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.498 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:06.498 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.665322+0000", 2024-09-21T22:08:06.498 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.037050650999999997 2024-09-21T22:08:06.498 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.498 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.498 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:06.498 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.665407+0000", 2024-09-21T22:08:06.498 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.4761999999999999e-05 2024-09-21T22:08:06.498 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:06.499 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:06.499 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:06.499 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.499 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.499 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:149 2.5 e72/37 2:a70ae99f:::rbd_data.126483d75f2d.000000000000000c:head v 72'56, mlcod=72'54)", 2024-09-21T22:08:06.499 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.690377+0000", 2024-09-21T22:08:06.499 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 27.748516559999999, 2024-09-21T22:08:06.499 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.049254799000000002, 2024-09-21T22:08:06.499 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:06.499 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:06.499 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:06.500 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:06.500 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.500 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:06.500 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.690377+0000", 2024-09-21T22:08:06.500 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:06.500 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.500 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.500 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:06.501 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.690377+0000", 2024-09-21T22:08:06.501 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:06.501 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.501 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.501 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:06.501 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.690378+0000", 2024-09-21T22:08:06.501 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.344e-06 2024-09-21T22:08:06.501 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.501 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.501 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:06.501 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.691302+0000", 2024-09-21T22:08:06.501 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00092322400000000005 2024-09-21T22:08:06.501 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.501 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.502 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:06.502 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.691321+0000", 2024-09-21T22:08:06.502 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9928999999999998e-05 2024-09-21T22:08:06.502 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.502 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.502 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:06.502 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.691345+0000", 2024-09-21T22:08:06.502 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.372e-05 2024-09-21T22:08:06.502 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.502 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.502 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:06.502 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.691442+0000", 2024-09-21T22:08:06.502 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.7270999999999998e-05 2024-09-21T22:08:06.502 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.503 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.503 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:06.503 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.691512+0000", 2024-09-21T22:08:06.503 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.9183000000000005e-05 2024-09-21T22:08:06.503 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.503 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.503 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:06.503 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.739545+0000", 2024-09-21T22:08:06.503 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.048033657 2024-09-21T22:08:06.503 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.503 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.503 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:06.503 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.739632+0000", 2024-09-21T22:08:06.503 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.6471000000000006e-05 2024-09-21T22:08:06.503 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:06.504 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:06.504 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:06.504 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.504 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.504 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:156 2.1 e72/53 2:837ddd83:::rbd_data.126483d75f2d.000000000000000d:head v 72'28, mlcod=72'26)", 2024-09-21T22:08:06.504 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.824856+0000", 2024-09-21T22:08:06.504 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 27.614037121999999, 2024-09-21T22:08:06.504 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.034320985999999998, 2024-09-21T22:08:06.504 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:06.504 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:06.504 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:06.504 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:06.504 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.504 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:06.505 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.824856+0000", 2024-09-21T22:08:06.505 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:06.505 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.505 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.505 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:06.505 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.824856+0000", 2024-09-21T22:08:06.505 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:06.505 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.505 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.505 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:06.505 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.824858+0000", 2024-09-21T22:08:06.505 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2979999999999999e-06 2024-09-21T22:08:06.505 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.506 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.506 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:06.506 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.825174+0000", 2024-09-21T22:08:06.506 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000316335 2024-09-21T22:08:06.506 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.506 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.506 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:06.506 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.825193+0000", 2024-09-21T22:08:06.506 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9436000000000002e-05 2024-09-21T22:08:06.506 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.506 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.506 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:06.506 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.825202+0000", 2024-09-21T22:08:06.506 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.8359999999999998e-06 2024-09-21T22:08:06.507 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.507 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.507 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:06.507 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.825286+0000", 2024-09-21T22:08:06.507 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.3683999999999994e-05 2024-09-21T22:08:06.507 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.507 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.507 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:06.507 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.825314+0000", 2024-09-21T22:08:06.507 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.7653000000000001e-05 2024-09-21T22:08:06.507 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.507 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.507 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:06.507 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.859122+0000", 2024-09-21T22:08:06.507 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.033808624000000002 2024-09-21T22:08:06.508 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.508 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.508 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:06.508 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.859177+0000", 2024-09-21T22:08:06.508 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.5120000000000001e-05 2024-09-21T22:08:06.508 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:06.508 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:06.508 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:06.508 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.508 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.508 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:158 2.1 e72/53 2:837ddd83:::rbd_data.126483d75f2d.000000000000000d:head v 72'30, mlcod=72'28)", 2024-09-21T22:08:06.508 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.869277+0000", 2024-09-21T22:08:06.508 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 27.569616924999998, 2024-09-21T22:08:06.508 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.040508390999999998, 2024-09-21T22:08:06.508 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:06.509 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:06.509 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:06.509 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:06.509 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.509 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:06.509 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.869277+0000", 2024-09-21T22:08:06.509 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:06.509 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.509 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.509 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:06.509 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.869277+0000", 2024-09-21T22:08:06.509 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:06.509 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.509 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.509 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:06.510 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.869278+0000", 2024-09-21T22:08:06.510 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5269999999999999e-06 2024-09-21T22:08:06.510 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.510 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.510 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:06.510 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.869902+0000", 2024-09-21T22:08:06.510 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00062399000000000005 2024-09-21T22:08:06.510 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.510 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.510 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:06.510 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.869924+0000", 2024-09-21T22:08:06.510 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1875999999999999e-05 2024-09-21T22:08:06.510 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.510 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.511 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:06.511 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.869937+0000", 2024-09-21T22:08:06.511 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2685e-05 2024-09-21T22:08:06.511 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.511 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.511 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:06.511 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.870058+0000", 2024-09-21T22:08:06.511 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000120862 2024-09-21T22:08:06.511 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.511 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.511 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:06.511 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.870093+0000", 2024-09-21T22:08:06.511 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.5896000000000003e-05 2024-09-21T22:08:06.511 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.512 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.512 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:06.512 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.909728+0000", 2024-09-21T22:08:06.512 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.039634461000000003 2024-09-21T22:08:06.512 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.512 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.512 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:06.512 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.909785+0000", 2024-09-21T22:08:06.512 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.7093999999999999e-05 2024-09-21T22:08:06.512 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:06.512 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:06.512 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:06.512 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.512 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.512 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:161 2.1 e72/53 2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head v 72'32, mlcod=72'30)", 2024-09-21T22:08:06.513 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.924455+0000", 2024-09-21T22:08:06.513 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 27.514438562999999, 2024-09-21T22:08:06.513 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.038366863000000001, 2024-09-21T22:08:06.513 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:06.513 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:06.513 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:06.513 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:06.513 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.513 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:06.513 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.924455+0000", 2024-09-21T22:08:06.513 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:06.513 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.513 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.513 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:06.513 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.924455+0000", 2024-09-21T22:08:06.514 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:06.514 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.514 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.514 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:06.514 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.924456+0000", 2024-09-21T22:08:06.514 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.003e-06 2024-09-21T22:08:06.514 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.514 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.514 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:06.514 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.924811+0000", 2024-09-21T22:08:06.514 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00035519499999999999 2024-09-21T22:08:06.514 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.514 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.514 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:06.515 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.924826+0000", 2024-09-21T22:08:06.515 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5296999999999999e-05 2024-09-21T22:08:06.515 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.515 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.515 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:06.515 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.924834+0000", 2024-09-21T22:08:06.515 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.5379999999999997e-06 2024-09-21T22:08:06.515 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.515 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.515 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:06.515 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.924910+0000", 2024-09-21T22:08:06.515 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.6488000000000004e-05 2024-09-21T22:08:06.515 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.515 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.516 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:06.516 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.924934+0000", 2024-09-21T22:08:06.516 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3598999999999999e-05 2024-09-21T22:08:06.516 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.516 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.516 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:06.516 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.962739+0000", 2024-09-21T22:08:06.516 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.037805370999999997 2024-09-21T22:08:06.516 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.516 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.516 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:06.516 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.962822+0000", 2024-09-21T22:08:06.516 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.2372000000000003e-05 2024-09-21T22:08:06.516 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:06.516 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:06.517 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:06.517 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.517 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.517 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:163 2.1 e72/53 2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head v 72'34, mlcod=72'32)", 2024-09-21T22:08:06.517 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.971683+0000", 2024-09-21T22:08:06.517 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 27.467210023, 2024-09-21T22:08:06.517 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.043171320999999999, 2024-09-21T22:08:06.517 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:06.517 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:06.517 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:06.517 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:06.517 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.517 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:06.517 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.971683+0000", 2024-09-21T22:08:06.517 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:06.518 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.518 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.518 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:06.518 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.971683+0000", 2024-09-21T22:08:06.518 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:06.518 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.518 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.518 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:06.518 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.971684+0000", 2024-09-21T22:08:06.518 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.6600000000000005e-07 2024-09-21T22:08:06.518 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.518 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.518 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:06.518 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.971945+0000", 2024-09-21T22:08:06.519 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00026101899999999998 2024-09-21T22:08:06.519 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.519 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.519 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:06.519 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.971959+0000", 2024-09-21T22:08:06.519 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4054e-05 2024-09-21T22:08:06.519 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.519 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.519 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:06.519 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.971969+0000", 2024-09-21T22:08:06.519 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.499e-06 2024-09-21T22:08:06.519 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.519 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.519 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:06.519 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.972037+0000", 2024-09-21T22:08:06.520 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.8448000000000006e-05 2024-09-21T22:08:06.520 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.520 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.520 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:06.520 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.972057+0000", 2024-09-21T22:08:06.520 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9817999999999999e-05 2024-09-21T22:08:06.520 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.520 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.520 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:06.520 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.014770+0000", 2024-09-21T22:08:06.520 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.042712464999999998 2024-09-21T22:08:06.520 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.520 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.520 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:06.520 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.014855+0000", 2024-09-21T22:08:06.521 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.5152000000000001e-05 2024-09-21T22:08:06.521 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:06.521 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:06.521 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:06.521 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.521 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.521 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:165 2.1 e72/53 2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head v 72'36, mlcod=72'34)", 2024-09-21T22:08:06.521 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:39.028631+0000", 2024-09-21T22:08:06.521 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 27.410262545999998, 2024-09-21T22:08:06.521 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.033430561999999997, 2024-09-21T22:08:06.521 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:06.521 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:06.521 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:06.521 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:06.521 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.522 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:06.522 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.028631+0000", 2024-09-21T22:08:06.522 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:06.522 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.522 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.522 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:06.522 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.028631+0000", 2024-09-21T22:08:06.522 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:06.522 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.522 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.522 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:06.522 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.028633+0000", 2024-09-21T22:08:06.522 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5799999999999999e-06 2024-09-21T22:08:06.522 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.523 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.523 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:06.523 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.029209+0000", 2024-09-21T22:08:06.523 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00057670799999999999 2024-09-21T22:08:06.523 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.523 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.523 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:06.523 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.029238+0000", 2024-09-21T22:08:06.523 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8943000000000001e-05 2024-09-21T22:08:06.523 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.523 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.523 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:06.523 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.029252+0000", 2024-09-21T22:08:06.523 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3672999999999999e-05 2024-09-21T22:08:06.523 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.524 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.524 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:06.524 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.029328+0000", 2024-09-21T22:08:06.524 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.6497999999999998e-05 2024-09-21T22:08:06.524 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.524 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.524 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:06.524 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.029353+0000", 2024-09-21T22:08:06.524 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5060999999999999e-05 2024-09-21T22:08:06.524 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.524 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.524 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:06.524 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.061993+0000", 2024-09-21T22:08:06.524 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.032639286000000003 2024-09-21T22:08:06.525 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.525 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.525 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:06.525 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.062062+0000", 2024-09-21T22:08:06.525 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.8813000000000007e-05 2024-09-21T22:08:06.525 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:06.525 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:06.525 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:06.525 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.525 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.525 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:167 2.1 e72/53 2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head v 72'38, mlcod=72'36)", 2024-09-21T22:08:06.525 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:39.077324+0000", 2024-09-21T22:08:06.525 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 27.361569732, 2024-09-21T22:08:06.525 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.036031961000000001, 2024-09-21T22:08:06.525 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:06.526 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:06.526 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:06.526 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:06.526 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.526 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:06.526 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.077324+0000", 2024-09-21T22:08:06.526 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:06.526 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.526 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.526 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:06.526 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.077324+0000", 2024-09-21T22:08:06.526 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:06.526 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.526 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.527 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:06.527 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.077326+0000", 2024-09-21T22:08:06.527 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.75e-06 2024-09-21T22:08:06.527 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.527 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.527 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:06.527 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.077719+0000", 2024-09-21T22:08:06.527 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00039322599999999999 2024-09-21T22:08:06.527 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.527 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.527 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:06.527 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.077749+0000", 2024-09-21T22:08:06.527 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9895000000000001e-05 2024-09-21T22:08:06.527 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.527 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.528 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:06.528 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.077763+0000", 2024-09-21T22:08:06.528 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4517e-05 2024-09-21T22:08:06.528 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.528 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.528 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:06.528 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.077843+0000", 2024-09-21T22:08:06.528 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.9994999999999996e-05 2024-09-21T22:08:06.528 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.528 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.528 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:06.528 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.077867+0000", 2024-09-21T22:08:06.528 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3640000000000001e-05 2024-09-21T22:08:06.528 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.529 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.529 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:06.529 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.113216+0000", 2024-09-21T22:08:06.529 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.035349168 2024-09-21T22:08:06.529 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:06.529 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:06.529 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:06.529 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.113356+0000", 2024-09-21T22:08:06.529 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013977 2024-09-21T22:08:06.529 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:06.529 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:06.529 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:06.529 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:06.529 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:06.530 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:08:06.531 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T22:08:06.531 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:08:06.528+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:08:06.632 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:08:06.632+0000 7f2075a06640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31720) UID: 0 2024-09-21T22:08:06.702 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:08:06.702 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:08:06.702 INFO:teuthology.orchestra.run.smithi039.stdout: "num_ops": 0 2024-09-21T22:08:06.702 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:08:06.717 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T22:08:06.764 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:08:06.761+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:08:06.834 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:08:06.829+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:08:06.836 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:08:06.836 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:08:06.836 INFO:teuthology.orchestra.run.smithi039.stdout: "complaint_time": 30, 2024-09-21T22:08:06.836 INFO:teuthology.orchestra.run.smithi039.stdout: "num_blocked_ops": 0 2024-09-21T22:08:06.836 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:08:06.851 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T22:08:06.936 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:08:06.932+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:08:07.016 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:08:07.016 INFO:teuthology.orchestra.run.smithi039.stdout: "size": 20, 2024-09-21T22:08:07.016 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 600, 2024-09-21T22:08:07.016 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [ 2024-09-21T22:08:07.016 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.016 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.3 55/54 [PushOp(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head, version: 52'2, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head@52'2, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T22:08:07.017 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.084379+0000", 2024-09-21T22:08:07.017 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 54.927824129999998, 2024-09-21T22:08:07.017 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.051020674000000002, 2024-09-21T22:08:07.017 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:07.017 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:07.017 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:08:07.017 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:07.017 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.017 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:07.017 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.084379+0000", 2024-09-21T22:08:07.017 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:07.017 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.017 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.017 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:07.018 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.084379+0000", 2024-09-21T22:08:07.018 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:07.018 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.018 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.018 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:07.018 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.084381+0000", 2024-09-21T22:08:07.018 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.73e-06 2024-09-21T22:08:07.018 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.018 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.018 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:07.018 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.085494+0000", 2024-09-21T22:08:07.018 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011131559999999999 2024-09-21T22:08:07.018 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.018 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.019 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:07.019 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.085542+0000", 2024-09-21T22:08:07.019 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8334999999999997e-05 2024-09-21T22:08:07.019 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.019 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.019 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:07.019 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.085591+0000", 2024-09-21T22:08:07.019 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.9165000000000001e-05 2024-09-21T22:08:07.019 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.019 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.019 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:07.019 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.085777+0000", 2024-09-21T22:08:07.019 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000186152 2024-09-21T22:08:07.019 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.019 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.020 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:07.020 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.085796+0000", 2024-09-21T22:08:07.020 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8612999999999999e-05 2024-09-21T22:08:07.020 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.020 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.020 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:07.020 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.135399+0000", 2024-09-21T22:08:07.020 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.049603522999999997 2024-09-21T22:08:07.020 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:07.020 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:07.020 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:07.020 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.020 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.020 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.3 55/54 [PushOp(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head, version: 52'2, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head@52'2, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:08:07.021 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.139554+0000", 2024-09-21T22:08:07.021 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 54.872649027000001, 2024-09-21T22:08:07.021 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.056215389999999997, 2024-09-21T22:08:07.021 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:07.021 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:07.021 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:08:07.021 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:07.021 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.021 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:07.021 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.139554+0000", 2024-09-21T22:08:07.021 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:07.021 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.021 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.021 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:07.021 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.139554+0000", 2024-09-21T22:08:07.022 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:07.022 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.022 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.022 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:07.022 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.139556+0000", 2024-09-21T22:08:07.022 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7999999999999999e-06 2024-09-21T22:08:07.022 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.022 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.022 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:07.022 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.140671+0000", 2024-09-21T22:08:07.022 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001115612 2024-09-21T22:08:07.022 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.022 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.022 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:07.023 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.140720+0000", 2024-09-21T22:08:07.023 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8631999999999998e-05 2024-09-21T22:08:07.023 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.023 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.023 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:07.023 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.140752+0000", 2024-09-21T22:08:07.023 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.2199000000000001e-05 2024-09-21T22:08:07.023 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.023 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.023 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:07.023 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.140900+0000", 2024-09-21T22:08:07.023 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00014786000000000001 2024-09-21T22:08:07.023 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.023 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.023 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:07.024 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.140917+0000", 2024-09-21T22:08:07.024 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6753999999999999e-05 2024-09-21T22:08:07.024 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.024 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.024 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:07.024 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.195769+0000", 2024-09-21T22:08:07.024 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.054852533000000002 2024-09-21T22:08:07.024 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:07.024 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:07.024 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:07.024 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.024 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.024 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.3 55/54 [PushOp(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head, version: 52'2, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head@52'2, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:08:07.024 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.199187+0000", 2024-09-21T22:08:07.024 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 54.813015483000001, 2024-09-21T22:08:07.025 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.051543975999999998, 2024-09-21T22:08:07.025 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:07.025 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:07.025 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:08:07.025 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:07.025 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.025 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:07.025 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.199187+0000", 2024-09-21T22:08:07.025 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:07.026 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.026 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.026 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:07.026 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.199187+0000", 2024-09-21T22:08:07.026 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:07.026 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.026 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.026 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:07.026 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.199189+0000", 2024-09-21T22:08:07.026 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7969999999999999e-06 2024-09-21T22:08:07.026 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.026 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.027 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:07.027 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.200355+0000", 2024-09-21T22:08:07.027 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011660139999999999 2024-09-21T22:08:07.027 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.027 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.027 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:07.027 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.200405+0000", 2024-09-21T22:08:07.027 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.0124000000000003e-05 2024-09-21T22:08:07.027 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.027 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.027 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:07.027 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.200457+0000", 2024-09-21T22:08:07.028 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.1177999999999997e-05 2024-09-21T22:08:07.028 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.028 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.028 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:07.028 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.200635+0000", 2024-09-21T22:08:07.028 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00017841299999999999 2024-09-21T22:08:07.028 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.028 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.028 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:07.028 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.200653+0000", 2024-09-21T22:08:07.028 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7944999999999998e-05 2024-09-21T22:08:07.029 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.029 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.029 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:07.029 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.250731+0000", 2024-09-21T22:08:07.029 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.050078505000000002 2024-09-21T22:08:07.029 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:07.029 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:07.029 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:07.029 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.029 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.029 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.3 55/54 [PushOp(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head, version: 52'3, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head@52'3, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:08:07.029 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.343754+0000", 2024-09-21T22:08:07.030 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 54.668448771999998, 2024-09-21T22:08:07.030 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.063560607000000005, 2024-09-21T22:08:07.030 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:07.030 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:07.030 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:08:07.030 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:07.030 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.030 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:07.030 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.343754+0000", 2024-09-21T22:08:07.030 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:07.030 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.030 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.031 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:07.031 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.343754+0000", 2024-09-21T22:08:07.031 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:07.031 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.031 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.031 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:07.031 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.343756+0000", 2024-09-21T22:08:07.031 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7069999999999999e-06 2024-09-21T22:08:07.031 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.031 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.031 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:07.031 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.344864+0000", 2024-09-21T22:08:07.032 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011085159999999999 2024-09-21T22:08:07.032 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.032 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.032 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:07.032 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.344914+0000", 2024-09-21T22:08:07.032 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.9462000000000002e-05 2024-09-21T22:08:07.032 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.032 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.032 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:07.032 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.344961+0000", 2024-09-21T22:08:07.032 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.7633999999999998e-05 2024-09-21T22:08:07.033 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.033 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.033 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:07.033 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.345171+0000", 2024-09-21T22:08:07.033 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000209652 2024-09-21T22:08:07.033 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.033 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.033 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:07.033 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.345193+0000", 2024-09-21T22:08:07.033 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1985999999999999e-05 2024-09-21T22:08:07.033 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.033 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.034 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:07.034 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.407315+0000", 2024-09-21T22:08:07.034 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.06212165 2024-09-21T22:08:07.034 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:07.034 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:07.034 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:07.034 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.034 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.034 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.3 55/54 [PushOp(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head, version: 52'3, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head@52'3, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:08:07.034 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.410551+0000", 2024-09-21T22:08:07.034 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 54.601651640999997, 2024-09-21T22:08:07.034 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.054517522999999998, 2024-09-21T22:08:07.035 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:07.035 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:07.035 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:08:07.035 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:07.035 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.035 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:07.035 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.410551+0000", 2024-09-21T22:08:07.035 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:07.035 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.035 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.035 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:07.036 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.410551+0000", 2024-09-21T22:08:07.036 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:07.036 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.036 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.036 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:07.036 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.410553+0000", 2024-09-21T22:08:07.036 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5200000000000001e-06 2024-09-21T22:08:07.036 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.036 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.036 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:07.036 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.411747+0000", 2024-09-21T22:08:07.036 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011946089999999999 2024-09-21T22:08:07.037 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.037 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.037 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:07.037 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.411796+0000", 2024-09-21T22:08:07.037 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8717999999999997e-05 2024-09-21T22:08:07.037 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.037 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.037 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:07.037 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.411844+0000", 2024-09-21T22:08:07.037 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8291999999999998e-05 2024-09-21T22:08:07.037 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.037 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.037 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:07.038 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.412066+0000", 2024-09-21T22:08:07.038 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00022115899999999999 2024-09-21T22:08:07.038 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.038 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.038 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:07.038 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.412081+0000", 2024-09-21T22:08:07.038 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5654e-05 2024-09-21T22:08:07.038 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.038 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.038 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:07.038 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.465069+0000", 2024-09-21T22:08:07.038 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.052987570999999997 2024-09-21T22:08:07.039 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:07.039 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:07.039 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:07.039 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.039 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.039 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.3 55/54 [PushOp(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head, version: 52'3, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head@52'3, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 4194304, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:08:07.039 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.468339+0000", 2024-09-21T22:08:07.039 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 54.543863977000001, 2024-09-21T22:08:07.039 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.064010024999999998, 2024-09-21T22:08:07.039 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:07.039 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:07.039 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:08:07.040 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:07.040 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.040 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:07.040 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.468339+0000", 2024-09-21T22:08:07.040 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:07.040 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.040 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.040 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:07.040 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.468339+0000", 2024-09-21T22:08:07.040 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:07.040 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.040 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.041 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:07.041 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.468340+0000", 2024-09-21T22:08:07.041 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4330000000000001e-06 2024-09-21T22:08:07.041 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.041 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.041 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:07.041 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.469484+0000", 2024-09-21T22:08:07.041 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001143535 2024-09-21T22:08:07.041 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.041 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.041 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:07.041 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.469533+0000", 2024-09-21T22:08:07.042 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8980999999999997e-05 2024-09-21T22:08:07.042 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.042 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.042 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:07.042 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.469581+0000", 2024-09-21T22:08:07.042 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8551999999999999e-05 2024-09-21T22:08:07.042 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.042 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.042 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:07.042 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.469804+0000", 2024-09-21T22:08:07.042 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00022287699999999999 2024-09-21T22:08:07.043 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.043 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.043 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:07.043 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.469829+0000", 2024-09-21T22:08:07.043 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4766e-05 2024-09-21T22:08:07.043 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.043 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.043 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:07.043 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.532349+0000", 2024-09-21T22:08:07.043 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.062519880999999999 2024-09-21T22:08:07.043 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:07.043 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:07.044 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:07.044 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.044 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.044 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4840.0:668 2.3 2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:08:07.044 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.409114+0000", 2024-09-21T22:08:07.044 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 42.603088771000003, 2024-09-21T22:08:07.044 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.048703758999999999, 2024-09-21T22:08:07.044 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:07.044 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:07.044 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:07.044 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-21T22:08:07.044 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4840", 2024-09-21T22:08:07.045 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:08:07.045 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 668 2024-09-21T22:08:07.045 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.045 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:07.045 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.045 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:07.045 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.409114+0000", 2024-09-21T22:08:07.045 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:07.045 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.045 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.045 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:07.045 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.409114+0000", 2024-09-21T22:08:07.046 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:07.046 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.046 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.046 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:07.046 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.409115+0000", 2024-09-21T22:08:07.046 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.114e-06 2024-09-21T22:08:07.046 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.046 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.046 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:07.046 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.410308+0000", 2024-09-21T22:08:07.046 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011930949999999999 2024-09-21T22:08:07.047 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.047 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.047 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:07.047 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.410325+0000", 2024-09-21T22:08:07.047 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7088e-05 2024-09-21T22:08:07.047 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.047 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.047 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:07.047 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.410334+0000", 2024-09-21T22:08:07.047 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.2549999999999997e-06 2024-09-21T22:08:07.047 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.048 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.048 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:07.048 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.410393+0000", 2024-09-21T22:08:07.048 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.9327999999999998e-05 2024-09-21T22:08:07.048 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.048 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.048 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:07.048 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.410482+0000", 2024-09-21T22:08:07.048 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.9019999999999998e-05 2024-09-21T22:08:07.048 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.048 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.048 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-21T22:08:07.049 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.410617+0000", 2024-09-21T22:08:07.049 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000135232 2024-09-21T22:08:07.049 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.049 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.049 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:07.049 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.457051+0000", 2024-09-21T22:08:07.049 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.04643415 2024-09-21T22:08:07.049 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.049 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.049 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-21T22:08:07.049 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.457675+0000", 2024-09-21T22:08:07.049 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00062332700000000002 2024-09-21T22:08:07.049 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.050 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.050 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:07.050 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.457735+0000", 2024-09-21T22:08:07.050 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.029e-05 2024-09-21T22:08:07.050 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.050 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.050 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:07.050 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.457818+0000", 2024-09-21T22:08:07.050 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.2860000000000003e-05 2024-09-21T22:08:07.050 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:07.050 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:07.050 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:07.051 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.051 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.051 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4840.0:670 2.3 2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:08:07.051 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.466980+0000", 2024-09-21T22:08:07.051 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 42.545222703, 2024-09-21T22:08:07.051 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.050913930000000003, 2024-09-21T22:08:07.051 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:07.051 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:07.051 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:07.051 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-21T22:08:07.051 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4840", 2024-09-21T22:08:07.051 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:08:07.052 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 670 2024-09-21T22:08:07.052 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.052 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:07.052 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.052 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:07.052 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.466980+0000", 2024-09-21T22:08:07.052 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:07.052 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.052 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.052 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:07.052 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.466980+0000", 2024-09-21T22:08:07.052 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:07.052 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.053 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.053 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:07.053 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.466981+0000", 2024-09-21T22:08:07.053 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0839999999999999e-06 2024-09-21T22:08:07.053 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.053 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.053 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:07.053 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.467815+0000", 2024-09-21T22:08:07.053 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00083330399999999999 2024-09-21T22:08:07.053 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.053 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.053 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:07.054 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.467832+0000", 2024-09-21T22:08:07.054 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7601e-05 2024-09-21T22:08:07.054 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.054 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.054 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:07.054 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.467840+0000", 2024-09-21T22:08:07.054 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.2220000000000002e-06 2024-09-21T22:08:07.054 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.054 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.054 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:07.054 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.467974+0000", 2024-09-21T22:08:07.054 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000133411 2024-09-21T22:08:07.055 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.055 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.055 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:07.055 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.468223+0000", 2024-09-21T22:08:07.055 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00024906400000000001 2024-09-21T22:08:07.055 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.055 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.055 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-21T22:08:07.055 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.468616+0000", 2024-09-21T22:08:07.055 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000393433 2024-09-21T22:08:07.055 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.055 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.055 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:07.056 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.517016+0000", 2024-09-21T22:08:07.056 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.048399708 2024-09-21T22:08:07.056 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.056 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.056 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-21T22:08:07.056 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.517693+0000", 2024-09-21T22:08:07.056 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00067681000000000002 2024-09-21T22:08:07.056 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.056 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.056 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:07.056 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.517789+0000", 2024-09-21T22:08:07.056 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.5778000000000004e-05 2024-09-21T22:08:07.057 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.057 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.057 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:07.057 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.517894+0000", 2024-09-21T22:08:07.057 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000105515 2024-09-21T22:08:07.057 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:07.057 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:07.057 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:07.057 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.057 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.057 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4840.0:672 2.3 2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:08:07.057 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.527127+0000", 2024-09-21T22:08:07.058 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 42.485076241000002, 2024-09-21T22:08:07.058 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.047345602000000001, 2024-09-21T22:08:07.058 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:07.058 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:07.058 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:07.058 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-21T22:08:07.058 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4840", 2024-09-21T22:08:07.058 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:08:07.058 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 672 2024-09-21T22:08:07.058 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.058 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:07.058 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.059 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:07.059 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.527127+0000", 2024-09-21T22:08:07.059 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:07.059 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.059 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.059 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:07.059 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.527127+0000", 2024-09-21T22:08:07.059 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:07.059 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.059 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.059 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:07.060 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.527128+0000", 2024-09-21T22:08:07.060 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2810000000000001e-06 2024-09-21T22:08:07.060 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.060 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.060 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:07.060 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.527962+0000", 2024-09-21T22:08:07.060 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00083445199999999996 2024-09-21T22:08:07.060 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.060 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.060 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:07.060 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.527981+0000", 2024-09-21T22:08:07.060 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9009e-05 2024-09-21T22:08:07.060 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.061 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.061 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:07.061 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.527990+0000", 2024-09-21T22:08:07.061 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.8619999999999992e-06 2024-09-21T22:08:07.061 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.061 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.061 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:07.061 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.528061+0000", 2024-09-21T22:08:07.061 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.0226000000000002e-05 2024-09-21T22:08:07.061 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.061 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.061 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:07.062 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.528156+0000", 2024-09-21T22:08:07.062 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.5724999999999996e-05 2024-09-21T22:08:07.062 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.062 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.062 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-21T22:08:07.062 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.528303+0000", 2024-09-21T22:08:07.062 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000146374 2024-09-21T22:08:07.062 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.062 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.062 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:07.062 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.573772+0000", 2024-09-21T22:08:07.062 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.045469745999999998 2024-09-21T22:08:07.063 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.063 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.063 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-21T22:08:07.063 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.574299+0000", 2024-09-21T22:08:07.063 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00052649200000000004 2024-09-21T22:08:07.063 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.063 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.063 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:07.063 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.574366+0000", 2024-09-21T22:08:07.063 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.6872000000000006e-05 2024-09-21T22:08:07.063 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.063 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.064 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:07.064 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.574472+0000", 2024-09-21T22:08:07.064 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000106563 2024-09-21T22:08:07.064 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:07.064 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:07.064 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:07.064 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.064 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.064 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4840.0:686 2.3 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:08:07.064 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.943561+0000", 2024-09-21T22:08:07.064 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 42.068641964999998, 2024-09-21T22:08:07.064 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.049021493999999999, 2024-09-21T22:08:07.064 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:07.065 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:07.065 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:07.065 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-21T22:08:07.065 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4840", 2024-09-21T22:08:07.065 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:08:07.065 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 686 2024-09-21T22:08:07.065 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.065 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:07.065 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.065 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:07.065 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.943561+0000", 2024-09-21T22:08:07.065 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:07.066 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.066 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.066 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:07.066 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.943561+0000", 2024-09-21T22:08:07.066 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:07.066 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.066 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.066 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:07.066 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.943563+0000", 2024-09-21T22:08:07.066 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9420000000000002e-06 2024-09-21T22:08:07.066 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.066 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.066 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:07.067 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.944546+0000", 2024-09-21T22:08:07.067 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00098357099999999992 2024-09-21T22:08:07.067 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.067 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.067 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:07.067 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.944565+0000", 2024-09-21T22:08:07.067 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8621e-05 2024-09-21T22:08:07.067 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.067 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.067 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:07.067 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.944575+0000", 2024-09-21T22:08:07.068 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.4639999999999995e-06 2024-09-21T22:08:07.068 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.068 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.068 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:07.068 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.944647+0000", 2024-09-21T22:08:07.068 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.2786000000000004e-05 2024-09-21T22:08:07.068 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.068 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.068 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:07.068 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.944746+0000", 2024-09-21T22:08:07.068 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.8196999999999995e-05 2024-09-21T22:08:07.068 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.068 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.069 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-21T22:08:07.069 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.944890+0000", 2024-09-21T22:08:07.069 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000144804 2024-09-21T22:08:07.069 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.069 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.069 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:07.069 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.992404+0000", 2024-09-21T22:08:07.069 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.047513211999999999 2024-09-21T22:08:07.069 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.069 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.069 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-21T22:08:07.069 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.992480+0000", 2024-09-21T22:08:07.070 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.6491999999999999e-05 2024-09-21T22:08:07.070 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.070 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.070 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:07.070 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.992521+0000", 2024-09-21T22:08:07.070 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.0608000000000002e-05 2024-09-21T22:08:07.070 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.070 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.070 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:07.070 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.992582+0000", 2024-09-21T22:08:07.070 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.1797000000000005e-05 2024-09-21T22:08:07.071 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:07.071 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:07.071 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:07.071 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.071 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.071 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4840.0:720 2.1 2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:08:07.071 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.803827+0000", 2024-09-21T22:08:07.071 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 41.208375689, 2024-09-21T22:08:07.071 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.049890510999999998, 2024-09-21T22:08:07.071 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:07.071 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:07.071 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:07.072 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-21T22:08:07.072 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4840", 2024-09-21T22:08:07.072 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:08:07.072 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 720 2024-09-21T22:08:07.072 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.072 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:07.072 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.072 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:07.072 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.803827+0000", 2024-09-21T22:08:07.072 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:07.072 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.072 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.073 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:07.073 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.803827+0000", 2024-09-21T22:08:07.073 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:07.073 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.073 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.073 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:07.073 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.803828+0000", 2024-09-21T22:08:07.073 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.569999999999999e-07 2024-09-21T22:08:07.073 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.073 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.073 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:07.074 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.804689+0000", 2024-09-21T22:08:07.074 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00086121799999999997 2024-09-21T22:08:07.074 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.074 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.074 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:07.074 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.804706+0000", 2024-09-21T22:08:07.074 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6266999999999999e-05 2024-09-21T22:08:07.074 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.074 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.074 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:07.074 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.804713+0000", 2024-09-21T22:08:07.074 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.1010000000000003e-06 2024-09-21T22:08:07.075 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.075 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.075 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:07.075 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.804781+0000", 2024-09-21T22:08:07.075 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.7871000000000001e-05 2024-09-21T22:08:07.075 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.075 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.075 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:07.075 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.804859+0000", 2024-09-21T22:08:07.075 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.7891000000000004e-05 2024-09-21T22:08:07.075 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.075 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.075 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 0", 2024-09-21T22:08:07.076 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.804971+0000", 2024-09-21T22:08:07.076 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00011275100000000001 2024-09-21T22:08:07.076 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.076 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.076 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:07.076 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.852704+0000", 2024-09-21T22:08:07.076 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.047732361000000001 2024-09-21T22:08:07.076 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.076 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.076 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-21T22:08:07.076 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.853446+0000", 2024-09-21T22:08:07.076 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00074258799999999997 2024-09-21T22:08:07.077 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.077 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.077 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:07.077 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.853608+0000", 2024-09-21T22:08:07.077 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00016190900000000001 2024-09-21T22:08:07.077 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.077 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.077 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:07.077 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.853718+0000", 2024-09-21T22:08:07.077 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000109597 2024-09-21T22:08:07.077 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:07.077 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:07.078 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:07.078 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.078 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.078 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4840.0:723 2.1 2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:08:07.078 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.860543+0000", 2024-09-21T22:08:07.078 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 41.151660255000003, 2024-09-21T22:08:07.078 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.062354957000000003, 2024-09-21T22:08:07.078 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:07.078 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:07.078 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:07.078 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-21T22:08:07.078 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4840", 2024-09-21T22:08:07.079 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:08:07.079 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 723 2024-09-21T22:08:07.079 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.079 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:07.079 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.079 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:07.079 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.860543+0000", 2024-09-21T22:08:07.079 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:07.079 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.079 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.079 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:07.079 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.860543+0000", 2024-09-21T22:08:07.080 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:07.080 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.080 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.080 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:07.080 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.860544+0000", 2024-09-21T22:08:07.080 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.1119999999999999e-06 2024-09-21T22:08:07.080 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.080 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.080 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:07.080 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.861400+0000", 2024-09-21T22:08:07.080 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000856393 2024-09-21T22:08:07.081 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.081 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.081 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:07.081 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.861419+0000", 2024-09-21T22:08:07.081 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8992e-05 2024-09-21T22:08:07.081 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.081 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.081 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:07.081 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.861428+0000", 2024-09-21T22:08:07.081 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.551e-06 2024-09-21T22:08:07.081 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.081 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.081 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:07.082 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.861555+0000", 2024-09-21T22:08:07.082 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000127286 2024-09-21T22:08:07.082 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.082 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.082 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:07.082 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.861816+0000", 2024-09-21T22:08:07.082 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00026100899999999998 2024-09-21T22:08:07.082 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.082 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.082 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 0", 2024-09-21T22:08:07.082 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.862189+0000", 2024-09-21T22:08:07.083 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00037261899999999998 2024-09-21T22:08:07.083 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.083 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.083 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:07.083 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.921514+0000", 2024-09-21T22:08:07.083 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.059325210000000003 2024-09-21T22:08:07.083 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.083 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.083 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-21T22:08:07.083 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.922721+0000", 2024-09-21T22:08:07.083 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0012066819999999999 2024-09-21T22:08:07.083 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.083 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.084 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:07.084 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.922797+0000", 2024-09-21T22:08:07.084 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.6162999999999996e-05 2024-09-21T22:08:07.084 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.084 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.084 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:07.084 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.922898+0000", 2024-09-21T22:08:07.084 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00010094000000000001 2024-09-21T22:08:07.084 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:07.084 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:07.084 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:07.084 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.085 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.085 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:725 2.0 e63/16 2:08b4f96a:::rbd_data.126483d75f2d.000000000000000f:head v 63'33, mlcod=63'31)", 2024-09-21T22:08:07.085 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.936778+0000", 2024-09-21T22:08:07.085 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 41.075424513000002, 2024-09-21T22:08:07.085 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.047671937999999997, 2024-09-21T22:08:07.085 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:07.085 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:07.085 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:07.085 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:07.085 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.085 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:07.085 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.936778+0000", 2024-09-21T22:08:07.086 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:07.086 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.086 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.086 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:07.086 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.936778+0000", 2024-09-21T22:08:07.086 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:07.086 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.086 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.086 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:07.086 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.936780+0000", 2024-09-21T22:08:07.086 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7799999999999999e-06 2024-09-21T22:08:07.087 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.087 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.087 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:07.087 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.938475+0000", 2024-09-21T22:08:07.087 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0016951749999999999 2024-09-21T22:08:07.087 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.087 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.087 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:07.087 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.938506+0000", 2024-09-21T22:08:07.087 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.0865999999999999e-05 2024-09-21T22:08:07.087 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.087 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.087 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:07.088 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.938520+0000", 2024-09-21T22:08:07.088 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3825999999999999e-05 2024-09-21T22:08:07.088 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.088 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.088 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:07.088 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.938624+0000", 2024-09-21T22:08:07.088 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00010391900000000001 2024-09-21T22:08:07.088 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.088 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.088 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:07.088 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.938651+0000", 2024-09-21T22:08:07.088 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.7481999999999999e-05 2024-09-21T22:08:07.089 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.089 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.089 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:07.089 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.984369+0000", 2024-09-21T22:08:07.089 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.045717595 2024-09-21T22:08:07.089 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.089 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.089 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:07.089 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.984450+0000", 2024-09-21T22:08:07.089 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.1295e-05 2024-09-21T22:08:07.089 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:07.089 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:07.090 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:07.090 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.090 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.090 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:727 2.0 e63/16 2:08b4f96a:::rbd_data.126483d75f2d.000000000000000f:head v 63'35, mlcod=63'33)", 2024-09-21T22:08:07.090 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.998090+0000", 2024-09-21T22:08:07.090 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 41.014113336999998, 2024-09-21T22:08:07.090 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.052205704999999998, 2024-09-21T22:08:07.090 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:07.090 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:07.090 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:07.090 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:07.090 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.091 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:07.091 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.998090+0000", 2024-09-21T22:08:07.091 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:07.091 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.091 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.091 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:07.091 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.998090+0000", 2024-09-21T22:08:07.091 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:07.091 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.091 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.091 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:07.091 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.998091+0000", 2024-09-21T22:08:07.091 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6929999999999999e-06 2024-09-21T22:08:07.092 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.092 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.092 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:07.092 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.998940+0000", 2024-09-21T22:08:07.092 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00084875200000000004 2024-09-21T22:08:07.092 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.092 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.092 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:07.092 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.998970+0000", 2024-09-21T22:08:07.092 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9709999999999998e-05 2024-09-21T22:08:07.092 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.092 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.093 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:07.093 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.998984+0000", 2024-09-21T22:08:07.093 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3739e-05 2024-09-21T22:08:07.093 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.093 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.093 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:07.093 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.999097+0000", 2024-09-21T22:08:07.093 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000113484 2024-09-21T22:08:07.093 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.093 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.093 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:07.093 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.999135+0000", 2024-09-21T22:08:07.093 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.8405999999999998e-05 2024-09-21T22:08:07.094 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.094 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.094 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:07.094 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.050245+0000", 2024-09-21T22:08:07.094 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.051109935000000002 2024-09-21T22:08:07.094 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.094 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.094 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:07.094 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.050295+0000", 2024-09-21T22:08:07.094 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.9985999999999999e-05 2024-09-21T22:08:07.094 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:07.094 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:07.095 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:07.095 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.095 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.095 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:729 2.0 e63/16 2:08b4f96a:::rbd_data.126483d75f2d.000000000000000f:head v 63'37, mlcod=63'35)", 2024-09-21T22:08:07.095 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:26.062424+0000", 2024-09-21T22:08:07.095 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 40.949778715000001, 2024-09-21T22:08:07.095 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.047312186999999999, 2024-09-21T22:08:07.095 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:07.095 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:07.095 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:07.095 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:07.095 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.096 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:07.096 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.062424+0000", 2024-09-21T22:08:07.096 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:07.096 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.096 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.096 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:07.096 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.062424+0000", 2024-09-21T22:08:07.096 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:07.096 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.096 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.096 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:07.096 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.062426+0000", 2024-09-21T22:08:07.096 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7129999999999999e-06 2024-09-21T22:08:07.097 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.097 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.097 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:07.097 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.063211+0000", 2024-09-21T22:08:07.097 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00078488500000000003 2024-09-21T22:08:07.097 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.097 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.097 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:07.097 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.063241+0000", 2024-09-21T22:08:07.097 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.0056000000000002e-05 2024-09-21T22:08:07.097 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.097 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.098 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:07.098 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.063255+0000", 2024-09-21T22:08:07.098 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4022999999999999e-05 2024-09-21T22:08:07.098 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.098 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.098 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:07.098 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.063384+0000", 2024-09-21T22:08:07.098 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00012920600000000001 2024-09-21T22:08:07.098 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.098 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.098 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:07.098 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.063421+0000", 2024-09-21T22:08:07.099 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.7242e-05 2024-09-21T22:08:07.099 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.099 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.099 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:07.099 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.109666+0000", 2024-09-21T22:08:07.099 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.046244446000000002 2024-09-21T22:08:07.099 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.099 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.099 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:07.099 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.109736+0000", 2024-09-21T22:08:07.099 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.0616000000000003e-05 2024-09-21T22:08:07.099 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:07.099 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:07.100 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:07.100 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.100 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.100 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:731 2.0 e63/16 2:08b4f96a:::rbd_data.126483d75f2d.000000000000000f:head v 63'39, mlcod=63'37)", 2024-09-21T22:08:07.100 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:26.135946+0000", 2024-09-21T22:08:07.100 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 40.876257207999998, 2024-09-21T22:08:07.100 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.054290359000000003, 2024-09-21T22:08:07.100 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:07.100 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:07.100 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:07.100 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:07.100 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.100 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:07.101 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.135946+0000", 2024-09-21T22:08:07.101 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:07.101 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.101 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.101 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:07.101 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.135946+0000", 2024-09-21T22:08:07.101 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:07.101 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.101 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.101 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:07.101 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.135948+0000", 2024-09-21T22:08:07.102 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7999999999999999e-06 2024-09-21T22:08:07.102 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.102 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.102 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:07.102 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.136730+0000", 2024-09-21T22:08:07.102 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00078271899999999997 2024-09-21T22:08:07.102 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.102 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.102 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:07.103 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.136760+0000", 2024-09-21T22:08:07.103 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9876e-05 2024-09-21T22:08:07.103 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.103 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.103 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:07.103 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.136774+0000", 2024-09-21T22:08:07.103 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3478999999999999e-05 2024-09-21T22:08:07.103 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.103 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.103 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:07.104 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.136944+0000", 2024-09-21T22:08:07.104 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000170626 2024-09-21T22:08:07.104 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.104 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.104 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:07.104 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.136985+0000", 2024-09-21T22:08:07.104 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.0377999999999999e-05 2024-09-21T22:08:07.104 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.104 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.104 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:07.105 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.190137+0000", 2024-09-21T22:08:07.105 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.053152438000000003 2024-09-21T22:08:07.105 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.105 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.105 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:07.105 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.190236+0000", 2024-09-21T22:08:07.105 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.9042999999999993e-05 2024-09-21T22:08:07.105 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:07.105 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:07.105 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:07.105 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.106 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.106 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:125 2.3 e71/66 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head v 71'31, mlcod=71'29)", 2024-09-21T22:08:07.106 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.066855+0000", 2024-09-21T22:08:07.106 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 28.945347773000002, 2024-09-21T22:08:07.106 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.059856059000000003, 2024-09-21T22:08:07.106 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:07.106 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:07.106 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:07.106 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:07.106 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.106 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:07.107 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.066855+0000", 2024-09-21T22:08:07.107 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:07.107 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.107 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.107 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:07.107 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.066855+0000", 2024-09-21T22:08:07.107 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:07.107 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.107 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.107 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:07.107 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.066857+0000", 2024-09-21T22:08:07.107 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.807e-06 2024-09-21T22:08:07.108 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.108 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.108 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:07.108 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.068365+0000", 2024-09-21T22:08:07.108 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0015077910000000001 2024-09-21T22:08:07.108 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.108 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.108 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:07.108 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.068395+0000", 2024-09-21T22:08:07.108 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.0326000000000001e-05 2024-09-21T22:08:07.108 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.108 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.109 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:07.109 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.068415+0000", 2024-09-21T22:08:07.109 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9871999999999999e-05 2024-09-21T22:08:07.109 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.109 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.109 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:07.109 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.068580+0000", 2024-09-21T22:08:07.109 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000165317 2024-09-21T22:08:07.109 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.109 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.109 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:07.109 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.068618+0000", 2024-09-21T22:08:07.109 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.7753000000000003e-05 2024-09-21T22:08:07.110 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.110 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.110 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:07.110 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.126623+0000", 2024-09-21T22:08:07.110 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.058004642000000002 2024-09-21T22:08:07.110 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.110 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.110 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:07.110 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.126711+0000", 2024-09-21T22:08:07.110 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.8551e-05 2024-09-21T22:08:07.111 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:07.111 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:07.111 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:07.111 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.111 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.111 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:127 2.3 e71/66 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head v 71'33, mlcod=71'31)", 2024-09-21T22:08:07.111 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.140864+0000", 2024-09-21T22:08:07.111 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 28.871338744999999, 2024-09-21T22:08:07.111 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.052512380999999997, 2024-09-21T22:08:07.111 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:07.111 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:07.111 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:07.111 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:07.112 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.112 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:07.112 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.140864+0000", 2024-09-21T22:08:07.112 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:07.112 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.112 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.112 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:07.112 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.140864+0000", 2024-09-21T22:08:07.112 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:07.112 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.112 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.112 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:07.113 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.140866+0000", 2024-09-21T22:08:07.113 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.694e-06 2024-09-21T22:08:07.113 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.113 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.113 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:07.113 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.141712+0000", 2024-09-21T22:08:07.113 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00084581499999999998 2024-09-21T22:08:07.113 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.113 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.113 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:07.113 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.141742+0000", 2024-09-21T22:08:07.113 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9865999999999998e-05 2024-09-21T22:08:07.114 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.114 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.114 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:07.114 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.141761+0000", 2024-09-21T22:08:07.114 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9473000000000001e-05 2024-09-21T22:08:07.114 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.114 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.114 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:07.114 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.141919+0000", 2024-09-21T22:08:07.114 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00015758300000000001 2024-09-21T22:08:07.114 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.114 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.115 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:07.115 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.141955+0000", 2024-09-21T22:08:07.115 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.6613000000000003e-05 2024-09-21T22:08:07.115 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.115 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.115 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:07.115 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.193333+0000", 2024-09-21T22:08:07.115 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.051377260000000001 2024-09-21T22:08:07.115 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.115 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.115 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:07.115 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.193377+0000", 2024-09-21T22:08:07.116 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.4076999999999998e-05 2024-09-21T22:08:07.116 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:07.116 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:07.116 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:07.116 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.116 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.116 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:129 2.3 e71/66 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head v 71'35, mlcod=71'33)", 2024-09-21T22:08:07.116 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.209458+0000", 2024-09-21T22:08:07.116 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 28.802744996000001, 2024-09-21T22:08:07.116 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.056595578000000001, 2024-09-21T22:08:07.116 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:07.116 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:07.117 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:07.117 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:07.117 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.117 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:07.117 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.209458+0000", 2024-09-21T22:08:07.117 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:07.117 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.117 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.117 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:07.117 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.209458+0000", 2024-09-21T22:08:07.117 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:07.118 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.118 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.118 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:07.118 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.209460+0000", 2024-09-21T22:08:07.118 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7600000000000001e-06 2024-09-21T22:08:07.118 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.118 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.118 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:07.118 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.210317+0000", 2024-09-21T22:08:07.119 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00085771700000000003 2024-09-21T22:08:07.119 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.119 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.119 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:07.119 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.210346+0000", 2024-09-21T22:08:07.119 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8334e-05 2024-09-21T22:08:07.119 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.119 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.119 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:07.119 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.210365+0000", 2024-09-21T22:08:07.120 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9439000000000001e-05 2024-09-21T22:08:07.120 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.120 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.120 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:07.120 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.210544+0000", 2024-09-21T22:08:07.120 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00017845599999999999 2024-09-21T22:08:07.120 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.120 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.120 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:07.120 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.210584+0000", 2024-09-21T22:08:07.120 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.0423000000000002e-05 2024-09-21T22:08:07.121 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.121 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.121 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:07.121 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.265970+0000", 2024-09-21T22:08:07.121 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.055385765000000003 2024-09-21T22:08:07.121 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.121 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.121 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:07.121 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.266054+0000", 2024-09-21T22:08:07.121 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.3683999999999994e-05 2024-09-21T22:08:07.122 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:07.122 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:07.122 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:07.122 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.122 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.122 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:131 2.3 e71/66 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head v 71'37, mlcod=71'35)", 2024-09-21T22:08:07.122 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.279550+0000", 2024-09-21T22:08:07.122 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 28.732653158000002, 2024-09-21T22:08:07.122 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.052290419999999997, 2024-09-21T22:08:07.122 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:07.122 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:07.122 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:07.123 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:07.123 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.123 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:07.123 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.279550+0000", 2024-09-21T22:08:07.123 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:07.123 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.123 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.123 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:07.123 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.279550+0000", 2024-09-21T22:08:07.123 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:07.123 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.123 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.124 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:07.124 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.279552+0000", 2024-09-21T22:08:07.124 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9439999999999999e-06 2024-09-21T22:08:07.124 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.124 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.124 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:07.124 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.280296+0000", 2024-09-21T22:08:07.124 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00074417000000000005 2024-09-21T22:08:07.124 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.124 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.124 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:07.124 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.280324+0000", 2024-09-21T22:08:07.125 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8045999999999999e-05 2024-09-21T22:08:07.125 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.125 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.125 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:07.125 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.280345+0000", 2024-09-21T22:08:07.125 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.0849999999999999e-05 2024-09-21T22:08:07.125 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.125 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.125 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:07.125 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.280497+0000", 2024-09-21T22:08:07.125 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00015252500000000001 2024-09-21T22:08:07.125 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.125 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.126 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:07.126 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.280531+0000", 2024-09-21T22:08:07.126 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.4034e-05 2024-09-21T22:08:07.126 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.126 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.126 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:07.126 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.331769+0000", 2024-09-21T22:08:07.126 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.051237999999999999 2024-09-21T22:08:07.126 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:07.126 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:07.126 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:07.127 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.331840+0000", 2024-09-21T22:08:07.127 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.0851000000000003e-05 2024-09-21T22:08:07.127 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:07.127 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:07.127 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:07.127 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:07.127 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:07.127 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:08:07.128 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-21T22:08:07.129 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:08:07.124+0000 7f2075a06640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31720) UID: 0 2024-09-21T22:08:07.130 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T22:08:07.151 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T22:08:07.229 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:08:07.224+0000 7f2075a06640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31720) UID: 0 2024-09-21T22:08:07.281 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:08:07.281 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-21T22:08:07.281 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:07.281 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.3 87/86 [PushOp(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head, version: 71'29, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head@71'29, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 4194304, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:08:07.281 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:08:07.260481+0000", 2024-09-21T22:08:07.281 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.021547120999999999, 2024-09-21T22:08:07.281 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021635904000000001, 2024-09-21T22:08:07.282 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:07.282 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:07.282 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:08:07.282 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:07.282 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:07.282 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:07.282 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.260481+0000", 2024-09-21T22:08:07.282 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:07.282 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:07.282 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:07.282 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:07.282 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.260481+0000", 2024-09-21T22:08:07.283 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:07.283 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:07.283 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:07.283 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:07.283 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.260483+0000", 2024-09-21T22:08:07.283 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.466e-06 2024-09-21T22:08:07.283 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:07.283 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:07.283 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:07.283 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.261175+0000", 2024-09-21T22:08:07.283 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00069265500000000003 2024-09-21T22:08:07.283 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:07.283 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:07.283 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:07.284 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.261197+0000", 2024-09-21T22:08:07.284 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2189e-05 2024-09-21T22:08:07.284 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:07.284 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:07.284 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:07.284 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.261211+0000", 2024-09-21T22:08:07.284 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3308e-05 2024-09-21T22:08:07.284 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:07.284 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:07.284 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:07.284 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.261292+0000", 2024-09-21T22:08:07.284 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1519000000000003e-05 2024-09-21T22:08:07.284 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:07.284 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:07.284 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:07.285 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.261301+0000", 2024-09-21T22:08:07.285 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4640000000000006e-06 2024-09-21T22:08:07.285 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:07.285 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:07.285 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:07.285 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:07.285 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-21T22:08:07.285 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 1 2024-09-21T22:08:07.285 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:08:07.300 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T22:08:07.330 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:08:07.325+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:08:07.331 INFO:teuthology.orchestra.run.smithi039.stdout:osd.0: {} 2024-09-21T22:08:07.331 INFO:teuthology.orchestra.run.smithi039.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T22:08:07.349 INFO:teuthology.orchestra.run.smithi039.stdout:osd.1: {} 2024-09-21T22:08:07.349 INFO:teuthology.orchestra.run.smithi039.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T22:08:07.349 INFO:teuthology.orchestra.run.smithi039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-21T22:08:07.367 INFO:teuthology.orchestra.run.smithi039.stdout:osd.3: {} 2024-09-21T22:08:07.368 INFO:teuthology.orchestra.run.smithi039.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T22:08:07.386 INFO:teuthology.orchestra.run.smithi039.stdout:osd.4: {} 2024-09-21T22:08:07.387 INFO:teuthology.orchestra.run.smithi039.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T22:08:07.405 INFO:teuthology.orchestra.run.smithi039.stdout:osd.5: {} 2024-09-21T22:08:07.406 INFO:teuthology.orchestra.run.smithi039.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T22:08:07.410 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T22:08:07.410 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T22:08:07.430 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:08:07.428+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:08:07.471 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:07.471 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:07.487 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T22:08:07.531 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:08:07.525+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:08:07.632 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:08:07.628+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:08:07.655 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:07.655 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:07.676 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T22:08:07.762 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:08:07.760+0000 7f2075a06640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31720) UID: 0 2024-09-21T22:08:07.834 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:08:07.832+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:08:07.849 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:07.849 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:07.866 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T22:08:07.954 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:08:07.952+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:08:08.037 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:08:08.033+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:08:08.038 INFO:tasks.workunit.client.0.smithi157.stderr:+ rbd device --device-type nbd list 2024-09-21T22:08:08.038 INFO:tasks.workunit.client.0.smithi157.stderr:+ expect_false grep '^23672\b' 2024-09-21T22:08:08.038 INFO:tasks.workunit.client.0.smithi157.stderr:+ grep '^23672\b' 2024-09-21T22:08:08.040 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:08.040 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:08.055 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T22:08:08.064 INFO:teuthology.orchestra.run.smithi039.stderr:noscrub is unset 2024-09-21T22:08:08.106 INFO:tasks.workunit.client.0.smithi157.stderr:+ return 0 2024-09-21T22:08:08.106 INFO:tasks.workunit.client.0.smithi157.stderr:+ ps -C rbd-nbd 2024-09-21T22:08:08.107 INFO:tasks.workunit.client.0.smithi157.stderr:+ expect_false grep '^ *23672\b' 2024-09-21T22:08:08.107 INFO:tasks.workunit.client.0.smithi157.stderr:+ grep '^ *23672\b' 2024-09-21T22:08:08.113 INFO:tasks.workunit.client.0.smithi157.stderr:+ return 0 2024-09-21T22:08:08.113 INFO:tasks.workunit.client.0.smithi157.stderr:+ sleep 0.5 2024-09-21T22:08:08.138 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:08:08.133+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:08:08.238 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:08:08.236+0000 7f2075a06640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31720) UID: 0 2024-09-21T22:08:08.241 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:08.241 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:08.256 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T22:08:08.340 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:08:08.337+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:08:08.441 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:08:08.436+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:08:08.449 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:08.449 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:08.464 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T22:08:08.550 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:08:08.548+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:08:08.614 INFO:tasks.workunit.client.0.smithi157.stderr:+ LOG_FILE=/tmp/tmp.GqKuv7rsZi/rbd-nbd.log 2024-09-21T22:08:08.615 INFO:tasks.workunit.client.0.smithi157.stderr:+ '[' -n '' ']' 2024-09-21T22:08:08.615 INFO:tasks.workunit.client.0.smithi157.stderr:++ _sudo rbd device --device-type nbd map --quiesce rbd/testrbdnbd21553 --log-file=/tmp/tmp.GqKuv7rsZi/rbd-nbd.log 2024-09-21T22:08:08.615 INFO:tasks.workunit.client.0.smithi157.stderr:++ local cmd 2024-09-21T22:08:08.616 INFO:tasks.workunit.client.0.smithi157.stderr:+++ id -u 2024-09-21T22:08:08.617 INFO:tasks.workunit.client.0.smithi157.stderr:++ '[' 1000 -eq 0 ']' 2024-09-21T22:08:08.617 INFO:tasks.workunit.client.0.smithi157.stderr:+++ which rbd 2024-09-21T22:08:08.619 INFO:tasks.workunit.client.0.smithi157.stderr:++ cmd=/usr/bin/rbd 2024-09-21T22:08:08.619 INFO:tasks.workunit.client.0.smithi157.stderr:++ shift 2024-09-21T22:08:08.619 INFO:tasks.workunit.client.0.smithi157.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map --quiesce rbd/testrbdnbd21553 --log-file=/tmp/tmp.GqKuv7rsZi/rbd-nbd.log 2024-09-21T22:08:08.635 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:08.635 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:08.643 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:08:08.640+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:08:08.649 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T22:08:08.744 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:08:08.740+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:08:08.757 INFO:tasks.workunit.client.0.smithi157.stderr:+ DEV=/dev/nbd1 2024-09-21T22:08:08.757 INFO:tasks.workunit.client.0.smithi157.stderr:+ get_pid rbd 2024-09-21T22:08:08.757 INFO:tasks.workunit.client.0.smithi157.stderr:+ local pool=rbd 2024-09-21T22:08:08.757 INFO:tasks.workunit.client.0.smithi157.stderr:+ local ns= 2024-09-21T22:08:08.758 INFO:tasks.workunit.client.0.smithi157.stderr:++ rbd device --device-type nbd --format xml list 2024-09-21T22:08:08.758 INFO:tasks.workunit.client.0.smithi157.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd21553'\''][device='\''/dev/nbd1'\'']/id' 2024-09-21T22:08:08.829 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:08.829 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:08.843 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T22:08:08.844 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:08:08.841+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:08:08.846 INFO:tasks.workunit.client.0.smithi157.stderr:+ PID=23853 2024-09-21T22:08:08.846 INFO:tasks.workunit.client.0.smithi157.stderr:+ test -n 23853 2024-09-21T22:08:08.846 INFO:tasks.workunit.client.0.smithi157.stderr:+ ps -p 23853 -C rbd-nbd 2024-09-21T22:08:08.854 INFO:tasks.workunit.client.0.smithi157.stdout: PID TTY TIME CMD 2024-09-21T22:08:08.854 INFO:tasks.workunit.client.0.smithi157.stdout: 23853 ? 00:00:00 rbd-nbd 2024-09-21T22:08:08.854 INFO:tasks.workunit.client.0.smithi157.stderr:+ _sudo mkfs /dev/nbd1 2024-09-21T22:08:08.854 INFO:tasks.workunit.client.0.smithi157.stderr:+ local cmd 2024-09-21T22:08:08.855 INFO:tasks.workunit.client.0.smithi157.stderr:++ id -u 2024-09-21T22:08:08.856 INFO:tasks.workunit.client.0.smithi157.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T22:08:08.857 INFO:tasks.workunit.client.0.smithi157.stderr:++ which mkfs 2024-09-21T22:08:08.859 INFO:tasks.workunit.client.0.smithi157.stderr:+ cmd=/usr/sbin/mkfs 2024-09-21T22:08:08.859 INFO:tasks.workunit.client.0.smithi157.stderr:+ shift 2024-09-21T22:08:08.859 INFO:tasks.workunit.client.0.smithi157.stderr:+ sudo -nE /usr/sbin/mkfs /dev/nbd1 2024-09-21T22:08:08.890 INFO:tasks.workunit.client.0.smithi157.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-21T22:08:08.945 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:08:08.940+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:08:08.975 INFO:tasks.workunit.client.0.smithi157.stdout:Discarding device blocks: 0/16384 done 2024-09-21T22:08:08.975 INFO:tasks.workunit.client.0.smithi157.stdout:Creating filesystem with 16384 4k blocks and 16384 inodes 2024-09-21T22:08:08.975 INFO:tasks.workunit.client.0.smithi157.stdout: 2024-09-21T22:08:08.975 INFO:tasks.workunit.client.0.smithi157.stdout:Allocating group tables: 0/1 done 2024-09-21T22:08:08.977 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:08.978 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:08.994 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T22:08:08.999 INFO:tasks.workunit.client.0.smithi157.stdout:Writing inode tables: 0/1 done 2024-09-21T22:08:09.046 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:08:09.041+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:08:09.077 INFO:tasks.workunit.client.0.smithi157.stdout:Writing superblocks and filesystem accounting information: 0/1 done 2024-09-21T22:08:09.077 INFO:tasks.workunit.client.0.smithi157.stdout: 2024-09-21T22:08:09.079 INFO:tasks.workunit.client.0.smithi157.stderr:+ mkdir /tmp/tmp.GqKuv7rsZi/mnt 2024-09-21T22:08:09.081 INFO:tasks.workunit.client.0.smithi157.stderr:+ _sudo mount /dev/nbd1 /tmp/tmp.GqKuv7rsZi/mnt 2024-09-21T22:08:09.082 INFO:tasks.workunit.client.0.smithi157.stderr:+ local cmd 2024-09-21T22:08:09.082 INFO:tasks.workunit.client.0.smithi157.stderr:++ id -u 2024-09-21T22:08:09.083 INFO:tasks.workunit.client.0.smithi157.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T22:08:09.084 INFO:tasks.workunit.client.0.smithi157.stderr:++ which mount 2024-09-21T22:08:09.086 INFO:tasks.workunit.client.0.smithi157.stderr:+ cmd=/usr/bin/mount 2024-09-21T22:08:09.086 INFO:tasks.workunit.client.0.smithi157.stderr:+ shift 2024-09-21T22:08:09.086 INFO:tasks.workunit.client.0.smithi157.stderr:+ sudo -nE /usr/bin/mount /dev/nbd1 /tmp/tmp.GqKuv7rsZi/mnt 2024-09-21T22:08:09.114 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:09.114 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:09.131 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T22:08:09.141 INFO:tasks.workunit.client.0.smithi157.stderr:+ rbd snap create rbd/testrbdnbd21553@quiesce3 2024-09-21T22:08:09.146 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:08:09.144+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:08:09.247 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:08:09.241+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:08:09.299 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:09.299 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:09.314 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T22:08:09.360 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:08:09.357+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:08:09.411 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T22:08:09.449 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:08:09.448+0000 7f2075a06640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31720) UID: 0 2024-09-21T22:08:09.481 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:09.481 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:09.496 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T22:08:09.550 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:08:09.548+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:08:09.554 INFO:teuthology.orchestra.run.smithi039.stdout:osd.0: {} 2024-09-21T22:08:09.555 INFO:teuthology.orchestra.run.smithi039.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T22:08:09.572 INFO:teuthology.orchestra.run.smithi039.stdout:osd.1: {} 2024-09-21T22:08:09.572 INFO:teuthology.orchestra.run.smithi039.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T22:08:09.572 INFO:teuthology.orchestra.run.smithi039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-21T22:08:09.593 INFO:teuthology.orchestra.run.smithi039.stdout:osd.3: {} 2024-09-21T22:08:09.593 INFO:teuthology.orchestra.run.smithi039.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T22:08:09.614 INFO:teuthology.orchestra.run.smithi039.stdout:osd.4: {} 2024-09-21T22:08:09.614 INFO:teuthology.orchestra.run.smithi039.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T22:08:09.622 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:08:09.622 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:08:09.622 INFO:teuthology.orchestra.run.smithi039.stdout: "complaint_time": 30, 2024-09-21T22:08:09.622 INFO:teuthology.orchestra.run.smithi039.stdout: "num_blocked_ops": 0 2024-09-21T22:08:09.622 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:08:09.635 INFO:teuthology.orchestra.run.smithi039.stdout:osd.5: {} 2024-09-21T22:08:09.635 INFO:teuthology.orchestra.run.smithi039.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T22:08:09.636 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T22:08:09.639 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T22:08:09.639 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T22:08:09.650 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:08:09.648+0000 7f2075a06640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31720) UID: 0 2024-09-21T22:08:09.752 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:08:09.748+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:08:09.757 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:08:09.757 INFO:teuthology.orchestra.run.smithi039.stdout: "size": 20, 2024-09-21T22:08:09.757 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 600, 2024-09-21T22:08:09.757 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [ 2024-09-21T22:08:09.757 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.757 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.3 55/54 [PushOp(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head, version: 52'2, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head@52'2, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T22:08:09.757 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.084379+0000", 2024-09-21T22:08:09.757 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 57.668728497000004, 2024-09-21T22:08:09.757 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.051020674000000002, 2024-09-21T22:08:09.758 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:09.758 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:09.758 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:08:09.758 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:09.758 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.758 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:09.758 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.084379+0000", 2024-09-21T22:08:09.758 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:09.758 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.758 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.758 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:09.758 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.084379+0000", 2024-09-21T22:08:09.758 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:09.759 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.759 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.759 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:09.759 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.084381+0000", 2024-09-21T22:08:09.759 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.73e-06 2024-09-21T22:08:09.759 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.759 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.759 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:09.759 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.085494+0000", 2024-09-21T22:08:09.759 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011131559999999999 2024-09-21T22:08:09.759 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.759 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.759 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:09.759 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.085542+0000", 2024-09-21T22:08:09.759 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8334999999999997e-05 2024-09-21T22:08:09.760 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.760 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.760 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:09.760 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.085591+0000", 2024-09-21T22:08:09.760 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.9165000000000001e-05 2024-09-21T22:08:09.760 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.760 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.760 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:09.760 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.085777+0000", 2024-09-21T22:08:09.760 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000186152 2024-09-21T22:08:09.760 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.760 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.760 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:09.760 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.085796+0000", 2024-09-21T22:08:09.760 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8612999999999999e-05 2024-09-21T22:08:09.760 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.761 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.761 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:09.761 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.135399+0000", 2024-09-21T22:08:09.761 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.049603522999999997 2024-09-21T22:08:09.761 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:09.761 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:09.761 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:09.761 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.761 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.761 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.3 55/54 [PushOp(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head, version: 52'2, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head@52'2, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:08:09.761 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.139554+0000", 2024-09-21T22:08:09.761 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 57.613553394, 2024-09-21T22:08:09.761 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.056215389999999997, 2024-09-21T22:08:09.761 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:09.762 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:09.762 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:08:09.762 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:09.762 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.762 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:09.762 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.139554+0000", 2024-09-21T22:08:09.762 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:09.762 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.762 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.762 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:09.762 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.139554+0000", 2024-09-21T22:08:09.762 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:09.762 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.762 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.763 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:09.763 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.139556+0000", 2024-09-21T22:08:09.763 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7999999999999999e-06 2024-09-21T22:08:09.763 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.763 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.763 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:09.763 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.140671+0000", 2024-09-21T22:08:09.763 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001115612 2024-09-21T22:08:09.763 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.763 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.763 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:09.763 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.140720+0000", 2024-09-21T22:08:09.763 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8631999999999998e-05 2024-09-21T22:08:09.764 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.764 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.764 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:09.764 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.140752+0000", 2024-09-21T22:08:09.764 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.2199000000000001e-05 2024-09-21T22:08:09.764 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.764 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.764 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:09.764 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.140900+0000", 2024-09-21T22:08:09.764 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00014786000000000001 2024-09-21T22:08:09.764 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.764 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.764 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:09.764 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.140917+0000", 2024-09-21T22:08:09.764 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6753999999999999e-05 2024-09-21T22:08:09.765 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.765 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.765 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:09.765 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.195769+0000", 2024-09-21T22:08:09.765 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.054852533000000002 2024-09-21T22:08:09.765 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:09.765 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:09.765 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:09.765 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.765 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.765 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.3 55/54 [PushOp(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head, version: 52'2, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head@52'2, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:08:09.765 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.199187+0000", 2024-09-21T22:08:09.765 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 57.55391985, 2024-09-21T22:08:09.765 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.051543975999999998, 2024-09-21T22:08:09.765 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:09.766 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:09.766 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:08:09.766 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:09.766 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.766 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:09.766 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.199187+0000", 2024-09-21T22:08:09.766 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:09.766 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.766 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.766 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:09.766 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.199187+0000", 2024-09-21T22:08:09.766 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:09.766 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.766 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.767 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:09.767 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.199189+0000", 2024-09-21T22:08:09.767 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7969999999999999e-06 2024-09-21T22:08:09.767 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.767 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.767 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:09.767 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.200355+0000", 2024-09-21T22:08:09.767 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011660139999999999 2024-09-21T22:08:09.767 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.767 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.767 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:09.767 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.200405+0000", 2024-09-21T22:08:09.767 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.0124000000000003e-05 2024-09-21T22:08:09.767 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.767 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.768 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:09.768 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.200457+0000", 2024-09-21T22:08:09.768 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.1177999999999997e-05 2024-09-21T22:08:09.768 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.768 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.768 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:09.768 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.200635+0000", 2024-09-21T22:08:09.768 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00017841299999999999 2024-09-21T22:08:09.768 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.768 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.768 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:09.768 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.200653+0000", 2024-09-21T22:08:09.768 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7944999999999998e-05 2024-09-21T22:08:09.769 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.769 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.769 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:09.769 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.250731+0000", 2024-09-21T22:08:09.769 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.050078505000000002 2024-09-21T22:08:09.769 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:09.769 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:09.769 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:09.769 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.769 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.769 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.3 55/54 [PushOp(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head, version: 52'3, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head@52'3, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:08:09.769 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.343754+0000", 2024-09-21T22:08:09.769 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 57.409353138999997, 2024-09-21T22:08:09.769 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.063560607000000005, 2024-09-21T22:08:09.770 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:09.770 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:09.770 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:08:09.770 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:09.770 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.770 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:09.770 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.343754+0000", 2024-09-21T22:08:09.770 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:09.770 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.770 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.770 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:09.770 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.343754+0000", 2024-09-21T22:08:09.770 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:09.770 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.771 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.771 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:09.771 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.343756+0000", 2024-09-21T22:08:09.771 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7069999999999999e-06 2024-09-21T22:08:09.771 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.771 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.771 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:09.771 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.344864+0000", 2024-09-21T22:08:09.771 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011085159999999999 2024-09-21T22:08:09.771 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.771 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.771 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:09.771 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.344914+0000", 2024-09-21T22:08:09.771 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.9462000000000002e-05 2024-09-21T22:08:09.771 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.771 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.772 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:09.772 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.344961+0000", 2024-09-21T22:08:09.772 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.7633999999999998e-05 2024-09-21T22:08:09.772 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.772 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.772 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:09.772 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.345171+0000", 2024-09-21T22:08:09.772 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000209652 2024-09-21T22:08:09.772 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.772 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.772 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:09.772 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.345193+0000", 2024-09-21T22:08:09.772 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1985999999999999e-05 2024-09-21T22:08:09.772 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.772 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.773 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:09.773 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.407315+0000", 2024-09-21T22:08:09.773 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.06212165 2024-09-21T22:08:09.773 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:09.773 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:09.773 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:09.773 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.773 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.773 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.3 55/54 [PushOp(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head, version: 52'3, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head@52'3, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:08:09.773 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.410551+0000", 2024-09-21T22:08:09.773 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 57.342556008000003, 2024-09-21T22:08:09.773 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.054517522999999998, 2024-09-21T22:08:09.773 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:09.773 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:09.774 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:08:09.774 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:09.774 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.774 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:09.774 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.410551+0000", 2024-09-21T22:08:09.774 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:09.774 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.774 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.774 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:09.774 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.410551+0000", 2024-09-21T22:08:09.774 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:09.774 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.774 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.774 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:09.775 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.410553+0000", 2024-09-21T22:08:09.775 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5200000000000001e-06 2024-09-21T22:08:09.775 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.775 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.775 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:09.775 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.411747+0000", 2024-09-21T22:08:09.775 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011946089999999999 2024-09-21T22:08:09.775 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.775 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.775 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:09.775 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.411796+0000", 2024-09-21T22:08:09.775 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8717999999999997e-05 2024-09-21T22:08:09.775 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.775 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.775 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:09.776 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.411844+0000", 2024-09-21T22:08:09.776 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8291999999999998e-05 2024-09-21T22:08:09.776 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.776 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.776 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:09.776 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.412066+0000", 2024-09-21T22:08:09.776 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00022115899999999999 2024-09-21T22:08:09.776 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.776 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.776 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:09.776 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.412081+0000", 2024-09-21T22:08:09.776 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5654e-05 2024-09-21T22:08:09.776 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.776 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.776 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:09.777 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.465069+0000", 2024-09-21T22:08:09.777 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.052987570999999997 2024-09-21T22:08:09.777 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:09.777 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:09.777 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:09.777 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.777 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.777 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.3 55/54 [PushOp(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head, version: 52'3, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head@52'3, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 4194304, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:08:09.777 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.468339+0000", 2024-09-21T22:08:09.777 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 57.284768344, 2024-09-21T22:08:09.777 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.064010024999999998, 2024-09-21T22:08:09.777 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:09.777 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:09.777 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:08:09.777 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:09.778 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.778 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:09.778 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.468339+0000", 2024-09-21T22:08:09.778 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:09.778 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.778 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.778 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:09.778 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.468339+0000", 2024-09-21T22:08:09.778 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:09.778 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.778 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.778 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:09.778 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.468340+0000", 2024-09-21T22:08:09.778 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4330000000000001e-06 2024-09-21T22:08:09.779 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.779 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.779 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:09.779 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.469484+0000", 2024-09-21T22:08:09.779 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001143535 2024-09-21T22:08:09.779 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.779 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.779 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:09.779 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.469533+0000", 2024-09-21T22:08:09.779 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8980999999999997e-05 2024-09-21T22:08:09.779 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.779 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.779 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:09.779 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.469581+0000", 2024-09-21T22:08:09.780 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8551999999999999e-05 2024-09-21T22:08:09.780 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.780 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.780 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:09.780 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.469804+0000", 2024-09-21T22:08:09.780 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00022287699999999999 2024-09-21T22:08:09.780 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.780 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.780 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:09.780 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.469829+0000", 2024-09-21T22:08:09.780 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4766e-05 2024-09-21T22:08:09.780 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.780 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.780 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:09.780 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.532349+0000", 2024-09-21T22:08:09.781 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.062519880999999999 2024-09-21T22:08:09.781 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:09.781 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:09.781 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:09.781 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.781 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.781 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4840.0:668 2.3 2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:08:09.781 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.409114+0000", 2024-09-21T22:08:09.781 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 45.343993138000002, 2024-09-21T22:08:09.781 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.048703758999999999, 2024-09-21T22:08:09.781 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:09.781 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:09.781 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:09.781 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-21T22:08:09.781 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4840", 2024-09-21T22:08:09.782 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:08:09.782 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 668 2024-09-21T22:08:09.782 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.782 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:09.782 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.782 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:09.782 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.409114+0000", 2024-09-21T22:08:09.782 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:09.782 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.782 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.782 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:09.782 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.409114+0000", 2024-09-21T22:08:09.782 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:09.782 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.783 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.783 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:09.783 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.409115+0000", 2024-09-21T22:08:09.783 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.114e-06 2024-09-21T22:08:09.783 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.783 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.783 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:09.783 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.410308+0000", 2024-09-21T22:08:09.783 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011930949999999999 2024-09-21T22:08:09.783 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.783 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.783 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:09.783 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.410325+0000", 2024-09-21T22:08:09.783 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7088e-05 2024-09-21T22:08:09.784 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.784 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.784 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:09.784 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.410334+0000", 2024-09-21T22:08:09.784 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.2549999999999997e-06 2024-09-21T22:08:09.784 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.784 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.784 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:09.784 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.410393+0000", 2024-09-21T22:08:09.784 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.9327999999999998e-05 2024-09-21T22:08:09.784 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.784 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.784 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:09.784 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.410482+0000", 2024-09-21T22:08:09.784 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.9019999999999998e-05 2024-09-21T22:08:09.785 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.785 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.785 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-21T22:08:09.785 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.410617+0000", 2024-09-21T22:08:09.785 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000135232 2024-09-21T22:08:09.785 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.785 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.785 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:09.785 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.457051+0000", 2024-09-21T22:08:09.785 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.04643415 2024-09-21T22:08:09.785 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.785 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.785 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-21T22:08:09.785 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.457675+0000", 2024-09-21T22:08:09.785 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00062332700000000002 2024-09-21T22:08:09.785 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.786 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.786 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:09.786 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.457735+0000", 2024-09-21T22:08:09.786 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.029e-05 2024-09-21T22:08:09.786 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.786 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.786 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:09.786 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.457818+0000", 2024-09-21T22:08:09.786 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.2860000000000003e-05 2024-09-21T22:08:09.786 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:09.786 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:09.786 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:09.786 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.786 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.786 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4840.0:670 2.3 2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:08:09.787 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.466980+0000", 2024-09-21T22:08:09.787 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 45.286127069999999, 2024-09-21T22:08:09.787 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.050913930000000003, 2024-09-21T22:08:09.787 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:09.787 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:09.787 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:09.787 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-21T22:08:09.787 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4840", 2024-09-21T22:08:09.787 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:08:09.787 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 670 2024-09-21T22:08:09.787 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.787 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:09.787 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.787 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:09.788 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.466980+0000", 2024-09-21T22:08:09.788 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:09.788 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.788 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.788 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:09.788 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.466980+0000", 2024-09-21T22:08:09.788 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:09.788 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.788 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.788 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:09.788 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.466981+0000", 2024-09-21T22:08:09.788 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0839999999999999e-06 2024-09-21T22:08:09.788 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.788 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.788 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:09.789 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.467815+0000", 2024-09-21T22:08:09.789 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00083330399999999999 2024-09-21T22:08:09.789 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.789 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.789 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:09.789 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.467832+0000", 2024-09-21T22:08:09.789 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7601e-05 2024-09-21T22:08:09.789 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.789 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.789 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:09.789 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.467840+0000", 2024-09-21T22:08:09.789 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.2220000000000002e-06 2024-09-21T22:08:09.789 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.789 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.789 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:09.790 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.467974+0000", 2024-09-21T22:08:09.790 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000133411 2024-09-21T22:08:09.790 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.790 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.790 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:09.790 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.468223+0000", 2024-09-21T22:08:09.790 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00024906400000000001 2024-09-21T22:08:09.790 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.790 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.790 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-21T22:08:09.790 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.468616+0000", 2024-09-21T22:08:09.790 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000393433 2024-09-21T22:08:09.790 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.790 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.790 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:09.791 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.517016+0000", 2024-09-21T22:08:09.791 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.048399708 2024-09-21T22:08:09.791 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.791 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.791 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-21T22:08:09.791 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.517693+0000", 2024-09-21T22:08:09.791 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00067681000000000002 2024-09-21T22:08:09.791 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.791 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.791 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:09.791 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.517789+0000", 2024-09-21T22:08:09.791 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.5778000000000004e-05 2024-09-21T22:08:09.791 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.791 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.791 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:09.792 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.517894+0000", 2024-09-21T22:08:09.792 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000105515 2024-09-21T22:08:09.792 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:09.792 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:09.792 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:09.792 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.792 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.792 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4840.0:672 2.3 2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:08:09.792 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.527127+0000", 2024-09-21T22:08:09.792 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 45.225980608, 2024-09-21T22:08:09.792 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.047345602000000001, 2024-09-21T22:08:09.792 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:09.792 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:09.792 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:09.793 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-21T22:08:09.793 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4840", 2024-09-21T22:08:09.793 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:08:09.793 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 672 2024-09-21T22:08:09.793 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.793 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:09.793 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.793 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:09.793 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.527127+0000", 2024-09-21T22:08:09.793 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:09.793 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.793 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.793 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:09.793 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.527127+0000", 2024-09-21T22:08:09.794 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:09.794 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.794 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.794 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:09.794 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.527128+0000", 2024-09-21T22:08:09.794 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2810000000000001e-06 2024-09-21T22:08:09.794 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.794 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.794 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:09.794 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.527962+0000", 2024-09-21T22:08:09.794 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00083445199999999996 2024-09-21T22:08:09.794 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.794 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.794 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:09.795 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.527981+0000", 2024-09-21T22:08:09.795 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9009e-05 2024-09-21T22:08:09.795 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.795 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.795 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:09.795 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.527990+0000", 2024-09-21T22:08:09.795 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.8619999999999992e-06 2024-09-21T22:08:09.795 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.795 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.795 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:09.795 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.528061+0000", 2024-09-21T22:08:09.795 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.0226000000000002e-05 2024-09-21T22:08:09.795 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.795 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.795 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:09.796 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.528156+0000", 2024-09-21T22:08:09.796 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.5724999999999996e-05 2024-09-21T22:08:09.796 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.796 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.796 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-21T22:08:09.796 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.528303+0000", 2024-09-21T22:08:09.796 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000146374 2024-09-21T22:08:09.796 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.796 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.796 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:09.796 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.573772+0000", 2024-09-21T22:08:09.796 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.045469745999999998 2024-09-21T22:08:09.796 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.796 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.797 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-21T22:08:09.797 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.574299+0000", 2024-09-21T22:08:09.797 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00052649200000000004 2024-09-21T22:08:09.797 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.797 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.797 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:09.797 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.574366+0000", 2024-09-21T22:08:09.797 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.6872000000000006e-05 2024-09-21T22:08:09.797 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.797 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.797 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:09.797 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.574472+0000", 2024-09-21T22:08:09.797 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000106563 2024-09-21T22:08:09.797 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:09.797 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:09.798 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:09.798 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.798 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.798 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4840.0:686 2.3 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:08:09.798 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.943561+0000", 2024-09-21T22:08:09.798 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 44.809546331999996, 2024-09-21T22:08:09.798 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.049021493999999999, 2024-09-21T22:08:09.798 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:09.798 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:09.798 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:09.798 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-21T22:08:09.798 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4840", 2024-09-21T22:08:09.798 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:08:09.798 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 686 2024-09-21T22:08:09.799 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.799 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:09.799 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.799 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:09.799 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.943561+0000", 2024-09-21T22:08:09.799 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:09.799 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.799 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.799 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:09.799 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.943561+0000", 2024-09-21T22:08:09.799 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:09.799 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.799 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.799 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:09.800 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.943563+0000", 2024-09-21T22:08:09.800 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9420000000000002e-06 2024-09-21T22:08:09.800 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.800 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.800 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:09.800 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.944546+0000", 2024-09-21T22:08:09.800 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00098357099999999992 2024-09-21T22:08:09.800 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.800 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.800 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:09.800 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.944565+0000", 2024-09-21T22:08:09.800 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8621e-05 2024-09-21T22:08:09.800 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.800 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.800 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:09.801 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.944575+0000", 2024-09-21T22:08:09.801 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.4639999999999995e-06 2024-09-21T22:08:09.801 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.801 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.801 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:09.801 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.944647+0000", 2024-09-21T22:08:09.801 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.2786000000000004e-05 2024-09-21T22:08:09.801 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.801 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.801 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:09.801 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.944746+0000", 2024-09-21T22:08:09.801 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.8196999999999995e-05 2024-09-21T22:08:09.801 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.801 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.801 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-21T22:08:09.802 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.944890+0000", 2024-09-21T22:08:09.802 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000144804 2024-09-21T22:08:09.802 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.802 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.802 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:09.802 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.992404+0000", 2024-09-21T22:08:09.802 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.047513211999999999 2024-09-21T22:08:09.802 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.802 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.802 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-21T22:08:09.802 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.992480+0000", 2024-09-21T22:08:09.802 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.6491999999999999e-05 2024-09-21T22:08:09.802 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.802 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.803 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:09.803 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.992521+0000", 2024-09-21T22:08:09.803 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.0608000000000002e-05 2024-09-21T22:08:09.803 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.803 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.803 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:09.803 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.992582+0000", 2024-09-21T22:08:09.803 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.1797000000000005e-05 2024-09-21T22:08:09.803 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:09.803 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:09.803 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:09.803 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.803 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.803 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4840.0:720 2.1 2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:08:09.804 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.803827+0000", 2024-09-21T22:08:09.804 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 43.949280055999999, 2024-09-21T22:08:09.804 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.049890510999999998, 2024-09-21T22:08:09.804 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:09.804 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:09.804 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:09.804 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-21T22:08:09.804 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4840", 2024-09-21T22:08:09.804 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:08:09.804 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 720 2024-09-21T22:08:09.804 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.804 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:09.804 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.804 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:09.804 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.803827+0000", 2024-09-21T22:08:09.805 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:09.805 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.805 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.805 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:09.805 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.803827+0000", 2024-09-21T22:08:09.805 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:09.805 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.805 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.805 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:09.805 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.803828+0000", 2024-09-21T22:08:09.805 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.569999999999999e-07 2024-09-21T22:08:09.805 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.805 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.805 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:09.805 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.804689+0000", 2024-09-21T22:08:09.806 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00086121799999999997 2024-09-21T22:08:09.806 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.806 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.806 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:09.806 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.804706+0000", 2024-09-21T22:08:09.806 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6266999999999999e-05 2024-09-21T22:08:09.806 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.806 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.806 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:09.806 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.804713+0000", 2024-09-21T22:08:09.806 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.1010000000000003e-06 2024-09-21T22:08:09.806 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.806 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.806 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:09.807 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.804781+0000", 2024-09-21T22:08:09.807 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.7871000000000001e-05 2024-09-21T22:08:09.807 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.807 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.807 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:09.807 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.804859+0000", 2024-09-21T22:08:09.807 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.7891000000000004e-05 2024-09-21T22:08:09.807 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.807 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.807 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 0", 2024-09-21T22:08:09.807 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.804971+0000", 2024-09-21T22:08:09.807 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00011275100000000001 2024-09-21T22:08:09.807 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.807 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.807 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:09.808 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.852704+0000", 2024-09-21T22:08:09.808 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.047732361000000001 2024-09-21T22:08:09.808 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.808 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.808 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-21T22:08:09.808 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.853446+0000", 2024-09-21T22:08:09.808 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00074258799999999997 2024-09-21T22:08:09.808 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.808 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.808 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:09.808 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.853608+0000", 2024-09-21T22:08:09.808 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00016190900000000001 2024-09-21T22:08:09.808 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.808 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.808 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:09.809 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.853718+0000", 2024-09-21T22:08:09.809 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000109597 2024-09-21T22:08:09.809 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:09.809 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:09.809 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:09.809 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.809 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.809 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4840.0:723 2.1 2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:08:09.809 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.860543+0000", 2024-09-21T22:08:09.809 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 43.892564622000002, 2024-09-21T22:08:09.809 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.062354957000000003, 2024-09-21T22:08:09.809 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:09.809 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:09.809 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:09.809 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-21T22:08:09.810 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4840", 2024-09-21T22:08:09.810 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:08:09.810 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 723 2024-09-21T22:08:09.810 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.810 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:09.810 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.810 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:09.810 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.860543+0000", 2024-09-21T22:08:09.810 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:09.810 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.810 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.810 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:09.810 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.860543+0000", 2024-09-21T22:08:09.810 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:09.811 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.811 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.811 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:09.811 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.860544+0000", 2024-09-21T22:08:09.811 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.1119999999999999e-06 2024-09-21T22:08:09.811 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.811 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.811 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:09.811 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.861400+0000", 2024-09-21T22:08:09.811 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000856393 2024-09-21T22:08:09.811 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.811 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.811 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:09.811 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.861419+0000", 2024-09-21T22:08:09.812 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8992e-05 2024-09-21T22:08:09.812 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.812 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.812 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:09.812 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.861428+0000", 2024-09-21T22:08:09.812 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.551e-06 2024-09-21T22:08:09.812 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.812 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.812 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:09.812 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.861555+0000", 2024-09-21T22:08:09.812 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000127286 2024-09-21T22:08:09.812 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.812 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.812 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:09.812 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.861816+0000", 2024-09-21T22:08:09.813 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00026100899999999998 2024-09-21T22:08:09.813 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.813 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.813 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 0", 2024-09-21T22:08:09.813 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.862189+0000", 2024-09-21T22:08:09.813 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00037261899999999998 2024-09-21T22:08:09.813 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.813 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.813 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:09.813 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.921514+0000", 2024-09-21T22:08:09.813 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.059325210000000003 2024-09-21T22:08:09.813 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.813 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.813 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-21T22:08:09.814 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.922721+0000", 2024-09-21T22:08:09.814 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0012066819999999999 2024-09-21T22:08:09.814 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.814 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.814 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:09.814 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.922797+0000", 2024-09-21T22:08:09.814 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.6162999999999996e-05 2024-09-21T22:08:09.814 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.814 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.814 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:09.814 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.922898+0000", 2024-09-21T22:08:09.814 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00010094000000000001 2024-09-21T22:08:09.814 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:09.814 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:09.815 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:09.815 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.815 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.815 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:725 2.0 e63/16 2:08b4f96a:::rbd_data.126483d75f2d.000000000000000f:head v 63'33, mlcod=63'31)", 2024-09-21T22:08:09.815 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.936778+0000", 2024-09-21T22:08:09.815 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 43.81632888, 2024-09-21T22:08:09.815 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.047671937999999997, 2024-09-21T22:08:09.815 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:09.815 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:09.815 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:09.815 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:09.815 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.815 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:09.815 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.936778+0000", 2024-09-21T22:08:09.816 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:09.816 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.816 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.816 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:09.816 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.936778+0000", 2024-09-21T22:08:09.816 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:09.816 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.816 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.816 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:09.816 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.936780+0000", 2024-09-21T22:08:09.816 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7799999999999999e-06 2024-09-21T22:08:09.816 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.816 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.817 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:09.817 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.938475+0000", 2024-09-21T22:08:09.817 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0016951749999999999 2024-09-21T22:08:09.817 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.817 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.817 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:09.817 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.938506+0000", 2024-09-21T22:08:09.817 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.0865999999999999e-05 2024-09-21T22:08:09.817 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.817 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.817 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:09.817 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.938520+0000", 2024-09-21T22:08:09.817 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3825999999999999e-05 2024-09-21T22:08:09.817 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.817 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.818 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:09.818 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.938624+0000", 2024-09-21T22:08:09.818 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00010391900000000001 2024-09-21T22:08:09.818 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.818 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.818 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:09.818 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.938651+0000", 2024-09-21T22:08:09.818 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.7481999999999999e-05 2024-09-21T22:08:09.818 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.818 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.818 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:09.818 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.984369+0000", 2024-09-21T22:08:09.818 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.045717595 2024-09-21T22:08:09.819 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.819 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.819 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:09.819 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.984450+0000", 2024-09-21T22:08:09.819 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.1295e-05 2024-09-21T22:08:09.819 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:09.819 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:09.819 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:09.819 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.819 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.819 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:727 2.0 e63/16 2:08b4f96a:::rbd_data.126483d75f2d.000000000000000f:head v 63'35, mlcod=63'33)", 2024-09-21T22:08:09.819 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.998090+0000", 2024-09-21T22:08:09.819 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 43.755017703999997, 2024-09-21T22:08:09.820 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.052205704999999998, 2024-09-21T22:08:09.820 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:09.820 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:09.820 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:09.820 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:09.820 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.820 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:09.820 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.998090+0000", 2024-09-21T22:08:09.820 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:09.820 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.820 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.820 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:09.820 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.998090+0000", 2024-09-21T22:08:09.820 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:09.820 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.821 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.821 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:09.821 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.998091+0000", 2024-09-21T22:08:09.821 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6929999999999999e-06 2024-09-21T22:08:09.821 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.821 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.821 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:09.821 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.998940+0000", 2024-09-21T22:08:09.821 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00084875200000000004 2024-09-21T22:08:09.821 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.821 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.821 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:09.821 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.998970+0000", 2024-09-21T22:08:09.821 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9709999999999998e-05 2024-09-21T22:08:09.822 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.822 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.822 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:09.822 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.998984+0000", 2024-09-21T22:08:09.822 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3739e-05 2024-09-21T22:08:09.822 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.822 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.822 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:09.822 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.999097+0000", 2024-09-21T22:08:09.822 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000113484 2024-09-21T22:08:09.822 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.822 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.822 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:09.822 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.999135+0000", 2024-09-21T22:08:09.823 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.8405999999999998e-05 2024-09-21T22:08:09.823 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.823 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.823 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:09.823 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.050245+0000", 2024-09-21T22:08:09.823 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.051109935000000002 2024-09-21T22:08:09.823 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.823 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.823 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:09.823 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.050295+0000", 2024-09-21T22:08:09.823 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.9985999999999999e-05 2024-09-21T22:08:09.823 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:09.823 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:09.823 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:09.824 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.824 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.824 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:729 2.0 e63/16 2:08b4f96a:::rbd_data.126483d75f2d.000000000000000f:head v 63'37, mlcod=63'35)", 2024-09-21T22:08:09.824 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:26.062424+0000", 2024-09-21T22:08:09.824 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 43.690683082, 2024-09-21T22:08:09.824 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.047312186999999999, 2024-09-21T22:08:09.824 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:09.824 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:09.824 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:09.824 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:09.824 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.824 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:09.824 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.062424+0000", 2024-09-21T22:08:09.824 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:09.825 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.825 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.825 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:09.825 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.062424+0000", 2024-09-21T22:08:09.825 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:09.825 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.825 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.825 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:09.825 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.062426+0000", 2024-09-21T22:08:09.825 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7129999999999999e-06 2024-09-21T22:08:09.825 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.825 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.825 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:09.825 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.063211+0000", 2024-09-21T22:08:09.825 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00078488500000000003 2024-09-21T22:08:09.826 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.826 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.826 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:09.826 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.063241+0000", 2024-09-21T22:08:09.826 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.0056000000000002e-05 2024-09-21T22:08:09.826 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.826 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.826 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:09.826 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.063255+0000", 2024-09-21T22:08:09.826 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4022999999999999e-05 2024-09-21T22:08:09.826 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.826 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.826 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:09.826 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.063384+0000", 2024-09-21T22:08:09.827 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00012920600000000001 2024-09-21T22:08:09.827 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.827 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.827 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:09.827 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.063421+0000", 2024-09-21T22:08:09.827 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.7242e-05 2024-09-21T22:08:09.827 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.827 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.827 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:09.827 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.109666+0000", 2024-09-21T22:08:09.827 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.046244446000000002 2024-09-21T22:08:09.827 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.827 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.827 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:09.828 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.109736+0000", 2024-09-21T22:08:09.828 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.0616000000000003e-05 2024-09-21T22:08:09.828 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:09.828 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:09.828 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:09.828 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.828 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.828 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:731 2.0 e63/16 2:08b4f96a:::rbd_data.126483d75f2d.000000000000000f:head v 63'39, mlcod=63'37)", 2024-09-21T22:08:09.828 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:26.135946+0000", 2024-09-21T22:08:09.828 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 43.617161574999997, 2024-09-21T22:08:09.828 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.054290359000000003, 2024-09-21T22:08:09.828 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:09.828 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:09.828 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:09.829 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:09.829 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.829 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:09.829 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.135946+0000", 2024-09-21T22:08:09.829 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:09.829 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.829 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.829 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:09.829 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.135946+0000", 2024-09-21T22:08:09.829 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:09.829 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.829 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.829 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:09.829 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.135948+0000", 2024-09-21T22:08:09.830 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7999999999999999e-06 2024-09-21T22:08:09.830 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.830 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.830 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:09.830 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.136730+0000", 2024-09-21T22:08:09.830 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00078271899999999997 2024-09-21T22:08:09.830 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.830 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.830 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:09.830 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.136760+0000", 2024-09-21T22:08:09.830 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9876e-05 2024-09-21T22:08:09.830 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.830 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.830 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:09.831 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.136774+0000", 2024-09-21T22:08:09.831 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3478999999999999e-05 2024-09-21T22:08:09.831 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.831 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.831 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:09.831 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.136944+0000", 2024-09-21T22:08:09.831 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000170626 2024-09-21T22:08:09.831 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.831 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.831 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:09.831 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.136985+0000", 2024-09-21T22:08:09.831 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.0377999999999999e-05 2024-09-21T22:08:09.831 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.831 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.831 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:09.832 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.190137+0000", 2024-09-21T22:08:09.832 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.053152438000000003 2024-09-21T22:08:09.832 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.832 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.832 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:09.832 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.190236+0000", 2024-09-21T22:08:09.832 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.9042999999999993e-05 2024-09-21T22:08:09.832 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:09.832 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:09.832 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:09.832 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.832 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.832 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:125 2.3 e71/66 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head v 71'31, mlcod=71'29)", 2024-09-21T22:08:09.832 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.066855+0000", 2024-09-21T22:08:09.833 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 31.686252140000001, 2024-09-21T22:08:09.833 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.059856059000000003, 2024-09-21T22:08:09.833 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:09.833 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:09.833 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:09.833 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:09.833 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.833 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:09.833 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.066855+0000", 2024-09-21T22:08:09.833 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:09.833 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.833 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.833 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:09.834 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.066855+0000", 2024-09-21T22:08:09.834 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:09.834 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.834 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.834 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:09.834 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.066857+0000", 2024-09-21T22:08:09.834 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.807e-06 2024-09-21T22:08:09.834 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.834 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.834 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:09.834 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.068365+0000", 2024-09-21T22:08:09.834 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0015077910000000001 2024-09-21T22:08:09.834 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.834 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.834 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:09.835 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.068395+0000", 2024-09-21T22:08:09.835 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.0326000000000001e-05 2024-09-21T22:08:09.835 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.835 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.835 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:09.835 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.068415+0000", 2024-09-21T22:08:09.835 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9871999999999999e-05 2024-09-21T22:08:09.835 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.835 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.835 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:09.835 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.068580+0000", 2024-09-21T22:08:09.835 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000165317 2024-09-21T22:08:09.835 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.835 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.836 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:09.836 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.068618+0000", 2024-09-21T22:08:09.836 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.7753000000000003e-05 2024-09-21T22:08:09.836 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.836 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.836 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:09.836 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.126623+0000", 2024-09-21T22:08:09.836 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.058004642000000002 2024-09-21T22:08:09.836 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.836 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.836 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:09.836 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.126711+0000", 2024-09-21T22:08:09.836 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.8551e-05 2024-09-21T22:08:09.836 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:09.837 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:09.837 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:09.837 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.837 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.837 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:127 2.3 e71/66 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head v 71'33, mlcod=71'31)", 2024-09-21T22:08:09.837 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.140864+0000", 2024-09-21T22:08:09.837 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 31.612243112000002, 2024-09-21T22:08:09.837 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.052512380999999997, 2024-09-21T22:08:09.837 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:09.837 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:09.837 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:09.837 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:09.837 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.837 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:09.837 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.140864+0000", 2024-09-21T22:08:09.838 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:09.838 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.838 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.838 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:09.838 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.140864+0000", 2024-09-21T22:08:09.838 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:09.838 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.838 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.838 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:09.838 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.140866+0000", 2024-09-21T22:08:09.838 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.694e-06 2024-09-21T22:08:09.838 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.838 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.838 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:09.839 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.141712+0000", 2024-09-21T22:08:09.839 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00084581499999999998 2024-09-21T22:08:09.839 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.839 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.839 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:09.839 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.141742+0000", 2024-09-21T22:08:09.839 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9865999999999998e-05 2024-09-21T22:08:09.839 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.839 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.839 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:09.839 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.141761+0000", 2024-09-21T22:08:09.839 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9473000000000001e-05 2024-09-21T22:08:09.839 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.839 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.840 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:09.840 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.141919+0000", 2024-09-21T22:08:09.840 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00015758300000000001 2024-09-21T22:08:09.840 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.840 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.840 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:09.840 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.141955+0000", 2024-09-21T22:08:09.840 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.6613000000000003e-05 2024-09-21T22:08:09.840 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.840 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.840 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:09.840 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.193333+0000", 2024-09-21T22:08:09.840 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.051377260000000001 2024-09-21T22:08:09.840 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.840 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.841 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:09.841 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.193377+0000", 2024-09-21T22:08:09.841 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.4076999999999998e-05 2024-09-21T22:08:09.841 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:09.841 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:09.841 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:09.841 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.841 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.841 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:129 2.3 e71/66 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head v 71'35, mlcod=71'33)", 2024-09-21T22:08:09.841 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.209458+0000", 2024-09-21T22:08:09.841 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 31.543649363, 2024-09-21T22:08:09.841 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.056595578000000001, 2024-09-21T22:08:09.841 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:09.841 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:09.841 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:09.842 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:09.842 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.842 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:09.842 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.209458+0000", 2024-09-21T22:08:09.842 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:09.842 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.842 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.842 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:09.842 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.209458+0000", 2024-09-21T22:08:09.842 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:09.842 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.842 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.842 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:09.842 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.209460+0000", 2024-09-21T22:08:09.843 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7600000000000001e-06 2024-09-21T22:08:09.843 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.843 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.843 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:09.843 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.210317+0000", 2024-09-21T22:08:09.843 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00085771700000000003 2024-09-21T22:08:09.843 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.843 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.843 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:09.843 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.210346+0000", 2024-09-21T22:08:09.843 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8334e-05 2024-09-21T22:08:09.843 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.843 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.843 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:09.844 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.210365+0000", 2024-09-21T22:08:09.844 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9439000000000001e-05 2024-09-21T22:08:09.844 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.844 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.844 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:09.844 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.210544+0000", 2024-09-21T22:08:09.844 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00017845599999999999 2024-09-21T22:08:09.844 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.844 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.844 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:09.844 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.210584+0000", 2024-09-21T22:08:09.844 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.0423000000000002e-05 2024-09-21T22:08:09.844 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.844 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.844 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:09.844 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.265970+0000", 2024-09-21T22:08:09.845 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.055385765000000003 2024-09-21T22:08:09.845 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.845 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.845 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:09.845 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.266054+0000", 2024-09-21T22:08:09.845 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.3683999999999994e-05 2024-09-21T22:08:09.845 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:09.845 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:09.845 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:09.845 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.845 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.845 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:131 2.3 e71/66 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head v 71'37, mlcod=71'35)", 2024-09-21T22:08:09.845 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.279550+0000", 2024-09-21T22:08:09.846 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 31.473557525, 2024-09-21T22:08:09.846 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.052290419999999997, 2024-09-21T22:08:09.846 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:09.846 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:09.846 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:09.846 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:09.846 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.846 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:09.846 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.279550+0000", 2024-09-21T22:08:09.846 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:09.846 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.846 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.846 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:09.846 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.279550+0000", 2024-09-21T22:08:09.846 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:09.847 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.847 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.847 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:09.847 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.279552+0000", 2024-09-21T22:08:09.847 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9439999999999999e-06 2024-09-21T22:08:09.847 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.847 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.847 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:09.847 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.280296+0000", 2024-09-21T22:08:09.847 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00074417000000000005 2024-09-21T22:08:09.847 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.847 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.847 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:09.847 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.280324+0000", 2024-09-21T22:08:09.848 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8045999999999999e-05 2024-09-21T22:08:09.848 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.848 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.848 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:09.848 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.280345+0000", 2024-09-21T22:08:09.848 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.0849999999999999e-05 2024-09-21T22:08:09.848 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.848 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.848 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:09.848 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.280497+0000", 2024-09-21T22:08:09.848 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00015252500000000001 2024-09-21T22:08:09.848 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.848 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.848 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:09.849 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.280531+0000", 2024-09-21T22:08:09.849 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.4034e-05 2024-09-21T22:08:09.849 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.849 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.849 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:09.849 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.331769+0000", 2024-09-21T22:08:09.849 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.051237999999999999 2024-09-21T22:08:09.849 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:09.849 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:09.849 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:09.849 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.331840+0000", 2024-09-21T22:08:09.849 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.0851000000000003e-05 2024-09-21T22:08:09.849 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:09.849 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:09.849 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:09.850 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:09.850 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:09.850 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:08:09.851 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T22:08:09.852 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:08:09.849+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:08:09.953 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:08:09.948+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:08:09.984 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:08:09.985 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:08:09.985 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-09-21T22:08:09.985 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:08:09.999 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T22:08:10.091 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-21T22:08:10.094 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:08:10.092+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:08:10.114 INFO:tasks.workunit.client.0.smithi157.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-09-21T22:08:10.123 INFO:tasks.workunit.client.0.smithi157.stderr:+ _sudo dd if=/tmp/tmp.GqKuv7rsZi/data of=/tmp/tmp.GqKuv7rsZi/mnt/test bs=1M count=1 oflag=direct 2024-09-21T22:08:10.123 INFO:tasks.workunit.client.0.smithi157.stderr:+ local cmd 2024-09-21T22:08:10.123 INFO:tasks.workunit.client.0.smithi157.stderr:++ id -u 2024-09-21T22:08:10.126 INFO:tasks.workunit.client.0.smithi157.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T22:08:10.126 INFO:tasks.workunit.client.0.smithi157.stderr:++ which dd 2024-09-21T22:08:10.127 INFO:tasks.workunit.client.0.smithi157.stderr:+ cmd=/usr/bin/dd 2024-09-21T22:08:10.127 INFO:tasks.workunit.client.0.smithi157.stderr:+ shift 2024-09-21T22:08:10.127 INFO:tasks.workunit.client.0.smithi157.stderr:+ sudo -nE /usr/bin/dd if=/tmp/tmp.GqKuv7rsZi/data of=/tmp/tmp.GqKuv7rsZi/mnt/test bs=1M count=1 oflag=direct 2024-09-21T22:08:10.139 INFO:tasks.workunit.client.0.smithi157.stderr:1+0 records in 2024-09-21T22:08:10.139 INFO:tasks.workunit.client.0.smithi157.stderr:1+0 records out 2024-09-21T22:08:10.139 INFO:tasks.workunit.client.0.smithi157.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00221132 s, 474 MB/s 2024-09-21T22:08:10.140 INFO:tasks.workunit.client.0.smithi157.stderr:+ _sudo umount /tmp/tmp.GqKuv7rsZi/mnt 2024-09-21T22:08:10.140 INFO:tasks.workunit.client.0.smithi157.stderr:+ local cmd 2024-09-21T22:08:10.140 INFO:tasks.workunit.client.0.smithi157.stderr:++ id -u 2024-09-21T22:08:10.141 INFO:tasks.workunit.client.0.smithi157.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T22:08:10.142 INFO:tasks.workunit.client.0.smithi157.stderr:++ which umount 2024-09-21T22:08:10.143 INFO:tasks.workunit.client.0.smithi157.stderr:+ cmd=/usr/bin/umount 2024-09-21T22:08:10.143 INFO:tasks.workunit.client.0.smithi157.stderr:+ shift 2024-09-21T22:08:10.143 INFO:tasks.workunit.client.0.smithi157.stderr:+ sudo -nE /usr/bin/umount /tmp/tmp.GqKuv7rsZi/mnt 2024-09-21T22:08:10.155 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:08:10.152+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:08:10.197 INFO:tasks.workunit.client.0.smithi157.stderr:+ unmap_device /dev/nbd1 23853 2024-09-21T22:08:10.197 INFO:tasks.workunit.client.0.smithi157.stderr:+ local args=/dev/nbd1 2024-09-21T22:08:10.197 INFO:tasks.workunit.client.0.smithi157.stderr:+ local pid=23853 2024-09-21T22:08:10.197 INFO:tasks.workunit.client.0.smithi157.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd1 2024-09-21T22:08:10.198 INFO:tasks.workunit.client.0.smithi157.stderr:+ local cmd 2024-09-21T22:08:10.198 INFO:tasks.workunit.client.0.smithi157.stderr:++ id -u 2024-09-21T22:08:10.198 INFO:tasks.workunit.client.0.smithi157.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T22:08:10.199 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:08:10.199 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:08:10.199 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-21T22:08:10.199 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-21T22:08:10.199 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:08:10.199 INFO:tasks.workunit.client.0.smithi157.stderr:++ which rbd 2024-09-21T22:08:10.200 INFO:tasks.workunit.client.0.smithi157.stderr:+ cmd=/usr/bin/rbd 2024-09-21T22:08:10.200 INFO:tasks.workunit.client.0.smithi157.stderr:+ shift 2024-09-21T22:08:10.200 INFO:tasks.workunit.client.0.smithi157.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd1 2024-09-21T22:08:10.213 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T22:08:10.256 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:08:10.252+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:08:10.313 INFO:tasks.workunit.client.0.smithi157.stderr:+ rbd device --device-type nbd list 2024-09-21T22:08:10.313 INFO:tasks.workunit.client.0.smithi157.stderr:+ expect_false grep '^23853\b' 2024-09-21T22:08:10.313 INFO:tasks.workunit.client.0.smithi157.stderr:+ grep '^23853\b' 2024-09-21T22:08:10.356 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:08:10.352+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:08:10.381 INFO:tasks.workunit.client.0.smithi157.stderr:+ return 0 2024-09-21T22:08:10.381 INFO:tasks.workunit.client.0.smithi157.stderr:+ ps -C rbd-nbd 2024-09-21T22:08:10.382 INFO:tasks.workunit.client.0.smithi157.stderr:+ expect_false grep '^ *23853\b' 2024-09-21T22:08:10.382 INFO:tasks.workunit.client.0.smithi157.stderr:+ grep '^ *23853\b' 2024-09-21T22:08:10.385 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:08:10.385 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-21T22:08:10.385 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-21T22:08:10.385 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-21T22:08:10.385 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.385 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:08:10.385 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.036355+0000", 2024-09-21T22:08:10.385 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 128.34588528800001, 2024-09-21T22:08:10.385 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.062441934999999997, 2024-09-21T22:08:10.385 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:10.386 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:10.386 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:10.386 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:10.386 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:08:10.386 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:08:10.386 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1 2024-09-21T22:08:10.386 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.386 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:10.386 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.386 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:10.386 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036355+0000", 2024-09-21T22:08:10.386 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:10.386 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.386 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.387 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:10.387 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036355+0000", 2024-09-21T22:08:10.387 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:10.387 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.387 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.387 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:10.387 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036357+0000", 2024-09-21T22:08:10.387 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.446e-06 2024-09-21T22:08:10.387 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.387 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.387 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:10.387 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036373+0000", 2024-09-21T22:08:10.387 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5447e-05 2024-09-21T22:08:10.387 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.387 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.388 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:10.388 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036399+0000", 2024-09-21T22:08:10.388 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6423000000000001e-05 2024-09-21T22:08:10.388 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.388 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.388 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:10.388 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036420+0000", 2024-09-21T22:08:10.388 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1129999999999999e-05 2024-09-21T22:08:10.388 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.388 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.388 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:10.388 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036557+0000", 2024-09-21T22:08:10.388 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013686899999999999 2024-09-21T22:08:10.388 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.388 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.389 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-21T22:08:10.389 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036585+0000", 2024-09-21T22:08:10.389 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8362999999999999e-05 2024-09-21T22:08:10.389 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.389 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.389 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:10.389 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.095068+0000", 2024-09-21T22:08:10.389 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058482972000000001 2024-09-21T22:08:10.389 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.389 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.390 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:10.390 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.095516+0000", 2024-09-21T22:08:10.390 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00044802999999999998 2024-09-21T22:08:10.390 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.390 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.390 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:08:10.390 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.095869+0000", 2024-09-21T22:08:10.390 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000352104 2024-09-21T22:08:10.390 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.390 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.390 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:10.390 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.097055+0000", 2024-09-21T22:08:10.390 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011865599999999999 2024-09-21T22:08:10.391 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.391 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.391 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:10.391 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.098637+0000", 2024-09-21T22:08:10.391 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001581458 2024-09-21T22:08:10.391 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.391 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.391 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:10.391 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.098728+0000", 2024-09-21T22:08:10.391 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1043000000000002e-05 2024-09-21T22:08:10.391 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.391 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.391 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:10.392 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.098797+0000", 2024-09-21T22:08:10.392 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9090000000000004e-05 2024-09-21T22:08:10.393 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:10.393 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:10.393 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:10.393 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.393 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.393 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4840.0:680 2.7 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:08:10.393 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.744506+0000", 2024-09-21T22:08:10.393 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 45.637734434000002, 2024-09-21T22:08:10.393 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.063865055000000004, 2024-09-21T22:08:10.393 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:10.393 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:10.393 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:10.393 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:10.394 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4840", 2024-09-21T22:08:10.394 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:08:10.394 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 680 2024-09-21T22:08:10.394 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.394 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:10.394 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.394 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:10.394 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.744506+0000", 2024-09-21T22:08:10.394 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:10.394 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.394 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.394 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:10.394 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.744506+0000", 2024-09-21T22:08:10.394 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:10.394 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.395 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.395 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:10.395 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.744508+0000", 2024-09-21T22:08:10.395 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.103e-06 2024-09-21T22:08:10.395 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.395 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.395 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:10.395 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.745371+0000", 2024-09-21T22:08:10.395 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00086322199999999997 2024-09-21T22:08:10.395 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.395 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.395 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:10.395 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.745401+0000", 2024-09-21T22:08:10.395 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9955999999999999e-05 2024-09-21T22:08:10.396 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.396 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.396 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:10.396 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.745416+0000", 2024-09-21T22:08:10.396 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4873e-05 2024-09-21T22:08:10.396 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.396 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.396 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:10.396 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.745509+0000", 2024-09-21T22:08:10.396 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3171000000000006e-05 2024-09-21T22:08:10.396 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.396 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.396 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:10.396 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.745632+0000", 2024-09-21T22:08:10.396 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012272699999999999 2024-09-21T22:08:10.397 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.397 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.397 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-09-21T22:08:10.397 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.745789+0000", 2024-09-21T22:08:10.397 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015781399999999999 2024-09-21T22:08:10.397 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.397 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.397 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:10.397 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.785494+0000", 2024-09-21T22:08:10.397 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.039704401 2024-09-21T22:08:10.397 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.397 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.397 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:10.397 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.808090+0000", 2024-09-21T22:08:10.397 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022596207 2024-09-21T22:08:10.398 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.398 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.398 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:10.398 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.808246+0000", 2024-09-21T22:08:10.398 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015567200000000001 2024-09-21T22:08:10.398 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.398 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.398 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:10.398 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.808371+0000", 2024-09-21T22:08:10.398 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012490900000000001 2024-09-21T22:08:10.398 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:10.398 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:10.398 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:10.398 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.398 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.399 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4840.0:682 2.7 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:08:10.399 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.822415+0000", 2024-09-21T22:08:10.399 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 45.559824871000004, 2024-09-21T22:08:10.399 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.065044633000000004, 2024-09-21T22:08:10.399 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:10.399 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:10.399 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:10.399 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:10.399 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4840", 2024-09-21T22:08:10.399 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:08:10.399 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 682 2024-09-21T22:08:10.400 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.400 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:10.400 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.400 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:10.400 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.822415+0000", 2024-09-21T22:08:10.400 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:10.400 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.400 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.400 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:10.400 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.822415+0000", 2024-09-21T22:08:10.400 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:10.400 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.400 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.400 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:10.401 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.822417+0000", 2024-09-21T22:08:10.401 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8029999999999999e-06 2024-09-21T22:08:10.401 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.401 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.401 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:10.401 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.823308+0000", 2024-09-21T22:08:10.401 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00089146500000000003 2024-09-21T22:08:10.401 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.401 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.401 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:10.401 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.823339+0000", 2024-09-21T22:08:10.401 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0493000000000001e-05 2024-09-21T22:08:10.401 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.402 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.402 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:10.402 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.823354+0000", 2024-09-21T22:08:10.402 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5472999999999999e-05 2024-09-21T22:08:10.402 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.402 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.402 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:10.402 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.823476+0000", 2024-09-21T22:08:10.402 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000121209 2024-09-21T22:08:10.402 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.402 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.402 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:10.402 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.823650+0000", 2024-09-21T22:08:10.402 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017394599999999999 2024-09-21T22:08:10.403 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.403 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.403 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-09-21T22:08:10.403 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.823873+0000", 2024-09-21T22:08:10.403 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022314899999999999 2024-09-21T22:08:10.403 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.403 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.403 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:10.403 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.884396+0000", 2024-09-21T22:08:10.403 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.060523674999999999 2024-09-21T22:08:10.403 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.403 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.403 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:10.403 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.887238+0000", 2024-09-21T22:08:10.403 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028420450000000001 2024-09-21T22:08:10.404 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.404 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.404 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:10.404 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.887369+0000", 2024-09-21T22:08:10.404 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013010699999999999 2024-09-21T22:08:10.404 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.404 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.404 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:10.404 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.887460+0000", 2024-09-21T22:08:10.404 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1267999999999994e-05 2024-09-21T22:08:10.404 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:10.404 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:10.404 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:10.404 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.404 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.405 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4840.0:692 2.5 2:b1d55633:::rbd_data.126483d75f2d.000000000000000b:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:08:10.405 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:25.091048+0000", 2024-09-21T22:08:10.405 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 45.291192336999998, 2024-09-21T22:08:10.405 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.060917092999999999, 2024-09-21T22:08:10.405 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:10.405 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:10.405 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:10.405 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:10.405 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4840", 2024-09-21T22:08:10.405 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:08:10.405 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 692 2024-09-21T22:08:10.405 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.405 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:10.406 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.406 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:10.406 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.091048+0000", 2024-09-21T22:08:10.406 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:10.406 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.406 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.406 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:10.406 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.091048+0000", 2024-09-21T22:08:10.406 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:10.406 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.406 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.406 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:10.406 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.091050+0000", 2024-09-21T22:08:10.406 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.03e-06 2024-09-21T22:08:10.407 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.407 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.407 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:10.407 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.091989+0000", 2024-09-21T22:08:10.407 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00093932100000000001 2024-09-21T22:08:10.407 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.407 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.407 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:10.407 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.092023+0000", 2024-09-21T22:08:10.407 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4310000000000002e-05 2024-09-21T22:08:10.407 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.407 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.407 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:10.407 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.092040+0000", 2024-09-21T22:08:10.408 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6426000000000002e-05 2024-09-21T22:08:10.408 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.408 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.408 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:10.408 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.092175+0000", 2024-09-21T22:08:10.408 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013500299999999999 2024-09-21T22:08:10.408 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.408 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.408 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:10.408 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.092307+0000", 2024-09-21T22:08:10.408 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000132562 2024-09-21T22:08:10.408 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.408 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.408 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:08:10.409 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.092499+0000", 2024-09-21T22:08:10.409 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000191972 2024-09-21T22:08:10.409 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.409 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.409 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:10.409 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.151065+0000", 2024-09-21T22:08:10.409 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058565514999999999 2024-09-21T22:08:10.409 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.409 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.409 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:10.409 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.151844+0000", 2024-09-21T22:08:10.409 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00077930800000000004 2024-09-21T22:08:10.409 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.410 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.410 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:10.410 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.151916+0000", 2024-09-21T22:08:10.410 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1921999999999993e-05 2024-09-21T22:08:10.410 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.410 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.410 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:10.410 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.151965+0000", 2024-09-21T22:08:10.410 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8724000000000003e-05 2024-09-21T22:08:10.410 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:10.410 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:10.410 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:10.410 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.410 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.411 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4840.0:696 2.5 2:b1d55633:::rbd_data.126483d75f2d.000000000000000b:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:08:10.411 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:25.218641+0000", 2024-09-21T22:08:10.411 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 45.163599069, 2024-09-21T22:08:10.411 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058363669999999999, 2024-09-21T22:08:10.411 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:10.411 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:10.411 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:10.411 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:10.411 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4840", 2024-09-21T22:08:10.411 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:08:10.411 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 696 2024-09-21T22:08:10.411 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.411 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:10.412 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.412 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:10.412 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.218641+0000", 2024-09-21T22:08:10.412 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:10.412 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.412 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.412 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:10.412 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.218641+0000", 2024-09-21T22:08:10.412 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:10.412 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.412 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.412 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:10.412 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.218642+0000", 2024-09-21T22:08:10.413 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.266e-06 2024-09-21T22:08:10.413 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.413 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.413 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:10.413 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.219432+0000", 2024-09-21T22:08:10.413 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00078986300000000002 2024-09-21T22:08:10.413 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.413 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.413 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:10.414 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.219450+0000", 2024-09-21T22:08:10.414 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7526000000000001e-05 2024-09-21T22:08:10.414 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.414 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.414 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:10.414 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.219459+0000", 2024-09-21T22:08:10.414 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3000000000000007e-06 2024-09-21T22:08:10.414 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.414 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.414 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:10.414 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.219655+0000", 2024-09-21T22:08:10.415 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019599099999999999 2024-09-21T22:08:10.415 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.415 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.415 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:10.415 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.219917+0000", 2024-09-21T22:08:10.415 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026198200000000001 2024-09-21T22:08:10.415 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.415 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.415 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:08:10.415 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.220198+0000", 2024-09-21T22:08:10.415 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00028154500000000002 2024-09-21T22:08:10.415 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.415 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.415 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:10.416 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.268856+0000", 2024-09-21T22:08:10.416 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.048657687999999998 2024-09-21T22:08:10.416 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.416 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.416 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:10.416 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.276903+0000", 2024-09-21T22:08:10.416 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0080472249999999999 2024-09-21T22:08:10.416 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.416 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.416 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:10.416 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.276958+0000", 2024-09-21T22:08:10.417 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4184999999999997e-05 2024-09-21T22:08:10.417 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.417 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.417 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:10.417 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.277005+0000", 2024-09-21T22:08:10.417 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7098999999999997e-05 2024-09-21T22:08:10.417 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:10.417 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:10.417 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:10.417 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.417 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.417 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4840.0:729 2.0 2:08b4f96a:::rbd_data.126483d75f2d.000000000000000f:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:08:10.417 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:26.062012+0000", 2024-09-21T22:08:10.417 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 44.320227836000001, 2024-09-21T22:08:10.417 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.060656823999999998, 2024-09-21T22:08:10.418 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:10.418 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:10.418 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:10.418 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:10.418 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4840", 2024-09-21T22:08:10.418 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:08:10.418 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 729 2024-09-21T22:08:10.418 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.418 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:10.418 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.418 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:10.418 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.062012+0000", 2024-09-21T22:08:10.418 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:10.418 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.419 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.419 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:10.419 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.062012+0000", 2024-09-21T22:08:10.419 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:10.419 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.419 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.419 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:10.419 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.062013+0000", 2024-09-21T22:08:10.419 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.243e-06 2024-09-21T22:08:10.419 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.419 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.419 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:10.419 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.062827+0000", 2024-09-21T22:08:10.419 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00081374799999999999 2024-09-21T22:08:10.420 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.420 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.420 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:10.420 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.062845+0000", 2024-09-21T22:08:10.420 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.808e-05 2024-09-21T22:08:10.420 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.420 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.420 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:10.420 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.062854+0000", 2024-09-21T22:08:10.420 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1039999999999996e-06 2024-09-21T22:08:10.420 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.420 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.420 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:10.420 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.062970+0000", 2024-09-21T22:08:10.421 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011601 2024-09-21T22:08:10.421 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.421 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.421 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:10.421 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.063230+0000", 2024-09-21T22:08:10.421 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025918799999999999 2024-09-21T22:08:10.421 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.421 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.421 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-09-21T22:08:10.421 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.063641+0000", 2024-09-21T22:08:10.421 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00041152799999999998 2024-09-21T22:08:10.421 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.421 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.421 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:10.422 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.122015+0000", 2024-09-21T22:08:10.422 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058374246999999997 2024-09-21T22:08:10.422 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.422 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.422 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:10.422 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.122451+0000", 2024-09-21T22:08:10.422 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000435771 2024-09-21T22:08:10.422 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.422 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.422 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:10.422 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.122576+0000", 2024-09-21T22:08:10.422 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012469099999999999 2024-09-21T22:08:10.422 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.422 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.423 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:10.423 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.122669+0000", 2024-09-21T22:08:10.423 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3214000000000006e-05 2024-09-21T22:08:10.423 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:10.423 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:10.423 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:10.423 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.423 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.423 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4840.0:731 2.0 2:08b4f96a:::rbd_data.126483d75f2d.000000000000000f:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:08:10.423 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:26.135603+0000", 2024-09-21T22:08:10.423 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 44.246637335999999, 2024-09-21T22:08:10.423 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.056838474999999999, 2024-09-21T22:08:10.423 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:10.423 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:10.424 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:10.424 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:10.424 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4840", 2024-09-21T22:08:10.424 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:08:10.424 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 731 2024-09-21T22:08:10.424 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.424 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:10.424 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.424 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:10.424 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.135603+0000", 2024-09-21T22:08:10.424 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:10.424 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.424 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.424 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:10.425 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.135603+0000", 2024-09-21T22:08:10.425 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:10.425 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.425 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.425 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:10.425 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.135605+0000", 2024-09-21T22:08:10.425 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9700000000000002e-06 2024-09-21T22:08:10.425 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.425 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.425 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:10.425 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.136554+0000", 2024-09-21T22:08:10.425 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00094886700000000005 2024-09-21T22:08:10.425 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.425 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.425 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:10.426 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.136582+0000", 2024-09-21T22:08:10.426 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8393000000000001e-05 2024-09-21T22:08:10.426 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.426 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.426 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:10.426 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.136596+0000", 2024-09-21T22:08:10.426 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.43e-05 2024-09-21T22:08:10.426 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.426 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.426 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:10.426 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.136691+0000", 2024-09-21T22:08:10.426 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4843e-05 2024-09-21T22:08:10.426 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.426 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.427 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:10.427 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.136873+0000", 2024-09-21T22:08:10.427 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018237099999999999 2024-09-21T22:08:10.427 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.427 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.427 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-09-21T22:08:10.427 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.137140+0000", 2024-09-21T22:08:10.427 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026642700000000002 2024-09-21T22:08:10.427 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.427 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.427 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:10.427 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.180580+0000", 2024-09-21T22:08:10.427 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.043440078 2024-09-21T22:08:10.428 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.428 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.428 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:10.428 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.192175+0000", 2024-09-21T22:08:10.428 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011594959 2024-09-21T22:08:10.428 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.428 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.428 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:10.428 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.192353+0000", 2024-09-21T22:08:10.428 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017850200000000001 2024-09-21T22:08:10.428 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.428 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.428 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:10.428 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.192441+0000", 2024-09-21T22:08:10.428 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7764999999999997e-05 2024-09-21T22:08:10.429 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:10.429 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:10.429 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:10.429 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.429 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.429 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5207.0:34 2.7 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head [write 262144~131072] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e78)", 2024-09-21T22:08:10.429 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:52.587708+0000", 2024-09-21T22:08:10.429 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 17.794532315000001, 2024-09-21T22:08:10.429 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.056690213000000003, 2024-09-21T22:08:10.429 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:10.429 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:10.429 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:10.429 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:10.429 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5207", 2024-09-21T22:08:10.429 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/357731919", 2024-09-21T22:08:10.430 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 34 2024-09-21T22:08:10.430 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.430 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:10.430 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.430 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:10.430 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.587708+0000", 2024-09-21T22:08:10.430 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:10.430 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.430 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.430 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:10.430 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.587708+0000", 2024-09-21T22:08:10.430 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:10.430 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.431 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.431 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:10.431 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.587709+0000", 2024-09-21T22:08:10.431 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.72e-06 2024-09-21T22:08:10.431 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.431 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.431 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:10.431 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588017+0000", 2024-09-21T22:08:10.431 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00030767499999999998 2024-09-21T22:08:10.431 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.431 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.431 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:10.431 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588037+0000", 2024-09-21T22:08:10.431 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9865999999999999e-05 2024-09-21T22:08:10.431 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.432 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.432 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:10.432 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588048+0000", 2024-09-21T22:08:10.432 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1384e-05 2024-09-21T22:08:10.432 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.432 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.432 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:10.432 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.614263+0000", 2024-09-21T22:08:10.432 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026214232000000001 2024-09-21T22:08:10.432 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.432 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.432 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:10.432 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.614397+0000", 2024-09-21T22:08:10.432 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013396299999999999 2024-09-21T22:08:10.433 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.433 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.433 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-09-21T22:08:10.433 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.614670+0000", 2024-09-21T22:08:10.433 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027295500000000002 2024-09-21T22:08:10.433 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.433 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.433 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:10.433 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.644046+0000", 2024-09-21T22:08:10.433 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029376973000000001 2024-09-21T22:08:10.433 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.433 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.433 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:10.433 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.644208+0000", 2024-09-21T22:08:10.433 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016169900000000001 2024-09-21T22:08:10.434 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.434 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.434 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:10.434 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.644335+0000", 2024-09-21T22:08:10.434 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012656000000000001 2024-09-21T22:08:10.434 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.434 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.434 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:10.434 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.644398+0000", 2024-09-21T22:08:10.434 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3186000000000001e-05 2024-09-21T22:08:10.434 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:10.434 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:10.434 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:10.434 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.435 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.435 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5207.0:35 2.7 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head [write 393216~131072] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e78)", 2024-09-21T22:08:10.435 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:52.588070+0000", 2024-09-21T22:08:10.435 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 17.794170417, 2024-09-21T22:08:10.435 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.076411232999999995, 2024-09-21T22:08:10.435 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:10.435 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:10.435 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:10.435 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:10.435 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5207", 2024-09-21T22:08:10.435 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/357731919", 2024-09-21T22:08:10.435 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 35 2024-09-21T22:08:10.436 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.436 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:10.436 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.436 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:10.436 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588070+0000", 2024-09-21T22:08:10.436 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:10.436 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.436 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.436 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:10.437 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588070+0000", 2024-09-21T22:08:10.437 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:10.437 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.437 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.437 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:10.437 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588071+0000", 2024-09-21T22:08:10.437 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1299999999999998e-07 2024-09-21T22:08:10.437 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.437 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.437 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:10.437 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588327+0000", 2024-09-21T22:08:10.437 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025630199999999997 2024-09-21T22:08:10.437 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.438 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.438 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:10.438 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588343+0000", 2024-09-21T22:08:10.438 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6140000000000001e-05 2024-09-21T22:08:10.438 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.438 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.438 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:10.438 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588353+0000", 2024-09-21T22:08:10.438 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0043e-05 2024-09-21T22:08:10.438 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.438 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.438 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:10.438 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.623978+0000", 2024-09-21T22:08:10.438 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.035625281000000002 2024-09-21T22:08:10.439 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.439 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.439 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:10.439 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.624104+0000", 2024-09-21T22:08:10.439 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012591300000000001 2024-09-21T22:08:10.439 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.439 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.439 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-09-21T22:08:10.439 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.624296+0000", 2024-09-21T22:08:10.439 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019226200000000001 2024-09-21T22:08:10.439 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.440 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.440 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:10.440 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.654488+0000", 2024-09-21T22:08:10.440 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.030191551 2024-09-21T22:08:10.440 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.440 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.440 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:10.440 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.664329+0000", 2024-09-21T22:08:10.440 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0098405119999999992 2024-09-21T22:08:10.440 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.440 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.441 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:10.441 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.664414+0000", 2024-09-21T22:08:10.441 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5385999999999999e-05 2024-09-21T22:08:10.441 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.441 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.441 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:10.441 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.664481+0000", 2024-09-21T22:08:10.441 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6929999999999998e-05 2024-09-21T22:08:10.441 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:10.441 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:10.442 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:10.442 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.442 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.442 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5207.0:36 2.7 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head [write 524288~131072] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e78)", 2024-09-21T22:08:10.442 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:52.588370+0000", 2024-09-21T22:08:10.442 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 17.793869639, 2024-09-21T22:08:10.442 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.076552001999999994, 2024-09-21T22:08:10.442 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:10.442 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:10.442 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:10.443 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:10.443 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5207", 2024-09-21T22:08:10.443 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/357731919", 2024-09-21T22:08:10.443 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 36 2024-09-21T22:08:10.443 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.443 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:10.443 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.443 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:10.443 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588370+0000", 2024-09-21T22:08:10.443 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:10.443 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.443 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.444 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:10.444 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588370+0000", 2024-09-21T22:08:10.444 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:10.444 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.444 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.444 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:10.444 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588372+0000", 2024-09-21T22:08:10.444 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.153e-06 2024-09-21T22:08:10.444 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.444 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.444 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:10.444 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588716+0000", 2024-09-21T22:08:10.444 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00034479200000000002 2024-09-21T22:08:10.444 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.445 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.445 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:10.445 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588752+0000", 2024-09-21T22:08:10.445 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5865999999999998e-05 2024-09-21T22:08:10.445 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.445 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.445 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:10.445 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588777+0000", 2024-09-21T22:08:10.445 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4583000000000002e-05 2024-09-21T22:08:10.445 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.445 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.445 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:10.445 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.633886+0000", 2024-09-21T22:08:10.445 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.045108941999999999 2024-09-21T22:08:10.446 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.446 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.446 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:10.446 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.634059+0000", 2024-09-21T22:08:10.446 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000173046 2024-09-21T22:08:10.446 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.446 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.446 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-09-21T22:08:10.446 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.634314+0000", 2024-09-21T22:08:10.446 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025482400000000001 2024-09-21T22:08:10.446 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.446 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.446 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:10.446 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.664541+0000", 2024-09-21T22:08:10.447 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.030227222000000002 2024-09-21T22:08:10.447 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.447 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.447 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:10.447 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.664653+0000", 2024-09-21T22:08:10.447 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011192900000000001 2024-09-21T22:08:10.447 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.447 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.447 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:10.447 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.664798+0000", 2024-09-21T22:08:10.447 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014499900000000001 2024-09-21T22:08:10.447 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.447 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.447 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:10.448 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.664922+0000", 2024-09-21T22:08:10.448 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012464600000000001 2024-09-21T22:08:10.448 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:10.448 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:10.448 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:10.448 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.448 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.448 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5207.0:37 2.7 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head [write 655360~131072] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e78)", 2024-09-21T22:08:10.448 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:52.588813+0000", 2024-09-21T22:08:10.448 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 17.793426624999999, 2024-09-21T22:08:10.448 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.087074796999999995, 2024-09-21T22:08:10.448 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:10.448 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:10.448 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:10.448 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:10.449 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5207", 2024-09-21T22:08:10.449 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/357731919", 2024-09-21T22:08:10.449 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 37 2024-09-21T22:08:10.449 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.449 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:10.449 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.449 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:10.449 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588813+0000", 2024-09-21T22:08:10.449 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:10.449 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.449 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.449 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:10.449 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588813+0000", 2024-09-21T22:08:10.450 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:10.450 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.450 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.450 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:10.450 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588815+0000", 2024-09-21T22:08:10.450 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.643e-06 2024-09-21T22:08:10.450 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.450 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.450 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:10.450 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.589071+0000", 2024-09-21T22:08:10.450 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025580200000000002 2024-09-21T22:08:10.450 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.450 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.450 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:10.451 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.589098+0000", 2024-09-21T22:08:10.451 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7336000000000001e-05 2024-09-21T22:08:10.451 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.451 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.451 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:10.451 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.589121+0000", 2024-09-21T22:08:10.451 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2373000000000001e-05 2024-09-21T22:08:10.451 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.451 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.451 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:10.451 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.644491+0000", 2024-09-21T22:08:10.451 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.055369994999999998 2024-09-21T22:08:10.451 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.451 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.452 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:10.452 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.644707+0000", 2024-09-21T22:08:10.452 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000216488 2024-09-21T22:08:10.452 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.452 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.452 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-09-21T22:08:10.452 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.644908+0000", 2024-09-21T22:08:10.452 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000201196 2024-09-21T22:08:10.452 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.452 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.452 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:10.452 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.675419+0000", 2024-09-21T22:08:10.452 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.03051071 2024-09-21T22:08:10.452 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.453 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.453 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:10.453 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.675591+0000", 2024-09-21T22:08:10.453 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000172545 2024-09-21T22:08:10.453 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.453 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.453 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:10.453 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.675761+0000", 2024-09-21T22:08:10.453 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016942300000000001 2024-09-21T22:08:10.453 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.453 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.453 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:10.454 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.675888+0000", 2024-09-21T22:08:10.454 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000127286 2024-09-21T22:08:10.454 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:10.454 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:10.454 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:10.454 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.454 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.454 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5207.0:38 2.7 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head [write 786432~131072] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e78)", 2024-09-21T22:08:10.454 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:52.589156+0000", 2024-09-21T22:08:10.454 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 17.793083720999999, 2024-09-21T22:08:10.454 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.087126880000000004, 2024-09-21T22:08:10.454 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:10.455 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:10.455 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:10.455 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:10.455 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5207", 2024-09-21T22:08:10.455 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/357731919", 2024-09-21T22:08:10.455 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 38 2024-09-21T22:08:10.455 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.456 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:10.456 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.456 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:10.456 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.589156+0000", 2024-09-21T22:08:10.456 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:10.456 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.456 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.456 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:10.456 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.589156+0000", 2024-09-21T22:08:10.456 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:10.456 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.456 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.456 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:10.456 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.589158+0000", 2024-09-21T22:08:10.457 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.717e-06 2024-09-21T22:08:10.457 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.457 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.457 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:10.457 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.589417+0000", 2024-09-21T22:08:10.457 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025860500000000002 2024-09-21T22:08:10.457 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.457 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.457 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:10.457 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.589447+0000", 2024-09-21T22:08:10.457 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9916e-05 2024-09-21T22:08:10.457 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.457 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.457 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:10.458 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.589467+0000", 2024-09-21T22:08:10.458 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0894e-05 2024-09-21T22:08:10.458 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.458 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.458 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:10.458 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.654576+0000", 2024-09-21T22:08:10.458 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.065108579999999999 2024-09-21T22:08:10.458 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.458 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.458 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:10.458 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.654694+0000", 2024-09-21T22:08:10.458 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000118083 2024-09-21T22:08:10.458 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.459 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.459 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-09-21T22:08:10.459 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.654880+0000", 2024-09-21T22:08:10.459 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000186069 2024-09-21T22:08:10.459 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.459 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.459 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:10.459 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.675453+0000", 2024-09-21T22:08:10.459 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020572498000000002 2024-09-21T22:08:10.459 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.459 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.459 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:10.459 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.676055+0000", 2024-09-21T22:08:10.459 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00060247000000000002 2024-09-21T22:08:10.459 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.460 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.460 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:10.460 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.676178+0000", 2024-09-21T22:08:10.460 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000122395 2024-09-21T22:08:10.460 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.460 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.460 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:10.460 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.676283+0000", 2024-09-21T22:08:10.460 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000105653 2024-09-21T22:08:10.461 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:10.461 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:10.461 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:10.461 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.461 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.461 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5207.0:39 2.7 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head [write 917504~131072] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e78)", 2024-09-21T22:08:10.461 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:52.589499+0000", 2024-09-21T22:08:10.461 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 17.792741113000002, 2024-09-21T22:08:10.461 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.087365371999999997, 2024-09-21T22:08:10.461 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:10.461 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:10.461 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:10.462 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:10.462 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5207", 2024-09-21T22:08:10.462 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/357731919", 2024-09-21T22:08:10.462 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 39 2024-09-21T22:08:10.462 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.462 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:10.462 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.462 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:10.462 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.589499+0000", 2024-09-21T22:08:10.462 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:10.462 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.462 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.462 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:10.462 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.589499+0000", 2024-09-21T22:08:10.463 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:10.463 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.463 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.463 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:10.463 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.589500+0000", 2024-09-21T22:08:10.463 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.547e-06 2024-09-21T22:08:10.463 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.463 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.463 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:10.463 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.589752+0000", 2024-09-21T22:08:10.463 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025112799999999998 2024-09-21T22:08:10.463 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.463 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.463 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:10.464 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.589783+0000", 2024-09-21T22:08:10.464 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1557000000000003e-05 2024-09-21T22:08:10.464 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.464 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.464 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:10.464 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.589832+0000", 2024-09-21T22:08:10.464 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9175999999999998e-05 2024-09-21T22:08:10.464 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.464 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.464 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:10.464 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.665028+0000", 2024-09-21T22:08:10.464 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.075195293999999996 2024-09-21T22:08:10.464 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.464 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.465 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:10.465 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.665169+0000", 2024-09-21T22:08:10.465 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000141299 2024-09-21T22:08:10.465 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.465 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.465 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-09-21T22:08:10.465 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.665464+0000", 2024-09-21T22:08:10.465 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00029459899999999999 2024-09-21T22:08:10.465 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.465 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.465 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:10.465 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.676551+0000", 2024-09-21T22:08:10.465 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011087107000000001 2024-09-21T22:08:10.465 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.466 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.466 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:10.466 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.676700+0000", 2024-09-21T22:08:10.466 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000149853 2024-09-21T22:08:10.466 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.466 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.466 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:10.466 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.676802+0000", 2024-09-21T22:08:10.466 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010194199999999999 2024-09-21T22:08:10.466 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.466 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.466 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:10.466 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.676864+0000", 2024-09-21T22:08:10.467 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1870000000000002e-05 2024-09-21T22:08:10.467 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:10.467 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:10.467 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:10.467 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.467 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.467 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5207.0:74 2.7 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head [write 524288~131072] snapc 5=[5,3] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-21T22:08:10.467 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:52.938124+0000", 2024-09-21T22:08:10.467 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 17.444115901, 2024-09-21T22:08:10.467 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.071435100000000001, 2024-09-21T22:08:10.467 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:10.467 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:10.467 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:10.467 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:10.467 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5207", 2024-09-21T22:08:10.468 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/357731919", 2024-09-21T22:08:10.468 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 74 2024-09-21T22:08:10.468 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.468 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:10.468 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.468 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:10.468 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938124+0000", 2024-09-21T22:08:10.468 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:10.468 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.468 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.468 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:10.469 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938124+0000", 2024-09-21T22:08:10.469 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:10.469 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.469 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.469 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:10.469 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938125+0000", 2024-09-21T22:08:10.469 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5600000000000005e-07 2024-09-21T22:08:10.469 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.469 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.469 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:10.470 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938218+0000", 2024-09-21T22:08:10.470 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3239999999999995e-05 2024-09-21T22:08:10.470 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.470 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.470 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:10.470 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938231+0000", 2024-09-21T22:08:10.470 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.272e-05 2024-09-21T22:08:10.470 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.470 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.470 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:10.470 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938239+0000", 2024-09-21T22:08:10.470 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5059999999999995e-06 2024-09-21T22:08:10.470 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.470 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.470 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:10.471 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.983312+0000", 2024-09-21T22:08:10.471 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.045072252 2024-09-21T22:08:10.471 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.471 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.471 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:10.471 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.983458+0000", 2024-09-21T22:08:10.471 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014628899999999999 2024-09-21T22:08:10.471 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.471 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.471 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-09-21T22:08:10.471 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.983682+0000", 2024-09-21T22:08:10.471 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022402899999999999 2024-09-21T22:08:10.471 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.471 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.472 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:10.472 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.993679+0000", 2024-09-21T22:08:10.472 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0099972740000000004 2024-09-21T22:08:10.472 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.472 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.472 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:10.472 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.009436+0000", 2024-09-21T22:08:10.472 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015756867000000001 2024-09-21T22:08:10.472 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.472 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.472 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:10.473 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.009506+0000", 2024-09-21T22:08:10.473 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0277999999999994e-05 2024-09-21T22:08:10.473 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.473 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.473 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:10.473 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.009559+0000", 2024-09-21T22:08:10.473 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2889000000000001e-05 2024-09-21T22:08:10.473 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:10.473 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:10.473 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:10.473 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.473 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.474 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5207.0:75 2.7 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head [write 655360~131072] snapc 5=[5,3] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-21T22:08:10.474 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:52.938253+0000", 2024-09-21T22:08:10.474 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 17.443986845000001, 2024-09-21T22:08:10.474 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082064041000000004, 2024-09-21T22:08:10.474 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:10.474 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:10.474 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:10.474 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:10.474 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5207", 2024-09-21T22:08:10.474 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/357731919", 2024-09-21T22:08:10.474 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 75 2024-09-21T22:08:10.475 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.475 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:10.475 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.475 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:10.475 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938253+0000", 2024-09-21T22:08:10.475 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:10.475 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.475 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.475 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:10.475 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938253+0000", 2024-09-21T22:08:10.475 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:10.475 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.475 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.475 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:10.476 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938254+0000", 2024-09-21T22:08:10.476 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8599999999999998e-07 2024-09-21T22:08:10.476 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.476 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.476 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:10.476 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938354+0000", 2024-09-21T22:08:10.476 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9696000000000002e-05 2024-09-21T22:08:10.476 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.476 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.476 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:10.476 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938381+0000", 2024-09-21T22:08:10.476 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.705e-05 2024-09-21T22:08:10.476 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.476 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.476 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:10.477 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938397+0000", 2024-09-21T22:08:10.477 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5996999999999999e-05 2024-09-21T22:08:10.477 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.477 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.477 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:10.477 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.993739+0000", 2024-09-21T22:08:10.477 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.055342823999999999 2024-09-21T22:08:10.477 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.477 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.477 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:10.477 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.993835+0000", 2024-09-21T22:08:10.477 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5919000000000001e-05 2024-09-21T22:08:10.478 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.478 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.478 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-09-21T22:08:10.478 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.993993+0000", 2024-09-21T22:08:10.478 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015729900000000001 2024-09-21T22:08:10.478 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.478 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.478 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:10.478 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.009579+0000", 2024-09-21T22:08:10.478 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015586723 2024-09-21T22:08:10.478 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.478 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.479 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:10.479 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.020134+0000", 2024-09-21T22:08:10.479 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010554908 2024-09-21T22:08:10.479 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.479 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.479 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:10.479 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.020241+0000", 2024-09-21T22:08:10.479 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000106246 2024-09-21T22:08:10.479 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.479 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.479 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:10.479 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.020317+0000", 2024-09-21T22:08:10.479 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6693000000000006e-05 2024-09-21T22:08:10.479 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:10.479 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:10.480 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:10.480 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.480 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.480 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5207.0:76 2.7 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head [write 786432~131072] snapc 5=[5,3] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-21T22:08:10.480 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:52.938410+0000", 2024-09-21T22:08:10.480 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 17.443829676, 2024-09-21T22:08:10.480 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.086926370000000003, 2024-09-21T22:08:10.480 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:10.480 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:10.480 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:10.480 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:10.480 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5207", 2024-09-21T22:08:10.480 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/357731919", 2024-09-21T22:08:10.481 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 76 2024-09-21T22:08:10.481 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.481 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:10.481 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.481 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:10.481 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938410+0000", 2024-09-21T22:08:10.481 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:10.481 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.481 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.481 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:10.481 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938410+0000", 2024-09-21T22:08:10.481 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:10.482 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.482 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.482 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:10.482 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938411+0000", 2024-09-21T22:08:10.482 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0299999999999998e-07 2024-09-21T22:08:10.482 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.482 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.482 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:10.482 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938502+0000", 2024-09-21T22:08:10.482 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0562000000000005e-05 2024-09-21T22:08:10.482 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.482 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.482 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:10.482 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938514+0000", 2024-09-21T22:08:10.482 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2327000000000001e-05 2024-09-21T22:08:10.483 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.483 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.483 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:10.483 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938522+0000", 2024-09-21T22:08:10.483 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1170000000000004e-06 2024-09-21T22:08:10.483 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.483 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.483 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:10.483 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.001427+0000", 2024-09-21T22:08:10.483 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.062904871000000001 2024-09-21T22:08:10.483 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.483 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.483 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:10.483 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.001519+0000", 2024-09-21T22:08:10.484 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1798000000000004e-05 2024-09-21T22:08:10.484 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.484 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.484 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-09-21T22:08:10.484 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.001671+0000", 2024-09-21T22:08:10.484 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015195900000000001 2024-09-21T22:08:10.484 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.484 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.484 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:10.484 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.020388+0000", 2024-09-21T22:08:10.484 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018717428000000001 2024-09-21T22:08:10.484 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.484 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.484 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:10.485 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.025041+0000", 2024-09-21T22:08:10.485 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0046524039999999997 2024-09-21T22:08:10.485 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.485 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.485 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:10.485 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.025213+0000", 2024-09-21T22:08:10.485 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017286499999999999 2024-09-21T22:08:10.485 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.485 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.485 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:10.485 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.025337+0000", 2024-09-21T22:08:10.485 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012333600000000001 2024-09-21T22:08:10.485 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:10.485 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:10.485 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:10.486 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.486 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.486 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5207.0:77 2.7 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head [write 917504~131072] snapc 5=[5,3] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-21T22:08:10.486 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:52.938534+0000", 2024-09-21T22:08:10.486 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 17.443705861000002, 2024-09-21T22:08:10.486 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.087709436000000002, 2024-09-21T22:08:10.486 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:10.486 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:10.486 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:10.486 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:10.486 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5207", 2024-09-21T22:08:10.486 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/357731919", 2024-09-21T22:08:10.486 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 77 2024-09-21T22:08:10.486 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.487 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:10.487 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.487 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:10.487 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938534+0000", 2024-09-21T22:08:10.487 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:10.487 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.487 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.487 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:10.487 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938534+0000", 2024-09-21T22:08:10.487 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:10.487 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.487 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.487 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:10.487 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938535+0000", 2024-09-21T22:08:10.488 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1399999999999996e-07 2024-09-21T22:08:10.488 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.488 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.488 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:10.488 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938623+0000", 2024-09-21T22:08:10.488 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8226000000000005e-05 2024-09-21T22:08:10.488 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.488 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.488 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:10.488 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938645+0000", 2024-09-21T22:08:10.488 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1948999999999999e-05 2024-09-21T22:08:10.488 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.488 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.488 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:10.488 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938660+0000", 2024-09-21T22:08:10.489 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4776999999999999e-05 2024-09-21T22:08:10.489 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.489 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.489 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:10.489 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.009669+0000", 2024-09-21T22:08:10.489 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.071009116999999997 2024-09-21T22:08:10.489 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.489 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.489 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:10.489 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.009949+0000", 2024-09-21T22:08:10.489 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027962899999999999 2024-09-21T22:08:10.489 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.489 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.490 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-09-21T22:08:10.490 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.010355+0000", 2024-09-21T22:08:10.490 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000406334 2024-09-21T22:08:10.490 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.490 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.490 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:10.490 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.020875+0000", 2024-09-21T22:08:10.490 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010520144 2024-09-21T22:08:10.490 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.490 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.490 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:10.490 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.026086+0000", 2024-09-21T22:08:10.490 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0052107569999999999 2024-09-21T22:08:10.490 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.490 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.491 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:10.491 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.026178+0000", 2024-09-21T22:08:10.491 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2113000000000004e-05 2024-09-21T22:08:10.491 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.491 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.491 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:10.491 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.026244+0000", 2024-09-21T22:08:10.491 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5675999999999999e-05 2024-09-21T22:08:10.491 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:10.491 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:10.491 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:10.491 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.491 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.491 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5207.0:93 2.7 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head [write 131072~917504] snapc 5=[5,3] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-21T22:08:10.492 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:53.147757+0000", 2024-09-21T22:08:10.492 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 17.234482683, 2024-09-21T22:08:10.492 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.062493043999999998, 2024-09-21T22:08:10.492 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:10.492 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:10.492 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:10.492 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:10.492 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5207", 2024-09-21T22:08:10.492 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/357731919", 2024-09-21T22:08:10.492 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 93 2024-09-21T22:08:10.492 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.492 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:10.492 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.492 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:10.492 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.147757+0000", 2024-09-21T22:08:10.493 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:10.493 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.493 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.493 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:10.493 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.147757+0000", 2024-09-21T22:08:10.493 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:10.493 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.493 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.493 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:10.493 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.147759+0000", 2024-09-21T22:08:10.493 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0159999999999998e-06 2024-09-21T22:08:10.493 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.493 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.493 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:10.494 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.149009+0000", 2024-09-21T22:08:10.494 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001250082 2024-09-21T22:08:10.494 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.494 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.494 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:10.494 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.149042+0000", 2024-09-21T22:08:10.494 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2617000000000003e-05 2024-09-21T22:08:10.494 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.494 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.494 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:10.494 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.149058+0000", 2024-09-21T22:08:10.494 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6113e-05 2024-09-21T22:08:10.494 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.494 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.494 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:10.495 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.149237+0000", 2024-09-21T22:08:10.495 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017834900000000001 2024-09-21T22:08:10.495 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.495 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.495 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:10.495 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.149378+0000", 2024-09-21T22:08:10.495 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014099199999999999 2024-09-21T22:08:10.495 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.495 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.495 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-09-21T22:08:10.495 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.149582+0000", 2024-09-21T22:08:10.495 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020414900000000001 2024-09-21T22:08:10.495 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.495 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.495 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:10.496 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.208648+0000", 2024-09-21T22:08:10.496 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.059066397999999999 2024-09-21T22:08:10.496 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.496 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.496 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:10.496 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.210125+0000", 2024-09-21T22:08:10.496 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001477119 2024-09-21T22:08:10.496 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.496 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.496 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:10.496 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.210200+0000", 2024-09-21T22:08:10.496 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4973000000000002e-05 2024-09-21T22:08:10.496 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.496 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.496 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:10.497 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.210250+0000", 2024-09-21T22:08:10.497 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0235999999999998e-05 2024-09-21T22:08:10.497 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:10.497 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:10.497 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:10.497 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.497 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.497 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5207.0:128 2.7 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head [write 131072~917504] snapc 6=[6,5,3] ondisk+write+known_if_redirected+supports_pool_eio e83)", 2024-09-21T22:08:10.497 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:08:01.056777+0000", 2024-09-21T22:08:10.497 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.3254632130000008, 2024-09-21T22:08:10.497 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.062294929999999998, 2024-09-21T22:08:10.497 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:10.497 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:10.497 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:10.498 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:10.498 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5207", 2024-09-21T22:08:10.498 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/357731919", 2024-09-21T22:08:10.498 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 128 2024-09-21T22:08:10.498 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.498 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:10.498 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.498 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:10.498 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.056777+0000", 2024-09-21T22:08:10.498 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:10.498 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.498 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.498 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:10.499 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.056777+0000", 2024-09-21T22:08:10.499 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:10.499 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.499 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.499 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:10.499 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.056779+0000", 2024-09-21T22:08:10.499 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4669999999999998e-06 2024-09-21T22:08:10.499 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.499 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.499 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:10.499 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.058094+0000", 2024-09-21T22:08:10.499 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001314602 2024-09-21T22:08:10.499 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.499 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.499 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:10.500 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.058126+0000", 2024-09-21T22:08:10.500 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1625999999999998e-05 2024-09-21T22:08:10.500 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.500 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.500 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:10.500 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.058141+0000", 2024-09-21T22:08:10.500 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5274e-05 2024-09-21T22:08:10.500 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.500 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.500 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:10.500 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.058267+0000", 2024-09-21T22:08:10.500 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000126682 2024-09-21T22:08:10.500 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.500 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.501 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:10.501 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.058420+0000", 2024-09-21T22:08:10.501 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015212599999999999 2024-09-21T22:08:10.501 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.501 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.501 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-09-21T22:08:10.501 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.058644+0000", 2024-09-21T22:08:10.501 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000224749 2024-09-21T22:08:10.501 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.501 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.501 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:10.501 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.117883+0000", 2024-09-21T22:08:10.501 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.059238617 2024-09-21T22:08:10.501 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.502 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.502 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:10.502 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.118942+0000", 2024-09-21T22:08:10.502 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0010588780000000001 2024-09-21T22:08:10.502 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.502 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.502 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:10.502 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.119011+0000", 2024-09-21T22:08:10.502 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8802999999999998e-05 2024-09-21T22:08:10.502 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.502 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.502 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:10.502 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.119072+0000", 2024-09-21T22:08:10.502 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1105999999999994e-05 2024-09-21T22:08:10.502 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:10.503 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:10.503 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:10.503 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.503 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.503 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.3 87/86 [PushOp(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head, version: 71'29, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head@71'29, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T22:08:10.503 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:08:07.107088+0000", 2024-09-21T22:08:10.503 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.2751524710000002, 2024-09-21T22:08:10.503 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.070273375999999999, 2024-09-21T22:08:10.503 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:10.503 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:10.503 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:08:10.503 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:10.503 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.503 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:10.503 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.107088+0000", 2024-09-21T22:08:10.504 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:10.504 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.504 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.504 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:10.504 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.107088+0000", 2024-09-21T22:08:10.504 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:10.504 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.504 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.504 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:10.504 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.107089+0000", 2024-09-21T22:08:10.504 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6559999999999999e-06 2024-09-21T22:08:10.504 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.504 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.504 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:10.505 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.108876+0000", 2024-09-21T22:08:10.505 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001786489 2024-09-21T22:08:10.505 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.505 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.505 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:10.505 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.108925+0000", 2024-09-21T22:08:10.505 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8843e-05 2024-09-21T22:08:10.505 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.505 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.505 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:10.505 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.108979+0000", 2024-09-21T22:08:10.505 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4945999999999999e-05 2024-09-21T22:08:10.505 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.505 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.505 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:10.506 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.109230+0000", 2024-09-21T22:08:10.506 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025086599999999998 2024-09-21T22:08:10.506 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.506 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.506 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:10.506 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.109269+0000", 2024-09-21T22:08:10.506 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8216000000000001e-05 2024-09-21T22:08:10.506 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.506 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.506 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:10.506 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.177361+0000", 2024-09-21T22:08:10.506 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.068092360000000005 2024-09-21T22:08:10.506 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:10.507 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:10.507 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:10.507 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:10.507 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:10.507 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:08:10.508 INFO:tasks.workunit.client.0.smithi157.stderr:+ return 0 2024-09-21T22:08:10.508 INFO:tasks.workunit.client.0.smithi157.stderr:+ sleep 0.5 2024-09-21T22:08:10.509 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T22:08:10.509 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:08:10.504+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:08:10.609 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:08:10.605+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:08:10.683 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:08:10.683 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:08:10.683 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-09-21T22:08:10.684 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:08:10.698 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T22:08:10.742 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:08:10.740+0000 7f2075a06640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31720) UID: 0 2024-09-21T22:08:10.811 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:08:10.808+0000 7f2075a06640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31720) UID: 0 2024-09-21T22:08:10.817 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:08:10.817 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:08:10.817 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-21T22:08:10.817 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-21T22:08:10.817 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:08:10.832 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T22:08:10.889 INFO:tasks.workunit.client.0.smithi157.stderr:+ DEV= 2024-09-21T22:08:10.889 INFO:tasks.workunit.client.0.smithi157.stderr:+ cat /tmp/tmp.GqKuv7rsZi/rbd-nbd.log 2024-09-21T22:08:10.890 INFO:tasks.workunit.client.0.smithi157.stdout:2024-09-21T22:08:08.697+0000 7f28bf60ba80 0 ceph version 19.1.1-373-g4e45e6b4 (4e45e6b490e6cbe082cdf155fea780438ec53fc7) squid (stable), process rbd-nbd, pid 23844 2024-09-21T22:08:10.890 INFO:tasks.workunit.client.0.smithi157.stdout:2024-09-21T22:08:08.701+0000 7f28bf60ba80 0 pidfile_write: ignore empty --pid-file 2024-09-21T22:08:10.890 INFO:tasks.workunit.client.0.smithi157.stdout:2024-09-21T22:08:08.997+0000 7f28bd8fe640 0 -- 172.21.15.157:0/3099046058 >> [v2:172.21.15.64:6800/1132198485,v1:172.21.15.64:6801/1132198485] conn(0x7f28a003b800 msgr2=0x7f28a003dbf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-21T22:08:10.891 INFO:tasks.workunit.client.0.smithi157.stdout:2024-09-21T22:08:09.029+0000 7f28bd8fe640 0 -- 172.21.15.157:0/3099046058 >> [v2:172.21.15.64:6800/1132198485,v1:172.21.15.64:6801/1132198485] conn(0x55c9a82ff8b0 msgr2=0x7f28a003e210 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-21T22:08:10.891 INFO:tasks.workunit.client.0.smithi157.stdout:2024-09-21T22:08:10.284+0000 7f289dffb640 0 rbd-nbd: disconnect request received 2024-09-21T22:08:10.891 INFO:tasks.workunit.client.0.smithi157.stderr:+ expect_false grep 'quiesce failed' /tmp/tmp.GqKuv7rsZi/rbd-nbd.log 2024-09-21T22:08:10.891 INFO:tasks.workunit.client.0.smithi157.stderr:+ grep 'quiesce failed' /tmp/tmp.GqKuv7rsZi/rbd-nbd.log 2024-09-21T22:08:10.892 INFO:tasks.workunit.client.0.smithi157.stderr:+ return 0 2024-09-21T22:08:10.893 INFO:tasks.workunit.client.0.smithi157.stderr:++ _sudo rbd device --device-type nbd --show-cookie map rbd/testrbdnbd21553 2024-09-21T22:08:10.893 INFO:tasks.workunit.client.0.smithi157.stderr:++ local cmd 2024-09-21T22:08:10.893 INFO:tasks.workunit.client.0.smithi157.stderr:+++ id -u 2024-09-21T22:08:10.895 INFO:tasks.workunit.client.0.smithi157.stderr:++ '[' 1000 -eq 0 ']' 2024-09-21T22:08:10.895 INFO:tasks.workunit.client.0.smithi157.stderr:+++ which rbd 2024-09-21T22:08:10.897 INFO:tasks.workunit.client.0.smithi157.stderr:++ cmd=/usr/bin/rbd 2024-09-21T22:08:10.897 INFO:tasks.workunit.client.0.smithi157.stderr:++ shift 2024-09-21T22:08:10.897 INFO:tasks.workunit.client.0.smithi157.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd --show-cookie map rbd/testrbdnbd21553 2024-09-21T22:08:10.912 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:08:10.909+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:08:10.996 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:08:10.997 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-21T22:08:10.997 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-21T22:08:10.997 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-21T22:08:10.997 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.997 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4840.0:676 2.7 e63/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 63'57, mlcod=63'55)", 2024-09-21T22:08:10.997 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.627276+0000", 2024-09-21T22:08:10.997 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 46.367704449000001, 2024-09-21T22:08:10.997 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.037500787000000001, 2024-09-21T22:08:10.997 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:10.997 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:10.997 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:10.998 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:10.998 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.998 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:10.998 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.627276+0000", 2024-09-21T22:08:10.998 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:10.998 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.998 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.998 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:10.998 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.627276+0000", 2024-09-21T22:08:10.998 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:10.998 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.998 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.998 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:10.998 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.627277+0000", 2024-09-21T22:08:10.999 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.068e-06 2024-09-21T22:08:10.999 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.999 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.999 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:10.999 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.628148+0000", 2024-09-21T22:08:10.999 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00087069999999999997 2024-09-21T22:08:10.999 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.999 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:10.999 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:10.999 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.628170+0000", 2024-09-21T22:08:10.999 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2147999999999999e-05 2024-09-21T22:08:10.999 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:10.999 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.000 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:11.000 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.628179+0000", 2024-09-21T22:08:11.000 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2450000000000004e-06 2024-09-21T22:08:11.000 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.000 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.000 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:11.000 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.628279+0000", 2024-09-21T22:08:11.000 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9298999999999999e-05 2024-09-21T22:08:11.000 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.000 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.000 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:11.000 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.628305+0000", 2024-09-21T22:08:11.000 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6316999999999999e-05 2024-09-21T22:08:11.001 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.001 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.001 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:11.001 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.664714+0000", 2024-09-21T22:08:11.001 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.036408923000000003 2024-09-21T22:08:11.001 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.001 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.001 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:11.001 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.664777+0000", 2024-09-21T22:08:11.001 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3087000000000001e-05 2024-09-21T22:08:11.001 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:11.001 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:11.001 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:11.001 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.002 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.002 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4840.0:678 2.7 e63/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 63'59, mlcod=63'57)", 2024-09-21T22:08:11.002 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.679689+0000", 2024-09-21T22:08:11.002 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 46.315291948999999, 2024-09-21T22:08:11.002 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053308406000000003, 2024-09-21T22:08:11.002 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:11.002 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:11.002 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:11.002 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:11.002 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.002 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:11.002 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.679689+0000", 2024-09-21T22:08:11.002 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:11.002 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.003 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.003 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:11.003 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.679689+0000", 2024-09-21T22:08:11.003 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:11.003 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.003 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.003 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:11.003 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.679691+0000", 2024-09-21T22:08:11.003 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9400000000000001e-06 2024-09-21T22:08:11.003 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.003 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.003 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:11.003 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.680245+0000", 2024-09-21T22:08:11.003 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00055382999999999995 2024-09-21T22:08:11.003 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.004 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.004 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:11.004 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.680276+0000", 2024-09-21T22:08:11.004 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1123e-05 2024-09-21T22:08:11.004 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.004 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.004 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:11.004 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.680290+0000", 2024-09-21T22:08:11.004 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4056999999999999e-05 2024-09-21T22:08:11.004 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.004 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.004 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:11.004 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.680418+0000", 2024-09-21T22:08:11.004 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000127792 2024-09-21T22:08:11.005 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.005 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.005 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:11.005 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.680465+0000", 2024-09-21T22:08:11.005 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7360000000000001e-05 2024-09-21T22:08:11.005 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.005 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.005 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:11.005 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.732923+0000", 2024-09-21T22:08:11.005 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.052458312999999999 2024-09-21T22:08:11.005 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.005 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.005 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:11.005 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.732997+0000", 2024-09-21T22:08:11.006 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3991000000000004e-05 2024-09-21T22:08:11.006 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:11.006 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:11.006 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:11.006 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.006 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.006 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4840.0:680 2.7 e63/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 63'61, mlcod=63'59)", 2024-09-21T22:08:11.006 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.745996+0000", 2024-09-21T22:08:11.006 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 46.248984436000001, 2024-09-21T22:08:11.006 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.061804132999999997, 2024-09-21T22:08:11.006 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:11.006 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:11.006 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:11.007 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:11.007 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.007 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:11.007 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.745996+0000", 2024-09-21T22:08:11.007 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:11.007 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.007 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.007 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:11.007 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.745996+0000", 2024-09-21T22:08:11.007 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:11.007 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.007 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.007 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:11.007 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.745998+0000", 2024-09-21T22:08:11.007 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.717e-06 2024-09-21T22:08:11.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.008 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:11.008 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.746523+0000", 2024-09-21T22:08:11.008 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00052525399999999996 2024-09-21T22:08:11.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.008 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:11.008 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.746554+0000", 2024-09-21T22:08:11.008 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0586000000000002e-05 2024-09-21T22:08:11.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.008 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:11.008 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.746577+0000", 2024-09-21T22:08:11.008 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3079e-05 2024-09-21T22:08:11.009 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.009 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.009 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:11.009 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.746689+0000", 2024-09-21T22:08:11.009 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011177 2024-09-21T22:08:11.009 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.009 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.009 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:11.009 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.746722+0000", 2024-09-21T22:08:11.009 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2870000000000002e-05 2024-09-21T22:08:11.009 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.009 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.009 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:11.010 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.807740+0000", 2024-09-21T22:08:11.010 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.061018612 2024-09-21T22:08:11.010 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.010 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.010 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:11.010 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.807800+0000", 2024-09-21T22:08:11.010 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0244999999999997e-05 2024-09-21T22:08:11.010 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:11.010 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:11.010 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:11.010 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.010 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.010 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4840.0:682 2.7 e63/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 63'63, mlcod=63'61)", 2024-09-21T22:08:11.011 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.824126+0000", 2024-09-21T22:08:11.011 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 46.170854867000003, 2024-09-21T22:08:11.011 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.038159327, 2024-09-21T22:08:11.011 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:11.011 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:11.011 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:11.011 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:11.011 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.011 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:11.011 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.824126+0000", 2024-09-21T22:08:11.011 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:11.011 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.011 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.012 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:11.012 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.824126+0000", 2024-09-21T22:08:11.012 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:11.012 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.012 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:11.012 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.824128+0000", 2024-09-21T22:08:11.012 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9659999999999999e-06 2024-09-21T22:08:11.012 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.012 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:11.012 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.824842+0000", 2024-09-21T22:08:11.013 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00071423999999999999 2024-09-21T22:08:11.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.013 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.013 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:11.013 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.824883+0000", 2024-09-21T22:08:11.013 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1131999999999999e-05 2024-09-21T22:08:11.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.013 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.013 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:11.013 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.824931+0000", 2024-09-21T22:08:11.013 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7327000000000003e-05 2024-09-21T22:08:11.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.014 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.014 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:11.014 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.825058+0000", 2024-09-21T22:08:11.014 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012790499999999999 2024-09-21T22:08:11.014 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.014 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.014 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:11.014 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.825087+0000", 2024-09-21T22:08:11.014 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8192e-05 2024-09-21T22:08:11.014 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.014 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.014 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:11.015 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.862225+0000", 2024-09-21T22:08:11.015 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.037138309000000001 2024-09-21T22:08:11.015 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.015 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.015 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:11.015 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.862285+0000", 2024-09-21T22:08:11.015 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0256e-05 2024-09-21T22:08:11.015 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:11.015 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:11.015 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:11.015 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.015 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.016 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5008.0:116 2.7 e71/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 71'148, mlcod=71'146)", 2024-09-21T22:08:11.016 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:37.867612+0000", 2024-09-21T22:08:11.016 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 33.127368371999999, 2024-09-21T22:08:11.016 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.045123155999999998, 2024-09-21T22:08:11.016 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:11.016 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:11.016 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:11.016 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:11.016 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.016 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:11.016 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.867612+0000", 2024-09-21T22:08:11.016 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:11.016 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.017 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.017 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:11.017 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.867612+0000", 2024-09-21T22:08:11.017 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:11.017 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.017 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.017 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:11.017 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.867613+0000", 2024-09-21T22:08:11.017 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1200000000000002e-07 2024-09-21T22:08:11.017 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.017 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.017 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:11.018 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.868135+0000", 2024-09-21T22:08:11.018 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00052132700000000003 2024-09-21T22:08:11.018 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.018 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.018 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:11.018 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.868150+0000", 2024-09-21T22:08:11.018 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5271e-05 2024-09-21T22:08:11.018 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.018 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.018 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:11.018 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.868156+0000", 2024-09-21T22:08:11.018 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4969999999999998e-06 2024-09-21T22:08:11.018 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.019 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.019 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:11.019 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.868213+0000", 2024-09-21T22:08:11.019 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7114999999999998e-05 2024-09-21T22:08:11.019 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.019 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.019 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:11.019 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.868238+0000", 2024-09-21T22:08:11.019 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4987999999999999e-05 2024-09-21T22:08:11.019 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.019 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.020 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:11.020 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.912677+0000", 2024-09-21T22:08:11.020 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.044439019000000003 2024-09-21T22:08:11.020 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.020 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.020 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:11.020 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.912736+0000", 2024-09-21T22:08:11.020 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8127000000000001e-05 2024-09-21T22:08:11.020 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:11.020 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:11.020 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:11.020 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.020 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.021 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5008.0:118 2.7 e71/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 71'150, mlcod=71'148)", 2024-09-21T22:08:11.021 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:37.924700+0000", 2024-09-21T22:08:11.021 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 33.070281180999999, 2024-09-21T22:08:11.021 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.034650561000000003, 2024-09-21T22:08:11.021 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:11.021 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:11.021 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:11.021 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:11.021 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.021 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:11.021 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.924700+0000", 2024-09-21T22:08:11.021 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:11.021 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.021 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.021 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:11.022 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.924700+0000", 2024-09-21T22:08:11.022 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:11.022 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.022 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.022 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:11.022 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.924702+0000", 2024-09-21T22:08:11.022 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9460000000000001e-06 2024-09-21T22:08:11.022 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.022 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.022 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:11.022 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.925132+0000", 2024-09-21T22:08:11.022 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00043021200000000002 2024-09-21T22:08:11.022 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.022 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.023 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:11.023 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.925156+0000", 2024-09-21T22:08:11.023 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4451999999999999e-05 2024-09-21T22:08:11.023 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.023 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.023 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:11.023 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.925168+0000", 2024-09-21T22:08:11.023 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2249000000000001e-05 2024-09-21T22:08:11.023 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.023 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.023 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:11.023 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.925241+0000", 2024-09-21T22:08:11.023 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2502000000000005e-05 2024-09-21T22:08:11.023 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.023 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.024 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:11.024 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.925267+0000", 2024-09-21T22:08:11.024 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6471000000000001e-05 2024-09-21T22:08:11.024 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.024 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.024 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:11.024 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.959217+0000", 2024-09-21T22:08:11.024 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.033949569999999998 2024-09-21T22:08:11.024 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.024 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.024 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:11.024 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.959350+0000", 2024-09-21T22:08:11.024 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013315899999999999 2024-09-21T22:08:11.024 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:11.025 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:11.025 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:11.025 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.025 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.025 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5008.0:120 2.7 e71/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 71'152, mlcod=71'150)", 2024-09-21T22:08:11.025 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:37.970837+0000", 2024-09-21T22:08:11.025 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 33.024143930000001, 2024-09-21T22:08:11.025 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.03044943, 2024-09-21T22:08:11.025 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:11.025 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:11.025 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:11.025 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:11.025 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.025 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:11.025 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.970837+0000", 2024-09-21T22:08:11.026 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:11.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.026 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.026 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:11.026 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.970837+0000", 2024-09-21T22:08:11.026 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:11.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.026 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.026 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:11.026 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.970838+0000", 2024-09-21T22:08:11.026 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4649999999999999e-06 2024-09-21T22:08:11.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.026 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.026 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:11.027 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.971265+0000", 2024-09-21T22:08:11.027 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00042665800000000001 2024-09-21T22:08:11.027 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.027 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.027 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:11.027 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.971285+0000", 2024-09-21T22:08:11.027 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9639999999999998e-05 2024-09-21T22:08:11.027 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.027 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.027 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:11.027 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.971294+0000", 2024-09-21T22:08:11.027 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7210000000000006e-06 2024-09-21T22:08:11.027 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.027 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.028 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:11.028 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.971354+0000", 2024-09-21T22:08:11.028 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9613e-05 2024-09-21T22:08:11.028 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.028 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.028 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:11.028 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.971381+0000", 2024-09-21T22:08:11.028 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7538999999999998e-05 2024-09-21T22:08:11.028 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.028 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.028 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:11.028 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.001199+0000", 2024-09-21T22:08:11.028 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029817537000000002 2024-09-21T22:08:11.028 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.028 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.029 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:11.029 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.001286+0000", 2024-09-21T22:08:11.029 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7256999999999995e-05 2024-09-21T22:08:11.029 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:11.029 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:11.029 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:11.029 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.029 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.029 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5008.0:122 2.7 e71/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 71'154, mlcod=71'152)", 2024-09-21T22:08:11.029 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:38.014709+0000", 2024-09-21T22:08:11.029 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 32.980272055, 2024-09-21T22:08:11.029 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029567408999999999, 2024-09-21T22:08:11.029 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:11.029 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:11.030 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:11.030 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:11.030 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.030 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:11.030 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.014709+0000", 2024-09-21T22:08:11.030 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:11.030 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.030 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.030 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:11.030 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.014709+0000", 2024-09-21T22:08:11.030 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:11.030 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.030 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.030 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:11.031 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.014709+0000", 2024-09-21T22:08:11.031 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9599999999999999e-07 2024-09-21T22:08:11.031 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.031 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.031 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:11.031 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.015201+0000", 2024-09-21T22:08:11.031 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00049175599999999999 2024-09-21T22:08:11.031 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.031 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.031 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:11.031 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.015215+0000", 2024-09-21T22:08:11.031 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3692999999999999e-05 2024-09-21T22:08:11.031 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.031 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.032 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:11.032 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.015225+0000", 2024-09-21T22:08:11.032 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.815e-06 2024-09-21T22:08:11.032 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.032 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.032 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:11.032 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.015287+0000", 2024-09-21T22:08:11.032 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1858000000000003e-05 2024-09-21T22:08:11.032 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.032 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.032 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:11.032 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.015315+0000", 2024-09-21T22:08:11.033 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8490999999999999e-05 2024-09-21T22:08:11.033 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.033 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.033 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:11.033 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.044182+0000", 2024-09-21T22:08:11.033 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.028866936999999999 2024-09-21T22:08:11.033 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.033 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.033 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:11.033 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.044276+0000", 2024-09-21T22:08:11.033 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4163e-05 2024-09-21T22:08:11.033 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:11.033 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:11.033 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:11.033 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.034 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.034 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5207.0:70 2.7 e79/16 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head v 79'184, mlcod=79'181)", 2024-09-21T22:08:11.034 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:52.938385+0000", 2024-09-21T22:08:11.034 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 18.056596149000001, 2024-09-21T22:08:11.034 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023928324000000001, 2024-09-21T22:08:11.034 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:11.034 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:11.034 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:11.034 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:11.034 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.034 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:11.034 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938385+0000", 2024-09-21T22:08:11.035 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:11.035 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.035 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:11.035 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938385+0000", 2024-09-21T22:08:11.035 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:11.035 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.035 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:11.035 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938387+0000", 2024-09-21T22:08:11.035 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1639999999999999e-06 2024-09-21T22:08:11.035 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.036 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:11.036 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938511+0000", 2024-09-21T22:08:11.036 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000124485 2024-09-21T22:08:11.036 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.036 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.036 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:11.036 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938549+0000", 2024-09-21T22:08:11.036 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7347e-05 2024-09-21T22:08:11.036 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.036 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.036 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:11.036 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938570+0000", 2024-09-21T22:08:11.036 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1013e-05 2024-09-21T22:08:11.036 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.037 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.037 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:11.037 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938718+0000", 2024-09-21T22:08:11.037 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000148789 2024-09-21T22:08:11.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.037 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.037 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:11.037 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938769+0000", 2024-09-21T22:08:11.037 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1002999999999999e-05 2024-09-21T22:08:11.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.037 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.037 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:11.037 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.962199+0000", 2024-09-21T22:08:11.037 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023429783999999999 2024-09-21T22:08:11.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.038 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.038 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:11.038 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.962313+0000", 2024-09-21T22:08:11.038 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000113739 2024-09-21T22:08:11.038 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:11.038 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:11.038 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:11.038 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.038 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.038 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5207.0:71 2.7 e79/16 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head v 79'185, mlcod=79'181)", 2024-09-21T22:08:11.038 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:52.951522+0000", 2024-09-21T22:08:11.038 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 18.043458257000001, 2024-09-21T22:08:11.038 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021108028000000001, 2024-09-21T22:08:11.038 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:11.039 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:11.039 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:11.039 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:11.039 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.039 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:11.039 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.951522+0000", 2024-09-21T22:08:11.039 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:11.039 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.039 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.039 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:11.039 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.951522+0000", 2024-09-21T22:08:11.039 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:11.039 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.039 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.039 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:11.040 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.951524+0000", 2024-09-21T22:08:11.040 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9199999999999998e-06 2024-09-21T22:08:11.040 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.040 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.040 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:11.040 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.951686+0000", 2024-09-21T22:08:11.040 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016202499999999999 2024-09-21T22:08:11.040 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.040 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.040 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:11.040 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.951713+0000", 2024-09-21T22:08:11.040 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7007000000000001e-05 2024-09-21T22:08:11.040 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.040 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.041 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:11.041 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.951734+0000", 2024-09-21T22:08:11.041 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0228999999999999e-05 2024-09-21T22:08:11.041 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.041 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.041 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:11.041 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.952740+0000", 2024-09-21T22:08:11.041 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001006631 2024-09-21T22:08:11.041 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.041 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.041 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:11.041 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.952778+0000", 2024-09-21T22:08:11.041 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7316e-05 2024-09-21T22:08:11.041 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.041 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.042 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:11.042 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.972505+0000", 2024-09-21T22:08:11.042 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019727776999999998 2024-09-21T22:08:11.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.042 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.042 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:11.042 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.972631+0000", 2024-09-21T22:08:11.042 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012512299999999999 2024-09-21T22:08:11.042 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:11.042 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:11.042 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:11.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.042 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.043 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5207.0:76 2.7 e79/16 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head v 79'190, mlcod=79'186)", 2024-09-21T22:08:11.043 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:53.001917+0000", 2024-09-21T22:08:11.043 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 17.993063807999999, 2024-09-21T22:08:11.043 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022783040000000001, 2024-09-21T22:08:11.043 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:11.043 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:11.043 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:11.043 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:11.043 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.043 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:11.043 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.001917+0000", 2024-09-21T22:08:11.043 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:11.043 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.043 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.043 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:11.044 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.001917+0000", 2024-09-21T22:08:11.044 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:11.044 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.044 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.044 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:11.044 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.001919+0000", 2024-09-21T22:08:11.044 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.703e-06 2024-09-21T22:08:11.044 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.044 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.044 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:11.044 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.002006+0000", 2024-09-21T22:08:11.044 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6978999999999995e-05 2024-09-21T22:08:11.044 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.044 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.045 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:11.045 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.002042+0000", 2024-09-21T22:08:11.045 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6847000000000001e-05 2024-09-21T22:08:11.045 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.045 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.045 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:11.045 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.002057+0000", 2024-09-21T22:08:11.045 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.419e-05 2024-09-21T22:08:11.045 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.045 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.045 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:11.045 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.004387+0000", 2024-09-21T22:08:11.045 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002330365 2024-09-21T22:08:11.045 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.045 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.046 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:11.046 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.004420+0000", 2024-09-21T22:08:11.046 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3170000000000003e-05 2024-09-21T22:08:11.046 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.046 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.046 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:11.046 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.024610+0000", 2024-09-21T22:08:11.046 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020190084000000001 2024-09-21T22:08:11.046 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.046 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.046 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:11.046 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.024700+0000", 2024-09-21T22:08:11.046 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9702000000000003e-05 2024-09-21T22:08:11.046 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:11.047 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:11.047 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:11.047 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.047 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.047 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5207.0:93 2.7 e79/16 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head v 79'193, mlcod=79'191)", 2024-09-21T22:08:11.047 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:53.149813+0000", 2024-09-21T22:08:11.047 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 17.845167749000002, 2024-09-21T22:08:11.047 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.060053738000000002, 2024-09-21T22:08:11.047 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:11.047 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:11.047 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:11.047 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:11.047 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.047 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:11.047 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.149813+0000", 2024-09-21T22:08:11.048 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:11.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.048 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:11.048 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.149813+0000", 2024-09-21T22:08:11.048 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:11.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.048 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:11.048 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.149815+0000", 2024-09-21T22:08:11.048 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9099999999999999e-06 2024-09-21T22:08:11.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.048 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:11.049 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.150828+0000", 2024-09-21T22:08:11.049 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0010129099999999999 2024-09-21T22:08:11.049 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.049 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.049 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:11.049 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.150860+0000", 2024-09-21T22:08:11.049 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2443000000000001e-05 2024-09-21T22:08:11.049 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.049 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.049 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:11.049 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.150874+0000", 2024-09-21T22:08:11.049 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3627e-05 2024-09-21T22:08:11.049 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.049 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.049 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:11.050 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.151010+0000", 2024-09-21T22:08:11.050 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013656599999999999 2024-09-21T22:08:11.050 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.050 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.050 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:11.050 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.151047+0000", 2024-09-21T22:08:11.050 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6495999999999998e-05 2024-09-21T22:08:11.050 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.050 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.050 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:11.050 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.209791+0000", 2024-09-21T22:08:11.050 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058744004000000002 2024-09-21T22:08:11.050 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.051 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.051 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:11.051 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.209867+0000", 2024-09-21T22:08:11.051 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5782000000000001e-05 2024-09-21T22:08:11.051 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:11.051 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:11.051 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:11.051 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.051 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.051 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5207.0:128 2.7 e83/16 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head v 83'198, mlcod=83'195)", 2024-09-21T22:08:11.051 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:08:01.058831+0000", 2024-09-21T22:08:11.051 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 9.9361501140000001, 2024-09-21T22:08:11.051 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058096534999999998, 2024-09-21T22:08:11.051 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:11.052 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:11.052 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:11.052 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:11.052 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.052 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:11.052 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.058831+0000", 2024-09-21T22:08:11.052 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:11.052 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.052 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.052 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:11.052 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.058831+0000", 2024-09-21T22:08:11.052 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:11.052 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.052 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.053 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:11.053 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.058833+0000", 2024-09-21T22:08:11.053 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0660000000000002e-06 2024-09-21T22:08:11.053 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.053 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.053 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:11.053 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.059877+0000", 2024-09-21T22:08:11.053 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0010447 2024-09-21T22:08:11.053 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.053 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.053 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:11.053 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.059908+0000", 2024-09-21T22:08:11.053 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0800000000000003e-05 2024-09-21T22:08:11.053 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.054 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.054 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:11.054 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.059923+0000", 2024-09-21T22:08:11.054 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4497e-05 2024-09-21T22:08:11.054 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.054 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.054 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:11.054 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.060053+0000", 2024-09-21T22:08:11.054 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013002900000000001 2024-09-21T22:08:11.054 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.054 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.054 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:11.054 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.060091+0000", 2024-09-21T22:08:11.054 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8389999999999997e-05 2024-09-21T22:08:11.055 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.055 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.055 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:11.055 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.116840+0000", 2024-09-21T22:08:11.055 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.056749253999999999 2024-09-21T22:08:11.055 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.055 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.055 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:11.055 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.116927+0000", 2024-09-21T22:08:11.055 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6798999999999993e-05 2024-09-21T22:08:11.055 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:11.055 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:11.055 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:11.055 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.056 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.056 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.6 87/86 [PushOp(2:7e2189ff:ns::rbd_data.127f79bc213d.0000000000000001:head, version: 57'4, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:7e2189ff:ns::rbd_data.127f79bc213d.0000000000000001:head@57'4, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T22:08:11.056 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:08:07.095630+0000", 2024-09-21T22:08:11.056 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.899351158, 2024-09-21T22:08:11.056 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.063566191999999994, 2024-09-21T22:08:11.056 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:11.056 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:11.056 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:08:11.056 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:11.056 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.056 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:11.056 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.095630+0000", 2024-09-21T22:08:11.056 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:11.056 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.056 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.057 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:11.057 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.095630+0000", 2024-09-21T22:08:11.057 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:11.057 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.057 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.057 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:11.057 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.095631+0000", 2024-09-21T22:08:11.057 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3069999999999999e-06 2024-09-21T22:08:11.057 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.057 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.057 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:11.057 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.097075+0000", 2024-09-21T22:08:11.057 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001443831 2024-09-21T22:08:11.058 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.058 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.058 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:11.058 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.097153+0000", 2024-09-21T22:08:11.058 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8485999999999994e-05 2024-09-21T22:08:11.058 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.058 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.058 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:11.058 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.097208+0000", 2024-09-21T22:08:11.058 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4716000000000003e-05 2024-09-21T22:08:11.058 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.058 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.058 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:11.058 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.097459+0000", 2024-09-21T22:08:11.059 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025113799999999998 2024-09-21T22:08:11.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.059 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.059 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:11.059 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.097481+0000", 2024-09-21T22:08:11.059 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2344000000000002e-05 2024-09-21T22:08:11.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.059 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.059 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:11.059 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.159196+0000", 2024-09-21T22:08:11.059 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.061714369999999998 2024-09-21T22:08:11.059 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:11.059 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:11.059 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:11.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.060 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.060 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.6 87/86 [PushOp(2:7e2189ff:ns::rbd_data.127f79bc213d.0000000000000001:head, version: 57'4, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:7e2189ff:ns::rbd_data.127f79bc213d.0000000000000001:head@57'4, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:08:11.060 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:08:07.163314+0000", 2024-09-21T22:08:11.060 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.8316671040000001, 2024-09-21T22:08:11.060 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.044796923000000002, 2024-09-21T22:08:11.060 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:11.060 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:11.060 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:08:11.060 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:11.060 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.060 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:11.060 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.163314+0000", 2024-09-21T22:08:11.060 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:11.060 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.060 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.061 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:11.061 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.163314+0000", 2024-09-21T22:08:11.061 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:11.061 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.061 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.061 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:11.061 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.163315+0000", 2024-09-21T22:08:11.061 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7120000000000001e-06 2024-09-21T22:08:11.061 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.061 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.061 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:11.061 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.164270+0000", 2024-09-21T22:08:11.062 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000955034 2024-09-21T22:08:11.062 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.062 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.062 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:11.062 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.164326+0000", 2024-09-21T22:08:11.062 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5627999999999997e-05 2024-09-21T22:08:11.062 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.062 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.062 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:11.062 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.164351+0000", 2024-09-21T22:08:11.062 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5219999999999999e-05 2024-09-21T22:08:11.062 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.062 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.063 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:11.063 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.164571+0000", 2024-09-21T22:08:11.063 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000219996 2024-09-21T22:08:11.063 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.063 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.063 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:11.063 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.164596+0000", 2024-09-21T22:08:11.063 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4564000000000001e-05 2024-09-21T22:08:11.063 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.063 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.063 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:11.063 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.208111+0000", 2024-09-21T22:08:11.063 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.043514769000000002 2024-09-21T22:08:11.063 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:11.064 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:11.064 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:11.064 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.064 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.064 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.6 87/86 [PushOp(2:7e2189ff:ns::rbd_data.127f79bc213d.0000000000000001:head, version: 57'4, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:7e2189ff:ns::rbd_data.127f79bc213d.0000000000000001:head@57'4, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:08:11.064 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:08:07.212243+0000", 2024-09-21T22:08:11.064 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.7827376610000001, 2024-09-21T22:08:11.064 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.051665924000000002, 2024-09-21T22:08:11.064 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:11.064 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:11.064 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:08:11.064 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:11.064 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.064 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:11.064 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.212243+0000", 2024-09-21T22:08:11.065 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:11.065 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.065 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:11.065 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.212243+0000", 2024-09-21T22:08:11.065 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:11.065 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.065 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:11.065 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.212244+0000", 2024-09-21T22:08:11.065 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2839999999999999e-06 2024-09-21T22:08:11.065 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.065 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:11.065 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.212993+0000", 2024-09-21T22:08:11.066 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00074894100000000004 2024-09-21T22:08:11.066 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.066 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.066 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:11.066 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.213032+0000", 2024-09-21T22:08:11.066 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8948e-05 2024-09-21T22:08:11.066 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.066 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.066 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:11.066 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.213054+0000", 2024-09-21T22:08:11.066 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1937999999999999e-05 2024-09-21T22:08:11.066 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.066 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.066 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:11.067 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.213194+0000", 2024-09-21T22:08:11.067 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013942699999999999 2024-09-21T22:08:11.067 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.067 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.067 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:11.067 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.213210+0000", 2024-09-21T22:08:11.067 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5883999999999999e-05 2024-09-21T22:08:11.067 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.067 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.067 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:11.067 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.263909+0000", 2024-09-21T22:08:11.067 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.050699502 2024-09-21T22:08:11.067 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:11.067 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:11.068 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:11.068 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.068 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.068 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.6 87/86 [PushOp(2:7e2189ff:ns::rbd_data.127f79bc213d.0000000000000001:head, version: 57'4, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:7e2189ff:ns::rbd_data.127f79bc213d.0000000000000001:head@57'4, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 4194304, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:08:11.068 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:08:07.268604+0000", 2024-09-21T22:08:11.068 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.7263763889999999, 2024-09-21T22:08:11.068 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.038386754000000002, 2024-09-21T22:08:11.068 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:11.068 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:11.068 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:08:11.068 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:11.068 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.068 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:11.068 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.268604+0000", 2024-09-21T22:08:11.069 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:11.069 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.069 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.069 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:11.069 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.268604+0000", 2024-09-21T22:08:11.069 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:11.069 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.069 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.069 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:11.069 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.268606+0000", 2024-09-21T22:08:11.069 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2020000000000001e-06 2024-09-21T22:08:11.069 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.069 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.069 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:11.070 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.269149+0000", 2024-09-21T22:08:11.070 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00054327900000000003 2024-09-21T22:08:11.070 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.070 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.070 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:11.070 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.269173+0000", 2024-09-21T22:08:11.070 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4046999999999999e-05 2024-09-21T22:08:11.070 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.070 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.070 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:11.070 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.269187+0000", 2024-09-21T22:08:11.070 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4185999999999999e-05 2024-09-21T22:08:11.070 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.070 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.071 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:11.071 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.269303+0000", 2024-09-21T22:08:11.071 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000115831 2024-09-21T22:08:11.071 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.071 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.071 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:11.071 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.269320+0000", 2024-09-21T22:08:11.071 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6878999999999999e-05 2024-09-21T22:08:11.071 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.071 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.071 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:11.071 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.306991+0000", 2024-09-21T22:08:11.071 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.037671330000000003 2024-09-21T22:08:11.071 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:11.072 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:11.072 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:11.072 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.072 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.072 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.6 87/86 [PushOp(2:762dd302:ns::rbd_data.127f79bc213d.0000000000000007:head, version: 57'5, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:762dd302:ns::rbd_data.127f79bc213d.0000000000000007:head@57'5, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T22:08:11.072 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:08:07.310599+0000", 2024-09-21T22:08:11.072 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.684382195, 2024-09-21T22:08:11.072 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025565551999999998, 2024-09-21T22:08:11.072 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:11.072 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:11.072 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:08:11.072 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:11.072 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.072 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:11.073 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.310599+0000", 2024-09-21T22:08:11.073 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:11.073 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.073 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.073 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:11.073 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.310599+0000", 2024-09-21T22:08:11.073 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:11.073 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.073 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.073 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:11.073 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.310600+0000", 2024-09-21T22:08:11.073 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0559999999999999e-06 2024-09-21T22:08:11.073 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.073 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.074 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:11.074 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.311158+0000", 2024-09-21T22:08:11.074 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00055862200000000005 2024-09-21T22:08:11.074 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.074 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.074 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:11.074 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.311180+0000", 2024-09-21T22:08:11.074 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1577999999999999e-05 2024-09-21T22:08:11.074 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.074 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.074 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:11.074 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.311198+0000", 2024-09-21T22:08:11.074 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8359000000000002e-05 2024-09-21T22:08:11.074 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.075 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.075 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:11.075 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.311291+0000", 2024-09-21T22:08:11.075 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2887999999999996e-05 2024-09-21T22:08:11.075 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.075 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.075 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:11.075 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.311300+0000", 2024-09-21T22:08:11.075 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8640000000000002e-06 2024-09-21T22:08:11.075 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.075 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.075 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:11.075 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.336164+0000", 2024-09-21T22:08:11.075 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024864185 2024-09-21T22:08:11.075 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:11.076 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:11.076 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:11.076 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.076 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.076 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.6 87/86 [PushOp(2:762dd302:ns::rbd_data.127f79bc213d.0000000000000007:head, version: 57'5, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:762dd302:ns::rbd_data.127f79bc213d.0000000000000007:head@57'5, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:08:11.076 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:08:07.338463+0000", 2024-09-21T22:08:11.076 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 3.6565175839999999, 2024-09-21T22:08:11.076 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026221639000000001, 2024-09-21T22:08:11.076 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:11.076 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:11.076 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:08:11.076 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:11.077 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.077 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:11.077 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.338463+0000", 2024-09-21T22:08:11.077 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:11.077 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.077 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.077 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:11.077 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.338463+0000", 2024-09-21T22:08:11.077 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:11.077 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.077 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.077 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:11.077 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.338464+0000", 2024-09-21T22:08:11.077 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.035e-06 2024-09-21T22:08:11.078 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.078 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.078 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:11.078 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.339044+0000", 2024-09-21T22:08:11.078 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00057999699999999996 2024-09-21T22:08:11.078 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.078 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.078 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:11.078 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.339070+0000", 2024-09-21T22:08:11.078 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.565e-05 2024-09-21T22:08:11.078 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.078 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.078 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:11.079 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.339088+0000", 2024-09-21T22:08:11.079 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8189000000000002e-05 2024-09-21T22:08:11.079 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.079 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.079 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:11.079 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.339184+0000", 2024-09-21T22:08:11.079 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5475000000000003e-05 2024-09-21T22:08:11.079 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.079 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.079 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:11.079 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.339193+0000", 2024-09-21T22:08:11.079 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7790000000000007e-06 2024-09-21T22:08:11.079 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.080 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.080 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:11.080 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.364685+0000", 2024-09-21T22:08:11.080 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025491514 2024-09-21T22:08:11.080 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:11.080 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:11.080 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:11.080 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.080 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.080 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5398.0:204 2.6 e89/86 2:633e2f34:::rbd_data.126483d75f2d.0000000000000001:head v 89'32, mlcod=88'29)", 2024-09-21T22:08:11.080 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:08:10.148990+0000", 2024-09-21T22:08:11.080 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.84599103899999994, 2024-09-21T22:08:11.080 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.034897343999999997, 2024-09-21T22:08:11.080 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:11.081 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:11.081 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:11.081 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:11.081 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.081 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:11.081 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:10.148990+0000", 2024-09-21T22:08:11.081 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:11.081 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.081 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.081 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:11.081 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:10.148990+0000", 2024-09-21T22:08:11.081 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:11.081 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.081 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.081 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:11.081 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:10.148991+0000", 2024-09-21T22:08:11.082 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.61e-06 2024-09-21T22:08:11.082 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.082 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.082 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:11.082 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:10.149966+0000", 2024-09-21T22:08:11.082 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00097419800000000001 2024-09-21T22:08:11.082 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.082 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.082 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:11.082 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:10.149996+0000", 2024-09-21T22:08:11.082 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0762999999999997e-05 2024-09-21T22:08:11.082 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.082 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.082 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:11.083 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:10.150017+0000", 2024-09-21T22:08:11.083 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.109e-05 2024-09-21T22:08:11.083 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.083 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.083 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:11.083 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:10.150090+0000", 2024-09-21T22:08:11.083 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2835000000000003e-05 2024-09-21T22:08:11.083 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.083 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.083 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:11.083 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:10.150112+0000", 2024-09-21T22:08:11.083 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1746000000000002e-05 2024-09-21T22:08:11.083 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.083 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.083 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:11.084 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:10.183831+0000", 2024-09-21T22:08:11.084 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.033718571000000003 2024-09-21T22:08:11.084 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.084 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.084 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:11.084 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:10.183887+0000", 2024-09-21T22:08:11.084 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6530999999999998e-05 2024-09-21T22:08:11.084 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:11.084 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:11.084 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:11.084 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:11.084 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:11.084 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:08:11.086 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T22:08:11.087 INFO:tasks.workunit.client.0.smithi157.stderr:+ OUT='/dev/nbd1 15f2d4a2-c1e3-4632-98ee-ccbdf87d457c' 2024-09-21T22:08:11.087 INFO:tasks.workunit.client.0.smithi157.stderr:+ read DEV COOKIE 2024-09-21T22:08:11.087 INFO:tasks.workunit.client.0.smithi157.stderr:+ get_pid rbd 2024-09-21T22:08:11.087 INFO:tasks.workunit.client.0.smithi157.stderr:+ local pool=rbd 2024-09-21T22:08:11.087 INFO:tasks.workunit.client.0.smithi157.stderr:+ local ns= 2024-09-21T22:08:11.087 INFO:tasks.workunit.client.0.smithi157.stderr:++ rbd device --device-type nbd --format xml list 2024-09-21T22:08:11.087 INFO:tasks.workunit.client.0.smithi157.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd21553'\''][device='\''/dev/nbd1'\'']/id' 2024-09-21T22:08:11.087 INFO:tasks.thrashosds.thrasher:in_osds: [3, 4, 5, 0] out_osds: [2, 1] dead_osds: [2] live_osds: [0, 1, 3, 4, 5] 2024-09-21T22:08:11.087 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T22:08:11.088 INFO:tasks.thrashosds.thrasher:Reviving osd 2 2024-09-21T22:08:11.088 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-21T22:08:11.088 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-09-21T22:08:11.089 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:08:11.081+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:08:11.090 INFO:tasks.ceph.osd.2:Started 2024-09-21T22:08:11.091 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T22:08:11.098 INFO:teuthology.orchestra.run.smithi039.stderr:nodeep-scrub is unset 2024-09-21T22:08:11.141 INFO:tasks.workunit.client.0.smithi157.stderr:+ PID=23970 2024-09-21T22:08:11.142 INFO:tasks.workunit.client.0.smithi157.stderr:+ test -n 23970 2024-09-21T22:08:11.142 INFO:tasks.workunit.client.0.smithi157.stderr:+ ps -p 23970 -C rbd-nbd 2024-09-21T22:08:11.147 INFO:tasks.workunit.client.0.smithi157.stdout: PID TTY TIME CMD 2024-09-21T22:08:11.147 INFO:tasks.workunit.client.0.smithi157.stdout: 23970 ? 00:00:00 rbd-nbd 2024-09-21T22:08:11.148 INFO:tasks.workunit.client.0.smithi157.stderr:+ _sudo mount /dev/nbd1 /tmp/tmp.GqKuv7rsZi/mnt 2024-09-21T22:08:11.148 INFO:tasks.workunit.client.0.smithi157.stderr:+ local cmd 2024-09-21T22:08:11.148 INFO:tasks.workunit.client.0.smithi157.stderr:++ id -u 2024-09-21T22:08:11.149 INFO:tasks.workunit.client.0.smithi157.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T22:08:11.150 INFO:tasks.workunit.client.0.smithi157.stderr:++ which mount 2024-09-21T22:08:11.152 INFO:tasks.workunit.client.0.smithi157.stderr:+ cmd=/usr/bin/mount 2024-09-21T22:08:11.152 INFO:tasks.workunit.client.0.smithi157.stderr:+ shift 2024-09-21T22:08:11.152 INFO:tasks.workunit.client.0.smithi157.stderr:+ sudo -nE /usr/bin/mount /dev/nbd1 /tmp/tmp.GqKuv7rsZi/mnt 2024-09-21T22:08:11.186 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:08:11.184+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:08:11.210 INFO:tasks.workunit.client.0.smithi157.stderr:+ _sudo rbd device detach rbd/testrbdnbd21553 --device-type nbd 2024-09-21T22:08:11.211 INFO:tasks.workunit.client.0.smithi157.stderr:+ local cmd 2024-09-21T22:08:11.211 INFO:tasks.workunit.client.0.smithi157.stderr:++ id -u 2024-09-21T22:08:11.213 INFO:tasks.workunit.client.0.smithi157.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T22:08:11.213 INFO:tasks.workunit.client.0.smithi157.stderr:++ which rbd 2024-09-21T22:08:11.214 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:08:11.215 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:08:11.215 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-09-21T22:08:11.215 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:08:11.215 INFO:tasks.workunit.client.0.smithi157.stderr:+ cmd=/usr/bin/rbd 2024-09-21T22:08:11.215 INFO:tasks.workunit.client.0.smithi157.stderr:+ shift 2024-09-21T22:08:11.215 INFO:tasks.workunit.client.0.smithi157.stderr:+ sudo -nE /usr/bin/rbd device detach rbd/testrbdnbd21553 --device-type nbd 2024-09-21T22:08:11.220 INFO:teuthology.orchestra.run.smithi039.stderr:no valid command found; 10 closest matches: 2024-09-21T22:08:11.220 INFO:teuthology.orchestra.run.smithi039.stderr:0 2024-09-21T22:08:11.220 INFO:teuthology.orchestra.run.smithi039.stderr:1 2024-09-21T22:08:11.221 INFO:teuthology.orchestra.run.smithi039.stderr:2 2024-09-21T22:08:11.221 INFO:teuthology.orchestra.run.smithi039.stderr:abort 2024-09-21T22:08:11.221 INFO:teuthology.orchestra.run.smithi039.stderr:assert 2024-09-21T22:08:11.221 INFO:teuthology.orchestra.run.smithi039.stderr:config diff 2024-09-21T22:08:11.221 INFO:teuthology.orchestra.run.smithi039.stderr:config diff get 2024-09-21T22:08:11.221 INFO:teuthology.orchestra.run.smithi039.stderr:config get 2024-09-21T22:08:11.221 INFO:teuthology.orchestra.run.smithi039.stderr:config help [] 2024-09-21T22:08:11.221 INFO:teuthology.orchestra.run.smithi039.stderr:config set ... 2024-09-21T22:08:11.221 INFO:teuthology.orchestra.run.smithi039.stderr:admin_socket: invalid command 2024-09-21T22:08:11.223 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-21T22:08:11.223 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-09-21T22:08:11.229 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T22:08:11.314 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:08:11.312+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:08:11.325 INFO:tasks.workunit.client.0.smithi157.stderr:+ expect_false get_pid rbd 2024-09-21T22:08:11.325 INFO:tasks.workunit.client.0.smithi157.stderr:+ get_pid rbd 2024-09-21T22:08:11.325 INFO:tasks.workunit.client.0.smithi157.stderr:+ local pool=rbd 2024-09-21T22:08:11.325 INFO:tasks.workunit.client.0.smithi157.stderr:+ local ns= 2024-09-21T22:08:11.326 INFO:tasks.workunit.client.0.smithi157.stderr:++ rbd device --device-type nbd --format xml list 2024-09-21T22:08:11.326 INFO:tasks.workunit.client.0.smithi157.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd21553'\''][device='\''/dev/nbd1'\'']/id' 2024-09-21T22:08:11.388 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:08:11.385+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:08:11.393 INFO:tasks.workunit.client.0.smithi157.stderr:+ PID= 2024-09-21T22:08:11.393 INFO:tasks.workunit.client.0.smithi157.stderr:+ test -n '' 2024-09-21T22:08:11.394 INFO:tasks.workunit.client.0.smithi157.stderr:+ return 1 2024-09-21T22:08:11.394 INFO:tasks.workunit.client.0.smithi157.stderr:+ return 0 2024-09-21T22:08:11.394 INFO:tasks.workunit.client.0.smithi157.stderr:+ expect_false _sudo rbd device attach --device /dev/nbd1 rbd/testrbdnbd21553 --device-type nbd 2024-09-21T22:08:11.394 INFO:tasks.workunit.client.0.smithi157.stderr:+ _sudo rbd device attach --device /dev/nbd1 rbd/testrbdnbd21553 --device-type nbd 2024-09-21T22:08:11.394 INFO:tasks.workunit.client.0.smithi157.stderr:+ local cmd 2024-09-21T22:08:11.394 INFO:tasks.workunit.client.0.smithi157.stderr:++ id -u 2024-09-21T22:08:11.394 INFO:tasks.workunit.client.0.smithi157.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T22:08:11.395 INFO:tasks.workunit.client.0.smithi157.stderr:++ which rbd 2024-09-21T22:08:11.396 INFO:tasks.workunit.client.0.smithi157.stderr:+ cmd=/usr/bin/rbd 2024-09-21T22:08:11.396 INFO:tasks.workunit.client.0.smithi157.stderr:+ shift 2024-09-21T22:08:11.396 INFO:tasks.workunit.client.0.smithi157.stderr:+ sudo -nE /usr/bin/rbd device attach --device /dev/nbd1 rbd/testrbdnbd21553 --device-type nbd 2024-09-21T22:08:11.397 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:08:11.397 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:08:11.397 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-21T22:08:11.397 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-21T22:08:11.397 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:08:11.412 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T22:08:11.429 INFO:tasks.workunit.client.0.smithi157.stderr:rbd: could not validate attach request 2024-09-21T22:08:11.429 INFO:tasks.workunit.client.0.smithi157.stderr:rbd: mismatching the image and the device may lead to data corruption 2024-09-21T22:08:11.429 INFO:tasks.workunit.client.0.smithi157.stderr:rbd: must specify --cookie or --force to proceed 2024-09-21T22:08:11.431 INFO:tasks.workunit.client.0.smithi157.stderr:+ return 0 2024-09-21T22:08:11.431 INFO:tasks.workunit.client.0.smithi157.stderr:+ '[' -n 15f2d4a2-c1e3-4632-98ee-ccbdf87d457c ']' 2024-09-21T22:08:11.431 INFO:tasks.workunit.client.0.smithi157.stderr:+ _sudo rbd device attach --device /dev/nbd1 --cookie 15f2d4a2-c1e3-4632-98ee-ccbdf87d457c rbd/testrbdnbd21553 --device-type nbd 2024-09-21T22:08:11.431 INFO:tasks.workunit.client.0.smithi157.stderr:+ local cmd 2024-09-21T22:08:11.432 INFO:tasks.workunit.client.0.smithi157.stderr:++ id -u 2024-09-21T22:08:11.432 INFO:tasks.workunit.client.0.smithi157.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T22:08:11.433 INFO:tasks.workunit.client.0.smithi157.stderr:++ which rbd 2024-09-21T22:08:11.434 INFO:tasks.workunit.client.0.smithi157.stderr:+ cmd=/usr/bin/rbd 2024-09-21T22:08:11.434 INFO:tasks.workunit.client.0.smithi157.stderr:+ shift 2024-09-21T22:08:11.434 INFO:tasks.workunit.client.0.smithi157.stderr:+ sudo -nE /usr/bin/rbd device attach --device /dev/nbd1 --cookie 15f2d4a2-c1e3-4632-98ee-ccbdf87d457c rbd/testrbdnbd21553 --device-type nbd 2024-09-21T22:08:11.489 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:08:11.485+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:08:11.565 INFO:tasks.workunit.client.0.smithi157.stdout:/dev/nbd1 2024-09-21T22:08:11.571 INFO:tasks.workunit.client.0.smithi157.stderr:+ get_pid rbd 2024-09-21T22:08:11.571 INFO:tasks.workunit.client.0.smithi157.stderr:+ local pool=rbd 2024-09-21T22:08:11.572 INFO:tasks.workunit.client.0.smithi157.stderr:+ local ns= 2024-09-21T22:08:11.572 INFO:tasks.workunit.client.0.smithi157.stderr:++ rbd device --device-type nbd --format xml list 2024-09-21T22:08:11.572 INFO:tasks.workunit.client.0.smithi157.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd21553'\''][device='\''/dev/nbd1'\'']/id' 2024-09-21T22:08:11.583 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:08:11.584 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-21T22:08:11.584 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-21T22:08:11.584 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-21T22:08:11.584 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.584 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5031.0:3 2.2 2:53d5e6d0:::rbd_header.126483d75f2d:head [call rbd.get_size in=11b,call,call rbd.get_features in=15b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e65)", 2024-09-21T22:08:11.584 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:27.373436+0000", 2024-09-21T22:08:11.584 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 44.208232197000001, 2024-09-21T22:08:11.584 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.86524013700000002, 2024-09-21T22:08:11.584 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:11.584 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:11.584 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:08:11.584 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:11.584 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5031", 2024-09-21T22:08:11.585 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/3232705659", 2024-09-21T22:08:11.585 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3 2024-09-21T22:08:11.585 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.585 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:11.585 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.585 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:11.585 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:27.373436+0000", 2024-09-21T22:08:11.585 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:11.585 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.585 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.585 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:11.585 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:27.373436+0000", 2024-09-21T22:08:11.585 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:11.585 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.586 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.586 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:11.586 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:27.373439+0000", 2024-09-21T22:08:11.586 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5969999999999999e-06 2024-09-21T22:08:11.586 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.586 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.586 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:11.586 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:27.373450+0000", 2024-09-21T22:08:11.586 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1326e-05 2024-09-21T22:08:11.586 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.586 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.586 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:11.586 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:27.373471+0000", 2024-09-21T22:08:11.586 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0169999999999998e-05 2024-09-21T22:08:11.587 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.587 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.587 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:11.587 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:27.373486+0000", 2024-09-21T22:08:11.587 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5483000000000001e-05 2024-09-21T22:08:11.587 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.587 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.587 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:11.587 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:27.373602+0000", 2024-09-21T22:08:11.587 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000115603 2024-09-21T22:08:11.587 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.587 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.587 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-21T22:08:11.587 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:27.373617+0000", 2024-09-21T22:08:11.588 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4956999999999999e-05 2024-09-21T22:08:11.588 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.588 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.588 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:11.588 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.234578+0000", 2024-09-21T22:08:11.588 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.86096097900000002 2024-09-21T22:08:11.588 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.588 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.588 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for missing object", 2024-09-21T22:08:11.588 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.234771+0000", 2024-09-21T22:08:11.588 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019394700000000001 2024-09-21T22:08:11.588 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.588 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.588 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:11.588 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.237470+0000", 2024-09-21T22:08:11.589 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026981319999999998 2024-09-21T22:08:11.589 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.589 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.589 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:11.589 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.237791+0000", 2024-09-21T22:08:11.589 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00032138499999999999 2024-09-21T22:08:11.589 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.589 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.589 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:11.589 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.238677+0000", 2024-09-21T22:08:11.589 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00088555800000000005 2024-09-21T22:08:11.589 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:11.589 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:11.589 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:11.590 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.590 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.590 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.2 66/65 [PushOp(2:4481fad7:ns::rbd_data.127f79bc213d.0000000000000006:head, version: 57'9, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:4481fad7:ns::rbd_data.127f79bc213d.0000000000000006:head@57'9, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T22:08:11.590 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:28.247056+0000", 2024-09-21T22:08:11.590 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 43.334612534000001, 2024-09-21T22:08:11.590 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.045025012000000003, 2024-09-21T22:08:11.590 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:11.590 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:11.590 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:08:11.590 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:11.590 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.590 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:11.590 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.247056+0000", 2024-09-21T22:08:11.590 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:11.591 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.591 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.591 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:11.591 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.247056+0000", 2024-09-21T22:08:11.591 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:11.591 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.591 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.591 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:11.591 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.247057+0000", 2024-09-21T22:08:11.591 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2500000000000001e-06 2024-09-21T22:08:11.591 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.591 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.591 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:11.591 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.249384+0000", 2024-09-21T22:08:11.592 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0023265619999999999 2024-09-21T22:08:11.592 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.592 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.592 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:11.592 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.249429+0000", 2024-09-21T22:08:11.592 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5392999999999997e-05 2024-09-21T22:08:11.592 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.592 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.592 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:11.592 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.249476+0000", 2024-09-21T22:08:11.592 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6570000000000003e-05 2024-09-21T22:08:11.592 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.592 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.592 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:11.592 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.249611+0000", 2024-09-21T22:08:11.593 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000135143 2024-09-21T22:08:11.593 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.593 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.593 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:11.593 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.249626+0000", 2024-09-21T22:08:11.593 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.469e-05 2024-09-21T22:08:11.593 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.593 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.593 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:11.593 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.292081+0000", 2024-09-21T22:08:11.593 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.042455404000000002 2024-09-21T22:08:11.593 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:11.593 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:11.593 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:11.593 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.594 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.594 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5031.0:9 2.2 2:53d5e6d0:::rbd_header.126483d75f2d:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-21T22:08:11.594 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:28.250163+0000", 2024-09-21T22:08:11.594 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 43.331505139999997, 2024-09-21T22:08:11.594 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.042986430999999999, 2024-09-21T22:08:11.594 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:11.594 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:11.594 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:08:11.594 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:11.594 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5031", 2024-09-21T22:08:11.594 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/3232705659", 2024-09-21T22:08:11.594 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 9 2024-09-21T22:08:11.594 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.594 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:11.595 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.595 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:11.595 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.250163+0000", 2024-09-21T22:08:11.595 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:11.595 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.595 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.595 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:11.595 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.250163+0000", 2024-09-21T22:08:11.595 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:11.595 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.595 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.595 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:11.595 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.250166+0000", 2024-09-21T22:08:11.595 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7039999999999999e-06 2024-09-21T22:08:11.595 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.596 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.596 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:11.596 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.250175+0000", 2024-09-21T22:08:11.596 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7560000000000005e-06 2024-09-21T22:08:11.596 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.596 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.596 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:11.596 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.250199+0000", 2024-09-21T22:08:11.596 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4049999999999998e-05 2024-09-21T22:08:11.596 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.596 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.596 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:11.596 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.250218+0000", 2024-09-21T22:08:11.596 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9236e-05 2024-09-21T22:08:11.597 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.597 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.597 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:11.597 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.292193+0000", 2024-09-21T22:08:11.597 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.041974694 2024-09-21T22:08:11.597 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.597 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.597 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:11.597 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.292492+0000", 2024-09-21T22:08:11.597 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00029900100000000002 2024-09-21T22:08:11.597 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.597 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.597 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:11.597 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.293150+0000", 2024-09-21T22:08:11.598 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00065799000000000001 2024-09-21T22:08:11.598 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:11.598 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:11.598 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:11.598 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.598 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.598 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.2 66/65 [PushOp(2:4481fad7:ns::rbd_data.127f79bc213d.0000000000000006:head, version: 57'9, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:4481fad7:ns::rbd_data.127f79bc213d.0000000000000006:head@57'9, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:08:11.598 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:28.331166+0000", 2024-09-21T22:08:11.598 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 43.250502390000001, 2024-09-21T22:08:11.598 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.039995378999999998, 2024-09-21T22:08:11.598 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:11.598 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:11.598 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:08:11.598 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:11.599 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.599 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:11.599 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.331166+0000", 2024-09-21T22:08:11.599 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:11.599 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.599 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.599 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:11.599 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.331166+0000", 2024-09-21T22:08:11.599 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:11.599 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.599 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.599 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:11.599 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.331167+0000", 2024-09-21T22:08:11.599 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9199999999999999e-07 2024-09-21T22:08:11.599 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.600 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.600 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:11.600 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.332127+0000", 2024-09-21T22:08:11.600 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00095969800000000004 2024-09-21T22:08:11.600 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.600 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.600 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:11.600 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.332149+0000", 2024-09-21T22:08:11.600 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1974999999999999e-05 2024-09-21T22:08:11.600 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.600 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.600 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:11.600 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.332161+0000", 2024-09-21T22:08:11.600 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2276000000000001e-05 2024-09-21T22:08:11.600 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.601 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.601 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:11.601 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.332290+0000", 2024-09-21T22:08:11.601 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000129214 2024-09-21T22:08:11.601 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.601 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.601 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:11.601 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.332312+0000", 2024-09-21T22:08:11.601 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1308999999999999e-05 2024-09-21T22:08:11.601 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.601 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.601 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:11.601 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.371162+0000", 2024-09-21T22:08:11.601 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.038850015000000002 2024-09-21T22:08:11.602 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:11.602 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:11.602 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:11.602 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.602 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.602 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4811.0:3 2.3 2:c4c92e5a:::rbd_trash:head [call rbd.trash_list in=13b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-21T22:08:11.602 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:28.475306+0000", 2024-09-21T22:08:11.602 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 43.106362846000003, 2024-09-21T22:08:11.602 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.77996670099999998, 2024-09-21T22:08:11.602 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:11.602 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:11.602 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:08:11.602 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:11.602 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4811", 2024-09-21T22:08:11.603 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/2028736390", 2024-09-21T22:08:11.603 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3 2024-09-21T22:08:11.603 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.603 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:11.603 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.603 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:11.603 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.475306+0000", 2024-09-21T22:08:11.603 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:11.603 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.603 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.603 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:11.603 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.475306+0000", 2024-09-21T22:08:11.603 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:11.603 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.603 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.604 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:11.604 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.475308+0000", 2024-09-21T22:08:11.604 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8959999999999999e-06 2024-09-21T22:08:11.604 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.604 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.604 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:11.604 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.475314+0000", 2024-09-21T22:08:11.604 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2280000000000003e-06 2024-09-21T22:08:11.604 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.604 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.604 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:11.604 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.475329+0000", 2024-09-21T22:08:11.604 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5054e-05 2024-09-21T22:08:11.604 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.604 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.605 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:11.605 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.475338+0000", 2024-09-21T22:08:11.605 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3770000000000002e-06 2024-09-21T22:08:11.605 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.605 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.605 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:11.605 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.475484+0000", 2024-09-21T22:08:11.605 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014612300000000001 2024-09-21T22:08:11.605 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.605 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.605 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-21T22:08:11.605 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.475506+0000", 2024-09-21T22:08:11.605 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1319999999999999e-05 2024-09-21T22:08:11.605 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.606 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.606 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:11.606 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.254433+0000", 2024-09-21T22:08:11.606 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.778927708 2024-09-21T22:08:11.606 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.606 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.606 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:11.606 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.254979+0000", 2024-09-21T22:08:11.606 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00054573100000000004 2024-09-21T22:08:11.606 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.606 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.606 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:11.606 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.255272+0000", 2024-09-21T22:08:11.606 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000293264 2024-09-21T22:08:11.606 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:11.607 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:11.607 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:11.607 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.607 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.607 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.3 67/66 [PushOp(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head, version: 63'11, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head@63'11, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T22:08:11.607 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:29.262595+0000", 2024-09-21T22:08:11.607 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 42.319073662999998, 2024-09-21T22:08:11.607 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.036027272999999999, 2024-09-21T22:08:11.607 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:11.607 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:11.607 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:08:11.607 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:11.607 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.607 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:11.608 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.262595+0000", 2024-09-21T22:08:11.608 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:11.608 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.608 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.608 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:11.608 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.262595+0000", 2024-09-21T22:08:11.608 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:11.608 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.608 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.608 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:11.608 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.262596+0000", 2024-09-21T22:08:11.608 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4300000000000001e-06 2024-09-21T22:08:11.608 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.608 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.609 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:11.609 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.264505+0000", 2024-09-21T22:08:11.609 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019085650000000001 2024-09-21T22:08:11.609 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.609 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.609 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:11.609 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.264557+0000", 2024-09-21T22:08:11.609 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2040000000000003e-05 2024-09-21T22:08:11.609 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.609 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.609 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:11.609 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.264589+0000", 2024-09-21T22:08:11.609 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1636e-05 2024-09-21T22:08:11.609 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.609 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.610 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:11.610 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.264778+0000", 2024-09-21T22:08:11.610 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018935600000000001 2024-09-21T22:08:11.610 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.610 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.610 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:11.610 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.264803+0000", 2024-09-21T22:08:11.610 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4692999999999999e-05 2024-09-21T22:08:11.610 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.610 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.610 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:11.610 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.298622+0000", 2024-09-21T22:08:11.610 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.033819553000000002 2024-09-21T22:08:11.610 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:11.611 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:11.611 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:11.611 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.611 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.611 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.3 67/66 [PushOp(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head, version: 63'11, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head@63'11, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:08:11.611 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:29.301490+0000", 2024-09-21T22:08:11.611 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 42.280178876999997, 2024-09-21T22:08:11.611 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.041178293999999997, 2024-09-21T22:08:11.611 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:11.611 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:11.611 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:08:11.611 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:11.611 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.611 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:11.612 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.301490+0000", 2024-09-21T22:08:11.612 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:11.612 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.612 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.612 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:11.612 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.301490+0000", 2024-09-21T22:08:11.612 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:11.612 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.612 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.612 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:11.612 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.301491+0000", 2024-09-21T22:08:11.612 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.513e-06 2024-09-21T22:08:11.612 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.612 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.613 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:11.613 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.302815+0000", 2024-09-21T22:08:11.613 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001323722 2024-09-21T22:08:11.613 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.613 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.613 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:11.613 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.302863+0000", 2024-09-21T22:08:11.613 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8449999999999999e-05 2024-09-21T22:08:11.613 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.613 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.613 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:11.613 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.302895+0000", 2024-09-21T22:08:11.613 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1072999999999999e-05 2024-09-21T22:08:11.613 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.613 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.614 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:11.614 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.303014+0000", 2024-09-21T22:08:11.614 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011907400000000001 2024-09-21T22:08:11.614 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.614 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.614 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:11.614 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.303029+0000", 2024-09-21T22:08:11.614 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5322e-05 2024-09-21T22:08:11.614 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.614 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.614 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:11.614 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.342668+0000", 2024-09-21T22:08:11.614 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.039639140000000003 2024-09-21T22:08:11.614 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:11.615 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:11.615 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:11.615 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.615 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.615 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.3 67/66 [PushOp(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head, version: 63'11, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head@63'11, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:08:11.615 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:29.345626+0000", 2024-09-21T22:08:11.615 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 42.236042142999999, 2024-09-21T22:08:11.615 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.034820216000000001, 2024-09-21T22:08:11.615 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:11.615 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:11.615 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:08:11.615 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:11.615 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.615 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:11.615 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.345626+0000", 2024-09-21T22:08:11.616 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:11.616 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.616 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.616 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:11.616 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.345626+0000", 2024-09-21T22:08:11.616 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:11.616 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.616 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.616 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:11.616 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.345628+0000", 2024-09-21T22:08:11.616 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3799999999999999e-06 2024-09-21T22:08:11.616 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.616 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.616 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:11.617 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.347215+0000", 2024-09-21T22:08:11.617 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015868760000000001 2024-09-21T22:08:11.617 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.617 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.617 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:11.617 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.347283+0000", 2024-09-21T22:08:11.617 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8267000000000002e-05 2024-09-21T22:08:11.617 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.617 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.617 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:11.617 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.347336+0000", 2024-09-21T22:08:11.617 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3455999999999998e-05 2024-09-21T22:08:11.617 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.617 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.618 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:11.618 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.347463+0000", 2024-09-21T22:08:11.618 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012614399999999999 2024-09-21T22:08:11.618 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.618 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.618 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:11.618 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.347473+0000", 2024-09-21T22:08:11.618 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0462000000000001e-05 2024-09-21T22:08:11.618 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.618 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.618 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:11.618 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.380447+0000", 2024-09-21T22:08:11.619 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.032973631000000003 2024-09-21T22:08:11.619 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:11.619 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:11.619 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:11.619 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.619 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.619 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.3 67/66 [PushOp(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head, version: 63'19, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head@63'19, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T22:08:11.619 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:29.420098+0000", 2024-09-21T22:08:11.619 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 42.161570308000002, 2024-09-21T22:08:11.619 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.067283846999999994, 2024-09-21T22:08:11.619 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:11.619 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:11.619 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:08:11.619 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:11.620 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.620 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:11.620 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.420098+0000", 2024-09-21T22:08:11.620 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:11.620 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.620 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.620 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:11.620 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.420098+0000", 2024-09-21T22:08:11.620 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:11.620 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.620 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.620 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:11.620 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.420100+0000", 2024-09-21T22:08:11.620 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.716e-06 2024-09-21T22:08:11.621 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.621 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.621 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:11.621 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.421949+0000", 2024-09-21T22:08:11.621 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0018485190000000001 2024-09-21T22:08:11.621 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.621 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.621 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:11.621 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.421996+0000", 2024-09-21T22:08:11.621 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7089000000000003e-05 2024-09-21T22:08:11.621 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.621 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.621 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:11.621 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.422042+0000", 2024-09-21T22:08:11.621 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6010000000000002e-05 2024-09-21T22:08:11.622 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.622 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.622 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:11.622 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.422231+0000", 2024-09-21T22:08:11.622 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018960900000000001 2024-09-21T22:08:11.622 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.622 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.622 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:11.622 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.422254+0000", 2024-09-21T22:08:11.622 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2864e-05 2024-09-21T22:08:11.622 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.622 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.622 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:11.622 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.487382+0000", 2024-09-21T22:08:11.623 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.065128039999999998 2024-09-21T22:08:11.623 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:11.623 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:11.623 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:11.623 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.623 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.623 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.3 67/66 [PushOp(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head, version: 63'19, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head@63'19, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:08:11.623 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:29.491783+0000", 2024-09-21T22:08:11.623 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 42.089885963, 2024-09-21T22:08:11.623 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.04476596, 2024-09-21T22:08:11.623 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:11.623 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:11.623 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:08:11.623 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:11.623 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.624 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:11.624 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.491783+0000", 2024-09-21T22:08:11.624 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:11.624 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.624 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.624 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:11.624 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.491783+0000", 2024-09-21T22:08:11.624 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:11.624 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.624 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.624 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:11.624 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.491784+0000", 2024-09-21T22:08:11.624 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.17e-06 2024-09-21T22:08:11.624 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.625 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.625 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:11.625 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.493126+0000", 2024-09-21T22:08:11.625 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0013424489999999999 2024-09-21T22:08:11.625 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.625 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.625 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:11.625 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.493170+0000", 2024-09-21T22:08:11.625 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4153000000000002e-05 2024-09-21T22:08:11.625 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.625 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.625 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:11.625 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.493218+0000", 2024-09-21T22:08:11.625 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7392999999999998e-05 2024-09-21T22:08:11.626 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.626 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.626 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:11.626 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.493392+0000", 2024-09-21T22:08:11.626 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017392699999999999 2024-09-21T22:08:11.626 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.626 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.626 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:11.626 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.493416+0000", 2024-09-21T22:08:11.626 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4117e-05 2024-09-21T22:08:11.626 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.626 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.626 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:11.626 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.536549+0000", 2024-09-21T22:08:11.627 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.043132750999999997 2024-09-21T22:08:11.627 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:11.627 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:11.627 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:11.627 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.627 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.627 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.3 67/66 [PushOp(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head, version: 63'19, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head@63'19, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:08:11.627 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:29.539831+0000", 2024-09-21T22:08:11.627 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 42.041837203, 2024-09-21T22:08:11.627 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.036233222000000002, 2024-09-21T22:08:11.627 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:11.627 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:11.627 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:08:11.627 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:11.628 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.628 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:11.628 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.539831+0000", 2024-09-21T22:08:11.628 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:11.628 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.628 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.628 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:11.628 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.539831+0000", 2024-09-21T22:08:11.628 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:11.628 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.628 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.628 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:11.628 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.539833+0000", 2024-09-21T22:08:11.628 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5200000000000001e-06 2024-09-21T22:08:11.628 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.629 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.629 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:11.629 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.541130+0000", 2024-09-21T22:08:11.629 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0012972719999999999 2024-09-21T22:08:11.629 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.629 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.629 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:11.629 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.541179+0000", 2024-09-21T22:08:11.629 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8507000000000002e-05 2024-09-21T22:08:11.629 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.629 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.629 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:11.629 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.541226+0000", 2024-09-21T22:08:11.629 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7663000000000001e-05 2024-09-21T22:08:11.629 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.630 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.630 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:11.630 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.541361+0000", 2024-09-21T22:08:11.630 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000135065 2024-09-21T22:08:11.630 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.630 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.630 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:11.630 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.541374+0000", 2024-09-21T22:08:11.630 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2295e-05 2024-09-21T22:08:11.630 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.630 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.630 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:11.630 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.576065+0000", 2024-09-21T22:08:11.630 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.034690899999999997 2024-09-21T22:08:11.631 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:11.631 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:11.631 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:11.631 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.631 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.631 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.3 67/66 [PushOp(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head, version: 63'19, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head@63'19, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 4194304, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:08:11.631 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:29.579455+0000", 2024-09-21T22:08:11.631 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 42.002213894999997, 2024-09-21T22:08:11.631 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.034295594999999998, 2024-09-21T22:08:11.631 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:11.631 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:11.631 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:08:11.631 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:11.631 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.632 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:11.632 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.579455+0000", 2024-09-21T22:08:11.632 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:11.632 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.632 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.632 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:11.632 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.579455+0000", 2024-09-21T22:08:11.632 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:11.632 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.632 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.632 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:11.632 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.579456+0000", 2024-09-21T22:08:11.632 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.404e-06 2024-09-21T22:08:11.632 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.632 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.633 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:11.633 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.580763+0000", 2024-09-21T22:08:11.633 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001307325 2024-09-21T22:08:11.633 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.633 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.633 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:11.633 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.580811+0000", 2024-09-21T22:08:11.633 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7843000000000003e-05 2024-09-21T22:08:11.633 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.633 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.633 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:11.633 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.580860+0000", 2024-09-21T22:08:11.633 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9036999999999998e-05 2024-09-21T22:08:11.633 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.633 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.634 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:11.634 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.580988+0000", 2024-09-21T22:08:11.634 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000127529 2024-09-21T22:08:11.634 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.634 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.634 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:11.634 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.580998+0000", 2024-09-21T22:08:11.634 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0617e-05 2024-09-21T22:08:11.634 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.634 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.634 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:11.634 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.613750+0000", 2024-09-21T22:08:11.634 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.032751839999999997 2024-09-21T22:08:11.634 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:11.635 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:11.635 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:11.635 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.635 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.635 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5008.0:125 2.3 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-21T22:08:11.635 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:38.066298+0000", 2024-09-21T22:08:11.635 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 33.515370230999999, 2024-09-21T22:08:11.635 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.062817136999999995, 2024-09-21T22:08:11.635 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:11.635 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:11.635 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:11.635 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:11.635 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5008", 2024-09-21T22:08:11.635 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/1400114666", 2024-09-21T22:08:11.636 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 125 2024-09-21T22:08:11.636 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.636 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:11.636 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.636 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:11.636 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.066298+0000", 2024-09-21T22:08:11.636 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:11.636 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.636 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.636 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:11.636 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.066298+0000", 2024-09-21T22:08:11.636 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:11.636 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.636 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.636 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:11.637 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.066300+0000", 2024-09-21T22:08:11.637 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.474e-06 2024-09-21T22:08:11.637 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.637 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.637 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:11.637 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.067443+0000", 2024-09-21T22:08:11.637 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011428009999999999 2024-09-21T22:08:11.637 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.637 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.637 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:11.637 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.067477+0000", 2024-09-21T22:08:11.637 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4059e-05 2024-09-21T22:08:11.637 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.637 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.637 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:11.638 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.067492+0000", 2024-09-21T22:08:11.638 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5054e-05 2024-09-21T22:08:11.638 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.638 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.638 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:11.638 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.067638+0000", 2024-09-21T22:08:11.638 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014583900000000001 2024-09-21T22:08:11.638 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.638 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.638 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:11.638 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.067882+0000", 2024-09-21T22:08:11.638 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024415500000000002 2024-09-21T22:08:11.638 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.638 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.639 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-09-21T22:08:11.639 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.068247+0000", 2024-09-21T22:08:11.639 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00036570800000000002 2024-09-21T22:08:11.639 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.639 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.639 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:11.639 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.119971+0000", 2024-09-21T22:08:11.639 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.051723866 2024-09-21T22:08:11.639 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.639 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.639 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:11.639 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.128849+0000", 2024-09-21T22:08:11.639 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0088779489999999996 2024-09-21T22:08:11.640 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.640 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.640 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:11.640 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.129000+0000", 2024-09-21T22:08:11.640 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015062499999999999 2024-09-21T22:08:11.640 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.640 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.640 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:11.640 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.129116+0000", 2024-09-21T22:08:11.640 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011560700000000001 2024-09-21T22:08:11.640 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:11.640 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:11.640 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:11.640 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.640 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.641 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5008.0:127 2.3 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-21T22:08:11.641 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:38.140308+0000", 2024-09-21T22:08:11.641 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 33.441360129000003, 2024-09-21T22:08:11.641 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.055290512999999999, 2024-09-21T22:08:11.641 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:11.641 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:11.641 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:11.641 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:11.641 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5008", 2024-09-21T22:08:11.641 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/1400114666", 2024-09-21T22:08:11.641 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 127 2024-09-21T22:08:11.641 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.641 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:11.641 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.641 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:11.641 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.140308+0000", 2024-09-21T22:08:11.642 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:11.642 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.642 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.642 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:11.642 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.140308+0000", 2024-09-21T22:08:11.642 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:11.642 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.642 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.642 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:11.642 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.140310+0000", 2024-09-21T22:08:11.642 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6509999999999999e-06 2024-09-21T22:08:11.642 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.642 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.642 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:11.642 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.141504+0000", 2024-09-21T22:08:11.643 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011935750000000001 2024-09-21T22:08:11.643 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.643 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.643 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:11.643 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.141522+0000", 2024-09-21T22:08:11.643 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8349e-05 2024-09-21T22:08:11.643 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.643 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.643 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:11.643 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.141531+0000", 2024-09-21T22:08:11.643 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3239999999999992e-06 2024-09-21T22:08:11.643 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.643 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.643 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:11.644 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.141661+0000", 2024-09-21T22:08:11.644 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013001000000000001 2024-09-21T22:08:11.644 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.644 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.644 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:11.644 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.141907+0000", 2024-09-21T22:08:11.644 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024547499999999999 2024-09-21T22:08:11.644 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.644 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.644 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-09-21T22:08:11.644 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.142274+0000", 2024-09-21T22:08:11.644 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00036716400000000001 2024-09-21T22:08:11.644 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.644 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.644 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:11.645 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.189864+0000", 2024-09-21T22:08:11.645 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047589617000000001 2024-09-21T22:08:11.645 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.645 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.645 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:11.645 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.195406+0000", 2024-09-21T22:08:11.645 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0055425860000000004 2024-09-21T22:08:11.645 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.645 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.645 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:11.645 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.195507+0000", 2024-09-21T22:08:11.645 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000100862 2024-09-21T22:08:11.645 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.645 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.645 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:11.646 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.195599+0000", 2024-09-21T22:08:11.646 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1899999999999998e-05 2024-09-21T22:08:11.646 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:11.646 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:11.646 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:11.646 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.646 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.646 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5008.0:129 2.3 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head [write 2228224~917504 in=917504b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-21T22:08:11.646 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:38.209297+0000", 2024-09-21T22:08:11.646 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 33.372371248, 2024-09-21T22:08:11.646 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.060441054000000001, 2024-09-21T22:08:11.646 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:11.646 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:11.646 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:11.647 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:11.647 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5008", 2024-09-21T22:08:11.647 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/1400114666", 2024-09-21T22:08:11.647 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 129 2024-09-21T22:08:11.647 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.647 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:11.647 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.647 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:11.647 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.209297+0000", 2024-09-21T22:08:11.647 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:11.647 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.647 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.647 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:11.647 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.209297+0000", 2024-09-21T22:08:11.647 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:11.648 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.648 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.648 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:11.648 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.209299+0000", 2024-09-21T22:08:11.648 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.274e-06 2024-09-21T22:08:11.648 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.648 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.648 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:11.648 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.210077+0000", 2024-09-21T22:08:11.648 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00077849799999999997 2024-09-21T22:08:11.648 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.648 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.648 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:11.648 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.210094+0000", 2024-09-21T22:08:11.649 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6772000000000001e-05 2024-09-21T22:08:11.649 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.649 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.649 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:11.649 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.210103+0000", 2024-09-21T22:08:11.649 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6780000000000006e-06 2024-09-21T22:08:11.649 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.649 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.649 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:11.649 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.210254+0000", 2024-09-21T22:08:11.649 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015141699999999999 2024-09-21T22:08:11.649 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.649 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.649 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:11.649 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.210501+0000", 2024-09-21T22:08:11.650 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000246498 2024-09-21T22:08:11.650 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.650 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.650 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-09-21T22:08:11.650 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.210872+0000", 2024-09-21T22:08:11.650 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00037147500000000002 2024-09-21T22:08:11.650 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.650 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.650 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:11.650 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.269019+0000", 2024-09-21T22:08:11.650 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058147114 2024-09-21T22:08:11.650 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.650 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.650 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:11.650 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.269638+0000", 2024-09-21T22:08:11.651 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00061896499999999997 2024-09-21T22:08:11.651 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.651 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.651 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:11.651 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.269693+0000", 2024-09-21T22:08:11.651 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4786e-05 2024-09-21T22:08:11.651 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.651 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.651 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:11.651 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.269738+0000", 2024-09-21T22:08:11.651 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5577000000000001e-05 2024-09-21T22:08:11.651 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:11.651 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:11.651 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:11.652 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.652 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.652 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5008.0:131 2.3 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-21T22:08:11.652 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:38.279879+0000", 2024-09-21T22:08:11.652 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 33.301789558000003, 2024-09-21T22:08:11.652 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.054184296999999999, 2024-09-21T22:08:11.652 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:11.652 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:11.652 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:11.652 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:11.652 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5008", 2024-09-21T22:08:11.652 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/1400114666", 2024-09-21T22:08:11.652 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 131 2024-09-21T22:08:11.652 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.653 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:11.653 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.653 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:11.653 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.279879+0000", 2024-09-21T22:08:11.653 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:11.653 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.653 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.653 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:11.653 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.279879+0000", 2024-09-21T22:08:11.653 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:11.653 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.653 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.653 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:11.653 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.279880+0000", 2024-09-21T22:08:11.653 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.136e-06 2024-09-21T22:08:11.654 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.654 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.654 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:11.654 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.280636+0000", 2024-09-21T22:08:11.654 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00075625900000000001 2024-09-21T22:08:11.654 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.654 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.654 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:11.654 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.280648+0000", 2024-09-21T22:08:11.654 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2026e-05 2024-09-21T22:08:11.654 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.654 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.654 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:11.654 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.280654+0000", 2024-09-21T22:08:11.654 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8839999999999999e-06 2024-09-21T22:08:11.655 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.655 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.655 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:11.655 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.280739+0000", 2024-09-21T22:08:11.655 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4196000000000005e-05 2024-09-21T22:08:11.655 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.655 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.655 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:11.655 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.280808+0000", 2024-09-21T22:08:11.655 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9685999999999997e-05 2024-09-21T22:08:11.655 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.655 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.655 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-09-21T22:08:11.655 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.280900+0000", 2024-09-21T22:08:11.655 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1407000000000001e-05 2024-09-21T22:08:11.656 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.656 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.656 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:11.656 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.307730+0000", 2024-09-21T22:08:11.656 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026830568999999999 2024-09-21T22:08:11.656 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.656 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.656 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:11.656 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.333901+0000", 2024-09-21T22:08:11.656 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026170787000000001 2024-09-21T22:08:11.656 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.656 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.656 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:11.656 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.333996+0000", 2024-09-21T22:08:11.657 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4723999999999996e-05 2024-09-21T22:08:11.657 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.657 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.657 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:11.657 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.334063+0000", 2024-09-21T22:08:11.657 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7623e-05 2024-09-21T22:08:11.657 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:11.657 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:11.657 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:11.657 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.657 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.657 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.1 87/86 [PushOp(2:99eff0b8:ns::rbd_data.127f79bc213d.0000000000000002:head, version: 57'3, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:99eff0b8:ns::rbd_data.127f79bc213d.0000000000000002:head@57'3, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T22:08:11.657 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:08:07.091918+0000", 2024-09-21T22:08:11.657 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.4897511149999998, 2024-09-21T22:08:11.658 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.071321166000000005, 2024-09-21T22:08:11.658 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:11.658 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:11.658 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:08:11.658 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:11.658 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.658 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:11.658 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.091918+0000", 2024-09-21T22:08:11.658 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:11.658 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.658 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.658 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:11.658 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.091918+0000", 2024-09-21T22:08:11.659 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:11.659 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.659 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.659 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:11.659 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.091919+0000", 2024-09-21T22:08:11.659 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.57e-06 2024-09-21T22:08:11.659 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.659 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.659 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:11.659 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.093627+0000", 2024-09-21T22:08:11.659 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001708162 2024-09-21T22:08:11.659 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.659 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.659 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:11.660 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.093675+0000", 2024-09-21T22:08:11.660 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.761e-05 2024-09-21T22:08:11.660 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.660 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.660 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:11.660 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.093707+0000", 2024-09-21T22:08:11.660 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1847000000000002e-05 2024-09-21T22:08:11.660 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.660 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.660 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:11.660 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.093899+0000", 2024-09-21T22:08:11.660 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000192449 2024-09-21T22:08:11.660 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.660 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.660 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:11.661 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.093922+0000", 2024-09-21T22:08:11.661 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2976000000000002e-05 2024-09-21T22:08:11.661 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.661 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.661 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:11.661 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.163239+0000", 2024-09-21T22:08:11.661 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.069316552000000003 2024-09-21T22:08:11.661 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:11.661 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:11.661 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:11.661 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.661 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.661 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.1 87/86 [PushOp(2:99eff0b8:ns::rbd_data.127f79bc213d.0000000000000002:head, version: 57'3, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:99eff0b8:ns::rbd_data.127f79bc213d.0000000000000002:head@57'3, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:08:11.662 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:08:07.167681+0000", 2024-09-21T22:08:11.662 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.4139875289999999, 2024-09-21T22:08:11.662 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.043679153999999998, 2024-09-21T22:08:11.662 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:11.662 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:11.662 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:08:11.662 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:11.662 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.662 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:11.662 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.167681+0000", 2024-09-21T22:08:11.662 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:11.662 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.662 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.662 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:11.663 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.167681+0000", 2024-09-21T22:08:11.663 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:11.663 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.663 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.663 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:11.663 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.167683+0000", 2024-09-21T22:08:11.663 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.59e-06 2024-09-21T22:08:11.663 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.663 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.663 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:11.663 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.168793+0000", 2024-09-21T22:08:11.663 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011100229999999999 2024-09-21T22:08:11.663 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.664 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.664 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:11.664 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.168830+0000", 2024-09-21T22:08:11.664 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7580999999999998e-05 2024-09-21T22:08:11.664 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.664 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.664 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:11.664 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.168852+0000", 2024-09-21T22:08:11.664 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1766000000000001e-05 2024-09-21T22:08:11.664 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.664 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.664 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:11.664 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.168987+0000", 2024-09-21T22:08:11.664 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000135203 2024-09-21T22:08:11.664 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.665 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.665 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:11.665 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.169003+0000", 2024-09-21T22:08:11.665 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.543e-05 2024-09-21T22:08:11.665 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.665 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.665 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:11.665 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.211360+0000", 2024-09-21T22:08:11.665 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.042357561000000002 2024-09-21T22:08:11.665 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:11.665 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:11.665 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:11.665 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.665 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.666 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.1 87/86 [PushOp(2:99eff0b8:ns::rbd_data.127f79bc213d.0000000000000002:head, version: 57'3, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:99eff0b8:ns::rbd_data.127f79bc213d.0000000000000002:head@57'3, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:08:11.666 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:08:07.215689+0000", 2024-09-21T22:08:11.666 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.3659793750000002, 2024-09-21T22:08:11.666 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.052501277999999998, 2024-09-21T22:08:11.666 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:11.666 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:11.666 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:08:11.666 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:11.666 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.666 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:11.666 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.215689+0000", 2024-09-21T22:08:11.666 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:11.666 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.666 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.666 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:11.667 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.215689+0000", 2024-09-21T22:08:11.667 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:11.667 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.667 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.667 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:11.667 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.215690+0000", 2024-09-21T22:08:11.667 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1319999999999999e-06 2024-09-21T22:08:11.667 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.667 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.667 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:11.667 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.216711+0000", 2024-09-21T22:08:11.667 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0010210600000000001 2024-09-21T22:08:11.667 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.667 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.668 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:11.668 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.216748+0000", 2024-09-21T22:08:11.668 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6776000000000002e-05 2024-09-21T22:08:11.668 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.668 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.668 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:11.668 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.216769+0000", 2024-09-21T22:08:11.668 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1189999999999999e-05 2024-09-21T22:08:11.668 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.668 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.668 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:11.668 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.216932+0000", 2024-09-21T22:08:11.668 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016283700000000001 2024-09-21T22:08:11.668 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.669 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.669 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:11.669 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.216957+0000", 2024-09-21T22:08:11.669 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4896999999999999e-05 2024-09-21T22:08:11.669 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.669 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.669 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:11.669 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.268191+0000", 2024-09-21T22:08:11.669 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.051233385999999999 2024-09-21T22:08:11.669 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:11.669 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:11.669 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:11.669 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.669 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.669 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.1 87/86 [PushOp(2:922055e2:ns::rbd_data.127f79bc213d.0000000000000009:head, version: 57'4, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:922055e2:ns::rbd_data.127f79bc213d.0000000000000009:head@57'4, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T22:08:11.670 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:08:07.302602+0000", 2024-09-21T22:08:11.670 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 4.2790662399999997, 2024-09-21T22:08:11.670 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.038952326000000002, 2024-09-21T22:08:11.670 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:11.670 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:11.670 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:08:11.670 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:11.670 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.670 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:11.670 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.302602+0000", 2024-09-21T22:08:11.670 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:11.670 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.670 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.670 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:11.671 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.302602+0000", 2024-09-21T22:08:11.671 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:11.671 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.671 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.671 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:11.671 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.302604+0000", 2024-09-21T22:08:11.671 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9769999999999999e-06 2024-09-21T22:08:11.671 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.671 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.671 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:11.671 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.303971+0000", 2024-09-21T22:08:11.671 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001366719 2024-09-21T22:08:11.671 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.671 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.672 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:11.672 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.304022+0000", 2024-09-21T22:08:11.672 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1307999999999997e-05 2024-09-21T22:08:11.672 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.672 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.672 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:11.672 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.304065+0000", 2024-09-21T22:08:11.672 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.214e-05 2024-09-21T22:08:11.672 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.672 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.672 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:11.672 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.304200+0000", 2024-09-21T22:08:11.672 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000135038 2024-09-21T22:08:11.672 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.672 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.673 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:11.673 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.304217+0000", 2024-09-21T22:08:11.673 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7856e-05 2024-09-21T22:08:11.673 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:11.673 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:11.673 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:11.673 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.341555+0000", 2024-09-21T22:08:11.673 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.037337288000000003 2024-09-21T22:08:11.673 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:11.673 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:11.673 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:11.673 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:11.673 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:11.673 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:08:11.674 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T22:08:11.675 INFO:tasks.workunit.client.0.smithi157.stderr:+ PID=24040 2024-09-21T22:08:11.675 INFO:tasks.workunit.client.0.smithi157.stderr:+ test -n 24040 2024-09-21T22:08:11.675 INFO:tasks.workunit.client.0.smithi157.stderr:+ ps -p 24040 -C rbd-nbd 2024-09-21T22:08:11.676 INFO:tasks.workunit.client.0.smithi157.stdout: PID TTY TIME CMD 2024-09-21T22:08:11.676 INFO:tasks.workunit.client.0.smithi157.stdout: 24040 ? 00:00:00 rbd-nbd 2024-09-21T22:08:11.677 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:08:11.672+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:08:11.677 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T22:08:11.678 INFO:tasks.workunit.client.0.smithi157.stderr:+ _sudo rbd device detach /dev/nbd1 --device-type nbd 2024-09-21T22:08:11.678 INFO:tasks.workunit.client.0.smithi157.stderr:+ local cmd 2024-09-21T22:08:11.678 INFO:tasks.workunit.client.0.smithi157.stderr:++ id -u 2024-09-21T22:08:11.678 INFO:tasks.workunit.client.0.smithi157.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T22:08:11.678 INFO:tasks.workunit.client.0.smithi157.stderr:++ which rbd 2024-09-21T22:08:11.678 INFO:tasks.workunit.client.0.smithi157.stderr:+ cmd=/usr/bin/rbd 2024-09-21T22:08:11.678 INFO:tasks.workunit.client.0.smithi157.stderr:+ shift 2024-09-21T22:08:11.678 INFO:tasks.workunit.client.0.smithi157.stderr:+ sudo -nE /usr/bin/rbd device detach /dev/nbd1 --device-type nbd 2024-09-21T22:08:11.753 INFO:tasks.workunit.client.0.smithi157.stderr:+ expect_false get_pid rbd 2024-09-21T22:08:11.753 INFO:tasks.workunit.client.0.smithi157.stderr:+ get_pid rbd 2024-09-21T22:08:11.753 INFO:tasks.workunit.client.0.smithi157.stderr:+ local pool=rbd 2024-09-21T22:08:11.753 INFO:tasks.workunit.client.0.smithi157.stderr:+ local ns= 2024-09-21T22:08:11.753 INFO:tasks.workunit.client.0.smithi157.stderr:++ rbd device --device-type nbd --format xml list 2024-09-21T22:08:11.754 INFO:tasks.workunit.client.0.smithi157.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd21553'\''][device='\''/dev/nbd1'\'']/id' 2024-09-21T22:08:11.776 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:08:11.776+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:08:11.813 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:08:11.813 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:08:11.813 INFO:teuthology.orchestra.run.smithi039.stdout: "num_ops": 0 2024-09-21T22:08:11.813 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:08:11.824 INFO:tasks.workunit.client.0.smithi157.stderr:+ PID= 2024-09-21T22:08:11.824 INFO:tasks.workunit.client.0.smithi157.stderr:+ test -n '' 2024-09-21T22:08:11.824 INFO:tasks.workunit.client.0.smithi157.stderr:+ return 1 2024-09-21T22:08:11.824 INFO:tasks.workunit.client.0.smithi157.stderr:+ return 0 2024-09-21T22:08:11.824 INFO:tasks.workunit.client.0.smithi157.stderr:+ '[' -n 15f2d4a2-c1e3-4632-98ee-ccbdf87d457c ']' 2024-09-21T22:08:11.824 INFO:tasks.workunit.client.0.smithi157.stderr:+ _sudo rbd device attach --device /dev/nbd1 --cookie 15f2d4a2-c1e3-4632-98ee-ccbdf87d457c rbd/testrbdnbd21553 --device-type nbd 2024-09-21T22:08:11.824 INFO:tasks.workunit.client.0.smithi157.stderr:+ local cmd 2024-09-21T22:08:11.824 INFO:tasks.workunit.client.0.smithi157.stderr:++ id -u 2024-09-21T22:08:11.825 INFO:tasks.workunit.client.0.smithi157.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T22:08:11.825 INFO:tasks.workunit.client.0.smithi157.stderr:++ which rbd 2024-09-21T22:08:11.826 INFO:tasks.workunit.client.0.smithi157.stderr:+ cmd=/usr/bin/rbd 2024-09-21T22:08:11.826 INFO:tasks.workunit.client.0.smithi157.stderr:+ shift 2024-09-21T22:08:11.826 INFO:tasks.workunit.client.0.smithi157.stderr:+ sudo -nE /usr/bin/rbd device attach --device /dev/nbd1 --cookie 15f2d4a2-c1e3-4632-98ee-ccbdf87d457c rbd/testrbdnbd21553 --device-type nbd 2024-09-21T22:08:11.829 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T22:08:11.841 INFO:teuthology.orchestra.run.smithi039.stdout:osd.0: {} 2024-09-21T22:08:11.841 INFO:teuthology.orchestra.run.smithi039.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T22:08:11.872 INFO:teuthology.orchestra.run.smithi039.stdout:osd.1: {} 2024-09-21T22:08:11.873 INFO:teuthology.orchestra.run.smithi039.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T22:08:11.873 INFO:teuthology.orchestra.run.smithi039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-21T22:08:11.876 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:08:11.872+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:08:11.910 INFO:teuthology.orchestra.run.smithi039.stdout:osd.3: {} 2024-09-21T22:08:11.910 INFO:teuthology.orchestra.run.smithi039.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T22:08:11.948 INFO:teuthology.orchestra.run.smithi039.stdout:osd.4: {} 2024-09-21T22:08:11.949 INFO:teuthology.orchestra.run.smithi039.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T22:08:11.956 INFO:tasks.workunit.client.0.smithi157.stdout:/dev/nbd1 2024-09-21T22:08:11.963 INFO:tasks.workunit.client.0.smithi157.stderr:+ get_pid rbd 2024-09-21T22:08:11.963 INFO:tasks.workunit.client.0.smithi157.stderr:+ local pool=rbd 2024-09-21T22:08:11.963 INFO:tasks.workunit.client.0.smithi157.stderr:+ local ns= 2024-09-21T22:08:11.964 INFO:tasks.workunit.client.0.smithi157.stderr:++ rbd device --device-type nbd --format xml list 2024-09-21T22:08:11.964 INFO:tasks.workunit.client.0.smithi157.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd21553'\''][device='\''/dev/nbd1'\'']/id' 2024-09-21T22:08:11.977 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:08:11.972+0000 7f2075a06640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31720) UID: 0 2024-09-21T22:08:11.984 INFO:teuthology.orchestra.run.smithi039.stdout:osd.5: {} 2024-09-21T22:08:11.984 INFO:teuthology.orchestra.run.smithi039.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T22:08:11.987 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T22:08:11.987 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T22:08:11.992 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:11.992 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:12.006 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T22:08:12.043 INFO:tasks.workunit.client.0.smithi157.stderr:+ PID=24098 2024-09-21T22:08:12.044 INFO:tasks.workunit.client.0.smithi157.stderr:+ test -n 24098 2024-09-21T22:08:12.044 INFO:tasks.workunit.client.0.smithi157.stderr:+ ps -p 24098 -C rbd-nbd 2024-09-21T22:08:12.051 INFO:tasks.workunit.client.0.smithi157.stdout: PID TTY TIME CMD 2024-09-21T22:08:12.051 INFO:tasks.workunit.client.0.smithi157.stdout: 24098 ? 00:00:00 rbd-nbd 2024-09-21T22:08:12.051 INFO:tasks.workunit.client.0.smithi157.stderr:+ ls /tmp/tmp.GqKuv7rsZi/mnt/ 2024-09-21T22:08:12.054 INFO:tasks.workunit.client.0.smithi157.stdout:lost+found 2024-09-21T22:08:12.054 INFO:tasks.workunit.client.0.smithi157.stdout:test 2024-09-21T22:08:12.054 INFO:tasks.workunit.client.0.smithi157.stderr:+ dd if=/tmp/tmp.GqKuv7rsZi/mnt/test of=/dev/null bs=1M count=1 2024-09-21T22:08:12.067 INFO:tasks.workunit.client.0.smithi157.stderr:1+0 records in 2024-09-21T22:08:12.067 INFO:tasks.workunit.client.0.smithi157.stderr:1+0 records out 2024-09-21T22:08:12.067 INFO:tasks.workunit.client.0.smithi157.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0126104 s, 83.2 MB/s 2024-09-21T22:08:12.068 INFO:tasks.workunit.client.0.smithi157.stderr:+ _sudo dd if=/tmp/tmp.GqKuv7rsZi/data of=/tmp/tmp.GqKuv7rsZi/mnt/test1 bs=1M count=1 oflag=direct 2024-09-21T22:08:12.068 INFO:tasks.workunit.client.0.smithi157.stderr:+ local cmd 2024-09-21T22:08:12.068 INFO:tasks.workunit.client.0.smithi157.stderr:++ id -u 2024-09-21T22:08:12.068 INFO:tasks.workunit.client.0.smithi157.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T22:08:12.069 INFO:tasks.workunit.client.0.smithi157.stderr:++ which dd 2024-09-21T22:08:12.070 INFO:tasks.workunit.client.0.smithi157.stderr:+ cmd=/usr/bin/dd 2024-09-21T22:08:12.070 INFO:tasks.workunit.client.0.smithi157.stderr:+ shift 2024-09-21T22:08:12.071 INFO:tasks.workunit.client.0.smithi157.stderr:+ sudo -nE /usr/bin/dd if=/tmp/tmp.GqKuv7rsZi/data of=/tmp/tmp.GqKuv7rsZi/mnt/test1 bs=1M count=1 oflag=direct 2024-09-21T22:08:12.078 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:08:12.076+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:08:12.170 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:12.170 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:12.179 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:08:12.176+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:08:12.180 INFO:tasks.workunit.client.0.smithi157.stderr:1+0 records in 2024-09-21T22:08:12.180 INFO:tasks.workunit.client.0.smithi157.stderr:1+0 records out 2024-09-21T22:08:12.180 INFO:tasks.workunit.client.0.smithi157.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.102002 s, 10.3 MB/s 2024-09-21T22:08:12.182 INFO:tasks.workunit.client.0.smithi157.stderr:+ _sudo umount /tmp/tmp.GqKuv7rsZi/mnt 2024-09-21T22:08:12.182 INFO:tasks.workunit.client.0.smithi157.stderr:+ local cmd 2024-09-21T22:08:12.182 INFO:tasks.workunit.client.0.smithi157.stderr:++ id -u 2024-09-21T22:08:12.184 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T22:08:12.185 INFO:tasks.workunit.client.0.smithi157.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T22:08:12.185 INFO:tasks.workunit.client.0.smithi157.stderr:++ which umount 2024-09-21T22:08:12.187 INFO:tasks.workunit.client.0.smithi157.stderr:+ cmd=/usr/bin/umount 2024-09-21T22:08:12.187 INFO:tasks.workunit.client.0.smithi157.stderr:+ shift 2024-09-21T22:08:12.187 INFO:tasks.workunit.client.0.smithi157.stderr:+ sudo -nE /usr/bin/umount /tmp/tmp.GqKuv7rsZi/mnt 2024-09-21T22:08:12.210 INFO:tasks.workunit.client.0.smithi157.stderr:+ unmap_device /dev/nbd1 24098 2024-09-21T22:08:12.210 INFO:tasks.workunit.client.0.smithi157.stderr:+ local args=/dev/nbd1 2024-09-21T22:08:12.210 INFO:tasks.workunit.client.0.smithi157.stderr:+ local pid=24098 2024-09-21T22:08:12.210 INFO:tasks.workunit.client.0.smithi157.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd1 2024-09-21T22:08:12.210 INFO:tasks.workunit.client.0.smithi157.stderr:+ local cmd 2024-09-21T22:08:12.210 INFO:tasks.workunit.client.0.smithi157.stderr:++ id -u 2024-09-21T22:08:12.211 INFO:tasks.workunit.client.0.smithi157.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T22:08:12.211 INFO:tasks.workunit.client.0.smithi157.stderr:++ which rbd 2024-09-21T22:08:12.213 INFO:tasks.workunit.client.0.smithi157.stderr:+ cmd=/usr/bin/rbd 2024-09-21T22:08:12.213 INFO:tasks.workunit.client.0.smithi157.stderr:+ shift 2024-09-21T22:08:12.213 INFO:tasks.workunit.client.0.smithi157.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd1 2024-09-21T22:08:12.280 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:08:12.276+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:08:12.313 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:12.313 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:12.322 INFO:tasks.workunit.client.0.smithi157.stderr:+ expect_false grep '^24098\b' 2024-09-21T22:08:12.322 INFO:tasks.workunit.client.0.smithi157.stderr:+ rbd device --device-type nbd list 2024-09-21T22:08:12.322 INFO:tasks.workunit.client.0.smithi157.stderr:+ grep '^24098\b' 2024-09-21T22:08:12.328 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T22:08:12.381 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:08:12.376+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:08:12.381 INFO:tasks.workunit.client.0.smithi157.stderr:+ return 0 2024-09-21T22:08:12.382 INFO:tasks.workunit.client.0.smithi157.stderr:+ ps -C rbd-nbd 2024-09-21T22:08:12.382 INFO:tasks.workunit.client.0.smithi157.stderr:+ expect_false grep '^ *24098\b' 2024-09-21T22:08:12.382 INFO:tasks.workunit.client.0.smithi157.stderr:+ grep '^ *24098\b' 2024-09-21T22:08:12.388 INFO:tasks.workunit.client.0.smithi157.stderr:+ return 0 2024-09-21T22:08:12.388 INFO:tasks.workunit.client.0.smithi157.stderr:+ sleep 0.5 2024-09-21T22:08:12.457 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:12.457 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:12.471 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T22:08:12.505 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:08:12.501+0000 7fe38df737c0 -1 Falling back to public interface 2024-09-21T22:08:12.505 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:08:12.501+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:08:12.583 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:08:12.580+0000 7f2075a06640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31720) UID: 0 2024-09-21T22:08:12.598 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:12.599 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:12.614 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T22:08:12.684 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:08:12.680+0000 7f2075a06640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31720) UID: 0 2024-09-21T22:08:12.751 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:12.751 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:12.765 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T22:08:12.810 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:08:12.808+0000 7f2075a06640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31720) UID: 0 2024-09-21T22:08:12.886 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:08:12.884+0000 7f2075a06640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31720) UID: 0 2024-09-21T22:08:12.889 INFO:tasks.workunit.client.0.smithi157.stderr:+ '[' -n 15f2d4a2-c1e3-4632-98ee-ccbdf87d457c ']' 2024-09-21T22:08:12.889 INFO:tasks.workunit.client.0.smithi157.stderr:++ _sudo rbd device --device-type nbd --show-cookie --cookie 'abc de' map rbd/testrbdnbd21553 2024-09-21T22:08:12.890 INFO:tasks.workunit.client.0.smithi157.stderr:++ local cmd 2024-09-21T22:08:12.890 INFO:tasks.workunit.client.0.smithi157.stderr:+++ id -u 2024-09-21T22:08:12.892 INFO:tasks.workunit.client.0.smithi157.stderr:++ '[' 1000 -eq 0 ']' 2024-09-21T22:08:12.892 INFO:tasks.workunit.client.0.smithi157.stderr:+++ which rbd 2024-09-21T22:08:12.895 INFO:tasks.workunit.client.0.smithi157.stderr:++ cmd=/usr/bin/rbd 2024-09-21T22:08:12.895 INFO:tasks.workunit.client.0.smithi157.stderr:++ shift 2024-09-21T22:08:12.895 INFO:tasks.workunit.client.0.smithi157.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd --show-cookie --cookie 'abc de' map rbd/testrbdnbd21553 2024-09-21T22:08:12.938 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:12.939 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:12.954 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T22:08:12.987 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:08:12.981+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:08:13.053 INFO:tasks.workunit.client.0.smithi157.stderr:+ OUT='/dev/nbd1 abc de' 2024-09-21T22:08:13.053 INFO:tasks.workunit.client.0.smithi157.stderr:+ read DEV ANOTHER_COOKIE 2024-09-21T22:08:13.053 INFO:tasks.workunit.client.0.smithi157.stderr:+ get_pid rbd 2024-09-21T22:08:13.053 INFO:tasks.workunit.client.0.smithi157.stderr:+ local pool=rbd 2024-09-21T22:08:13.054 INFO:tasks.workunit.client.0.smithi157.stderr:+ local ns= 2024-09-21T22:08:13.054 INFO:tasks.workunit.client.0.smithi157.stderr:++ rbd device --device-type nbd --format xml list 2024-09-21T22:08:13.054 INFO:tasks.workunit.client.0.smithi157.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd21553'\''][device='\''/dev/nbd1'\'']/id' 2024-09-21T22:08:13.087 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:08:13.081+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:08:13.126 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-21T22:08:13.126 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:13.127 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:13.142 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T22:08:13.149 INFO:tasks.workunit.client.0.smithi157.stderr:+ PID=24172 2024-09-21T22:08:13.149 INFO:tasks.workunit.client.0.smithi157.stderr:+ test -n 24172 2024-09-21T22:08:13.149 INFO:tasks.workunit.client.0.smithi157.stderr:+ ps -p 24172 -C rbd-nbd 2024-09-21T22:08:13.157 INFO:tasks.workunit.client.0.smithi157.stdout: PID TTY TIME CMD 2024-09-21T22:08:13.157 INFO:tasks.workunit.client.0.smithi157.stdout: 24172 ? 00:00:00 rbd-nbd 2024-09-21T22:08:13.158 INFO:tasks.workunit.client.0.smithi157.stderr:+ test 'abc de' = 'abc de' 2024-09-21T22:08:13.158 INFO:tasks.workunit.client.0.smithi157.stderr:+ unmap_device /dev/nbd1 24172 2024-09-21T22:08:13.158 INFO:tasks.workunit.client.0.smithi157.stderr:+ local args=/dev/nbd1 2024-09-21T22:08:13.158 INFO:tasks.workunit.client.0.smithi157.stderr:+ local pid=24172 2024-09-21T22:08:13.158 INFO:tasks.workunit.client.0.smithi157.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd1 2024-09-21T22:08:13.158 INFO:tasks.workunit.client.0.smithi157.stderr:+ local cmd 2024-09-21T22:08:13.158 INFO:tasks.workunit.client.0.smithi157.stderr:++ id -u 2024-09-21T22:08:13.159 INFO:tasks.workunit.client.0.smithi157.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T22:08:13.159 INFO:tasks.workunit.client.0.smithi157.stderr:++ which rbd 2024-09-21T22:08:13.160 INFO:tasks.workunit.client.0.smithi157.stderr:+ cmd=/usr/bin/rbd 2024-09-21T22:08:13.160 INFO:tasks.workunit.client.0.smithi157.stderr:+ shift 2024-09-21T22:08:13.160 INFO:tasks.workunit.client.0.smithi157.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd1 2024-09-21T22:08:13.190 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:08:13.187+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:08:13.271 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:13.271 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:13.272 INFO:tasks.workunit.client.0.smithi157.stderr:+ rbd device --device-type nbd list 2024-09-21T22:08:13.272 INFO:tasks.workunit.client.0.smithi157.stderr:+ expect_false grep '^24172\b' 2024-09-21T22:08:13.272 INFO:tasks.workunit.client.0.smithi157.stderr:+ grep '^24172\b' 2024-09-21T22:08:13.285 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T22:08:13.290 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:08:13.285+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:08:13.337 INFO:tasks.workunit.client.0.smithi157.stderr:+ return 0 2024-09-21T22:08:13.337 INFO:tasks.workunit.client.0.smithi157.stderr:+ ps -C rbd-nbd 2024-09-21T22:08:13.337 INFO:tasks.workunit.client.0.smithi157.stderr:+ expect_false grep '^ *24172\b' 2024-09-21T22:08:13.338 INFO:tasks.workunit.client.0.smithi157.stderr:+ grep '^ *24172\b' 2024-09-21T22:08:13.343 INFO:tasks.workunit.client.0.smithi157.stderr:+ return 0 2024-09-21T22:08:13.343 INFO:tasks.workunit.client.0.smithi157.stderr:+ sleep 0.5 2024-09-21T22:08:13.391 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:08:13.385+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:08:13.450 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:13.451 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:13.465 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T22:08:13.510 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:08:13.507+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:08:13.585 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:13.585 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:13.593 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:08:13.587+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:08:13.599 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T22:08:13.694 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:08:13.691+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:08:13.771 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:13.771 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:13.785 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T22:08:13.830 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:08:13.827+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:08:13.845 INFO:tasks.workunit.client.0.smithi157.stderr:+ DEV= 2024-09-21T22:08:13.846 INFO:tasks.workunit.client.0.smithi157.stderr:++ rbd snap ls rbd/testrbdnbd21553 2024-09-21T22:08:13.846 INFO:tasks.workunit.client.0.smithi157.stderr:++ awk '$2 == "snap" {print $1}' 2024-09-21T22:08:13.895 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:08:13.889+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:08:13.903 INFO:tasks.workunit.client.0.smithi157.stderr:+ SNAPID=3 2024-09-21T22:08:13.904 INFO:tasks.workunit.client.0.smithi157.stderr:++ _sudo rbd device --device-type nbd --show-cookie map --snap-id 3 rbd/testrbdnbd21553 2024-09-21T22:08:13.904 INFO:tasks.workunit.client.0.smithi157.stderr:++ local cmd 2024-09-21T22:08:13.904 INFO:tasks.workunit.client.0.smithi157.stderr:+++ id -u 2024-09-21T22:08:13.905 INFO:tasks.workunit.client.0.smithi157.stderr:++ '[' 1000 -eq 0 ']' 2024-09-21T22:08:13.905 INFO:tasks.workunit.client.0.smithi157.stderr:+++ which rbd 2024-09-21T22:08:13.907 INFO:tasks.workunit.client.0.smithi157.stderr:++ cmd=/usr/bin/rbd 2024-09-21T22:08:13.907 INFO:tasks.workunit.client.0.smithi157.stderr:++ shift 2024-09-21T22:08:13.907 INFO:tasks.workunit.client.0.smithi157.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd --show-cookie map --snap-id 3 rbd/testrbdnbd21553 2024-09-21T22:08:13.961 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:13.961 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:13.976 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T22:08:13.988 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T22:08:14.023 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:08:14.019+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:08:14.052 INFO:tasks.workunit.client.0.smithi157.stderr:+ OUT='/dev/nbd1 9895fa88-5033-4b00-b87d-8b30bdf612c2' 2024-09-21T22:08:14.053 INFO:tasks.workunit.client.0.smithi157.stderr:+ read DEV COOKIE 2024-09-21T22:08:14.053 INFO:tasks.workunit.client.0.smithi157.stderr:+ get_pid rbd 2024-09-21T22:08:14.053 INFO:tasks.workunit.client.0.smithi157.stderr:+ local pool=rbd 2024-09-21T22:08:14.053 INFO:tasks.workunit.client.0.smithi157.stderr:+ local ns= 2024-09-21T22:08:14.053 INFO:tasks.workunit.client.0.smithi157.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd21553'\''][device='\''/dev/nbd1'\'']/id' 2024-09-21T22:08:14.053 INFO:tasks.workunit.client.0.smithi157.stderr:++ rbd device --device-type nbd --format xml list 2024-09-21T22:08:14.097 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:08:14.093+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:08:14.104 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:14.104 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:14.119 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T22:08:14.145 INFO:teuthology.orchestra.run.smithi039.stderr:noscrub is set 2024-09-21T22:08:14.146 INFO:teuthology.orchestra.run.smithi039.stdout:osd.0: {} 2024-09-21T22:08:14.146 INFO:teuthology.orchestra.run.smithi039.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T22:08:14.151 INFO:tasks.workunit.client.0.smithi157.stderr:+ PID=24256 2024-09-21T22:08:14.151 INFO:tasks.workunit.client.0.smithi157.stderr:+ test -n 24256 2024-09-21T22:08:14.151 INFO:tasks.workunit.client.0.smithi157.stderr:+ ps -p 24256 -C rbd-nbd 2024-09-21T22:08:14.159 INFO:tasks.workunit.client.0.smithi157.stdout: PID TTY TIME CMD 2024-09-21T22:08:14.159 INFO:tasks.workunit.client.0.smithi157.stdout: 24256 ? 00:00:00 rbd-nbd 2024-09-21T22:08:14.160 INFO:tasks.workunit.client.0.smithi157.stderr:+ _sudo rbd device detach rbd/testrbdnbd21553 --snap-id 3 --device-type nbd 2024-09-21T22:08:14.160 INFO:tasks.workunit.client.0.smithi157.stderr:+ local cmd 2024-09-21T22:08:14.160 INFO:tasks.workunit.client.0.smithi157.stderr:++ id -u 2024-09-21T22:08:14.160 INFO:tasks.workunit.client.0.smithi157.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T22:08:14.161 INFO:tasks.workunit.client.0.smithi157.stderr:++ which rbd 2024-09-21T22:08:14.163 INFO:tasks.workunit.client.0.smithi157.stderr:+ cmd=/usr/bin/rbd 2024-09-21T22:08:14.163 INFO:tasks.workunit.client.0.smithi157.stderr:+ shift 2024-09-21T22:08:14.163 INFO:tasks.workunit.client.0.smithi157.stderr:+ sudo -nE /usr/bin/rbd device detach rbd/testrbdnbd21553 --snap-id 3 --device-type nbd 2024-09-21T22:08:14.176 INFO:teuthology.orchestra.run.smithi039.stdout:osd.1: {} 2024-09-21T22:08:14.176 INFO:teuthology.orchestra.run.smithi039.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T22:08:14.176 INFO:teuthology.orchestra.run.smithi039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-21T22:08:14.195 INFO:teuthology.orchestra.run.smithi039.stdout:osd.3: {} 2024-09-21T22:08:14.195 INFO:teuthology.orchestra.run.smithi039.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T22:08:14.197 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:08:14.193+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:08:14.213 INFO:teuthology.orchestra.run.smithi039.stdout:osd.4: {} 2024-09-21T22:08:14.214 INFO:teuthology.orchestra.run.smithi039.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T22:08:14.235 INFO:teuthology.orchestra.run.smithi039.stdout:osd.5: {} 2024-09-21T22:08:14.235 INFO:teuthology.orchestra.run.smithi039.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T22:08:14.238 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T22:08:14.238 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T22:08:14.241 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:08:14.241 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:08:14.241 INFO:teuthology.orchestra.run.smithi039.stdout: "num_ops": 0 2024-09-21T22:08:14.241 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:08:14.256 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T22:08:14.287 INFO:tasks.workunit.client.0.smithi157.stderr:+ expect_false get_pid rbd 2024-09-21T22:08:14.287 INFO:tasks.workunit.client.0.smithi157.stderr:+ get_pid rbd 2024-09-21T22:08:14.288 INFO:tasks.workunit.client.0.smithi157.stderr:+ local pool=rbd 2024-09-21T22:08:14.288 INFO:tasks.workunit.client.0.smithi157.stderr:+ local ns= 2024-09-21T22:08:14.288 INFO:tasks.workunit.client.0.smithi157.stderr:++ rbd device --device-type nbd --format xml list 2024-09-21T22:08:14.289 INFO:tasks.workunit.client.0.smithi157.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd21553'\''][device='\''/dev/nbd1'\'']/id' 2024-09-21T22:08:14.300 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:08:14.297+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:08:14.353 INFO:tasks.workunit.client.0.smithi157.stderr:+ PID= 2024-09-21T22:08:14.353 INFO:tasks.workunit.client.0.smithi157.stderr:+ test -n '' 2024-09-21T22:08:14.353 INFO:tasks.workunit.client.0.smithi157.stderr:+ return 1 2024-09-21T22:08:14.353 INFO:tasks.workunit.client.0.smithi157.stderr:+ return 0 2024-09-21T22:08:14.354 INFO:tasks.workunit.client.0.smithi157.stderr:+ expect_false _sudo rbd device attach --device /dev/nbd1 --snap-id 3 rbd/testrbdnbd21553 --device-type nbd 2024-09-21T22:08:14.354 INFO:tasks.workunit.client.0.smithi157.stderr:+ _sudo rbd device attach --device /dev/nbd1 --snap-id 3 rbd/testrbdnbd21553 --device-type nbd 2024-09-21T22:08:14.354 INFO:tasks.workunit.client.0.smithi157.stderr:+ local cmd 2024-09-21T22:08:14.354 INFO:tasks.workunit.client.0.smithi157.stderr:++ id -u 2024-09-21T22:08:14.354 INFO:tasks.workunit.client.0.smithi157.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T22:08:14.354 INFO:tasks.workunit.client.0.smithi157.stderr:++ which rbd 2024-09-21T22:08:14.356 INFO:tasks.workunit.client.0.smithi157.stderr:+ cmd=/usr/bin/rbd 2024-09-21T22:08:14.356 INFO:tasks.workunit.client.0.smithi157.stderr:+ shift 2024-09-21T22:08:14.356 INFO:tasks.workunit.client.0.smithi157.stderr:+ sudo -nE /usr/bin/rbd device attach --device /dev/nbd1 --snap-id 3 rbd/testrbdnbd21553 --device-type nbd 2024-09-21T22:08:14.394 INFO:tasks.workunit.client.0.smithi157.stderr:rbd: could not validate attach request 2024-09-21T22:08:14.395 INFO:tasks.workunit.client.0.smithi157.stderr:rbd: mismatching the image and the device may lead to data corruption 2024-09-21T22:08:14.395 INFO:tasks.workunit.client.0.smithi157.stderr:rbd: must specify --cookie or --force to proceed 2024-09-21T22:08:14.397 INFO:tasks.workunit.client.0.smithi157.stderr:+ return 0 2024-09-21T22:08:14.397 INFO:tasks.workunit.client.0.smithi157.stderr:+ '[' -n 9895fa88-5033-4b00-b87d-8b30bdf612c2 ']' 2024-09-21T22:08:14.397 INFO:tasks.workunit.client.0.smithi157.stderr:+ _sudo rbd device attach --device /dev/nbd1 --cookie 9895fa88-5033-4b00-b87d-8b30bdf612c2 --snap-id 3 rbd/testrbdnbd21553 --device-type nbd 2024-09-21T22:08:14.397 INFO:tasks.workunit.client.0.smithi157.stderr:+ local cmd 2024-09-21T22:08:14.397 INFO:tasks.workunit.client.0.smithi157.stderr:++ id -u 2024-09-21T22:08:14.398 INFO:tasks.workunit.client.0.smithi157.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T22:08:14.399 INFO:tasks.workunit.client.0.smithi157.stderr:++ which rbd 2024-09-21T22:08:14.399 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:08:14.399+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:08:14.400 INFO:tasks.workunit.client.0.smithi157.stderr:+ cmd=/usr/bin/rbd 2024-09-21T22:08:14.400 INFO:tasks.workunit.client.0.smithi157.stderr:+ shift 2024-09-21T22:08:14.400 INFO:tasks.workunit.client.0.smithi157.stderr:+ sudo -nE /usr/bin/rbd device attach --device /dev/nbd1 --cookie 9895fa88-5033-4b00-b87d-8b30bdf612c2 --snap-id 3 rbd/testrbdnbd21553 --device-type nbd 2024-09-21T22:08:14.421 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:08:14.421 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:08:14.421 INFO:teuthology.orchestra.run.smithi039.stdout: "complaint_time": 30, 2024-09-21T22:08:14.421 INFO:teuthology.orchestra.run.smithi039.stdout: "num_blocked_ops": 0 2024-09-21T22:08:14.421 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:08:14.436 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T22:08:14.500 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:08:14.497+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:08:14.545 INFO:tasks.workunit.client.0.smithi157.stdout:/dev/nbd1 2024-09-21T22:08:14.550 INFO:tasks.workunit.client.0.smithi157.stderr:+ get_pid rbd 2024-09-21T22:08:14.551 INFO:tasks.workunit.client.0.smithi157.stderr:+ local pool=rbd 2024-09-21T22:08:14.551 INFO:tasks.workunit.client.0.smithi157.stderr:+ local ns= 2024-09-21T22:08:14.551 INFO:tasks.workunit.client.0.smithi157.stderr:++ rbd device --device-type nbd --format xml list 2024-09-21T22:08:14.551 INFO:tasks.workunit.client.0.smithi157.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd21553'\''][device='\''/dev/nbd1'\'']/id' 2024-09-21T22:08:14.567 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:08:14.567 INFO:teuthology.orchestra.run.smithi039.stdout: "size": 20, 2024-09-21T22:08:14.567 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 600, 2024-09-21T22:08:14.568 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [ 2024-09-21T22:08:14.568 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.568 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:694 2.5 e63/37 2:b1d55633:::rbd_data.126483d75f2d.000000000000000b:head v 63'26, mlcod=63'24)", 2024-09-21T22:08:14.568 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.163199+0000", 2024-09-21T22:08:14.568 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 49.401488383999997, 2024-09-21T22:08:14.568 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.034676672999999998, 2024-09-21T22:08:14.568 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:14.568 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:14.568 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:14.568 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:14.568 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.568 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:14.568 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.163199+0000", 2024-09-21T22:08:14.569 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:14.569 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.569 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.569 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:14.569 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.163199+0000", 2024-09-21T22:08:14.569 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:14.569 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.569 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.569 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:14.569 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.163200+0000", 2024-09-21T22:08:14.569 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.703e-06 2024-09-21T22:08:14.570 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.570 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.570 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:14.570 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.164013+0000", 2024-09-21T22:08:14.570 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000812447 2024-09-21T22:08:14.570 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.570 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.570 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:14.570 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.164042+0000", 2024-09-21T22:08:14.571 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9006e-05 2024-09-21T22:08:14.571 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.571 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.571 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:14.571 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.164064+0000", 2024-09-21T22:08:14.571 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1719999999999999e-05 2024-09-21T22:08:14.571 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.571 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.572 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:14.572 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.164143+0000", 2024-09-21T22:08:14.572 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.9283999999999995e-05 2024-09-21T22:08:14.572 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.572 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.572 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:14.572 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.164165+0000", 2024-09-21T22:08:14.572 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.2379e-05 2024-09-21T22:08:14.572 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.572 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.572 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:14.573 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.197827+0000", 2024-09-21T22:08:14.573 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.033661782000000001 2024-09-21T22:08:14.573 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.573 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.573 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:14.573 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.197875+0000", 2024-09-21T22:08:14.573 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8352000000000001e-05 2024-09-21T22:08:14.573 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:14.573 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:14.573 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:14.573 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.574 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.574 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:696 2.5 e63/37 2:b1d55633:::rbd_data.126483d75f2d.000000000000000b:head v 63'28, mlcod=63'26)", 2024-09-21T22:08:14.574 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.218913+0000", 2024-09-21T22:08:14.574 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 49.345773938000001, 2024-09-21T22:08:14.574 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.056187599999999997, 2024-09-21T22:08:14.574 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:14.574 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:14.574 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:14.574 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:14.574 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.575 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:14.575 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.218913+0000", 2024-09-21T22:08:14.575 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:14.575 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.575 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.575 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:14.575 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.218913+0000", 2024-09-21T22:08:14.575 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:14.575 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.576 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.576 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:14.576 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.218914+0000", 2024-09-21T22:08:14.576 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.1930000000000001e-06 2024-09-21T22:08:14.576 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.576 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.576 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:14.576 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.219721+0000", 2024-09-21T22:08:14.577 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00080701500000000001 2024-09-21T22:08:14.577 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.577 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.577 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:14.577 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.219743+0000", 2024-09-21T22:08:14.577 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1364000000000001e-05 2024-09-21T22:08:14.577 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.577 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.577 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:14.577 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.219759+0000", 2024-09-21T22:08:14.577 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.611e-05 2024-09-21T22:08:14.577 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.577 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.578 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:14.578 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.219908+0000", 2024-09-21T22:08:14.578 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000149252 2024-09-21T22:08:14.578 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.578 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.578 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:14.578 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.219951+0000", 2024-09-21T22:08:14.578 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.2518999999999997e-05 2024-09-21T22:08:14.578 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.578 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.578 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:14.578 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.275031+0000", 2024-09-21T22:08:14.578 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.055080241000000002 2024-09-21T22:08:14.578 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.579 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.579 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:14.579 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.275101+0000", 2024-09-21T22:08:14.579 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.9906000000000005e-05 2024-09-21T22:08:14.579 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:14.579 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:14.579 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:14.579 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.579 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.579 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:698 2.5 e63/37 2:b1d55633:::rbd_data.126483d75f2d.000000000000000b:head v 63'30, mlcod=63'28)", 2024-09-21T22:08:14.579 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.289319+0000", 2024-09-21T22:08:14.579 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 49.275367729000003, 2024-09-21T22:08:14.579 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.053930944000000001, 2024-09-21T22:08:14.579 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:14.580 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:14.580 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:14.580 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:14.580 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.580 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:14.580 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.289319+0000", 2024-09-21T22:08:14.580 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:14.580 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.580 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.580 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:14.580 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.289319+0000", 2024-09-21T22:08:14.580 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:14.580 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.580 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.580 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:14.581 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.289321+0000", 2024-09-21T22:08:14.581 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.221e-06 2024-09-21T22:08:14.581 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.581 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.581 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:14.581 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.290091+0000", 2024-09-21T22:08:14.581 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00077069300000000003 2024-09-21T22:08:14.581 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.581 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.581 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:14.581 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.290109+0000", 2024-09-21T22:08:14.582 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7218999999999999e-05 2024-09-21T22:08:14.582 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.582 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.582 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:14.582 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.290122+0000", 2024-09-21T22:08:14.582 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3678e-05 2024-09-21T22:08:14.582 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.583 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.583 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:14.583 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.290274+0000", 2024-09-21T22:08:14.583 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00015205899999999999 2024-09-21T22:08:14.583 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.583 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.583 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:14.583 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.290313+0000", 2024-09-21T22:08:14.584 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.8636000000000001e-05 2024-09-21T22:08:14.584 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.584 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.584 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:14.584 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.343179+0000", 2024-09-21T22:08:14.584 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.052866199000000003 2024-09-21T22:08:14.585 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.585 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.585 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:14.585 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.343250+0000", 2024-09-21T22:08:14.585 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.1239e-05 2024-09-21T22:08:14.585 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:14.586 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:14.586 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:14.586 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.586 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.586 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:702 2.5 e63/37 2:a70ae99f:::rbd_data.126483d75f2d.000000000000000c:head v 63'34, mlcod=63'32)", 2024-09-21T22:08:14.586 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.392532+0000", 2024-09-21T22:08:14.587 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 49.172155140999998, 2024-09-21T22:08:14.587 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.035703990999999997, 2024-09-21T22:08:14.587 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:14.587 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:14.587 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:14.587 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:14.587 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.587 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:14.587 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.392532+0000", 2024-09-21T22:08:14.588 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:14.588 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.588 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.588 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:14.588 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.392532+0000", 2024-09-21T22:08:14.588 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:14.588 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.588 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.589 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:14.589 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.392534+0000", 2024-09-21T22:08:14.589 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.64e-06 2024-09-21T22:08:14.589 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.589 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.589 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:14.589 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.393321+0000", 2024-09-21T22:08:14.589 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00078705699999999997 2024-09-21T22:08:14.590 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.590 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.590 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:14.590 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.393347+0000", 2024-09-21T22:08:14.590 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.6735999999999999e-05 2024-09-21T22:08:14.590 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.590 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.590 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:14.591 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.393361+0000", 2024-09-21T22:08:14.591 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3346000000000001e-05 2024-09-21T22:08:14.591 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.591 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.591 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:14.591 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.393469+0000", 2024-09-21T22:08:14.591 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000108039 2024-09-21T22:08:14.591 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.592 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.592 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:14.592 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.393512+0000", 2024-09-21T22:08:14.592 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.2862999999999999e-05 2024-09-21T22:08:14.592 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.592 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.592 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:14.592 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.428170+0000", 2024-09-21T22:08:14.592 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.034658858000000001 2024-09-21T22:08:14.593 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.593 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.593 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:14.593 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.428236+0000", 2024-09-21T22:08:14.593 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.5451999999999996e-05 2024-09-21T22:08:14.593 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:14.593 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:14.593 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:14.594 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.594 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.594 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:708 2.1 e63/53 2:837ddd83:::rbd_data.126483d75f2d.000000000000000d:head v 63'6, mlcod=57'4)", 2024-09-21T22:08:14.594 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.517007+0000", 2024-09-21T22:08:14.594 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 49.047680215, 2024-09-21T22:08:14.594 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.032495584000000001, 2024-09-21T22:08:14.594 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:14.594 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:14.595 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:14.595 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:14.595 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.595 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:14.595 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.517007+0000", 2024-09-21T22:08:14.595 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:14.595 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.595 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.596 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:14.596 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.517007+0000", 2024-09-21T22:08:14.596 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:14.596 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.596 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.596 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:14.596 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.517008+0000", 2024-09-21T22:08:14.596 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.604e-06 2024-09-21T22:08:14.597 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.597 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.597 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:14.597 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.517603+0000", 2024-09-21T22:08:14.597 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00059461900000000001 2024-09-21T22:08:14.597 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.597 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.597 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:14.597 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.517632+0000", 2024-09-21T22:08:14.598 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8663e-05 2024-09-21T22:08:14.598 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.598 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.598 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:14.598 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.517647+0000", 2024-09-21T22:08:14.598 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4833e-05 2024-09-21T22:08:14.598 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.599 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.599 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:14.599 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.517714+0000", 2024-09-21T22:08:14.599 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.7669000000000006e-05 2024-09-21T22:08:14.599 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.599 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.599 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:14.599 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.517737+0000", 2024-09-21T22:08:14.599 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.2469000000000001e-05 2024-09-21T22:08:14.599 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.599 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.600 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:14.600 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.549370+0000", 2024-09-21T22:08:14.600 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.031633221000000003 2024-09-21T22:08:14.600 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.600 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.600 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:14.600 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.549502+0000", 2024-09-21T22:08:14.600 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013250600000000001 2024-09-21T22:08:14.600 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:14.600 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:14.600 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:14.600 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.600 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.600 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:712 2.1 e63/53 2:837ddd83:::rbd_data.126483d75f2d.000000000000000d:head v 63'10, mlcod=63'8)", 2024-09-21T22:08:14.601 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.604029+0000", 2024-09-21T22:08:14.601 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 48.960658275999997, 2024-09-21T22:08:14.601 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.037982371000000001, 2024-09-21T22:08:14.601 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:14.601 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:14.601 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:14.601 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:14.601 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.601 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:14.601 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.604029+0000", 2024-09-21T22:08:14.601 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:14.601 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.601 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.601 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:14.601 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.604029+0000", 2024-09-21T22:08:14.601 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:14.602 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.602 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.602 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:14.602 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.604031+0000", 2024-09-21T22:08:14.602 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8870000000000001e-06 2024-09-21T22:08:14.602 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.602 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.602 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:14.602 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.604395+0000", 2024-09-21T22:08:14.602 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00036422000000000001 2024-09-21T22:08:14.602 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.602 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.602 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:14.602 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.604424+0000", 2024-09-21T22:08:14.603 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8609e-05 2024-09-21T22:08:14.603 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.603 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.603 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:14.603 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.604437+0000", 2024-09-21T22:08:14.603 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3086e-05 2024-09-21T22:08:14.603 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.603 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.603 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:14.603 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.604509+0000", 2024-09-21T22:08:14.603 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.2448999999999997e-05 2024-09-21T22:08:14.603 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.603 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.603 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:14.604 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.604531+0000", 2024-09-21T22:08:14.604 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1602000000000001e-05 2024-09-21T22:08:14.604 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.604 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.604 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:14.604 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.641885+0000", 2024-09-21T22:08:14.604 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.037354545000000003 2024-09-21T22:08:14.604 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.604 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.604 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:14.604 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.642011+0000", 2024-09-21T22:08:14.604 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00012597300000000001 2024-09-21T22:08:14.604 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:14.604 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:14.604 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:14.605 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.605 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.605 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:720 2.1 e63/53 2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head v 63'18, mlcod=63'16)", 2024-09-21T22:08:14.605 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.805111+0000", 2024-09-21T22:08:14.605 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 48.759576215999999, 2024-09-21T22:08:14.605 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.046082864000000001, 2024-09-21T22:08:14.605 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:14.605 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:14.605 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:14.605 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:14.605 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.605 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:14.605 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.805111+0000", 2024-09-21T22:08:14.605 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:14.605 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.606 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.606 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:14.606 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.805111+0000", 2024-09-21T22:08:14.606 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:14.606 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.606 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.606 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:14.606 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.805112+0000", 2024-09-21T22:08:14.606 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.02e-06 2024-09-21T22:08:14.606 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.606 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.606 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:14.606 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.805429+0000", 2024-09-21T22:08:14.606 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00031740799999999998 2024-09-21T22:08:14.607 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.607 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.607 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:14.607 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.805459+0000", 2024-09-21T22:08:14.607 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.955e-05 2024-09-21T22:08:14.607 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.607 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.607 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:14.607 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.805472+0000", 2024-09-21T22:08:14.607 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3533000000000001e-05 2024-09-21T22:08:14.607 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.607 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.607 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:14.607 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.805581+0000", 2024-09-21T22:08:14.608 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000108194 2024-09-21T22:08:14.608 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.608 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.608 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:14.608 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.805615+0000", 2024-09-21T22:08:14.608 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.4310000000000002e-05 2024-09-21T22:08:14.608 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.608 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.608 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:14.608 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.851146+0000", 2024-09-21T22:08:14.608 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.045531092000000002 2024-09-21T22:08:14.608 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.608 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.608 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:14.609 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.851194+0000", 2024-09-21T22:08:14.609 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.7756999999999997e-05 2024-09-21T22:08:14.609 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:14.609 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:14.609 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:14.609 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.609 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.609 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:723 2.1 e63/53 2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head v 63'20, mlcod=63'18)", 2024-09-21T22:08:14.609 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.862341+0000", 2024-09-21T22:08:14.609 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 48.702346282000001, 2024-09-21T22:08:14.609 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.036454671000000001, 2024-09-21T22:08:14.609 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:14.609 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:14.609 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:14.609 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:14.610 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.610 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:14.610 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.862341+0000", 2024-09-21T22:08:14.610 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:14.610 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.610 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.610 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:14.610 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.862341+0000", 2024-09-21T22:08:14.610 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:14.610 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.610 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.610 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:14.610 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.862342+0000", 2024-09-21T22:08:14.611 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.0299999999999997e-07 2024-09-21T22:08:14.611 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.611 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.611 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:14.611 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.862721+0000", 2024-09-21T22:08:14.611 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00037889800000000002 2024-09-21T22:08:14.611 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.611 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.611 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:14.611 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.862738+0000", 2024-09-21T22:08:14.611 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6957999999999999e-05 2024-09-21T22:08:14.612 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.612 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.612 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:14.612 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.862747+0000", 2024-09-21T22:08:14.612 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.0729999999999994e-06 2024-09-21T22:08:14.612 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.612 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.612 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:14.612 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.862830+0000", 2024-09-21T22:08:14.612 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.3113000000000002e-05 2024-09-21T22:08:14.612 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.612 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.612 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:14.613 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.862860+0000", 2024-09-21T22:08:14.613 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9980000000000001e-05 2024-09-21T22:08:14.613 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.613 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.613 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:14.613 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.898705+0000", 2024-09-21T22:08:14.613 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.035845063000000003 2024-09-21T22:08:14.613 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.613 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.613 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:14.613 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.898795+0000", 2024-09-21T22:08:14.614 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.0682999999999999e-05 2024-09-21T22:08:14.614 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:14.614 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:14.614 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:14.614 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.614 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.614 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:136 2.5 e72/37 2:b1d55633:::rbd_data.126483d75f2d.000000000000000b:head v 72'44, mlcod=72'42)", 2024-09-21T22:08:14.614 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.386273+0000", 2024-09-21T22:08:14.614 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 36.178414167, 2024-09-21T22:08:14.614 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.033806744999999999, 2024-09-21T22:08:14.615 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:14.615 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:14.615 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:14.615 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:14.615 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.615 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:14.615 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.386273+0000", 2024-09-21T22:08:14.615 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:14.615 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.616 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.616 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:14.616 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.386273+0000", 2024-09-21T22:08:14.616 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:14.616 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.616 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.616 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:14.616 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.386274+0000", 2024-09-21T22:08:14.616 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.8299999999999996e-07 2024-09-21T22:08:14.617 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.617 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.617 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:14.617 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.387108+0000", 2024-09-21T22:08:14.617 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00083467800000000005 2024-09-21T22:08:14.617 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.617 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.617 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:14.617 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.387121+0000", 2024-09-21T22:08:14.618 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2823000000000001e-05 2024-09-21T22:08:14.618 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.618 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.618 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:14.618 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.387128+0000", 2024-09-21T22:08:14.618 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.404e-06 2024-09-21T22:08:14.618 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.618 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.618 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:14.618 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.387179+0000", 2024-09-21T22:08:14.619 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.1888000000000002e-05 2024-09-21T22:08:14.619 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.619 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.619 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:14.619 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.387201+0000", 2024-09-21T22:08:14.619 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.2019999999999999e-05 2024-09-21T22:08:14.619 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.619 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.619 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:14.619 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.419993+0000", 2024-09-21T22:08:14.620 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.032791182000000002 2024-09-21T22:08:14.620 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.620 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.620 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:14.620 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.420080+0000", 2024-09-21T22:08:14.620 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.7066999999999998e-05 2024-09-21T22:08:14.620 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:14.620 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:14.620 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:14.621 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.621 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.621 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:138 2.5 e72/37 2:b1d55633:::rbd_data.126483d75f2d.000000000000000b:head v 72'46, mlcod=72'44)", 2024-09-21T22:08:14.621 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.430104+0000", 2024-09-21T22:08:14.621 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 36.134582819000002, 2024-09-21T22:08:14.621 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.034087458000000001, 2024-09-21T22:08:14.621 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:14.621 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:14.621 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:14.621 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:14.621 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.622 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:14.622 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.430104+0000", 2024-09-21T22:08:14.622 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:14.622 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.622 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.622 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:14.622 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.430104+0000", 2024-09-21T22:08:14.622 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:14.622 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.622 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.622 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:14.622 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.430105+0000", 2024-09-21T22:08:14.623 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.1680000000000001e-06 2024-09-21T22:08:14.623 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.623 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.623 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:14.623 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.430864+0000", 2024-09-21T22:08:14.623 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00075819400000000001 2024-09-21T22:08:14.623 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.623 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.623 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:14.623 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.430881+0000", 2024-09-21T22:08:14.623 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7000999999999999e-05 2024-09-21T22:08:14.624 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.624 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.624 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:14.624 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.430889+0000", 2024-09-21T22:08:14.624 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.9540000000000004e-06 2024-09-21T22:08:14.624 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.624 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.624 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:14.624 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.430959+0000", 2024-09-21T22:08:14.624 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.0858999999999994e-05 2024-09-21T22:08:14.624 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.624 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.625 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:14.625 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.430996+0000", 2024-09-21T22:08:14.625 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.6443999999999999e-05 2024-09-21T22:08:14.625 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.625 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.625 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:14.625 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.464087+0000", 2024-09-21T22:08:14.625 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.033090799999999997 2024-09-21T22:08:14.625 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.625 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.625 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:14.626 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.464192+0000", 2024-09-21T22:08:14.626 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00010503800000000001 2024-09-21T22:08:14.626 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:14.626 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:14.626 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:14.626 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.626 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.626 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:140 2.5 e72/37 2:b1d55633:::rbd_data.126483d75f2d.000000000000000b:head v 72'48, mlcod=72'46)", 2024-09-21T22:08:14.626 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.475737+0000", 2024-09-21T22:08:14.626 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 36.088949870999997, 2024-09-21T22:08:14.626 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.037632025, 2024-09-21T22:08:14.626 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:14.627 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:14.627 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:14.627 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:14.627 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.627 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:14.627 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.475737+0000", 2024-09-21T22:08:14.627 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:14.627 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.627 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.627 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:14.627 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.475737+0000", 2024-09-21T22:08:14.627 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:14.627 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.628 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.628 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:14.628 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.475739+0000", 2024-09-21T22:08:14.628 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.666e-06 2024-09-21T22:08:14.628 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.628 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.628 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:14.628 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.476538+0000", 2024-09-21T22:08:14.628 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00079927399999999999 2024-09-21T22:08:14.628 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.628 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.628 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:14.628 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.476556+0000", 2024-09-21T22:08:14.628 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7666999999999999e-05 2024-09-21T22:08:14.628 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.629 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.629 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:14.629 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.476565+0000", 2024-09-21T22:08:14.629 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.0650000000000005e-06 2024-09-21T22:08:14.629 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.629 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.629 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:14.629 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.476651+0000", 2024-09-21T22:08:14.629 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.5916999999999997e-05 2024-09-21T22:08:14.629 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.629 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.629 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:14.629 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.476696+0000", 2024-09-21T22:08:14.629 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.5537000000000001e-05 2024-09-21T22:08:14.630 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.630 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.630 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:14.630 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.513300+0000", 2024-09-21T22:08:14.630 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.036603498999999998 2024-09-21T22:08:14.630 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.630 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.630 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:14.630 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.513369+0000", 2024-09-21T22:08:14.630 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.9400000000000006e-05 2024-09-21T22:08:14.630 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:14.630 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:14.630 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:14.630 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.631 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.631 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:145 2.5 e72/37 2:a70ae99f:::rbd_data.126483d75f2d.000000000000000c:head v 72'52, mlcod=72'50)", 2024-09-21T22:08:14.631 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.578171+0000", 2024-09-21T22:08:14.631 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 35.986516021, 2024-09-21T22:08:14.631 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.037181537000000001, 2024-09-21T22:08:14.631 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:14.631 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:14.631 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:14.631 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:14.631 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.631 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:14.631 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.578171+0000", 2024-09-21T22:08:14.631 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:14.632 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.632 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.632 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:14.632 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.578171+0000", 2024-09-21T22:08:14.632 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:14.632 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.632 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.633 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:14.633 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.578172+0000", 2024-09-21T22:08:14.634 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.017e-06 2024-09-21T22:08:14.634 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.634 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.634 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:14.634 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.579000+0000", 2024-09-21T22:08:14.634 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00082799499999999999 2024-09-21T22:08:14.634 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.634 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.634 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:14.634 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.579020+0000", 2024-09-21T22:08:14.634 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.0165000000000001e-05 2024-09-21T22:08:14.634 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.634 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.635 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:14.635 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.579036+0000", 2024-09-21T22:08:14.635 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5767e-05 2024-09-21T22:08:14.635 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.635 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.635 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:14.635 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.579106+0000", 2024-09-21T22:08:14.635 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.0091000000000004e-05 2024-09-21T22:08:14.635 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.635 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.635 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:14.635 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.579131+0000", 2024-09-21T22:08:14.635 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4978e-05 2024-09-21T22:08:14.635 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.635 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.636 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:14.636 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.615249+0000", 2024-09-21T22:08:14.636 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.036118355999999997 2024-09-21T22:08:14.636 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.636 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.636 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:14.636 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.615353+0000", 2024-09-21T22:08:14.636 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000103168 2024-09-21T22:08:14.636 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:14.636 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:14.637 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:14.637 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.637 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.637 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:147 2.5 e72/37 2:a70ae99f:::rbd_data.126483d75f2d.000000000000000c:head v 72'54, mlcod=72'52)", 2024-09-21T22:08:14.637 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.627203+0000", 2024-09-21T22:08:14.637 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 35.937483909000001, 2024-09-21T22:08:14.637 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.038203972000000003, 2024-09-21T22:08:14.637 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:14.637 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:14.637 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:14.637 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:14.637 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.637 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:14.638 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.627203+0000", 2024-09-21T22:08:14.638 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:14.638 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.638 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.638 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:14.638 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.627203+0000", 2024-09-21T22:08:14.638 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:14.638 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.638 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.638 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:14.639 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.627204+0000", 2024-09-21T22:08:14.639 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3069999999999999e-06 2024-09-21T22:08:14.639 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.639 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.639 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:14.639 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.628114+0000", 2024-09-21T22:08:14.639 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00090942099999999999 2024-09-21T22:08:14.639 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.639 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.639 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:14.640 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.628140+0000", 2024-09-21T22:08:14.640 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5882000000000001e-05 2024-09-21T22:08:14.640 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.640 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.640 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:14.640 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.628162+0000", 2024-09-21T22:08:14.640 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.2293000000000002e-05 2024-09-21T22:08:14.640 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.640 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.640 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:14.640 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.628250+0000", 2024-09-21T22:08:14.640 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.7973000000000006e-05 2024-09-21T22:08:14.641 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.641 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.641 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:14.641 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.628272+0000", 2024-09-21T22:08:14.641 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1682999999999999e-05 2024-09-21T22:08:14.641 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.641 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.641 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:14.641 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.665322+0000", 2024-09-21T22:08:14.641 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.037050650999999997 2024-09-21T22:08:14.641 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.642 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.642 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:14.642 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.665407+0000", 2024-09-21T22:08:14.642 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.4761999999999999e-05 2024-09-21T22:08:14.642 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:14.642 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:14.642 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:14.642 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.642 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.643 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:149 2.5 e72/37 2:a70ae99f:::rbd_data.126483d75f2d.000000000000000c:head v 72'56, mlcod=72'54)", 2024-09-21T22:08:14.643 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.690377+0000", 2024-09-21T22:08:14.643 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 35.874310166000001, 2024-09-21T22:08:14.643 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.049254799000000002, 2024-09-21T22:08:14.643 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:14.643 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:14.643 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:14.643 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:14.643 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.644 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:14.644 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.690377+0000", 2024-09-21T22:08:14.644 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:14.644 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.644 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.644 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:14.644 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.690377+0000", 2024-09-21T22:08:14.645 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:14.645 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.645 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.645 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:14.645 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.690378+0000", 2024-09-21T22:08:14.645 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.344e-06 2024-09-21T22:08:14.645 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.645 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.646 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:14.646 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.691302+0000", 2024-09-21T22:08:14.646 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00092322400000000005 2024-09-21T22:08:14.646 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.646 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.646 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:14.646 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.691321+0000", 2024-09-21T22:08:14.646 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9928999999999998e-05 2024-09-21T22:08:14.646 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.647 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.647 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:14.647 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.691345+0000", 2024-09-21T22:08:14.647 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.372e-05 2024-09-21T22:08:14.647 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.647 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.647 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:14.647 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.691442+0000", 2024-09-21T22:08:14.647 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.7270999999999998e-05 2024-09-21T22:08:14.648 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.648 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.648 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:14.648 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.691512+0000", 2024-09-21T22:08:14.648 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.9183000000000005e-05 2024-09-21T22:08:14.648 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.648 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.648 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:14.648 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.739545+0000", 2024-09-21T22:08:14.649 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.048033657 2024-09-21T22:08:14.649 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.649 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.649 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:14.649 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.739632+0000", 2024-09-21T22:08:14.649 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.6471000000000006e-05 2024-09-21T22:08:14.649 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:14.649 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:14.649 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:14.650 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.650 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.650 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:156 2.1 e72/53 2:837ddd83:::rbd_data.126483d75f2d.000000000000000d:head v 72'28, mlcod=72'26)", 2024-09-21T22:08:14.650 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.824856+0000", 2024-09-21T22:08:14.650 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 35.739830728000001, 2024-09-21T22:08:14.650 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.034320985999999998, 2024-09-21T22:08:14.650 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:14.650 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:14.650 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:14.650 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:14.650 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.650 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:14.651 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.824856+0000", 2024-09-21T22:08:14.651 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:14.651 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.651 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.651 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:14.651 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.824856+0000", 2024-09-21T22:08:14.651 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:14.651 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.651 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.651 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:14.651 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.824858+0000", 2024-09-21T22:08:14.651 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2979999999999999e-06 2024-09-21T22:08:14.652 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.652 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.652 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:14.652 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.825174+0000", 2024-09-21T22:08:14.652 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000316335 2024-09-21T22:08:14.652 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.652 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.652 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:14.652 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.825193+0000", 2024-09-21T22:08:14.652 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9436000000000002e-05 2024-09-21T22:08:14.653 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.653 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.653 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:14.653 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.825202+0000", 2024-09-21T22:08:14.653 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.8359999999999998e-06 2024-09-21T22:08:14.653 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.653 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.653 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:14.653 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.825286+0000", 2024-09-21T22:08:14.654 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.3683999999999994e-05 2024-09-21T22:08:14.654 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.654 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.654 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:14.654 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.825314+0000", 2024-09-21T22:08:14.654 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.7653000000000001e-05 2024-09-21T22:08:14.654 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.654 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.654 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:14.654 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.859122+0000", 2024-09-21T22:08:14.655 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.033808624000000002 2024-09-21T22:08:14.655 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.655 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.655 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:14.655 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.859177+0000", 2024-09-21T22:08:14.655 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.5120000000000001e-05 2024-09-21T22:08:14.655 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:14.655 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:14.655 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:14.655 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.655 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.656 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:158 2.1 e72/53 2:837ddd83:::rbd_data.126483d75f2d.000000000000000d:head v 72'30, mlcod=72'28)", 2024-09-21T22:08:14.656 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.869277+0000", 2024-09-21T22:08:14.656 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 35.695410531, 2024-09-21T22:08:14.656 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.040508390999999998, 2024-09-21T22:08:14.656 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:14.656 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:14.656 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:14.656 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:14.656 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.656 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:14.656 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.869277+0000", 2024-09-21T22:08:14.656 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:14.656 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.656 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.656 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:14.657 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.869277+0000", 2024-09-21T22:08:14.657 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:14.657 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.657 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.657 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:14.657 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.869278+0000", 2024-09-21T22:08:14.657 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5269999999999999e-06 2024-09-21T22:08:14.657 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.657 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.657 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:14.657 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.869902+0000", 2024-09-21T22:08:14.657 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00062399000000000005 2024-09-21T22:08:14.657 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.657 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.657 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:14.658 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.869924+0000", 2024-09-21T22:08:14.658 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1875999999999999e-05 2024-09-21T22:08:14.658 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.658 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.658 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:14.658 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.869937+0000", 2024-09-21T22:08:14.658 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2685e-05 2024-09-21T22:08:14.658 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.658 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.658 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:14.658 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.870058+0000", 2024-09-21T22:08:14.658 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000120862 2024-09-21T22:08:14.658 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.658 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.659 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:14.659 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.870093+0000", 2024-09-21T22:08:14.659 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.5896000000000003e-05 2024-09-21T22:08:14.659 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.659 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.659 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:14.659 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.909728+0000", 2024-09-21T22:08:14.659 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.039634461000000003 2024-09-21T22:08:14.659 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.659 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.659 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:14.659 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.909785+0000", 2024-09-21T22:08:14.659 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.7093999999999999e-05 2024-09-21T22:08:14.659 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:14.660 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:14.660 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:14.660 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.660 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.660 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:161 2.1 e72/53 2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head v 72'32, mlcod=72'30)", 2024-09-21T22:08:14.660 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.924455+0000", 2024-09-21T22:08:14.660 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 35.640232169000001, 2024-09-21T22:08:14.660 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.038366863000000001, 2024-09-21T22:08:14.660 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:14.660 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:14.660 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:14.660 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:14.660 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.660 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:14.660 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.924455+0000", 2024-09-21T22:08:14.661 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:14.661 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.661 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.661 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:14.661 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.924455+0000", 2024-09-21T22:08:14.661 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:14.661 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.661 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.661 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:14.661 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.924456+0000", 2024-09-21T22:08:14.661 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.003e-06 2024-09-21T22:08:14.661 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.661 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.661 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:14.661 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.924811+0000", 2024-09-21T22:08:14.662 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00035519499999999999 2024-09-21T22:08:14.662 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.662 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.662 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:14.662 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.924826+0000", 2024-09-21T22:08:14.662 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5296999999999999e-05 2024-09-21T22:08:14.662 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.662 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.662 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:14.662 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.924834+0000", 2024-09-21T22:08:14.662 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.5379999999999997e-06 2024-09-21T22:08:14.662 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.662 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.663 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:14.663 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.924910+0000", 2024-09-21T22:08:14.663 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.6488000000000004e-05 2024-09-21T22:08:14.663 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.663 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.663 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:14.663 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.924934+0000", 2024-09-21T22:08:14.663 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3598999999999999e-05 2024-09-21T22:08:14.663 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.663 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.663 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:14.663 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.962739+0000", 2024-09-21T22:08:14.663 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.037805370999999997 2024-09-21T22:08:14.663 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.663 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.664 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:14.664 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.962822+0000", 2024-09-21T22:08:14.664 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.2372000000000003e-05 2024-09-21T22:08:14.664 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:14.664 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:14.664 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:14.664 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.664 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.664 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:163 2.1 e72/53 2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head v 72'34, mlcod=72'32)", 2024-09-21T22:08:14.664 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.971683+0000", 2024-09-21T22:08:14.664 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 35.593003629000002, 2024-09-21T22:08:14.665 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.043171320999999999, 2024-09-21T22:08:14.665 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:14.665 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:14.665 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:14.665 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:14.665 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.665 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:14.665 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.971683+0000", 2024-09-21T22:08:14.665 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:14.665 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.665 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.665 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:14.666 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.971683+0000", 2024-09-21T22:08:14.666 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:14.666 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.666 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.666 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:14.666 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.971684+0000", 2024-09-21T22:08:14.666 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.6600000000000005e-07 2024-09-21T22:08:14.666 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.666 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.666 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:14.666 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.971945+0000", 2024-09-21T22:08:14.666 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00026101899999999998 2024-09-21T22:08:14.666 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.667 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.667 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:14.667 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.971959+0000", 2024-09-21T22:08:14.667 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4054e-05 2024-09-21T22:08:14.667 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.667 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.667 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:14.667 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.971969+0000", 2024-09-21T22:08:14.667 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.499e-06 2024-09-21T22:08:14.667 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.667 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.667 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:14.667 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.972037+0000", 2024-09-21T22:08:14.667 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.8448000000000006e-05 2024-09-21T22:08:14.667 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.668 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.668 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:14.668 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.972057+0000", 2024-09-21T22:08:14.668 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9817999999999999e-05 2024-09-21T22:08:14.668 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.668 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.668 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:14.668 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.014770+0000", 2024-09-21T22:08:14.668 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.042712464999999998 2024-09-21T22:08:14.668 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.668 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.668 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:14.669 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.014855+0000", 2024-09-21T22:08:14.669 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.5152000000000001e-05 2024-09-21T22:08:14.669 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:14.669 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:14.669 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:14.669 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.669 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.669 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:165 2.1 e72/53 2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head v 72'36, mlcod=72'34)", 2024-09-21T22:08:14.669 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:39.028631+0000", 2024-09-21T22:08:14.669 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 35.536056152, 2024-09-21T22:08:14.669 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.033430561999999997, 2024-09-21T22:08:14.670 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:14.670 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:14.670 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:14.670 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:14.670 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.670 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:14.670 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.028631+0000", 2024-09-21T22:08:14.670 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:14.670 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.670 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.670 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:14.671 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.028631+0000", 2024-09-21T22:08:14.671 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:14.671 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.671 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.671 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:14.671 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.028633+0000", 2024-09-21T22:08:14.671 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5799999999999999e-06 2024-09-21T22:08:14.671 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.671 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.672 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:14.672 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.029209+0000", 2024-09-21T22:08:14.672 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00057670799999999999 2024-09-21T22:08:14.672 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.672 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.672 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:14.672 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.029238+0000", 2024-09-21T22:08:14.672 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8943000000000001e-05 2024-09-21T22:08:14.672 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.672 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.672 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:14.673 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.029252+0000", 2024-09-21T22:08:14.673 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3672999999999999e-05 2024-09-21T22:08:14.673 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.673 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.673 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:14.673 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.029328+0000", 2024-09-21T22:08:14.673 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.6497999999999998e-05 2024-09-21T22:08:14.673 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.673 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.674 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:14.674 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.029353+0000", 2024-09-21T22:08:14.674 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5060999999999999e-05 2024-09-21T22:08:14.674 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.674 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.674 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:14.674 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.061993+0000", 2024-09-21T22:08:14.674 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.032639286000000003 2024-09-21T22:08:14.674 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.674 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.675 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:14.675 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.062062+0000", 2024-09-21T22:08:14.675 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.8813000000000007e-05 2024-09-21T22:08:14.675 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:14.675 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:14.675 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:14.675 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.675 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.675 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:167 2.1 e72/53 2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head v 72'38, mlcod=72'36)", 2024-09-21T22:08:14.676 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:39.077324+0000", 2024-09-21T22:08:14.676 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 35.487363338000002, 2024-09-21T22:08:14.676 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.036031961000000001, 2024-09-21T22:08:14.676 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:14.676 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:14.676 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:14.676 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:14.676 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.676 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:14.676 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.077324+0000", 2024-09-21T22:08:14.676 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:14.677 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.677 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.677 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:14.677 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.077324+0000", 2024-09-21T22:08:14.677 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:14.677 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.677 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.677 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:14.677 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.077326+0000", 2024-09-21T22:08:14.677 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.75e-06 2024-09-21T22:08:14.677 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.678 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.678 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:14.678 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.077719+0000", 2024-09-21T22:08:14.678 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00039322599999999999 2024-09-21T22:08:14.678 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.678 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.678 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:14.678 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.077749+0000", 2024-09-21T22:08:14.678 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9895000000000001e-05 2024-09-21T22:08:14.679 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.679 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.679 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:14.679 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.077763+0000", 2024-09-21T22:08:14.679 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4517e-05 2024-09-21T22:08:14.679 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.679 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.679 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:14.679 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.077843+0000", 2024-09-21T22:08:14.679 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.9994999999999996e-05 2024-09-21T22:08:14.679 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.680 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.680 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:14.680 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.077867+0000", 2024-09-21T22:08:14.680 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3640000000000001e-05 2024-09-21T22:08:14.680 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.680 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.680 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:14.680 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.113216+0000", 2024-09-21T22:08:14.680 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.035349168 2024-09-21T22:08:14.681 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:14.681 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:14.681 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:14.681 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.113356+0000", 2024-09-21T22:08:14.681 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013977 2024-09-21T22:08:14.681 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:14.681 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:14.681 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:14.682 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:14.682 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:14.682 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:08:14.684 INFO:tasks.workunit.client.0.smithi157.stderr:+ PID=24320 2024-09-21T22:08:14.684 INFO:tasks.workunit.client.0.smithi157.stderr:+ test -n 24320 2024-09-21T22:08:14.684 INFO:tasks.workunit.client.0.smithi157.stderr:+ ps -p 24320 -C rbd-nbd 2024-09-21T22:08:14.684 INFO:tasks.workunit.client.0.smithi157.stdout: PID TTY TIME CMD 2024-09-21T22:08:14.685 INFO:tasks.workunit.client.0.smithi157.stdout: 24320 ? 00:00:00 rbd-nbd 2024-09-21T22:08:14.685 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T22:08:14.686 INFO:tasks.workunit.client.0.smithi157.stderr:+ _sudo rbd device detach /dev/nbd1 --device-type nbd 2024-09-21T22:08:14.686 INFO:tasks.workunit.client.0.smithi157.stderr:+ local cmd 2024-09-21T22:08:14.686 INFO:tasks.workunit.client.0.smithi157.stderr:++ id -u 2024-09-21T22:08:14.686 INFO:tasks.workunit.client.0.smithi157.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T22:08:14.686 INFO:tasks.workunit.client.0.smithi157.stderr:++ which rbd 2024-09-21T22:08:14.686 INFO:tasks.workunit.client.0.smithi157.stderr:+ cmd=/usr/bin/rbd 2024-09-21T22:08:14.686 INFO:tasks.workunit.client.0.smithi157.stderr:+ shift 2024-09-21T22:08:14.687 INFO:tasks.workunit.client.0.smithi157.stderr:+ sudo -nE /usr/bin/rbd device detach /dev/nbd1 --device-type nbd 2024-09-21T22:08:14.687 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:08:14.679+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:08:14.747 INFO:tasks.workunit.client.0.smithi157.stderr:+ expect_false get_pid rbd 2024-09-21T22:08:14.747 INFO:tasks.workunit.client.0.smithi157.stderr:+ get_pid rbd 2024-09-21T22:08:14.747 INFO:tasks.workunit.client.0.smithi157.stderr:+ local pool=rbd 2024-09-21T22:08:14.747 INFO:tasks.workunit.client.0.smithi157.stderr:+ local ns= 2024-09-21T22:08:14.747 INFO:tasks.workunit.client.0.smithi157.stderr:++ rbd device --device-type nbd --format xml list 2024-09-21T22:08:14.747 INFO:tasks.workunit.client.0.smithi157.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd21553'\''][device='\''/dev/nbd1'\'']/id' 2024-09-21T22:08:14.784 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:08:14.779+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:08:14.815 INFO:tasks.workunit.client.0.smithi157.stderr:+ PID= 2024-09-21T22:08:14.815 INFO:tasks.workunit.client.0.smithi157.stderr:+ test -n '' 2024-09-21T22:08:14.815 INFO:tasks.workunit.client.0.smithi157.stderr:+ return 1 2024-09-21T22:08:14.815 INFO:tasks.workunit.client.0.smithi157.stderr:+ return 0 2024-09-21T22:08:14.816 INFO:tasks.workunit.client.0.smithi157.stderr:+ DEV= 2024-09-21T22:08:14.816 INFO:tasks.workunit.client.0.smithi157.stderr:+ rbd config image set rbd/testrbdnbd21553 rbd_discard_granularity_bytes 4096 2024-09-21T22:08:14.861 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:08:14.861 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:08:14.861 INFO:teuthology.orchestra.run.smithi039.stdout: "num_ops": 0 2024-09-21T22:08:14.861 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:08:14.876 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T22:08:14.911 INFO:tasks.workunit.client.0.smithi157.stderr:+ rbd feature enable rbd/testrbdnbd21553 journaling 2024-09-21T22:08:14.920 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:08:14.917+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:08:14.986 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:08:14.983+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:08:14.997 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:08:14.997 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:08:14.997 INFO:teuthology.orchestra.run.smithi039.stdout: "complaint_time": 30, 2024-09-21T22:08:14.997 INFO:teuthology.orchestra.run.smithi039.stdout: "num_blocked_ops": 0 2024-09-21T22:08:14.997 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:08:15.012 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T22:08:15.017 INFO:tasks.workunit.client.0.smithi157.stderr:++ _sudo rbd device --device-type nbd map rbd/testrbdnbd21553 2024-09-21T22:08:15.017 INFO:tasks.workunit.client.0.smithi157.stderr:++ local cmd 2024-09-21T22:08:15.017 INFO:tasks.workunit.client.0.smithi157.stderr:+++ id -u 2024-09-21T22:08:15.018 INFO:tasks.workunit.client.0.smithi157.stderr:++ '[' 1000 -eq 0 ']' 2024-09-21T22:08:15.019 INFO:tasks.workunit.client.0.smithi157.stderr:+++ which rbd 2024-09-21T22:08:15.021 INFO:tasks.workunit.client.0.smithi157.stderr:++ cmd=/usr/bin/rbd 2024-09-21T22:08:15.021 INFO:tasks.workunit.client.0.smithi157.stderr:++ shift 2024-09-21T22:08:15.021 INFO:tasks.workunit.client.0.smithi157.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map rbd/testrbdnbd21553 2024-09-21T22:08:15.100 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:08:15.097+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:08:15.182 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:08:15.182 INFO:teuthology.orchestra.run.smithi039.stdout: "size": 20, 2024-09-21T22:08:15.182 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 600, 2024-09-21T22:08:15.182 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [ 2024-09-21T22:08:15.182 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.182 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.3 55/54 [PushOp(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head, version: 52'2, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head@52'2, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T22:08:15.182 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.084379+0000", 2024-09-21T22:08:15.182 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 63.095002618999999, 2024-09-21T22:08:15.183 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.051020674000000002, 2024-09-21T22:08:15.183 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:15.183 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:15.183 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:08:15.183 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:15.183 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.183 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:15.183 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.084379+0000", 2024-09-21T22:08:15.183 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:15.183 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.183 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.183 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:15.183 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.084379+0000", 2024-09-21T22:08:15.183 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:15.184 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.184 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.184 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:15.184 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.084381+0000", 2024-09-21T22:08:15.184 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.73e-06 2024-09-21T22:08:15.184 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.184 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.184 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:15.184 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.085494+0000", 2024-09-21T22:08:15.184 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011131559999999999 2024-09-21T22:08:15.184 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.184 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.184 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:15.184 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.085542+0000", 2024-09-21T22:08:15.184 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8334999999999997e-05 2024-09-21T22:08:15.185 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.185 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.185 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:15.185 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.085591+0000", 2024-09-21T22:08:15.185 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.9165000000000001e-05 2024-09-21T22:08:15.185 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.185 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.185 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:15.185 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.085777+0000", 2024-09-21T22:08:15.186 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000186152 2024-09-21T22:08:15.186 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.186 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.186 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:15.186 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.085796+0000", 2024-09-21T22:08:15.186 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8612999999999999e-05 2024-09-21T22:08:15.186 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.186 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.186 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:15.186 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.135399+0000", 2024-09-21T22:08:15.187 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.049603522999999997 2024-09-21T22:08:15.187 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:15.187 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:15.187 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:15.187 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.187 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.187 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.3 55/54 [PushOp(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head, version: 52'2, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head@52'2, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:08:15.187 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.139554+0000", 2024-09-21T22:08:15.187 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 63.039827516000003, 2024-09-21T22:08:15.187 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.056215389999999997, 2024-09-21T22:08:15.187 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:15.187 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:15.187 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:08:15.188 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:15.188 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.188 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:15.188 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.139554+0000", 2024-09-21T22:08:15.188 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:15.188 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.188 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.188 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:15.188 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.139554+0000", 2024-09-21T22:08:15.189 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:15.189 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.189 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.189 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:15.189 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.139556+0000", 2024-09-21T22:08:15.189 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7999999999999999e-06 2024-09-21T22:08:15.189 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.190 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.190 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:15.190 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.140671+0000", 2024-09-21T22:08:15.190 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001115612 2024-09-21T22:08:15.190 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.190 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.190 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:15.190 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.140720+0000", 2024-09-21T22:08:15.190 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8631999999999998e-05 2024-09-21T22:08:15.190 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.191 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.191 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:15.191 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.140752+0000", 2024-09-21T22:08:15.191 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.2199000000000001e-05 2024-09-21T22:08:15.191 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.191 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.191 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:15.191 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.140900+0000", 2024-09-21T22:08:15.191 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00014786000000000001 2024-09-21T22:08:15.191 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.191 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.191 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:15.191 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.140917+0000", 2024-09-21T22:08:15.192 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6753999999999999e-05 2024-09-21T22:08:15.192 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.192 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.192 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:15.192 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.195769+0000", 2024-09-21T22:08:15.192 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.054852533000000002 2024-09-21T22:08:15.192 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:15.192 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:15.193 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:15.193 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.193 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.193 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.3 55/54 [PushOp(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head, version: 52'2, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head@52'2, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:08:15.193 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.199187+0000", 2024-09-21T22:08:15.193 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 62.980193972000002, 2024-09-21T22:08:15.193 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.051543975999999998, 2024-09-21T22:08:15.193 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:15.194 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:15.194 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:08:15.194 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:15.194 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.194 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:15.194 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.199187+0000", 2024-09-21T22:08:15.194 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:15.194 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.195 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.195 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:15.195 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.199187+0000", 2024-09-21T22:08:15.195 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:15.195 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.195 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.195 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:15.195 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.199189+0000", 2024-09-21T22:08:15.195 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7969999999999999e-06 2024-09-21T22:08:15.195 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.195 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.195 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:15.196 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.200355+0000", 2024-09-21T22:08:15.196 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011660139999999999 2024-09-21T22:08:15.196 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.196 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.196 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:15.196 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.200405+0000", 2024-09-21T22:08:15.196 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.0124000000000003e-05 2024-09-21T22:08:15.196 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.196 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.196 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:15.196 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.200457+0000", 2024-09-21T22:08:15.196 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.1177999999999997e-05 2024-09-21T22:08:15.196 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.196 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.197 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:15.197 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.200635+0000", 2024-09-21T22:08:15.197 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00017841299999999999 2024-09-21T22:08:15.197 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.197 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.197 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:15.197 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.200653+0000", 2024-09-21T22:08:15.197 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7944999999999998e-05 2024-09-21T22:08:15.197 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.197 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.197 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:15.197 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.250731+0000", 2024-09-21T22:08:15.197 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.050078505000000002 2024-09-21T22:08:15.198 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:15.198 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:15.198 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:15.198 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.198 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.198 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.3 55/54 [PushOp(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head, version: 52'3, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head@52'3, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:08:15.198 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.343754+0000", 2024-09-21T22:08:15.198 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 62.835627260999999, 2024-09-21T22:08:15.198 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.063560607000000005, 2024-09-21T22:08:15.198 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:15.198 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:15.198 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:08:15.198 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:15.199 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.199 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:15.199 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.343754+0000", 2024-09-21T22:08:15.199 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:15.199 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.199 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.199 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:15.199 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.343754+0000", 2024-09-21T22:08:15.199 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:15.199 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.199 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.199 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:15.200 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.343756+0000", 2024-09-21T22:08:15.200 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7069999999999999e-06 2024-09-21T22:08:15.200 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.200 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.200 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:15.200 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.344864+0000", 2024-09-21T22:08:15.200 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011085159999999999 2024-09-21T22:08:15.200 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.200 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.200 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:15.200 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.344914+0000", 2024-09-21T22:08:15.201 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.9462000000000002e-05 2024-09-21T22:08:15.201 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.201 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.201 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:15.201 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.344961+0000", 2024-09-21T22:08:15.201 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.7633999999999998e-05 2024-09-21T22:08:15.201 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.201 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.201 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:15.201 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.345171+0000", 2024-09-21T22:08:15.201 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000209652 2024-09-21T22:08:15.201 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.201 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.201 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:15.202 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.345193+0000", 2024-09-21T22:08:15.202 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1985999999999999e-05 2024-09-21T22:08:15.202 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.202 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.202 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:15.202 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.407315+0000", 2024-09-21T22:08:15.202 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.06212165 2024-09-21T22:08:15.202 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:15.203 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:15.203 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:15.203 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.203 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.203 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.3 55/54 [PushOp(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head, version: 52'3, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head@52'3, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:08:15.203 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.410551+0000", 2024-09-21T22:08:15.203 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 62.768830129999998, 2024-09-21T22:08:15.203 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.054517522999999998, 2024-09-21T22:08:15.203 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:15.203 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:15.203 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:08:15.203 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:15.203 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.204 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:15.204 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.410551+0000", 2024-09-21T22:08:15.204 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:15.204 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.204 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.204 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:15.204 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.410551+0000", 2024-09-21T22:08:15.204 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:15.204 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.205 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.205 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:15.205 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.410553+0000", 2024-09-21T22:08:15.205 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5200000000000001e-06 2024-09-21T22:08:15.205 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.205 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.205 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:15.205 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.411747+0000", 2024-09-21T22:08:15.205 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011946089999999999 2024-09-21T22:08:15.205 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.205 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.205 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:15.205 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.411796+0000", 2024-09-21T22:08:15.206 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8717999999999997e-05 2024-09-21T22:08:15.206 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.206 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.206 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:15.206 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.411844+0000", 2024-09-21T22:08:15.206 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8291999999999998e-05 2024-09-21T22:08:15.206 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.206 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.206 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:15.206 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.412066+0000", 2024-09-21T22:08:15.206 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00022115899999999999 2024-09-21T22:08:15.207 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.207 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.207 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:15.207 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.412081+0000", 2024-09-21T22:08:15.207 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5654e-05 2024-09-21T22:08:15.207 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.207 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.207 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:15.207 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.465069+0000", 2024-09-21T22:08:15.207 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.052987570999999997 2024-09-21T22:08:15.207 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:15.208 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:15.208 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:15.208 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.208 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.208 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.3 55/54 [PushOp(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head, version: 52'3, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head@52'3, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 4194304, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:08:15.208 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.468339+0000", 2024-09-21T22:08:15.208 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 62.711042466000002, 2024-09-21T22:08:15.208 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.064010024999999998, 2024-09-21T22:08:15.208 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:15.208 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:15.208 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:08:15.208 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:15.208 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.208 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:15.209 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.468339+0000", 2024-09-21T22:08:15.209 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:15.209 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.209 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.209 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:15.209 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.468339+0000", 2024-09-21T22:08:15.209 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:15.209 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.209 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.209 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:15.209 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.468340+0000", 2024-09-21T22:08:15.209 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4330000000000001e-06 2024-09-21T22:08:15.209 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.209 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.210 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:15.210 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.469484+0000", 2024-09-21T22:08:15.210 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001143535 2024-09-21T22:08:15.210 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.210 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.210 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:15.210 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.469533+0000", 2024-09-21T22:08:15.210 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8980999999999997e-05 2024-09-21T22:08:15.210 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.210 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.210 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:15.210 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.469581+0000", 2024-09-21T22:08:15.210 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8551999999999999e-05 2024-09-21T22:08:15.211 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.211 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.211 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:15.211 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.469804+0000", 2024-09-21T22:08:15.211 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00022287699999999999 2024-09-21T22:08:15.211 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.211 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.211 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:15.211 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.469829+0000", 2024-09-21T22:08:15.211 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4766e-05 2024-09-21T22:08:15.211 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.212 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.212 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:15.212 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.532349+0000", 2024-09-21T22:08:15.212 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.062519880999999999 2024-09-21T22:08:15.212 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:15.212 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:15.212 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:15.212 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.212 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.212 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4840.0:668 2.3 2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:08:15.212 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.409114+0000", 2024-09-21T22:08:15.212 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 50.770267259999997, 2024-09-21T22:08:15.212 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.048703758999999999, 2024-09-21T22:08:15.212 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:15.213 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:15.213 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:15.213 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-21T22:08:15.213 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4840", 2024-09-21T22:08:15.213 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:08:15.213 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 668 2024-09-21T22:08:15.213 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.213 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:15.213 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.213 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:15.213 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.409114+0000", 2024-09-21T22:08:15.213 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:15.213 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.214 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.214 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:15.214 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.409114+0000", 2024-09-21T22:08:15.214 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:15.214 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.214 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.214 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:15.214 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.409115+0000", 2024-09-21T22:08:15.214 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.114e-06 2024-09-21T22:08:15.214 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.214 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.214 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:15.215 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.410308+0000", 2024-09-21T22:08:15.215 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011930949999999999 2024-09-21T22:08:15.215 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.215 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.215 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:15.215 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.410325+0000", 2024-09-21T22:08:15.215 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7088e-05 2024-09-21T22:08:15.215 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.215 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.215 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:15.215 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.410334+0000", 2024-09-21T22:08:15.215 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.2549999999999997e-06 2024-09-21T22:08:15.215 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.216 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.216 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:15.216 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.410393+0000", 2024-09-21T22:08:15.216 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.9327999999999998e-05 2024-09-21T22:08:15.216 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.216 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.216 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:15.216 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.410482+0000", 2024-09-21T22:08:15.216 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.9019999999999998e-05 2024-09-21T22:08:15.216 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.216 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.216 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-21T22:08:15.216 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.410617+0000", 2024-09-21T22:08:15.217 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000135232 2024-09-21T22:08:15.217 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.217 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.217 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:15.217 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.457051+0000", 2024-09-21T22:08:15.217 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.04643415 2024-09-21T22:08:15.217 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.218 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.218 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-21T22:08:15.218 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.457675+0000", 2024-09-21T22:08:15.218 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00062332700000000002 2024-09-21T22:08:15.218 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.218 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.218 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:15.218 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.457735+0000", 2024-09-21T22:08:15.219 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.029e-05 2024-09-21T22:08:15.219 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.219 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.219 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:15.219 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.457818+0000", 2024-09-21T22:08:15.219 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.2860000000000003e-05 2024-09-21T22:08:15.219 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:15.219 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:15.219 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:15.219 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.219 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.219 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4840.0:670 2.3 2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:08:15.220 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.466980+0000", 2024-09-21T22:08:15.220 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 50.712401192000002, 2024-09-21T22:08:15.220 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.050913930000000003, 2024-09-21T22:08:15.220 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:15.220 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:15.220 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:15.220 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-21T22:08:15.220 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4840", 2024-09-21T22:08:15.220 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:08:15.220 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 670 2024-09-21T22:08:15.220 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.220 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:15.220 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.221 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:15.221 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.466980+0000", 2024-09-21T22:08:15.221 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:15.221 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.221 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.221 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:15.221 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.466980+0000", 2024-09-21T22:08:15.221 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:15.221 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.221 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.221 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:15.221 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.466981+0000", 2024-09-21T22:08:15.221 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0839999999999999e-06 2024-09-21T22:08:15.222 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.222 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.222 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:15.222 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.467815+0000", 2024-09-21T22:08:15.222 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00083330399999999999 2024-09-21T22:08:15.222 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.222 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.222 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:15.222 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.467832+0000", 2024-09-21T22:08:15.222 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7601e-05 2024-09-21T22:08:15.222 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.222 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.222 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:15.223 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.467840+0000", 2024-09-21T22:08:15.223 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.2220000000000002e-06 2024-09-21T22:08:15.223 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.223 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.223 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:15.223 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.467974+0000", 2024-09-21T22:08:15.223 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000133411 2024-09-21T22:08:15.223 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.223 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.223 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:15.223 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.468223+0000", 2024-09-21T22:08:15.223 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00024906400000000001 2024-09-21T22:08:15.223 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.224 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.224 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-21T22:08:15.224 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.468616+0000", 2024-09-21T22:08:15.224 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000393433 2024-09-21T22:08:15.224 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.224 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.224 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:15.224 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.517016+0000", 2024-09-21T22:08:15.224 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.048399708 2024-09-21T22:08:15.224 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.224 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.224 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-21T22:08:15.224 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.517693+0000", 2024-09-21T22:08:15.225 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00067681000000000002 2024-09-21T22:08:15.225 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.225 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.225 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:15.225 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.517789+0000", 2024-09-21T22:08:15.225 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.5778000000000004e-05 2024-09-21T22:08:15.225 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.225 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.225 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:15.225 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.517894+0000", 2024-09-21T22:08:15.225 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000105515 2024-09-21T22:08:15.225 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:15.226 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:15.226 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:15.226 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.226 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.226 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4840.0:672 2.3 2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:08:15.226 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.527127+0000", 2024-09-21T22:08:15.226 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 50.652254730000003, 2024-09-21T22:08:15.226 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.047345602000000001, 2024-09-21T22:08:15.226 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:15.226 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:15.227 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:15.227 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-21T22:08:15.227 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4840", 2024-09-21T22:08:15.227 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:08:15.227 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 672 2024-09-21T22:08:15.227 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.227 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:15.227 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.227 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:15.227 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.527127+0000", 2024-09-21T22:08:15.227 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:15.227 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.228 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.228 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:15.228 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.527127+0000", 2024-09-21T22:08:15.228 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:15.228 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.228 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.228 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:15.228 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.527128+0000", 2024-09-21T22:08:15.228 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2810000000000001e-06 2024-09-21T22:08:15.228 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.228 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.228 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:15.228 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.527962+0000", 2024-09-21T22:08:15.229 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00083445199999999996 2024-09-21T22:08:15.229 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.229 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.229 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:15.229 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.527981+0000", 2024-09-21T22:08:15.229 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9009e-05 2024-09-21T22:08:15.229 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.229 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.229 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:15.229 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.527990+0000", 2024-09-21T22:08:15.229 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.8619999999999992e-06 2024-09-21T22:08:15.229 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.229 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.230 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:15.230 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.528061+0000", 2024-09-21T22:08:15.230 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.0226000000000002e-05 2024-09-21T22:08:15.230 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.230 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.230 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:15.230 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.528156+0000", 2024-09-21T22:08:15.230 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.5724999999999996e-05 2024-09-21T22:08:15.230 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.230 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.230 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-21T22:08:15.231 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.528303+0000", 2024-09-21T22:08:15.231 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000146374 2024-09-21T22:08:15.231 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.231 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.231 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:15.231 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.573772+0000", 2024-09-21T22:08:15.231 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.045469745999999998 2024-09-21T22:08:15.232 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.232 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.232 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-21T22:08:15.232 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.574299+0000", 2024-09-21T22:08:15.232 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00052649200000000004 2024-09-21T22:08:15.232 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.232 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.232 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:15.233 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.574366+0000", 2024-09-21T22:08:15.233 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.6872000000000006e-05 2024-09-21T22:08:15.233 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.233 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.233 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:15.233 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.574472+0000", 2024-09-21T22:08:15.233 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000106563 2024-09-21T22:08:15.234 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:15.234 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:15.234 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:15.234 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.234 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.234 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4840.0:686 2.3 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:08:15.234 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.943561+0000", 2024-09-21T22:08:15.234 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 50.235820453999999, 2024-09-21T22:08:15.234 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.049021493999999999, 2024-09-21T22:08:15.235 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:15.235 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:15.235 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:15.235 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-21T22:08:15.235 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4840", 2024-09-21T22:08:15.235 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:08:15.235 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 686 2024-09-21T22:08:15.235 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.235 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:15.235 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.236 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:15.236 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.943561+0000", 2024-09-21T22:08:15.236 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:15.236 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.236 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.236 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:15.236 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.943561+0000", 2024-09-21T22:08:15.236 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:15.236 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.236 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.236 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:15.237 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.943563+0000", 2024-09-21T22:08:15.237 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9420000000000002e-06 2024-09-21T22:08:15.237 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.237 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.237 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:15.237 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.944546+0000", 2024-09-21T22:08:15.237 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00098357099999999992 2024-09-21T22:08:15.237 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.237 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.237 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:15.237 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.944565+0000", 2024-09-21T22:08:15.237 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8621e-05 2024-09-21T22:08:15.237 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.238 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.238 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:15.238 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.944575+0000", 2024-09-21T22:08:15.238 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.4639999999999995e-06 2024-09-21T22:08:15.238 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.238 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.238 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:15.238 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.944647+0000", 2024-09-21T22:08:15.238 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.2786000000000004e-05 2024-09-21T22:08:15.238 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.239 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.239 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:15.239 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.944746+0000", 2024-09-21T22:08:15.239 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.8196999999999995e-05 2024-09-21T22:08:15.239 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.239 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.239 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-21T22:08:15.239 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.944890+0000", 2024-09-21T22:08:15.239 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000144804 2024-09-21T22:08:15.239 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.239 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.239 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:15.239 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.992404+0000", 2024-09-21T22:08:15.239 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.047513211999999999 2024-09-21T22:08:15.240 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.240 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.240 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-21T22:08:15.240 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.992480+0000", 2024-09-21T22:08:15.240 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.6491999999999999e-05 2024-09-21T22:08:15.240 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.240 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.240 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:15.240 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.992521+0000", 2024-09-21T22:08:15.240 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.0608000000000002e-05 2024-09-21T22:08:15.240 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.241 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.241 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:15.241 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.992582+0000", 2024-09-21T22:08:15.241 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.1797000000000005e-05 2024-09-21T22:08:15.241 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:15.241 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:15.241 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:15.241 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.241 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.241 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4840.0:720 2.1 2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:08:15.241 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.803827+0000", 2024-09-21T22:08:15.241 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 49.375554178000002, 2024-09-21T22:08:15.241 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.049890510999999998, 2024-09-21T22:08:15.242 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:15.242 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:15.242 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:15.242 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-21T22:08:15.242 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4840", 2024-09-21T22:08:15.242 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:08:15.242 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 720 2024-09-21T22:08:15.242 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.242 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:15.242 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.242 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:15.242 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.803827+0000", 2024-09-21T22:08:15.242 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:15.243 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.243 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.243 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:15.243 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.803827+0000", 2024-09-21T22:08:15.243 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:15.243 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.243 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.243 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:15.243 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.803828+0000", 2024-09-21T22:08:15.243 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.569999999999999e-07 2024-09-21T22:08:15.243 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.243 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.244 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:15.244 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.804689+0000", 2024-09-21T22:08:15.244 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00086121799999999997 2024-09-21T22:08:15.244 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.245 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.246 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:15.246 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.804706+0000", 2024-09-21T22:08:15.246 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6266999999999999e-05 2024-09-21T22:08:15.246 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.246 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.246 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:15.246 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.804713+0000", 2024-09-21T22:08:15.246 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.1010000000000003e-06 2024-09-21T22:08:15.246 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.246 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.246 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:15.246 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.804781+0000", 2024-09-21T22:08:15.247 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.7871000000000001e-05 2024-09-21T22:08:15.247 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.247 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.247 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:15.247 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.804859+0000", 2024-09-21T22:08:15.247 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.7891000000000004e-05 2024-09-21T22:08:15.247 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.247 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.247 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 0", 2024-09-21T22:08:15.247 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.804971+0000", 2024-09-21T22:08:15.247 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00011275100000000001 2024-09-21T22:08:15.247 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.247 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.247 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:15.248 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.852704+0000", 2024-09-21T22:08:15.248 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.047732361000000001 2024-09-21T22:08:15.248 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.248 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.248 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-21T22:08:15.248 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.853446+0000", 2024-09-21T22:08:15.248 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00074258799999999997 2024-09-21T22:08:15.248 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.248 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.248 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:15.248 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.853608+0000", 2024-09-21T22:08:15.248 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00016190900000000001 2024-09-21T22:08:15.248 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.249 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.249 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:15.249 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.853718+0000", 2024-09-21T22:08:15.249 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000109597 2024-09-21T22:08:15.249 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:15.249 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:15.249 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:15.249 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.249 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.249 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4840.0:723 2.1 2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:08:15.249 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.860543+0000", 2024-09-21T22:08:15.249 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 49.318838743999997, 2024-09-21T22:08:15.249 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.062354957000000003, 2024-09-21T22:08:15.249 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:15.250 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:15.250 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:15.250 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-21T22:08:15.250 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4840", 2024-09-21T22:08:15.250 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:08:15.250 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 723 2024-09-21T22:08:15.250 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.250 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:15.250 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.250 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:15.250 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.860543+0000", 2024-09-21T22:08:15.250 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:15.250 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.250 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.251 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:15.251 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.860543+0000", 2024-09-21T22:08:15.251 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:15.251 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.251 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.251 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:15.251 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.860544+0000", 2024-09-21T22:08:15.251 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.1119999999999999e-06 2024-09-21T22:08:15.251 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.251 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.251 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:15.251 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.861400+0000", 2024-09-21T22:08:15.251 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000856393 2024-09-21T22:08:15.251 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.252 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.252 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:15.252 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.861419+0000", 2024-09-21T22:08:15.252 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8992e-05 2024-09-21T22:08:15.252 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.252 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.252 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:15.252 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.861428+0000", 2024-09-21T22:08:15.252 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.551e-06 2024-09-21T22:08:15.252 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.252 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.252 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:15.252 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.861555+0000", 2024-09-21T22:08:15.252 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000127286 2024-09-21T22:08:15.252 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.253 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.253 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:15.253 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.861816+0000", 2024-09-21T22:08:15.253 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00026100899999999998 2024-09-21T22:08:15.253 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.253 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.253 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 0", 2024-09-21T22:08:15.253 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.862189+0000", 2024-09-21T22:08:15.253 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00037261899999999998 2024-09-21T22:08:15.253 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.253 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.253 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:15.253 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.921514+0000", 2024-09-21T22:08:15.253 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.059325210000000003 2024-09-21T22:08:15.254 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.254 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.254 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-21T22:08:15.254 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.922721+0000", 2024-09-21T22:08:15.254 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0012066819999999999 2024-09-21T22:08:15.254 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.254 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.254 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:15.254 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.922797+0000", 2024-09-21T22:08:15.254 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.6162999999999996e-05 2024-09-21T22:08:15.254 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.254 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.254 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:15.255 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.922898+0000", 2024-09-21T22:08:15.255 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00010094000000000001 2024-09-21T22:08:15.255 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:15.255 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:15.255 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:15.255 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.255 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.255 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:725 2.0 e63/16 2:08b4f96a:::rbd_data.126483d75f2d.000000000000000f:head v 63'33, mlcod=63'31)", 2024-09-21T22:08:15.255 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.936778+0000", 2024-09-21T22:08:15.255 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 49.242603002000003, 2024-09-21T22:08:15.255 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.047671937999999997, 2024-09-21T22:08:15.255 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:15.255 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:15.255 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:15.255 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:15.256 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.256 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:15.256 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.936778+0000", 2024-09-21T22:08:15.256 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:15.256 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.256 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.256 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:15.256 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.936778+0000", 2024-09-21T22:08:15.256 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:15.256 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.256 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.256 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:15.256 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.936780+0000", 2024-09-21T22:08:15.256 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7799999999999999e-06 2024-09-21T22:08:15.256 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.257 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.257 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:15.257 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.938475+0000", 2024-09-21T22:08:15.257 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0016951749999999999 2024-09-21T22:08:15.257 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.257 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.257 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:15.257 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.938506+0000", 2024-09-21T22:08:15.257 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.0865999999999999e-05 2024-09-21T22:08:15.257 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.257 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.257 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:15.257 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.938520+0000", 2024-09-21T22:08:15.257 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3825999999999999e-05 2024-09-21T22:08:15.258 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.258 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.258 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:15.258 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.938624+0000", 2024-09-21T22:08:15.258 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00010391900000000001 2024-09-21T22:08:15.258 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.258 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.258 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:15.258 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.938651+0000", 2024-09-21T22:08:15.258 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.7481999999999999e-05 2024-09-21T22:08:15.258 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.258 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.258 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:15.258 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.984369+0000", 2024-09-21T22:08:15.259 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.045717595 2024-09-21T22:08:15.259 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.259 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.259 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:15.259 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.984450+0000", 2024-09-21T22:08:15.259 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.1295e-05 2024-09-21T22:08:15.259 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:15.259 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:15.259 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:15.259 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.259 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.259 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:727 2.0 e63/16 2:08b4f96a:::rbd_data.126483d75f2d.000000000000000f:head v 63'35, mlcod=63'33)", 2024-09-21T22:08:15.259 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.998090+0000", 2024-09-21T22:08:15.259 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 49.181291825999999, 2024-09-21T22:08:15.260 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.052205704999999998, 2024-09-21T22:08:15.260 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:15.260 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:15.260 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:15.260 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:15.260 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.260 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:15.260 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.998090+0000", 2024-09-21T22:08:15.260 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:15.260 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.260 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.260 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:15.260 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.998090+0000", 2024-09-21T22:08:15.260 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:15.261 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.261 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.261 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:15.261 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.998091+0000", 2024-09-21T22:08:15.261 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6929999999999999e-06 2024-09-21T22:08:15.261 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.261 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.261 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:15.261 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.998940+0000", 2024-09-21T22:08:15.261 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00084875200000000004 2024-09-21T22:08:15.261 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.261 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.261 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:15.261 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.998970+0000", 2024-09-21T22:08:15.262 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9709999999999998e-05 2024-09-21T22:08:15.262 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.262 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.262 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:15.262 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.998984+0000", 2024-09-21T22:08:15.262 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3739e-05 2024-09-21T22:08:15.262 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.262 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.262 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:15.262 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.999097+0000", 2024-09-21T22:08:15.262 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000113484 2024-09-21T22:08:15.262 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.262 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.262 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:15.263 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.999135+0000", 2024-09-21T22:08:15.263 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.8405999999999998e-05 2024-09-21T22:08:15.263 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.263 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.263 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:15.263 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.050245+0000", 2024-09-21T22:08:15.263 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.051109935000000002 2024-09-21T22:08:15.263 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.263 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.263 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:15.263 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.050295+0000", 2024-09-21T22:08:15.263 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.9985999999999999e-05 2024-09-21T22:08:15.263 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:15.264 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:15.264 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:15.264 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.264 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.264 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:729 2.0 e63/16 2:08b4f96a:::rbd_data.126483d75f2d.000000000000000f:head v 63'37, mlcod=63'35)", 2024-09-21T22:08:15.264 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:26.062424+0000", 2024-09-21T22:08:15.264 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 49.116957204000002, 2024-09-21T22:08:15.264 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.047312186999999999, 2024-09-21T22:08:15.264 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:15.264 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:15.264 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:15.264 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:15.264 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.264 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:15.265 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.062424+0000", 2024-09-21T22:08:15.265 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:15.265 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.265 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.265 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:15.265 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.062424+0000", 2024-09-21T22:08:15.265 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:15.265 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.265 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.265 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:15.265 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.062426+0000", 2024-09-21T22:08:15.265 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7129999999999999e-06 2024-09-21T22:08:15.265 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.265 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.266 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:15.266 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.063211+0000", 2024-09-21T22:08:15.266 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00078488500000000003 2024-09-21T22:08:15.266 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.266 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.266 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:15.266 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.063241+0000", 2024-09-21T22:08:15.266 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.0056000000000002e-05 2024-09-21T22:08:15.266 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.266 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.266 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:15.266 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.063255+0000", 2024-09-21T22:08:15.266 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4022999999999999e-05 2024-09-21T22:08:15.266 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.267 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.267 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:15.267 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.063384+0000", 2024-09-21T22:08:15.267 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00012920600000000001 2024-09-21T22:08:15.267 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.267 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.267 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:15.267 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.063421+0000", 2024-09-21T22:08:15.267 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.7242e-05 2024-09-21T22:08:15.267 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.267 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.267 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:15.267 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.109666+0000", 2024-09-21T22:08:15.267 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.046244446000000002 2024-09-21T22:08:15.267 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.268 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.268 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:15.268 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.109736+0000", 2024-09-21T22:08:15.268 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.0616000000000003e-05 2024-09-21T22:08:15.268 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:15.268 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:15.268 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:15.268 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.268 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.268 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:731 2.0 e63/16 2:08b4f96a:::rbd_data.126483d75f2d.000000000000000f:head v 63'39, mlcod=63'37)", 2024-09-21T22:08:15.268 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:26.135946+0000", 2024-09-21T22:08:15.268 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 49.043435697, 2024-09-21T22:08:15.268 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.054290359000000003, 2024-09-21T22:08:15.268 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:15.269 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:15.269 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:15.269 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:15.269 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.269 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:15.269 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.135946+0000", 2024-09-21T22:08:15.269 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:15.269 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.269 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.269 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:15.269 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.135946+0000", 2024-09-21T22:08:15.269 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:15.269 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.269 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.269 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:15.270 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.135948+0000", 2024-09-21T22:08:15.270 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7999999999999999e-06 2024-09-21T22:08:15.270 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.270 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.270 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:15.270 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.136730+0000", 2024-09-21T22:08:15.270 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00078271899999999997 2024-09-21T22:08:15.270 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.270 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.270 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:15.270 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.136760+0000", 2024-09-21T22:08:15.270 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9876e-05 2024-09-21T22:08:15.270 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.270 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.271 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:15.271 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.136774+0000", 2024-09-21T22:08:15.271 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3478999999999999e-05 2024-09-21T22:08:15.271 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.271 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.271 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:15.271 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.136944+0000", 2024-09-21T22:08:15.271 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000170626 2024-09-21T22:08:15.271 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.271 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.271 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:15.271 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.136985+0000", 2024-09-21T22:08:15.271 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.0377999999999999e-05 2024-09-21T22:08:15.271 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.271 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.272 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:15.272 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.190137+0000", 2024-09-21T22:08:15.272 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.053152438000000003 2024-09-21T22:08:15.272 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.272 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.272 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:15.272 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.190236+0000", 2024-09-21T22:08:15.272 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.9042999999999993e-05 2024-09-21T22:08:15.272 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:15.272 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:15.272 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:15.272 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.272 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.273 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:125 2.3 e71/66 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head v 71'31, mlcod=71'29)", 2024-09-21T22:08:15.273 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.066855+0000", 2024-09-21T22:08:15.273 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 37.112526262000003, 2024-09-21T22:08:15.273 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.059856059000000003, 2024-09-21T22:08:15.273 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:15.273 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:15.273 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:15.273 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:15.273 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.273 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:15.273 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.066855+0000", 2024-09-21T22:08:15.273 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:15.273 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.273 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.273 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:15.274 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.066855+0000", 2024-09-21T22:08:15.274 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:15.274 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.274 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.274 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:15.274 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.066857+0000", 2024-09-21T22:08:15.274 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.807e-06 2024-09-21T22:08:15.274 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.274 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.274 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:15.274 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.068365+0000", 2024-09-21T22:08:15.274 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0015077910000000001 2024-09-21T22:08:15.274 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.274 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.275 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:15.275 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.068395+0000", 2024-09-21T22:08:15.275 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.0326000000000001e-05 2024-09-21T22:08:15.275 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.275 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.275 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:15.275 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.068415+0000", 2024-09-21T22:08:15.275 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9871999999999999e-05 2024-09-21T22:08:15.275 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.275 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.275 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:15.275 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.068580+0000", 2024-09-21T22:08:15.275 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000165317 2024-09-21T22:08:15.275 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.276 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.276 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:15.276 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.068618+0000", 2024-09-21T22:08:15.276 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.7753000000000003e-05 2024-09-21T22:08:15.276 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.276 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.276 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:15.276 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.126623+0000", 2024-09-21T22:08:15.276 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.058004642000000002 2024-09-21T22:08:15.276 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.276 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.276 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:15.276 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.126711+0000", 2024-09-21T22:08:15.276 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.8551e-05 2024-09-21T22:08:15.276 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:15.277 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:15.277 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:15.277 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.277 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.277 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:127 2.3 e71/66 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head v 71'33, mlcod=71'31)", 2024-09-21T22:08:15.277 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.140864+0000", 2024-09-21T22:08:15.277 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 37.038517233999997, 2024-09-21T22:08:15.277 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.052512380999999997, 2024-09-21T22:08:15.277 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:15.277 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:15.277 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:15.277 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:15.277 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.277 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:15.278 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.140864+0000", 2024-09-21T22:08:15.278 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:15.278 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.278 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.278 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:15.278 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.140864+0000", 2024-09-21T22:08:15.278 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:15.278 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.278 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.278 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:15.278 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.140866+0000", 2024-09-21T22:08:15.278 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.694e-06 2024-09-21T22:08:15.278 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.278 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.279 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:15.279 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.141712+0000", 2024-09-21T22:08:15.279 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00084581499999999998 2024-09-21T22:08:15.279 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.279 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.279 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:15.279 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.141742+0000", 2024-09-21T22:08:15.279 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9865999999999998e-05 2024-09-21T22:08:15.279 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.279 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.279 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:15.279 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.141761+0000", 2024-09-21T22:08:15.279 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9473000000000001e-05 2024-09-21T22:08:15.279 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.280 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.280 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:15.280 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.141919+0000", 2024-09-21T22:08:15.280 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00015758300000000001 2024-09-21T22:08:15.280 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.280 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.280 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:15.280 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.141955+0000", 2024-09-21T22:08:15.280 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.6613000000000003e-05 2024-09-21T22:08:15.280 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.280 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.280 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:15.280 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.193333+0000", 2024-09-21T22:08:15.280 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.051377260000000001 2024-09-21T22:08:15.281 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.281 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.281 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:15.281 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.193377+0000", 2024-09-21T22:08:15.281 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.4076999999999998e-05 2024-09-21T22:08:15.281 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:15.281 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:15.281 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:15.281 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.281 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.281 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:129 2.3 e71/66 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head v 71'35, mlcod=71'33)", 2024-09-21T22:08:15.281 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.209458+0000", 2024-09-21T22:08:15.281 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 36.969923485000002, 2024-09-21T22:08:15.281 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.056595578000000001, 2024-09-21T22:08:15.282 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:15.282 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:15.282 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:15.282 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:15.282 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.282 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:15.282 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.209458+0000", 2024-09-21T22:08:15.282 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:15.282 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.282 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.282 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:15.282 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.209458+0000", 2024-09-21T22:08:15.282 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:15.282 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.282 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.283 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:15.283 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.209460+0000", 2024-09-21T22:08:15.283 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7600000000000001e-06 2024-09-21T22:08:15.283 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.283 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.283 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:15.283 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.210317+0000", 2024-09-21T22:08:15.283 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00085771700000000003 2024-09-21T22:08:15.283 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.283 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.283 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:15.283 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.210346+0000", 2024-09-21T22:08:15.283 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8334e-05 2024-09-21T22:08:15.283 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.284 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.284 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:15.284 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.210365+0000", 2024-09-21T22:08:15.284 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9439000000000001e-05 2024-09-21T22:08:15.284 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.284 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.284 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:15.284 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.210544+0000", 2024-09-21T22:08:15.284 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00017845599999999999 2024-09-21T22:08:15.284 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.284 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.284 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:15.284 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.210584+0000", 2024-09-21T22:08:15.284 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.0423000000000002e-05 2024-09-21T22:08:15.284 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.285 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.285 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:15.285 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.265970+0000", 2024-09-21T22:08:15.285 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.055385765000000003 2024-09-21T22:08:15.285 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.285 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.285 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:15.285 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.266054+0000", 2024-09-21T22:08:15.285 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.3683999999999994e-05 2024-09-21T22:08:15.285 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:15.285 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:15.285 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:15.285 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.285 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.286 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:131 2.3 e71/66 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head v 71'37, mlcod=71'35)", 2024-09-21T22:08:15.286 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.279550+0000", 2024-09-21T22:08:15.286 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 36.899831646999999, 2024-09-21T22:08:15.286 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.052290419999999997, 2024-09-21T22:08:15.286 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:15.286 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:15.286 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:15.286 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:15.286 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.286 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:15.286 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.279550+0000", 2024-09-21T22:08:15.286 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:15.286 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.286 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.287 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:15.287 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.279550+0000", 2024-09-21T22:08:15.287 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:15.287 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.287 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.287 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:15.287 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.279552+0000", 2024-09-21T22:08:15.287 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9439999999999999e-06 2024-09-21T22:08:15.287 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.287 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.287 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:15.287 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.280296+0000", 2024-09-21T22:08:15.287 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00074417000000000005 2024-09-21T22:08:15.288 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.288 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.288 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:15.288 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.280324+0000", 2024-09-21T22:08:15.288 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8045999999999999e-05 2024-09-21T22:08:15.288 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.288 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.288 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:15.288 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.280345+0000", 2024-09-21T22:08:15.288 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.0849999999999999e-05 2024-09-21T22:08:15.288 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.288 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.288 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:15.288 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.280497+0000", 2024-09-21T22:08:15.288 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00015252500000000001 2024-09-21T22:08:15.289 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.289 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.289 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:15.289 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.280531+0000", 2024-09-21T22:08:15.289 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.4034e-05 2024-09-21T22:08:15.289 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.289 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.289 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:15.289 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.331769+0000", 2024-09-21T22:08:15.289 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.051237999999999999 2024-09-21T22:08:15.289 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:15.289 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:15.289 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:15.289 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.331840+0000", 2024-09-21T22:08:15.289 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.0851000000000003e-05 2024-09-21T22:08:15.290 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:15.290 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:15.290 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:15.290 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:15.290 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:15.290 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:08:15.292 INFO:tasks.workunit.client.0.smithi157.stderr:+ DEV=/dev/nbd2 2024-09-21T22:08:15.292 INFO:tasks.workunit.client.0.smithi157.stderr:+ get_pid rbd 2024-09-21T22:08:15.292 INFO:tasks.workunit.client.0.smithi157.stderr:+ local pool=rbd 2024-09-21T22:08:15.292 INFO:tasks.workunit.client.0.smithi157.stderr:+ local ns= 2024-09-21T22:08:15.292 INFO:tasks.workunit.client.0.smithi157.stderr:++ rbd device --device-type nbd --format xml list 2024-09-21T22:08:15.292 INFO:tasks.workunit.client.0.smithi157.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd21553'\''][device='\''/dev/nbd2'\'']/id' 2024-09-21T22:08:15.292 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T22:08:15.293 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:08:15.287+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:08:15.306 INFO:tasks.workunit.client.0.smithi157.stderr:+ PID=24419 2024-09-21T22:08:15.306 INFO:tasks.workunit.client.0.smithi157.stderr:+ test -n 24419 2024-09-21T22:08:15.306 INFO:tasks.workunit.client.0.smithi157.stderr:+ ps -p 24419 -C rbd-nbd 2024-09-21T22:08:15.306 INFO:tasks.workunit.client.0.smithi157.stdout: PID TTY TIME CMD 2024-09-21T22:08:15.307 INFO:tasks.workunit.client.0.smithi157.stdout: 24419 ? 00:00:00 rbd-nbd 2024-09-21T22:08:15.308 INFO:tasks.workunit.client.0.smithi157.stderr:+ _sudo blkdiscard --offset 0 --length 3584 /dev/nbd2 2024-09-21T22:08:15.308 INFO:tasks.workunit.client.0.smithi157.stderr:+ local cmd 2024-09-21T22:08:15.308 INFO:tasks.workunit.client.0.smithi157.stderr:++ id -u 2024-09-21T22:08:15.309 INFO:tasks.workunit.client.0.smithi157.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T22:08:15.309 INFO:tasks.workunit.client.0.smithi157.stderr:++ which blkdiscard 2024-09-21T22:08:15.309 INFO:tasks.workunit.client.0.smithi157.stderr:+ cmd=/usr/sbin/blkdiscard 2024-09-21T22:08:15.309 INFO:tasks.workunit.client.0.smithi157.stderr:+ shift 2024-09-21T22:08:15.310 INFO:tasks.workunit.client.0.smithi157.stderr:+ sudo -nE /usr/sbin/blkdiscard --offset 0 --length 3584 /dev/nbd2 2024-09-21T22:08:15.329 INFO:tasks.workunit.client.0.smithi157.stderr:blkdiscard: /dev/nbd2 contains existing file system (ext2). 2024-09-21T22:08:15.329 INFO:tasks.workunit.client.0.smithi157.stderr:+ _sudo blkdiscard --offset 0 --length 4096 /dev/nbd2 2024-09-21T22:08:15.329 INFO:tasks.workunit.client.0.smithi157.stderr:+ local cmd 2024-09-21T22:08:15.329 INFO:tasks.workunit.client.0.smithi157.stderr:++ id -u 2024-09-21T22:08:15.330 INFO:tasks.workunit.client.0.smithi157.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T22:08:15.330 INFO:tasks.workunit.client.0.smithi157.stderr:++ which blkdiscard 2024-09-21T22:08:15.331 INFO:tasks.workunit.client.0.smithi157.stderr:+ cmd=/usr/sbin/blkdiscard 2024-09-21T22:08:15.332 INFO:tasks.workunit.client.0.smithi157.stderr:+ shift 2024-09-21T22:08:15.332 INFO:tasks.workunit.client.0.smithi157.stderr:+ sudo -nE /usr/sbin/blkdiscard --offset 0 --length 4096 /dev/nbd2 2024-09-21T22:08:15.351 INFO:tasks.workunit.client.0.smithi157.stderr:+ _sudo blkdiscard --offset 0 --length 4608 /dev/nbd2 2024-09-21T22:08:15.352 INFO:tasks.workunit.client.0.smithi157.stderr:+ local cmd 2024-09-21T22:08:15.352 INFO:tasks.workunit.client.0.smithi157.stderr:++ id -u 2024-09-21T22:08:15.352 INFO:tasks.workunit.client.0.smithi157.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T22:08:15.353 INFO:tasks.workunit.client.0.smithi157.stderr:++ which blkdiscard 2024-09-21T22:08:15.353 INFO:tasks.workunit.client.0.smithi157.stderr:+ cmd=/usr/sbin/blkdiscard 2024-09-21T22:08:15.354 INFO:tasks.workunit.client.0.smithi157.stderr:+ shift 2024-09-21T22:08:15.354 INFO:tasks.workunit.client.0.smithi157.stderr:+ sudo -nE /usr/sbin/blkdiscard --offset 0 --length 4608 /dev/nbd2 2024-09-21T22:08:15.369 INFO:tasks.workunit.client.0.smithi157.stderr:+ _sudo blkdiscard --offset 512 --length 7168 /dev/nbd2 2024-09-21T22:08:15.369 INFO:tasks.workunit.client.0.smithi157.stderr:+ local cmd 2024-09-21T22:08:15.369 INFO:tasks.workunit.client.0.smithi157.stderr:++ id -u 2024-09-21T22:08:15.370 INFO:tasks.workunit.client.0.smithi157.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T22:08:15.370 INFO:tasks.workunit.client.0.smithi157.stderr:++ which blkdiscard 2024-09-21T22:08:15.372 INFO:tasks.workunit.client.0.smithi157.stderr:+ cmd=/usr/sbin/blkdiscard 2024-09-21T22:08:15.373 INFO:tasks.workunit.client.0.smithi157.stderr:+ shift 2024-09-21T22:08:15.373 INFO:tasks.workunit.client.0.smithi157.stderr:+ sudo -nE /usr/sbin/blkdiscard --offset 512 --length 7168 /dev/nbd2 2024-09-21T22:08:15.381 INFO:tasks.workunit.client.0.smithi157.stderr:+ _sudo blkdiscard --offset 512 --length 7680 /dev/nbd2 2024-09-21T22:08:15.381 INFO:tasks.workunit.client.0.smithi157.stderr:+ local cmd 2024-09-21T22:08:15.382 INFO:tasks.workunit.client.0.smithi157.stderr:++ id -u 2024-09-21T22:08:15.382 INFO:tasks.workunit.client.0.smithi157.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T22:08:15.383 INFO:tasks.workunit.client.0.smithi157.stderr:++ which blkdiscard 2024-09-21T22:08:15.384 INFO:tasks.workunit.client.0.smithi157.stderr:+ cmd=/usr/sbin/blkdiscard 2024-09-21T22:08:15.384 INFO:tasks.workunit.client.0.smithi157.stderr:+ shift 2024-09-21T22:08:15.384 INFO:tasks.workunit.client.0.smithi157.stderr:+ sudo -nE /usr/sbin/blkdiscard --offset 512 --length 7680 /dev/nbd2 2024-09-21T22:08:15.397 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:08:15.391+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:08:15.398 INFO:tasks.workunit.client.0.smithi157.stderr:+ _sudo blkdiscard --offset 512 --length 8192 /dev/nbd2 2024-09-21T22:08:15.398 INFO:tasks.workunit.client.0.smithi157.stderr:+ local cmd 2024-09-21T22:08:15.398 INFO:tasks.workunit.client.0.smithi157.stderr:++ id -u 2024-09-21T22:08:15.399 INFO:tasks.workunit.client.0.smithi157.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T22:08:15.399 INFO:tasks.workunit.client.0.smithi157.stderr:++ which blkdiscard 2024-09-21T22:08:15.399 INFO:tasks.workunit.client.0.smithi157.stderr:+ cmd=/usr/sbin/blkdiscard 2024-09-21T22:08:15.399 INFO:tasks.workunit.client.0.smithi157.stderr:+ shift 2024-09-21T22:08:15.399 INFO:tasks.workunit.client.0.smithi157.stderr:+ sudo -nE /usr/sbin/blkdiscard --offset 512 --length 8192 /dev/nbd2 2024-09-21T22:08:15.413 INFO:tasks.workunit.client.0.smithi157.stderr:+ tries=0 2024-09-21T22:08:15.415 INFO:tasks.workunit.client.0.smithi157.stderr:++ rbd journal inspect --pool rbd --image testrbdnbd21553 2024-09-21T22:08:15.415 INFO:tasks.workunit.client.0.smithi157.stderr:++ awk '/entries inspected/ {print $1}' 2024-09-21T22:08:15.436 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:08:15.440 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:08:15.440 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-09-21T22:08:15.440 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:08:15.451 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T22:08:15.491 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-21T22:08:15.484+0000 7fd5df7fe640 -1 JournalMetadata: 0x556c4d4dcfd0 failed to locate client: INSPECT 2024-09-21T22:08:15.493 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:08:15.489+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:08:15.501 INFO:tasks.workunit.client.0.smithi157.stderr:+ queue_length=4 2024-09-21T22:08:15.501 INFO:tasks.workunit.client.0.smithi157.stderr:+ '[' 0 -lt 10 ']' 2024-09-21T22:08:15.501 INFO:tasks.workunit.client.0.smithi157.stderr:+ '[' 4 -gt 0 ']' 2024-09-21T22:08:15.501 INFO:tasks.workunit.client.0.smithi157.stderr:+ rbd journal inspect --pool rbd --image testrbdnbd21553 --verbose 2024-09-21T22:08:15.565 INFO:tasks.workunit.client.0.smithi157.stdout:Entry: tag_id=1, commit_tid=1 2024-09-21T22:08:15.565 INFO:tasks.workunit.client.0.smithi157.stdout:{ 2024-09-21T22:08:15.566 INFO:tasks.workunit.client.0.smithi157.stdout: "event_type": "AioDiscard", 2024-09-21T22:08:15.566 INFO:tasks.workunit.client.0.smithi157.stdout: "offset": 0, 2024-09-21T22:08:15.566 INFO:tasks.workunit.client.0.smithi157.stdout: "length": 4096, 2024-09-21T22:08:15.566 INFO:tasks.workunit.client.0.smithi157.stdout: "discard_granularity_bytes": 4096, 2024-09-21T22:08:15.566 INFO:tasks.workunit.client.0.smithi157.stdout: "timestamp": "2024-09-21T22:08:15.344115+0000" 2024-09-21T22:08:15.566 INFO:tasks.workunit.client.0.smithi157.stdout:} 2024-09-21T22:08:15.566 INFO:tasks.workunit.client.0.smithi157.stdout:Entry: tag_id=1, commit_tid=2 2024-09-21T22:08:15.566 INFO:tasks.workunit.client.0.smithi157.stdout:{ 2024-09-21T22:08:15.566 INFO:tasks.workunit.client.0.smithi157.stdout: "event_type": "AioDiscard", 2024-09-21T22:08:15.566 INFO:tasks.workunit.client.0.smithi157.stdout: "offset": 0, 2024-09-21T22:08:15.566 INFO:tasks.workunit.client.0.smithi157.stdout: "length": 4096, 2024-09-21T22:08:15.566 INFO:tasks.workunit.client.0.smithi157.stdout: "discard_granularity_bytes": 4096, 2024-09-21T22:08:15.567 INFO:tasks.workunit.client.0.smithi157.stdout: "timestamp": "2024-09-21T22:08:15.360522+0000" 2024-09-21T22:08:15.567 INFO:tasks.workunit.client.0.smithi157.stdout:} 2024-09-21T22:08:15.567 INFO:tasks.workunit.client.0.smithi157.stdout:Entry: tag_id=1, commit_tid=3 2024-09-21T22:08:15.567 INFO:tasks.workunit.client.0.smithi157.stdout:{ 2024-09-21T22:08:15.567 INFO:tasks.workunit.client.0.smithi157.stdout: "event_type": "AioDiscard", 2024-09-21T22:08:15.567 INFO:tasks.workunit.client.0.smithi157.stdout: "offset": 4096, 2024-09-21T22:08:15.567 INFO:tasks.workunit.client.0.smithi157.stdout: "length": 4096, 2024-09-21T22:08:15.567 INFO:tasks.workunit.client.0.smithi157.stdout: "discard_granularity_bytes": 4096, 2024-09-21T22:08:15.567 INFO:tasks.workunit.client.0.smithi157.stdout: "timestamp": "2024-09-21T22:08:15.387911+0000" 2024-09-21T22:08:15.567 INFO:tasks.workunit.client.0.smithi157.stdout:} 2024-09-21T22:08:15.567 INFO:tasks.workunit.client.0.smithi157.stdout:Entry: tag_id=1, commit_tid=4 2024-09-21T22:08:15.575 INFO:tasks.workunit.client.0.smithi157.stdout:{ 2024-09-21T22:08:15.575 INFO:tasks.workunit.client.0.smithi157.stdout: "event_type": "AioDiscard", 2024-09-21T22:08:15.575 INFO:tasks.workunit.client.0.smithi157.stdout: "offset": 4096, 2024-09-21T22:08:15.575 INFO:tasks.workunit.client.0.smithi157.stdout: "length": 4096, 2024-09-21T22:08:15.575 INFO:tasks.workunit.client.0.smithi157.stdout: "discard_granularity_bytes": 4096, 2024-09-21T22:08:15.575 INFO:tasks.workunit.client.0.smithi157.stdout: "timestamp": "2024-09-21T22:08:15.404762+0000" 2024-09-21T22:08:15.575 INFO:tasks.workunit.client.0.smithi157.stdout:} 2024-09-21T22:08:15.576 INFO:tasks.workunit.client.0.smithi157.stdout:Summary: 2024-09-21T22:08:15.576 INFO:tasks.workunit.client.0.smithi157.stdout: 4 entries inspected, 0 errors 2024-09-21T22:08:15.576 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-21T22:08:15.564+0000 7f7c3c87d640 -1 JournalMetadata: 0x55f3964c5460 failed to locate client: INSPECT 2024-09-21T22:08:15.577 INFO:tasks.workunit.client.0.smithi157.stderr:+ sleep 1 2024-09-21T22:08:15.594 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:08:15.591+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:08:15.631 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:08:15.631 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:08:15.631 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-21T22:08:15.631 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-21T22:08:15.632 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:08:15.647 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T22:08:15.695 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:08:15.689+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:08:15.795 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:08:15.791+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:08:15.814 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:08:15.815 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-21T22:08:15.815 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-21T22:08:15.815 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-21T22:08:15.815 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.815 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:08:15.815 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.036355+0000", 2024-09-21T22:08:15.815 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 133.77506739200001, 2024-09-21T22:08:15.816 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.062441934999999997, 2024-09-21T22:08:15.816 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:15.816 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:15.816 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:15.816 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:15.816 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:08:15.816 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:08:15.817 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1 2024-09-21T22:08:15.817 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.817 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:15.817 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.817 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:15.817 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036355+0000", 2024-09-21T22:08:15.817 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:15.818 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.818 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.818 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:15.818 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036355+0000", 2024-09-21T22:08:15.818 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:15.819 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.819 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.819 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:15.819 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036357+0000", 2024-09-21T22:08:15.819 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.446e-06 2024-09-21T22:08:15.819 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.819 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.819 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:15.820 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036373+0000", 2024-09-21T22:08:15.820 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5447e-05 2024-09-21T22:08:15.820 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.820 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.820 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:15.820 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036399+0000", 2024-09-21T22:08:15.820 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6423000000000001e-05 2024-09-21T22:08:15.821 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.821 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.821 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:15.821 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036420+0000", 2024-09-21T22:08:15.821 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1129999999999999e-05 2024-09-21T22:08:15.821 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.821 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.822 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:15.822 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036557+0000", 2024-09-21T22:08:15.822 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013686899999999999 2024-09-21T22:08:15.822 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.822 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.822 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-21T22:08:15.822 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036585+0000", 2024-09-21T22:08:15.822 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8362999999999999e-05 2024-09-21T22:08:15.823 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.823 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.823 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:15.823 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.095068+0000", 2024-09-21T22:08:15.823 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058482972000000001 2024-09-21T22:08:15.823 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.823 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.824 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:15.824 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.095516+0000", 2024-09-21T22:08:15.824 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00044802999999999998 2024-09-21T22:08:15.824 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.824 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.824 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:08:15.824 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.095869+0000", 2024-09-21T22:08:15.824 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000352104 2024-09-21T22:08:15.825 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.825 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.825 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:15.825 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.097055+0000", 2024-09-21T22:08:15.825 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011865599999999999 2024-09-21T22:08:15.825 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.825 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.826 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:15.826 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.098637+0000", 2024-09-21T22:08:15.826 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001581458 2024-09-21T22:08:15.826 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.826 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.826 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:15.826 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.098728+0000", 2024-09-21T22:08:15.827 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1043000000000002e-05 2024-09-21T22:08:15.827 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.827 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.827 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:15.827 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.098797+0000", 2024-09-21T22:08:15.827 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9090000000000004e-05 2024-09-21T22:08:15.827 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:15.827 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:15.828 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:15.828 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.828 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.828 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4840.0:680 2.7 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:08:15.828 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.744506+0000", 2024-09-21T22:08:15.828 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 51.066916538000001, 2024-09-21T22:08:15.828 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.063865055000000004, 2024-09-21T22:08:15.828 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:15.829 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:15.829 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:15.829 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:15.829 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4840", 2024-09-21T22:08:15.829 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:08:15.829 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 680 2024-09-21T22:08:15.829 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.830 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:15.830 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.830 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:15.830 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.744506+0000", 2024-09-21T22:08:15.830 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:15.830 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.830 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.830 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:15.831 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.744506+0000", 2024-09-21T22:08:15.831 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:15.831 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.831 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.831 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:15.831 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.744508+0000", 2024-09-21T22:08:15.831 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.103e-06 2024-09-21T22:08:15.832 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.832 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.832 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:15.832 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.745371+0000", 2024-09-21T22:08:15.832 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00086322199999999997 2024-09-21T22:08:15.832 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.832 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.833 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:15.833 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.745401+0000", 2024-09-21T22:08:15.833 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9955999999999999e-05 2024-09-21T22:08:15.833 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.833 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.833 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:15.833 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.745416+0000", 2024-09-21T22:08:15.833 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4873e-05 2024-09-21T22:08:15.834 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.834 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.834 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:15.834 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.745509+0000", 2024-09-21T22:08:15.834 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3171000000000006e-05 2024-09-21T22:08:15.834 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.834 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.835 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:15.835 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.745632+0000", 2024-09-21T22:08:15.835 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012272699999999999 2024-09-21T22:08:15.835 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.835 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.835 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-09-21T22:08:15.835 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.745789+0000", 2024-09-21T22:08:15.835 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015781399999999999 2024-09-21T22:08:15.835 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.836 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.836 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:15.836 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.785494+0000", 2024-09-21T22:08:15.836 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.039704401 2024-09-21T22:08:15.836 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.836 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.836 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:15.836 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.808090+0000", 2024-09-21T22:08:15.836 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022596207 2024-09-21T22:08:15.837 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.837 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.837 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:15.837 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.808246+0000", 2024-09-21T22:08:15.837 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015567200000000001 2024-09-21T22:08:15.837 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.838 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.838 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:15.838 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.808371+0000", 2024-09-21T22:08:15.838 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012490900000000001 2024-09-21T22:08:15.838 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:15.838 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:15.838 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:15.839 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.839 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.839 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4840.0:682 2.7 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:08:15.839 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.822415+0000", 2024-09-21T22:08:15.839 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 50.989006975000002, 2024-09-21T22:08:15.839 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.065044633000000004, 2024-09-21T22:08:15.839 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:15.839 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:15.840 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:15.840 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:15.840 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4840", 2024-09-21T22:08:15.840 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:08:15.840 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 682 2024-09-21T22:08:15.840 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.840 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:15.840 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.841 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:15.841 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.822415+0000", 2024-09-21T22:08:15.841 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:15.841 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.841 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.841 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:15.841 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.822415+0000", 2024-09-21T22:08:15.841 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:15.842 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.842 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.842 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:15.842 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.822417+0000", 2024-09-21T22:08:15.842 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8029999999999999e-06 2024-09-21T22:08:15.842 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.842 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.843 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:15.843 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.823308+0000", 2024-09-21T22:08:15.843 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00089146500000000003 2024-09-21T22:08:15.843 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.843 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.843 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:15.843 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.823339+0000", 2024-09-21T22:08:15.843 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0493000000000001e-05 2024-09-21T22:08:15.843 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.844 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.844 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:15.844 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.823354+0000", 2024-09-21T22:08:15.844 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5472999999999999e-05 2024-09-21T22:08:15.844 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.844 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.844 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:15.845 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.823476+0000", 2024-09-21T22:08:15.845 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000121209 2024-09-21T22:08:15.845 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.845 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.845 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:15.845 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.823650+0000", 2024-09-21T22:08:15.845 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017394599999999999 2024-09-21T22:08:15.846 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.846 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.846 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-09-21T22:08:15.846 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.823873+0000", 2024-09-21T22:08:15.846 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022314899999999999 2024-09-21T22:08:15.846 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.846 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.846 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:15.847 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.884396+0000", 2024-09-21T22:08:15.847 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.060523674999999999 2024-09-21T22:08:15.847 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.847 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.847 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:15.847 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.887238+0000", 2024-09-21T22:08:15.847 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028420450000000001 2024-09-21T22:08:15.848 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.848 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.848 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:15.848 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.887369+0000", 2024-09-21T22:08:15.848 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013010699999999999 2024-09-21T22:08:15.848 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.848 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.848 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:15.849 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.887460+0000", 2024-09-21T22:08:15.849 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1267999999999994e-05 2024-09-21T22:08:15.849 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:15.849 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:15.849 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:15.849 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.849 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.849 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4840.0:692 2.5 2:b1d55633:::rbd_data.126483d75f2d.000000000000000b:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:08:15.850 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:25.091048+0000", 2024-09-21T22:08:15.850 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 50.720374440999997, 2024-09-21T22:08:15.850 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.060917092999999999, 2024-09-21T22:08:15.850 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:15.850 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:15.850 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:15.850 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:15.851 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4840", 2024-09-21T22:08:15.851 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:08:15.851 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 692 2024-09-21T22:08:15.851 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.851 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:15.851 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.851 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:15.851 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.091048+0000", 2024-09-21T22:08:15.852 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:15.852 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.852 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.852 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:15.852 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.091048+0000", 2024-09-21T22:08:15.852 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:15.852 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.852 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.852 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:15.853 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.091050+0000", 2024-09-21T22:08:15.853 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.03e-06 2024-09-21T22:08:15.853 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.853 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.853 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:15.853 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.091989+0000", 2024-09-21T22:08:15.853 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00093932100000000001 2024-09-21T22:08:15.853 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.853 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.854 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:15.854 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.092023+0000", 2024-09-21T22:08:15.854 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4310000000000002e-05 2024-09-21T22:08:15.854 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.854 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.854 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:15.854 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.092040+0000", 2024-09-21T22:08:15.854 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6426000000000002e-05 2024-09-21T22:08:15.854 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.855 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.855 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:15.855 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.092175+0000", 2024-09-21T22:08:15.855 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013500299999999999 2024-09-21T22:08:15.855 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.856 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.856 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:15.856 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.092307+0000", 2024-09-21T22:08:15.856 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000132562 2024-09-21T22:08:15.856 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.856 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.856 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:08:15.857 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.092499+0000", 2024-09-21T22:08:15.857 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000191972 2024-09-21T22:08:15.857 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.857 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.857 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:15.857 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.151065+0000", 2024-09-21T22:08:15.857 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058565514999999999 2024-09-21T22:08:15.857 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.858 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.858 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:15.858 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.151844+0000", 2024-09-21T22:08:15.858 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00077930800000000004 2024-09-21T22:08:15.858 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.859 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.859 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:15.859 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.151916+0000", 2024-09-21T22:08:15.859 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1921999999999993e-05 2024-09-21T22:08:15.859 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.859 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.859 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:15.859 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.151965+0000", 2024-09-21T22:08:15.860 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8724000000000003e-05 2024-09-21T22:08:15.860 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:15.860 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:15.861 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:15.861 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.862 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.862 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4840.0:696 2.5 2:b1d55633:::rbd_data.126483d75f2d.000000000000000b:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:08:15.862 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:25.218641+0000", 2024-09-21T22:08:15.862 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 50.592781172999999, 2024-09-21T22:08:15.862 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058363669999999999, 2024-09-21T22:08:15.862 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:15.862 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:15.862 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:15.863 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:15.863 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4840", 2024-09-21T22:08:15.863 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:08:15.863 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 696 2024-09-21T22:08:15.863 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.863 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:15.863 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.863 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:15.864 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.218641+0000", 2024-09-21T22:08:15.864 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:15.864 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.864 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.864 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:15.864 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.218641+0000", 2024-09-21T22:08:15.864 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:15.864 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.865 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.865 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:15.865 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.218642+0000", 2024-09-21T22:08:15.865 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.266e-06 2024-09-21T22:08:15.865 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.865 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.865 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:15.865 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.219432+0000", 2024-09-21T22:08:15.866 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00078986300000000002 2024-09-21T22:08:15.866 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.866 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.866 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:15.866 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.219450+0000", 2024-09-21T22:08:15.866 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7526000000000001e-05 2024-09-21T22:08:15.867 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.867 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.867 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:15.867 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.219459+0000", 2024-09-21T22:08:15.867 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3000000000000007e-06 2024-09-21T22:08:15.867 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.867 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.867 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:15.868 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.219655+0000", 2024-09-21T22:08:15.868 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019599099999999999 2024-09-21T22:08:15.868 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.868 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.868 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:15.868 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.219917+0000", 2024-09-21T22:08:15.868 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026198200000000001 2024-09-21T22:08:15.869 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.869 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.869 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:08:15.869 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.220198+0000", 2024-09-21T22:08:15.869 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00028154500000000002 2024-09-21T22:08:15.869 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.869 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.869 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:15.870 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.268856+0000", 2024-09-21T22:08:15.870 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.048657687999999998 2024-09-21T22:08:15.870 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.870 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.870 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:15.870 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.276903+0000", 2024-09-21T22:08:15.870 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0080472249999999999 2024-09-21T22:08:15.871 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.871 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.871 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:15.871 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.276958+0000", 2024-09-21T22:08:15.871 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4184999999999997e-05 2024-09-21T22:08:15.871 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.871 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.872 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:15.872 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.277005+0000", 2024-09-21T22:08:15.872 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7098999999999997e-05 2024-09-21T22:08:15.872 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:15.872 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:15.872 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:15.872 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.872 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.873 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4840.0:729 2.0 2:08b4f96a:::rbd_data.126483d75f2d.000000000000000f:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:08:15.873 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:26.062012+0000", 2024-09-21T22:08:15.873 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 49.74940994, 2024-09-21T22:08:15.873 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.060656823999999998, 2024-09-21T22:08:15.873 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:15.873 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:15.873 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:15.874 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:15.874 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4840", 2024-09-21T22:08:15.874 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:08:15.874 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 729 2024-09-21T22:08:15.874 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.874 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:15.874 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.874 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:15.875 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.062012+0000", 2024-09-21T22:08:15.875 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:15.875 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.875 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.875 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:15.875 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.062012+0000", 2024-09-21T22:08:15.875 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:15.876 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.876 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.876 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:15.876 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.062013+0000", 2024-09-21T22:08:15.876 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.243e-06 2024-09-21T22:08:15.876 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.876 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.876 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:15.877 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.062827+0000", 2024-09-21T22:08:15.877 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00081374799999999999 2024-09-21T22:08:15.877 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.877 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.877 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:15.877 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.062845+0000", 2024-09-21T22:08:15.877 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.808e-05 2024-09-21T22:08:15.878 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.878 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.878 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:15.878 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.062854+0000", 2024-09-21T22:08:15.878 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1039999999999996e-06 2024-09-21T22:08:15.878 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.878 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.878 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:15.879 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.062970+0000", 2024-09-21T22:08:15.879 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011601 2024-09-21T22:08:15.879 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.879 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.879 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:15.879 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.063230+0000", 2024-09-21T22:08:15.879 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025918799999999999 2024-09-21T22:08:15.880 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.880 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.880 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-09-21T22:08:15.880 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.063641+0000", 2024-09-21T22:08:15.880 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00041152799999999998 2024-09-21T22:08:15.880 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.880 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.880 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:15.881 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.122015+0000", 2024-09-21T22:08:15.881 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058374246999999997 2024-09-21T22:08:15.881 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.881 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.881 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:15.881 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.122451+0000", 2024-09-21T22:08:15.881 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000435771 2024-09-21T22:08:15.882 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.882 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.882 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:15.882 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.122576+0000", 2024-09-21T22:08:15.882 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012469099999999999 2024-09-21T22:08:15.882 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.882 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.882 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:15.883 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.122669+0000", 2024-09-21T22:08:15.883 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3214000000000006e-05 2024-09-21T22:08:15.883 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:15.883 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:15.883 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:15.883 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.883 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.884 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4840.0:731 2.0 2:08b4f96a:::rbd_data.126483d75f2d.000000000000000f:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:08:15.884 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:26.135603+0000", 2024-09-21T22:08:15.884 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 49.675819439999998, 2024-09-21T22:08:15.884 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.056838474999999999, 2024-09-21T22:08:15.884 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:15.884 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:15.884 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:15.885 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:15.885 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4840", 2024-09-21T22:08:15.885 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:08:15.885 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 731 2024-09-21T22:08:15.885 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.885 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:15.885 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.885 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:15.886 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.135603+0000", 2024-09-21T22:08:15.886 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:15.886 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.886 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.886 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:15.886 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.135603+0000", 2024-09-21T22:08:15.886 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:15.887 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.887 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.887 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:15.887 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.135605+0000", 2024-09-21T22:08:15.887 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9700000000000002e-06 2024-09-21T22:08:15.887 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.887 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.887 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:15.888 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.136554+0000", 2024-09-21T22:08:15.888 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00094886700000000005 2024-09-21T22:08:15.888 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.888 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.888 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:15.888 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.136582+0000", 2024-09-21T22:08:15.888 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8393000000000001e-05 2024-09-21T22:08:15.888 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.889 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.889 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:15.889 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.136596+0000", 2024-09-21T22:08:15.889 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.43e-05 2024-09-21T22:08:15.889 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.889 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.889 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:15.890 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.136691+0000", 2024-09-21T22:08:15.890 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4843e-05 2024-09-21T22:08:15.890 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.890 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.890 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:15.890 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.136873+0000", 2024-09-21T22:08:15.890 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018237099999999999 2024-09-21T22:08:15.890 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.891 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.891 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-09-21T22:08:15.891 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.137140+0000", 2024-09-21T22:08:15.891 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026642700000000002 2024-09-21T22:08:15.891 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.891 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.891 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:15.892 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.180580+0000", 2024-09-21T22:08:15.892 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.043440078 2024-09-21T22:08:15.892 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.892 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.892 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:15.892 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.192175+0000", 2024-09-21T22:08:15.892 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011594959 2024-09-21T22:08:15.893 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.893 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.893 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:15.893 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.192353+0000", 2024-09-21T22:08:15.893 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017850200000000001 2024-09-21T22:08:15.893 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.893 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.894 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:15.894 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.192441+0000", 2024-09-21T22:08:15.894 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7764999999999997e-05 2024-09-21T22:08:15.894 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:15.894 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:15.894 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:15.894 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.894 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.895 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5207.0:34 2.7 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head [write 262144~131072] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e78)", 2024-09-21T22:08:15.895 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:52.587708+0000", 2024-09-21T22:08:15.895 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 23.223714419, 2024-09-21T22:08:15.895 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.056690213000000003, 2024-09-21T22:08:15.895 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:15.895 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:15.895 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:15.896 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:15.896 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5207", 2024-09-21T22:08:15.896 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/357731919", 2024-09-21T22:08:15.896 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 34 2024-09-21T22:08:15.896 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.896 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:15.896 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.896 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:15.896 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.587708+0000", 2024-09-21T22:08:15.896 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:15.896 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.896 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.897 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:15.897 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.587708+0000", 2024-09-21T22:08:15.897 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:15.897 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.897 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.897 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:15.897 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.587709+0000", 2024-09-21T22:08:15.897 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.72e-06 2024-09-21T22:08:15.897 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.897 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.897 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:15.897 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588017+0000", 2024-09-21T22:08:15.897 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00030767499999999998 2024-09-21T22:08:15.897 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.898 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.898 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:15.898 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588037+0000", 2024-09-21T22:08:15.898 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9865999999999999e-05 2024-09-21T22:08:15.898 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.898 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.898 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:15.898 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588048+0000", 2024-09-21T22:08:15.898 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1384e-05 2024-09-21T22:08:15.898 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.898 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.898 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:15.898 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.614263+0000", 2024-09-21T22:08:15.898 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026214232000000001 2024-09-21T22:08:15.899 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.899 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.899 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:15.899 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.614397+0000", 2024-09-21T22:08:15.899 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013396299999999999 2024-09-21T22:08:15.899 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.899 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.899 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-09-21T22:08:15.899 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.614670+0000", 2024-09-21T22:08:15.899 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027295500000000002 2024-09-21T22:08:15.899 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.899 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.899 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:15.899 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.644046+0000", 2024-09-21T22:08:15.900 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029376973000000001 2024-09-21T22:08:15.900 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.900 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.900 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:15.900 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.644208+0000", 2024-09-21T22:08:15.900 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016169900000000001 2024-09-21T22:08:15.900 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.900 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.900 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:15.900 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.644335+0000", 2024-09-21T22:08:15.900 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012656000000000001 2024-09-21T22:08:15.900 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.900 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.900 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:15.901 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.644398+0000", 2024-09-21T22:08:15.901 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3186000000000001e-05 2024-09-21T22:08:15.901 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:15.901 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:15.901 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:15.901 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.901 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.901 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5207.0:35 2.7 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head [write 393216~131072] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e78)", 2024-09-21T22:08:15.901 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:52.588070+0000", 2024-09-21T22:08:15.901 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 23.223352520999999, 2024-09-21T22:08:15.901 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.076411232999999995, 2024-09-21T22:08:15.901 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:15.901 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:15.901 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:15.902 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:15.902 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5207", 2024-09-21T22:08:15.902 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/357731919", 2024-09-21T22:08:15.902 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 35 2024-09-21T22:08:15.902 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.902 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:15.902 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.902 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:15.902 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588070+0000", 2024-09-21T22:08:15.902 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:15.902 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.902 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.902 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:15.902 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588070+0000", 2024-09-21T22:08:15.903 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:15.903 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.903 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.903 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:15.903 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588071+0000", 2024-09-21T22:08:15.903 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1299999999999998e-07 2024-09-21T22:08:15.903 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.903 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.903 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:15.903 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588327+0000", 2024-09-21T22:08:15.903 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025630199999999997 2024-09-21T22:08:15.903 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.903 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.903 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:15.903 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588343+0000", 2024-09-21T22:08:15.904 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6140000000000001e-05 2024-09-21T22:08:15.904 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.904 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.904 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:15.904 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588353+0000", 2024-09-21T22:08:15.904 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0043e-05 2024-09-21T22:08:15.904 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.904 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.904 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:15.904 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.623978+0000", 2024-09-21T22:08:15.904 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.035625281000000002 2024-09-21T22:08:15.904 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.904 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.904 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:15.905 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.624104+0000", 2024-09-21T22:08:15.905 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012591300000000001 2024-09-21T22:08:15.905 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.905 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.905 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-09-21T22:08:15.905 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.624296+0000", 2024-09-21T22:08:15.905 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019226200000000001 2024-09-21T22:08:15.905 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.905 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.905 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:15.905 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.654488+0000", 2024-09-21T22:08:15.905 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.030191551 2024-09-21T22:08:15.905 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.905 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.906 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:15.906 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.664329+0000", 2024-09-21T22:08:15.906 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0098405119999999992 2024-09-21T22:08:15.906 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.906 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.906 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:15.906 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.664414+0000", 2024-09-21T22:08:15.906 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5385999999999999e-05 2024-09-21T22:08:15.906 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.906 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.906 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:15.906 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.664481+0000", 2024-09-21T22:08:15.906 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6929999999999998e-05 2024-09-21T22:08:15.906 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:15.907 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:15.907 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:15.907 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.907 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.907 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5207.0:36 2.7 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head [write 524288~131072] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e78)", 2024-09-21T22:08:15.907 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:52.588370+0000", 2024-09-21T22:08:15.907 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 23.223051742999999, 2024-09-21T22:08:15.907 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.076552001999999994, 2024-09-21T22:08:15.907 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:15.907 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:15.907 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:15.907 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:15.907 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5207", 2024-09-21T22:08:15.907 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/357731919", 2024-09-21T22:08:15.908 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 36 2024-09-21T22:08:15.908 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.908 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:15.908 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.908 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:15.908 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588370+0000", 2024-09-21T22:08:15.908 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:15.908 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.908 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.908 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:15.908 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588370+0000", 2024-09-21T22:08:15.908 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:15.908 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.908 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.908 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:15.909 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588372+0000", 2024-09-21T22:08:15.909 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.153e-06 2024-09-21T22:08:15.909 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.909 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.909 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:15.909 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588716+0000", 2024-09-21T22:08:15.909 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00034479200000000002 2024-09-21T22:08:15.909 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.909 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.909 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:15.909 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588752+0000", 2024-09-21T22:08:15.909 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5865999999999998e-05 2024-09-21T22:08:15.909 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.909 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.910 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:15.910 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588777+0000", 2024-09-21T22:08:15.910 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4583000000000002e-05 2024-09-21T22:08:15.910 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.910 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.910 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:15.910 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.633886+0000", 2024-09-21T22:08:15.910 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.045108941999999999 2024-09-21T22:08:15.910 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.910 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.910 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:15.910 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.634059+0000", 2024-09-21T22:08:15.910 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000173046 2024-09-21T22:08:15.910 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.911 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.911 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-09-21T22:08:15.911 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.634314+0000", 2024-09-21T22:08:15.911 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025482400000000001 2024-09-21T22:08:15.911 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.911 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.911 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:15.911 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.664541+0000", 2024-09-21T22:08:15.911 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.030227222000000002 2024-09-21T22:08:15.911 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.911 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.911 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:15.911 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.664653+0000", 2024-09-21T22:08:15.911 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011192900000000001 2024-09-21T22:08:15.912 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.912 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.912 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:15.912 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.664798+0000", 2024-09-21T22:08:15.912 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014499900000000001 2024-09-21T22:08:15.912 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.912 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.912 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:15.912 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.664922+0000", 2024-09-21T22:08:15.912 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012464600000000001 2024-09-21T22:08:15.912 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:15.912 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:15.912 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:15.912 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.913 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.913 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5207.0:37 2.7 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head [write 655360~131072] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e78)", 2024-09-21T22:08:15.913 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:52.588813+0000", 2024-09-21T22:08:15.913 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 23.222608729000001, 2024-09-21T22:08:15.913 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.087074796999999995, 2024-09-21T22:08:15.913 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:15.913 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:15.913 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:15.913 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:15.913 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5207", 2024-09-21T22:08:15.913 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/357731919", 2024-09-21T22:08:15.913 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 37 2024-09-21T22:08:15.913 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.913 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:15.914 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.914 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:15.914 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588813+0000", 2024-09-21T22:08:15.914 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:15.914 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.914 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.914 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:15.914 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588813+0000", 2024-09-21T22:08:15.914 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:15.914 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.914 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.914 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:15.914 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588815+0000", 2024-09-21T22:08:15.914 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.643e-06 2024-09-21T22:08:15.914 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.915 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.915 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:15.915 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.589071+0000", 2024-09-21T22:08:15.915 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025580200000000002 2024-09-21T22:08:15.915 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.915 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.915 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:15.915 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.589098+0000", 2024-09-21T22:08:15.915 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7336000000000001e-05 2024-09-21T22:08:15.915 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.915 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.915 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:15.915 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.589121+0000", 2024-09-21T22:08:15.916 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2373000000000001e-05 2024-09-21T22:08:15.916 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.916 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.916 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:15.916 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.644491+0000", 2024-09-21T22:08:15.916 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.055369994999999998 2024-09-21T22:08:15.916 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.916 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.916 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:15.916 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.644707+0000", 2024-09-21T22:08:15.916 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000216488 2024-09-21T22:08:15.916 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.916 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.916 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-09-21T22:08:15.916 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.644908+0000", 2024-09-21T22:08:15.917 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000201196 2024-09-21T22:08:15.917 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.917 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.917 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:15.917 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.675419+0000", 2024-09-21T22:08:15.917 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.03051071 2024-09-21T22:08:15.917 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.917 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.917 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:15.917 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.675591+0000", 2024-09-21T22:08:15.917 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000172545 2024-09-21T22:08:15.917 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.917 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.917 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:15.918 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.675761+0000", 2024-09-21T22:08:15.918 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016942300000000001 2024-09-21T22:08:15.918 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.918 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.918 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:15.918 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.675888+0000", 2024-09-21T22:08:15.918 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000127286 2024-09-21T22:08:15.918 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:15.918 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:15.918 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:15.918 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.918 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.918 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5207.0:38 2.7 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head [write 786432~131072] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e78)", 2024-09-21T22:08:15.918 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:52.589156+0000", 2024-09-21T22:08:15.919 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 23.222265825000001, 2024-09-21T22:08:15.919 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.087126880000000004, 2024-09-21T22:08:15.919 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:15.919 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:15.919 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:15.919 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:15.919 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5207", 2024-09-21T22:08:15.919 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/357731919", 2024-09-21T22:08:15.919 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 38 2024-09-21T22:08:15.919 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.919 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:15.919 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.919 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:15.919 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.589156+0000", 2024-09-21T22:08:15.920 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:15.920 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.920 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.920 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:15.920 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.589156+0000", 2024-09-21T22:08:15.920 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:15.920 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.920 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.920 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:15.920 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.589158+0000", 2024-09-21T22:08:15.920 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.717e-06 2024-09-21T22:08:15.920 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.920 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.920 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:15.921 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.589417+0000", 2024-09-21T22:08:15.921 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025860500000000002 2024-09-21T22:08:15.921 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.921 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.921 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:15.921 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.589447+0000", 2024-09-21T22:08:15.921 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9916e-05 2024-09-21T22:08:15.921 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.921 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.921 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:15.921 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.589467+0000", 2024-09-21T22:08:15.921 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0894e-05 2024-09-21T22:08:15.921 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.921 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.922 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:15.922 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.654576+0000", 2024-09-21T22:08:15.922 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.065108579999999999 2024-09-21T22:08:15.922 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.922 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.922 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:15.922 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.654694+0000", 2024-09-21T22:08:15.922 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000118083 2024-09-21T22:08:15.922 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.922 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.922 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-09-21T22:08:15.922 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.654880+0000", 2024-09-21T22:08:15.922 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000186069 2024-09-21T22:08:15.922 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.923 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.923 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:15.923 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.675453+0000", 2024-09-21T22:08:15.923 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020572498000000002 2024-09-21T22:08:15.923 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.923 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.923 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:15.923 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.676055+0000", 2024-09-21T22:08:15.923 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00060247000000000002 2024-09-21T22:08:15.923 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.923 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.923 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:15.923 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.676178+0000", 2024-09-21T22:08:15.923 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000122395 2024-09-21T22:08:15.924 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.924 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.924 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:15.924 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.676283+0000", 2024-09-21T22:08:15.924 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000105653 2024-09-21T22:08:15.924 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:15.924 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:15.924 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:15.924 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.924 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.924 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5207.0:39 2.7 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head [write 917504~131072] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e78)", 2024-09-21T22:08:15.924 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:52.589499+0000", 2024-09-21T22:08:15.924 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 23.221923217000001, 2024-09-21T22:08:15.925 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.087365371999999997, 2024-09-21T22:08:15.925 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:15.925 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:15.925 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:15.925 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:15.925 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5207", 2024-09-21T22:08:15.925 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/357731919", 2024-09-21T22:08:15.925 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 39 2024-09-21T22:08:15.925 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.925 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:15.925 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.925 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:15.925 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.589499+0000", 2024-09-21T22:08:15.925 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:15.926 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.926 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.926 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:15.926 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.589499+0000", 2024-09-21T22:08:15.926 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:15.926 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.926 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.926 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:15.926 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.589500+0000", 2024-09-21T22:08:15.926 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.547e-06 2024-09-21T22:08:15.926 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.926 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.926 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:15.926 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.589752+0000", 2024-09-21T22:08:15.927 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025112799999999998 2024-09-21T22:08:15.927 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.927 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.927 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:15.927 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.589783+0000", 2024-09-21T22:08:15.927 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1557000000000003e-05 2024-09-21T22:08:15.927 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.927 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.927 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:15.927 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.589832+0000", 2024-09-21T22:08:15.927 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9175999999999998e-05 2024-09-21T22:08:15.927 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.927 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.927 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:15.928 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.665028+0000", 2024-09-21T22:08:15.928 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.075195293999999996 2024-09-21T22:08:15.928 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.928 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.928 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:15.928 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.665169+0000", 2024-09-21T22:08:15.928 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000141299 2024-09-21T22:08:15.928 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.928 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.928 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-09-21T22:08:15.928 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.665464+0000", 2024-09-21T22:08:15.928 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00029459899999999999 2024-09-21T22:08:15.928 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.928 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.929 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:15.929 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.676551+0000", 2024-09-21T22:08:15.929 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011087107000000001 2024-09-21T22:08:15.929 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.929 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.929 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:15.929 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.676700+0000", 2024-09-21T22:08:15.929 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000149853 2024-09-21T22:08:15.929 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.929 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.929 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:15.929 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.676802+0000", 2024-09-21T22:08:15.929 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010194199999999999 2024-09-21T22:08:15.929 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.929 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.930 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:15.930 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.676864+0000", 2024-09-21T22:08:15.930 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1870000000000002e-05 2024-09-21T22:08:15.930 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:15.930 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:15.930 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:15.930 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.930 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.930 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5207.0:74 2.7 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head [write 524288~131072] snapc 5=[5,3] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-21T22:08:15.930 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:52.938124+0000", 2024-09-21T22:08:15.930 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 22.873298004999999, 2024-09-21T22:08:15.930 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.071435100000000001, 2024-09-21T22:08:15.930 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:15.930 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:15.931 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:15.931 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:15.931 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5207", 2024-09-21T22:08:15.931 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/357731919", 2024-09-21T22:08:15.931 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 74 2024-09-21T22:08:15.931 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.931 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:15.931 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.931 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:15.931 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938124+0000", 2024-09-21T22:08:15.931 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:15.931 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.931 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.931 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:15.932 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938124+0000", 2024-09-21T22:08:15.932 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:15.932 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.932 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.932 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:15.932 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938125+0000", 2024-09-21T22:08:15.932 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5600000000000005e-07 2024-09-21T22:08:15.932 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.932 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.932 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:15.932 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938218+0000", 2024-09-21T22:08:15.932 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3239999999999995e-05 2024-09-21T22:08:15.932 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.932 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.932 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:15.933 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938231+0000", 2024-09-21T22:08:15.933 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.272e-05 2024-09-21T22:08:15.933 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.933 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.933 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:15.933 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938239+0000", 2024-09-21T22:08:15.933 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5059999999999995e-06 2024-09-21T22:08:15.933 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.933 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.933 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:15.933 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.983312+0000", 2024-09-21T22:08:15.933 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.045072252 2024-09-21T22:08:15.933 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.933 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.933 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:15.934 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.983458+0000", 2024-09-21T22:08:15.934 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014628899999999999 2024-09-21T22:08:15.934 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.934 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.934 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-09-21T22:08:15.934 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.983682+0000", 2024-09-21T22:08:15.934 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022402899999999999 2024-09-21T22:08:15.934 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.934 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.934 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:15.934 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.993679+0000", 2024-09-21T22:08:15.934 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0099972740000000004 2024-09-21T22:08:15.934 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.934 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.935 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:15.935 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.009436+0000", 2024-09-21T22:08:15.935 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015756867000000001 2024-09-21T22:08:15.935 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.935 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.935 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:15.935 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.009506+0000", 2024-09-21T22:08:15.935 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0277999999999994e-05 2024-09-21T22:08:15.935 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.935 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.935 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:15.935 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.009559+0000", 2024-09-21T22:08:15.935 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2889000000000001e-05 2024-09-21T22:08:15.935 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:15.936 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:15.936 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:15.936 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.936 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.936 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5207.0:75 2.7 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head [write 655360~131072] snapc 5=[5,3] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-21T22:08:15.936 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:52.938253+0000", 2024-09-21T22:08:15.936 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 22.873168949, 2024-09-21T22:08:15.936 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082064041000000004, 2024-09-21T22:08:15.936 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:15.936 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:15.936 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:15.936 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:15.936 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5207", 2024-09-21T22:08:15.936 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/357731919", 2024-09-21T22:08:15.937 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 75 2024-09-21T22:08:15.937 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.937 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:15.937 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.937 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:15.937 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938253+0000", 2024-09-21T22:08:15.937 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:15.937 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.937 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.937 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:15.937 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938253+0000", 2024-09-21T22:08:15.937 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:15.937 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.937 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.938 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:15.938 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938254+0000", 2024-09-21T22:08:15.938 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8599999999999998e-07 2024-09-21T22:08:15.938 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.938 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.938 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:15.938 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938354+0000", 2024-09-21T22:08:15.938 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9696000000000002e-05 2024-09-21T22:08:15.938 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.938 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.938 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:15.938 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938381+0000", 2024-09-21T22:08:15.938 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.705e-05 2024-09-21T22:08:15.938 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.939 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.939 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:15.939 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938397+0000", 2024-09-21T22:08:15.939 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5996999999999999e-05 2024-09-21T22:08:15.939 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.939 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.939 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:15.939 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.993739+0000", 2024-09-21T22:08:15.939 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.055342823999999999 2024-09-21T22:08:15.939 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.939 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.939 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:15.939 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.993835+0000", 2024-09-21T22:08:15.939 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5919000000000001e-05 2024-09-21T22:08:15.940 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.940 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.940 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-09-21T22:08:15.940 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.993993+0000", 2024-09-21T22:08:15.940 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015729900000000001 2024-09-21T22:08:15.940 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.940 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.940 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:15.940 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.009579+0000", 2024-09-21T22:08:15.940 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015586723 2024-09-21T22:08:15.940 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.940 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.940 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:15.940 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.020134+0000", 2024-09-21T22:08:15.941 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010554908 2024-09-21T22:08:15.941 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.941 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.941 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:15.941 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.020241+0000", 2024-09-21T22:08:15.941 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000106246 2024-09-21T22:08:15.941 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.941 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.941 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:15.941 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.020317+0000", 2024-09-21T22:08:15.941 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6693000000000006e-05 2024-09-21T22:08:15.941 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:15.941 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:15.941 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:15.941 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.942 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.942 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5207.0:76 2.7 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head [write 786432~131072] snapc 5=[5,3] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-21T22:08:15.942 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:52.938410+0000", 2024-09-21T22:08:15.942 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 22.873011779999999, 2024-09-21T22:08:15.942 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.086926370000000003, 2024-09-21T22:08:15.942 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:15.942 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:15.942 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:15.942 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:15.942 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5207", 2024-09-21T22:08:15.942 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/357731919", 2024-09-21T22:08:15.942 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 76 2024-09-21T22:08:15.942 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.943 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:15.943 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.943 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:15.943 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938410+0000", 2024-09-21T22:08:15.943 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:15.943 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.943 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.943 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:15.943 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938410+0000", 2024-09-21T22:08:15.943 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:15.943 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.943 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.943 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:15.943 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938411+0000", 2024-09-21T22:08:15.944 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0299999999999998e-07 2024-09-21T22:08:15.944 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.944 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.944 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:15.944 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938502+0000", 2024-09-21T22:08:15.944 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0562000000000005e-05 2024-09-21T22:08:15.944 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.944 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.944 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:15.944 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938514+0000", 2024-09-21T22:08:15.944 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2327000000000001e-05 2024-09-21T22:08:15.944 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.944 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.944 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:15.945 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938522+0000", 2024-09-21T22:08:15.945 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1170000000000004e-06 2024-09-21T22:08:15.945 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.945 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.945 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:15.945 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.001427+0000", 2024-09-21T22:08:15.945 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.062904871000000001 2024-09-21T22:08:15.945 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.945 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.945 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:15.945 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.001519+0000", 2024-09-21T22:08:15.945 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1798000000000004e-05 2024-09-21T22:08:15.945 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.945 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.946 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-09-21T22:08:15.946 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.001671+0000", 2024-09-21T22:08:15.946 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015195900000000001 2024-09-21T22:08:15.946 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.946 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.946 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:15.946 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.020388+0000", 2024-09-21T22:08:15.946 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018717428000000001 2024-09-21T22:08:15.946 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.946 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.946 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:15.946 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.025041+0000", 2024-09-21T22:08:15.946 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0046524039999999997 2024-09-21T22:08:15.946 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.947 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.947 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:15.947 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.025213+0000", 2024-09-21T22:08:15.947 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017286499999999999 2024-09-21T22:08:15.947 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.947 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.947 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:15.947 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.025337+0000", 2024-09-21T22:08:15.947 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012333600000000001 2024-09-21T22:08:15.947 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:15.947 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:15.947 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:15.947 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.947 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.947 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5207.0:77 2.7 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head [write 917504~131072] snapc 5=[5,3] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-21T22:08:15.948 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:52.938534+0000", 2024-09-21T22:08:15.948 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 22.872887965, 2024-09-21T22:08:15.948 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.087709436000000002, 2024-09-21T22:08:15.948 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:15.948 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:15.948 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:15.948 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:15.948 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5207", 2024-09-21T22:08:15.948 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/357731919", 2024-09-21T22:08:15.948 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 77 2024-09-21T22:08:15.948 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.948 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:15.948 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.948 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:15.949 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938534+0000", 2024-09-21T22:08:15.949 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:15.949 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.949 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.949 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:15.949 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938534+0000", 2024-09-21T22:08:15.949 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:15.949 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.949 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.949 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:15.949 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938535+0000", 2024-09-21T22:08:15.949 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1399999999999996e-07 2024-09-21T22:08:15.949 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.949 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.950 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:15.950 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938623+0000", 2024-09-21T22:08:15.950 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8226000000000005e-05 2024-09-21T22:08:15.950 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.950 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.950 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:15.950 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938645+0000", 2024-09-21T22:08:15.950 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1948999999999999e-05 2024-09-21T22:08:15.950 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.950 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.950 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:15.950 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938660+0000", 2024-09-21T22:08:15.950 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4776999999999999e-05 2024-09-21T22:08:15.950 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.951 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.951 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:15.951 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.009669+0000", 2024-09-21T22:08:15.951 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.071009116999999997 2024-09-21T22:08:15.951 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.951 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.951 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:15.951 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.009949+0000", 2024-09-21T22:08:15.951 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027962899999999999 2024-09-21T22:08:15.951 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.951 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.951 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-09-21T22:08:15.951 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.010355+0000", 2024-09-21T22:08:15.951 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000406334 2024-09-21T22:08:15.952 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.952 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.952 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:15.952 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.020875+0000", 2024-09-21T22:08:15.952 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010520144 2024-09-21T22:08:15.952 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.952 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.952 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:15.952 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.026086+0000", 2024-09-21T22:08:15.952 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0052107569999999999 2024-09-21T22:08:15.952 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.952 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.952 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:15.952 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.026178+0000", 2024-09-21T22:08:15.953 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2113000000000004e-05 2024-09-21T22:08:15.953 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.953 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.953 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:15.953 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.026244+0000", 2024-09-21T22:08:15.953 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5675999999999999e-05 2024-09-21T22:08:15.953 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:15.953 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:15.953 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:15.953 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.953 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.953 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5207.0:93 2.7 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head [write 131072~917504] snapc 5=[5,3] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-21T22:08:15.953 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:53.147757+0000", 2024-09-21T22:08:15.953 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 22.663664786999998, 2024-09-21T22:08:15.953 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.062493043999999998, 2024-09-21T22:08:15.954 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:15.954 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:15.954 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:15.954 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:15.954 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5207", 2024-09-21T22:08:15.954 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/357731919", 2024-09-21T22:08:15.954 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 93 2024-09-21T22:08:15.954 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.954 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:15.954 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.954 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:15.954 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.147757+0000", 2024-09-21T22:08:15.954 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:15.954 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.955 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.955 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:15.955 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.147757+0000", 2024-09-21T22:08:15.955 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:15.955 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.955 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.955 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:15.955 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.147759+0000", 2024-09-21T22:08:15.955 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0159999999999998e-06 2024-09-21T22:08:15.955 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.955 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.955 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:15.955 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.149009+0000", 2024-09-21T22:08:15.955 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001250082 2024-09-21T22:08:15.956 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.956 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.956 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:15.956 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.149042+0000", 2024-09-21T22:08:15.956 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2617000000000003e-05 2024-09-21T22:08:15.956 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.956 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.956 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:15.956 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.149058+0000", 2024-09-21T22:08:15.956 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6113e-05 2024-09-21T22:08:15.956 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.956 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.956 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:15.956 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.149237+0000", 2024-09-21T22:08:15.956 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017834900000000001 2024-09-21T22:08:15.957 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.957 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.957 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:15.957 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.149378+0000", 2024-09-21T22:08:15.957 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014099199999999999 2024-09-21T22:08:15.957 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.957 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.957 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-09-21T22:08:15.957 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.149582+0000", 2024-09-21T22:08:15.957 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020414900000000001 2024-09-21T22:08:15.957 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.957 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.957 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:15.957 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.208648+0000", 2024-09-21T22:08:15.958 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.059066397999999999 2024-09-21T22:08:15.958 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.958 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.958 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:15.958 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.210125+0000", 2024-09-21T22:08:15.958 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001477119 2024-09-21T22:08:15.958 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.958 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.958 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:15.958 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.210200+0000", 2024-09-21T22:08:15.958 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4973000000000002e-05 2024-09-21T22:08:15.958 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.958 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.958 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:15.959 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.210250+0000", 2024-09-21T22:08:15.959 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0235999999999998e-05 2024-09-21T22:08:15.959 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:15.959 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:15.959 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:15.959 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.959 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.959 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5207.0:128 2.7 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head [write 131072~917504] snapc 6=[6,5,3] ondisk+write+known_if_redirected+supports_pool_eio e83)", 2024-09-21T22:08:15.959 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:08:01.056777+0000", 2024-09-21T22:08:15.959 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 14.754645317, 2024-09-21T22:08:15.959 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.062294929999999998, 2024-09-21T22:08:15.959 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:15.959 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:15.959 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:15.959 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:15.960 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5207", 2024-09-21T22:08:15.960 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/357731919", 2024-09-21T22:08:15.960 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 128 2024-09-21T22:08:15.960 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.960 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:15.960 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.960 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:15.960 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.056777+0000", 2024-09-21T22:08:15.960 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:15.960 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.960 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.960 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:15.960 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.056777+0000", 2024-09-21T22:08:15.960 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:15.960 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.961 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.961 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:15.961 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.056779+0000", 2024-09-21T22:08:15.961 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4669999999999998e-06 2024-09-21T22:08:15.961 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.961 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.961 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:15.961 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.058094+0000", 2024-09-21T22:08:15.961 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001314602 2024-09-21T22:08:15.961 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.961 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.961 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:15.961 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.058126+0000", 2024-09-21T22:08:15.961 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1625999999999998e-05 2024-09-21T22:08:15.962 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.962 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.962 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:15.962 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.058141+0000", 2024-09-21T22:08:15.962 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5274e-05 2024-09-21T22:08:15.962 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.962 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.962 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:15.962 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.058267+0000", 2024-09-21T22:08:15.962 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000126682 2024-09-21T22:08:15.962 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.962 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.962 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:15.962 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.058420+0000", 2024-09-21T22:08:15.963 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015212599999999999 2024-09-21T22:08:15.963 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.963 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.963 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-09-21T22:08:15.963 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.058644+0000", 2024-09-21T22:08:15.963 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000224749 2024-09-21T22:08:15.963 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.963 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.963 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:15.963 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.117883+0000", 2024-09-21T22:08:15.963 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.059238617 2024-09-21T22:08:15.963 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.963 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.963 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:15.963 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.118942+0000", 2024-09-21T22:08:15.964 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0010588780000000001 2024-09-21T22:08:15.964 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.964 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.964 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:15.964 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.119011+0000", 2024-09-21T22:08:15.964 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8802999999999998e-05 2024-09-21T22:08:15.964 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.964 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.964 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:15.964 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.119072+0000", 2024-09-21T22:08:15.964 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1105999999999994e-05 2024-09-21T22:08:15.964 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:15.964 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:15.964 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:15.965 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.965 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.965 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.3 87/86 [PushOp(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head, version: 71'29, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head@71'29, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T22:08:15.965 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:08:07.107088+0000", 2024-09-21T22:08:15.965 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.7043345750000007, 2024-09-21T22:08:15.965 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.070273375999999999, 2024-09-21T22:08:15.965 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:15.965 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:15.965 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:08:15.965 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:15.965 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.965 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:15.965 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.107088+0000", 2024-09-21T22:08:15.965 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:15.965 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.966 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.966 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:15.966 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.107088+0000", 2024-09-21T22:08:15.966 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:15.966 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.966 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.966 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:15.966 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.107089+0000", 2024-09-21T22:08:15.966 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6559999999999999e-06 2024-09-21T22:08:15.966 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.966 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.966 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:15.966 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.108876+0000", 2024-09-21T22:08:15.966 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001786489 2024-09-21T22:08:15.967 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.967 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.967 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:15.968 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.108925+0000", 2024-09-21T22:08:15.968 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8843e-05 2024-09-21T22:08:15.968 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.968 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.968 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:15.968 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.108979+0000", 2024-09-21T22:08:15.968 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4945999999999999e-05 2024-09-21T22:08:15.968 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.969 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.969 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:15.969 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.109230+0000", 2024-09-21T22:08:15.969 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025086599999999998 2024-09-21T22:08:15.969 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.969 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.969 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:15.969 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.109269+0000", 2024-09-21T22:08:15.969 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8216000000000001e-05 2024-09-21T22:08:15.969 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:15.969 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:15.969 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:15.969 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.177361+0000", 2024-09-21T22:08:15.969 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.068092360000000005 2024-09-21T22:08:15.969 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:15.970 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:15.970 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:15.970 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:15.970 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:15.970 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:08:15.971 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T22:08:15.972 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:08:15.969+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:08:16.072 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:08:16.069+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:08:16.151 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:08:16.151 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:08:16.151 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-09-21T22:08:16.151 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:08:16.165 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T22:08:16.175 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-21T22:08:16.215 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:08:16.211+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:08:16.224 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T22:08:16.239 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T22:08:16.274 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:08:16.269+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:08:16.287 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:08:16.288 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:08:16.288 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-21T22:08:16.288 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-21T22:08:16.288 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:08:16.302 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T22:08:16.339 INFO:teuthology.orchestra.run.smithi039.stderr:no valid command found; 10 closest matches: 2024-09-21T22:08:16.339 INFO:teuthology.orchestra.run.smithi039.stderr:0 2024-09-21T22:08:16.339 INFO:teuthology.orchestra.run.smithi039.stderr:1 2024-09-21T22:08:16.340 INFO:teuthology.orchestra.run.smithi039.stderr:2 2024-09-21T22:08:16.340 INFO:teuthology.orchestra.run.smithi039.stderr:abort 2024-09-21T22:08:16.340 INFO:teuthology.orchestra.run.smithi039.stderr:assert 2024-09-21T22:08:16.340 INFO:teuthology.orchestra.run.smithi039.stderr:bluestore allocator dump block 2024-09-21T22:08:16.340 INFO:teuthology.orchestra.run.smithi039.stderr:bluestore allocator fragmentation block 2024-09-21T22:08:16.340 INFO:teuthology.orchestra.run.smithi039.stderr:bluestore allocator fragmentation histogram block [] [] 2024-09-21T22:08:16.340 INFO:teuthology.orchestra.run.smithi039.stderr:bluestore allocator score block 2024-09-21T22:08:16.340 INFO:teuthology.orchestra.run.smithi039.stderr:config diff 2024-09-21T22:08:16.340 INFO:teuthology.orchestra.run.smithi039.stderr:admin_socket: invalid command 2024-09-21T22:08:16.341 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-21T22:08:16.342 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-09-21T22:08:16.374 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:08:16.369+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:08:16.409 INFO:teuthology.orchestra.run.smithi039.stdout:osd.0: {} 2024-09-21T22:08:16.410 INFO:teuthology.orchestra.run.smithi039.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T22:08:16.426 INFO:teuthology.orchestra.run.smithi039.stdout:osd.1: {} 2024-09-21T22:08:16.427 INFO:teuthology.orchestra.run.smithi039.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T22:08:16.427 INFO:teuthology.orchestra.run.smithi039.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-21T22:08:16.447 INFO:teuthology.orchestra.run.smithi039.stdout:osd.3: {} 2024-09-21T22:08:16.448 INFO:teuthology.orchestra.run.smithi039.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T22:08:16.470 INFO:teuthology.orchestra.run.smithi039.stdout:osd.4: {} 2024-09-21T22:08:16.470 INFO:teuthology.orchestra.run.smithi039.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T22:08:16.470 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:16.470 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:16.476 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:08:16.473+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:08:16.485 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T22:08:16.491 INFO:teuthology.orchestra.run.smithi039.stdout:osd.5: {} 2024-09-21T22:08:16.491 INFO:teuthology.orchestra.run.smithi039.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T22:08:16.494 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T22:08:16.495 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T22:08:16.499 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:08:16.493+0000 7fe38df737c0 -1 osd.2 80 log_to_monitors true 2024-09-21T22:08:16.578 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:08:16.575+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:08:16.580 INFO:tasks.workunit.client.0.smithi157.stderr:++ rbd journal inspect --pool rbd --image testrbdnbd21553 2024-09-21T22:08:16.580 INFO:tasks.workunit.client.0.smithi157.stderr:++ awk '/entries inspected/ {print $1}' 2024-09-21T22:08:16.605 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:16.605 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:16.620 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T22:08:16.664 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-21T22:08:16.656+0000 7f3b3cb1c640 -1 JournalMetadata: 0x56345faf0040 failed to locate client: INSPECT 2024-09-21T22:08:16.668 INFO:tasks.workunit.client.0.smithi157.stderr:+ queue_length=4 2024-09-21T22:08:16.669 INFO:tasks.workunit.client.0.smithi157.stderr:+ tries=1 2024-09-21T22:08:16.669 INFO:tasks.workunit.client.0.smithi157.stderr:+ '[' 1 -lt 10 ']' 2024-09-21T22:08:16.669 INFO:tasks.workunit.client.0.smithi157.stderr:+ '[' 4 -gt 0 ']' 2024-09-21T22:08:16.669 INFO:tasks.workunit.client.0.smithi157.stderr:+ rbd journal inspect --pool rbd --image testrbdnbd21553 --verbose 2024-09-21T22:08:16.677 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:08:16.673+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:08:16.740 INFO:tasks.workunit.client.0.smithi157.stdout:Entry: tag_id=1, commit_tid=1 2024-09-21T22:08:16.740 INFO:tasks.workunit.client.0.smithi157.stdout:{ 2024-09-21T22:08:16.740 INFO:tasks.workunit.client.0.smithi157.stdout: "event_type": "AioDiscard", 2024-09-21T22:08:16.740 INFO:tasks.workunit.client.0.smithi157.stdout: "offset": 0, 2024-09-21T22:08:16.740 INFO:tasks.workunit.client.0.smithi157.stdout: "length": 4096, 2024-09-21T22:08:16.740 INFO:tasks.workunit.client.0.smithi157.stdout: "discard_granularity_bytes": 4096, 2024-09-21T22:08:16.740 INFO:tasks.workunit.client.0.smithi157.stdout: "timestamp": "2024-09-21T22:08:15.344115+0000" 2024-09-21T22:08:16.740 INFO:tasks.workunit.client.0.smithi157.stdout:} 2024-09-21T22:08:16.740 INFO:tasks.workunit.client.0.smithi157.stdout:Entry: tag_id=1, commit_tid=2 2024-09-21T22:08:16.740 INFO:tasks.workunit.client.0.smithi157.stdout:{ 2024-09-21T22:08:16.741 INFO:tasks.workunit.client.0.smithi157.stdout: "event_type": "AioDiscard", 2024-09-21T22:08:16.741 INFO:tasks.workunit.client.0.smithi157.stdout: "offset": 0, 2024-09-21T22:08:16.741 INFO:tasks.workunit.client.0.smithi157.stdout: "length": 4096, 2024-09-21T22:08:16.741 INFO:tasks.workunit.client.0.smithi157.stdout: "discard_granularity_bytes": 4096, 2024-09-21T22:08:16.741 INFO:tasks.workunit.client.0.smithi157.stdout: "timestamp": "2024-09-21T22:08:15.360522+0000" 2024-09-21T22:08:16.741 INFO:tasks.workunit.client.0.smithi157.stdout:} 2024-09-21T22:08:16.741 INFO:tasks.workunit.client.0.smithi157.stdout:Entry: tag_id=1, commit_tid=3 2024-09-21T22:08:16.741 INFO:tasks.workunit.client.0.smithi157.stdout:{ 2024-09-21T22:08:16.741 INFO:tasks.workunit.client.0.smithi157.stdout: "event_type": "AioDiscard", 2024-09-21T22:08:16.741 INFO:tasks.workunit.client.0.smithi157.stdout: "offset": 4096, 2024-09-21T22:08:16.741 INFO:tasks.workunit.client.0.smithi157.stdout: "length": 4096, 2024-09-21T22:08:16.741 INFO:tasks.workunit.client.0.smithi157.stdout: "discard_granularity_bytes": 4096, 2024-09-21T22:08:16.742 INFO:tasks.workunit.client.0.smithi157.stdout: "timestamp": "2024-09-21T22:08:15.387911+0000" 2024-09-21T22:08:16.742 INFO:tasks.workunit.client.0.smithi157.stdout:} 2024-09-21T22:08:16.742 INFO:tasks.workunit.client.0.smithi157.stdout:Entry: tag_id=1, commit_tid=4 2024-09-21T22:08:16.742 INFO:tasks.workunit.client.0.smithi157.stdout:{ 2024-09-21T22:08:16.742 INFO:tasks.workunit.client.0.smithi157.stdout: "event_type": "AioDiscard", 2024-09-21T22:08:16.742 INFO:tasks.workunit.client.0.smithi157.stdout: "offset": 4096, 2024-09-21T22:08:16.742 INFO:tasks.workunit.client.0.smithi157.stdout: "length": 4096, 2024-09-21T22:08:16.742 INFO:tasks.workunit.client.0.smithi157.stdout: "discard_granularity_bytes": 4096, 2024-09-21T22:08:16.742 INFO:tasks.workunit.client.0.smithi157.stdout: "timestamp": "2024-09-21T22:08:15.404762+0000" 2024-09-21T22:08:16.742 INFO:tasks.workunit.client.0.smithi157.stdout:} 2024-09-21T22:08:16.742 INFO:tasks.workunit.client.0.smithi157.stdout:Summary: 2024-09-21T22:08:16.742 INFO:tasks.workunit.client.0.smithi157.stdout: 4 entries inspected, 0 errors 2024-09-21T22:08:16.745 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-21T22:08:16.736+0000 7f0d7e077640 -1 JournalMetadata: 0x564254c39e50 failed to locate client: INSPECT 2024-09-21T22:08:16.751 INFO:tasks.workunit.client.0.smithi157.stderr:+ sleep 1 2024-09-21T22:08:16.778 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:08:16.775+0000 7f2075a06640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31720) UID: 0 2024-09-21T22:08:16.794 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:16.795 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:16.809 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T22:08:16.882 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:08:16.877+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:08:16.980 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:08:16.977+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:08:16.980 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:16.980 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:16.996 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T22:08:17.080 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:08:17.077+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:08:17.133 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:17.133 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:17.147 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T22:08:17.181 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:08:17.179+0000 7f2075a06640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31720) UID: 0 2024-09-21T22:08:17.282 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:08:17.277+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:08:17.316 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:17.316 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:17.331 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T22:08:17.383 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:08:17.379+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:08:17.469 INFO:teuthology.orchestra.run.smithi039.stderr:nodeep-scrub is set 2024-09-21T22:08:17.483 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:08:17.477+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:08:17.503 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:17.503 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:17.504 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:08:17.501+0000 7fe381d51640 -1 osd.2 80 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T22:08:17.518 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T22:08:17.585 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:08:17.579+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:08:17.683 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:17.684 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:17.686 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:08:17.681+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:08:17.697 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T22:08:17.754 INFO:tasks.workunit.client.0.smithi157.stderr:++ rbd journal inspect --pool rbd --image testrbdnbd21553 2024-09-21T22:08:17.754 INFO:tasks.workunit.client.0.smithi157.stderr:++ awk '/entries inspected/ {print $1}' 2024-09-21T22:08:17.787 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:08:17.783+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:08:17.847 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-21T22:08:17.840+0000 7fc845ffb640 -1 JournalMetadata: 0x559e9d592240 failed to locate client: INSPECT 2024-09-21T22:08:17.854 INFO:tasks.workunit.client.0.smithi157.stderr:+ queue_length=4 2024-09-21T22:08:17.854 INFO:tasks.workunit.client.0.smithi157.stderr:+ tries=2 2024-09-21T22:08:17.854 INFO:tasks.workunit.client.0.smithi157.stderr:+ '[' 2 -lt 10 ']' 2024-09-21T22:08:17.854 INFO:tasks.workunit.client.0.smithi157.stderr:+ '[' 4 -gt 0 ']' 2024-09-21T22:08:17.854 INFO:tasks.workunit.client.0.smithi157.stderr:+ rbd journal inspect --pool rbd --image testrbdnbd21553 --verbose 2024-09-21T22:08:17.864 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:17.865 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:17.879 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T22:08:17.924 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:08:17.921+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:08:17.936 INFO:tasks.workunit.client.0.smithi157.stdout:Entry: tag_id=1, commit_tid=1 2024-09-21T22:08:17.936 INFO:tasks.workunit.client.0.smithi157.stdout:{ 2024-09-21T22:08:17.936 INFO:tasks.workunit.client.0.smithi157.stdout: "event_type": "AioDiscard", 2024-09-21T22:08:17.936 INFO:tasks.workunit.client.0.smithi157.stdout: "offset": 0, 2024-09-21T22:08:17.936 INFO:tasks.workunit.client.0.smithi157.stdout: "length": 4096, 2024-09-21T22:08:17.936 INFO:tasks.workunit.client.0.smithi157.stdout: "discard_granularity_bytes": 4096, 2024-09-21T22:08:17.936 INFO:tasks.workunit.client.0.smithi157.stdout: "timestamp": "2024-09-21T22:08:15.344115+0000" 2024-09-21T22:08:17.936 INFO:tasks.workunit.client.0.smithi157.stdout:} 2024-09-21T22:08:17.936 INFO:tasks.workunit.client.0.smithi157.stdout:Entry: tag_id=1, commit_tid=2 2024-09-21T22:08:17.936 INFO:tasks.workunit.client.0.smithi157.stdout:{ 2024-09-21T22:08:17.936 INFO:tasks.workunit.client.0.smithi157.stdout: "event_type": "AioDiscard", 2024-09-21T22:08:17.936 INFO:tasks.workunit.client.0.smithi157.stdout: "offset": 0, 2024-09-21T22:08:17.937 INFO:tasks.workunit.client.0.smithi157.stdout: "length": 4096, 2024-09-21T22:08:17.937 INFO:tasks.workunit.client.0.smithi157.stdout: "discard_granularity_bytes": 4096, 2024-09-21T22:08:17.937 INFO:tasks.workunit.client.0.smithi157.stdout: "timestamp": "2024-09-21T22:08:15.360522+0000" 2024-09-21T22:08:17.937 INFO:tasks.workunit.client.0.smithi157.stdout:} 2024-09-21T22:08:17.937 INFO:tasks.workunit.client.0.smithi157.stdout:Entry: tag_id=1, commit_tid=3 2024-09-21T22:08:17.937 INFO:tasks.workunit.client.0.smithi157.stdout:{ 2024-09-21T22:08:17.937 INFO:tasks.workunit.client.0.smithi157.stdout: "event_type": "AioDiscard", 2024-09-21T22:08:17.937 INFO:tasks.workunit.client.0.smithi157.stdout: "offset": 4096, 2024-09-21T22:08:17.937 INFO:tasks.workunit.client.0.smithi157.stdout: "length": 4096, 2024-09-21T22:08:17.937 INFO:tasks.workunit.client.0.smithi157.stdout: "discard_granularity_bytes": 4096, 2024-09-21T22:08:17.937 INFO:tasks.workunit.client.0.smithi157.stdout: "timestamp": "2024-09-21T22:08:15.387911+0000" 2024-09-21T22:08:17.937 INFO:tasks.workunit.client.0.smithi157.stdout:} 2024-09-21T22:08:17.937 INFO:tasks.workunit.client.0.smithi157.stdout:Entry: tag_id=1, commit_tid=4 2024-09-21T22:08:17.938 INFO:tasks.workunit.client.0.smithi157.stdout:{ 2024-09-21T22:08:17.938 INFO:tasks.workunit.client.0.smithi157.stdout: "event_type": "AioDiscard", 2024-09-21T22:08:17.938 INFO:tasks.workunit.client.0.smithi157.stdout: "offset": 4096, 2024-09-21T22:08:17.938 INFO:tasks.workunit.client.0.smithi157.stdout: "length": 4096, 2024-09-21T22:08:17.938 INFO:tasks.workunit.client.0.smithi157.stdout: "discard_granularity_bytes": 4096, 2024-09-21T22:08:17.938 INFO:tasks.workunit.client.0.smithi157.stdout: "timestamp": "2024-09-21T22:08:15.404762+0000" 2024-09-21T22:08:17.938 INFO:tasks.workunit.client.0.smithi157.stdout:} 2024-09-21T22:08:17.938 INFO:tasks.workunit.client.0.smithi157.stdout:Summary: 2024-09-21T22:08:17.938 INFO:tasks.workunit.client.0.smithi157.stdout: 4 entries inspected, 0 errors 2024-09-21T22:08:17.946 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-21T22:08:17.940+0000 7f1d789f6640 -1 JournalMetadata: 0x562c38d55380 failed to locate client: INSPECT 2024-09-21T22:08:17.950 INFO:tasks.workunit.client.0.smithi157.stderr:+ sleep 1 2024-09-21T22:08:17.989 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:08:17.985+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:08:17.999 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:17.999 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:18.015 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T22:08:18.090 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:08:18.087+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:08:18.150 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:18.151 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:18.165 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T22:08:18.206 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:08:18.203+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:08:18.292 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:08:18.289+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:08:18.336 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:18.336 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:18.350 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T22:08:18.394 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:08:18.389+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:08:18.470 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:18.470 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:18.488 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T22:08:18.495 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T22:08:18.534 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:08:18.531+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:08:18.595 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:08:18.591+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:08:18.615 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:18.615 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:18.629 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T22:08:18.651 INFO:teuthology.orchestra.run.smithi039.stdout:osd.0: {} 2024-09-21T22:08:18.651 INFO:teuthology.orchestra.run.smithi039.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T22:08:18.667 INFO:teuthology.orchestra.run.smithi039.stdout:osd.1: {} 2024-09-21T22:08:18.668 INFO:teuthology.orchestra.run.smithi039.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T22:08:18.684 INFO:teuthology.orchestra.run.smithi039.stdout:osd.2: {} 2024-09-21T22:08:18.684 INFO:teuthology.orchestra.run.smithi039.stderr:osd.2: osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_enable_op_tracker = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' 2024-09-21T22:08:18.705 INFO:teuthology.orchestra.run.smithi039.stdout:osd.3: {} 2024-09-21T22:08:18.705 INFO:teuthology.orchestra.run.smithi039.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T22:08:18.718 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:08:18.715+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:08:18.725 INFO:teuthology.orchestra.run.smithi039.stdout:osd.4: {} 2024-09-21T22:08:18.726 INFO:teuthology.orchestra.run.smithi039.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T22:08:18.743 INFO:teuthology.orchestra.run.smithi039.stdout:osd.5: {} 2024-09-21T22:08:18.743 INFO:teuthology.orchestra.run.smithi039.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T22:08:18.797 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:08:18.791+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:08:18.800 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:08:18.800 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:08:18.800 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-21T22:08:18.800 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-21T22:08:18.800 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:08:18.815 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T22:08:18.906 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:08:18.903+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:08:18.952 INFO:tasks.workunit.client.0.smithi157.stderr:++ rbd journal inspect --pool rbd --image testrbdnbd21553 2024-09-21T22:08:18.953 INFO:tasks.workunit.client.0.smithi157.stderr:++ awk '/entries inspected/ {print $1}' 2024-09-21T22:08:18.983 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:08:18.983 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-21T22:08:18.983 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-21T22:08:18.983 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-21T22:08:18.984 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:18.984 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4840.0:676 2.7 e63/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 63'57, mlcod=63'55)", 2024-09-21T22:08:18.984 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.627276+0000", 2024-09-21T22:08:18.984 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 54.356146770000002, 2024-09-21T22:08:18.984 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.037500787000000001, 2024-09-21T22:08:18.984 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:18.984 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:18.984 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:18.984 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:18.984 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:18.984 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:18.984 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.627276+0000", 2024-09-21T22:08:18.984 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:18.985 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:18.985 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:18.985 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:18.985 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.627276+0000", 2024-09-21T22:08:18.985 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:18.985 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:18.985 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:18.985 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:18.985 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.627277+0000", 2024-09-21T22:08:18.985 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.068e-06 2024-09-21T22:08:18.985 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:18.985 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:18.985 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:18.985 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.628148+0000", 2024-09-21T22:08:18.986 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00087069999999999997 2024-09-21T22:08:18.986 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:18.986 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:18.986 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:18.986 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.628170+0000", 2024-09-21T22:08:18.986 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2147999999999999e-05 2024-09-21T22:08:18.986 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:18.986 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:18.986 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:18.986 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.628179+0000", 2024-09-21T22:08:18.986 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2450000000000004e-06 2024-09-21T22:08:18.986 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:18.986 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:18.986 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:18.987 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.628279+0000", 2024-09-21T22:08:18.987 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9298999999999999e-05 2024-09-21T22:08:18.987 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:18.987 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:18.987 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:18.987 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.628305+0000", 2024-09-21T22:08:18.987 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6316999999999999e-05 2024-09-21T22:08:18.987 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:18.987 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:18.987 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:18.987 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.664714+0000", 2024-09-21T22:08:18.987 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.036408923000000003 2024-09-21T22:08:18.987 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:18.987 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:18.987 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:18.988 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.664777+0000", 2024-09-21T22:08:18.988 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3087000000000001e-05 2024-09-21T22:08:18.988 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:18.988 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:18.988 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:18.988 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:18.988 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:18.988 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4840.0:678 2.7 e63/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 63'59, mlcod=63'57)", 2024-09-21T22:08:18.988 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.679689+0000", 2024-09-21T22:08:18.988 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 54.30373427, 2024-09-21T22:08:18.988 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053308406000000003, 2024-09-21T22:08:18.988 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:18.988 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:18.988 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:18.989 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:18.989 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:18.989 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:18.989 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.679689+0000", 2024-09-21T22:08:18.989 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:18.989 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:18.989 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:18.989 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:18.989 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.679689+0000", 2024-09-21T22:08:18.989 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:18.989 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:18.989 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:18.989 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:18.989 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.679691+0000", 2024-09-21T22:08:18.990 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9400000000000001e-06 2024-09-21T22:08:18.990 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:18.990 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:18.990 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:18.990 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.680245+0000", 2024-09-21T22:08:18.990 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00055382999999999995 2024-09-21T22:08:18.990 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:18.990 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:18.990 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:18.990 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.680276+0000", 2024-09-21T22:08:18.990 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1123e-05 2024-09-21T22:08:18.990 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:18.990 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:18.990 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:18.991 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.680290+0000", 2024-09-21T22:08:18.991 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4056999999999999e-05 2024-09-21T22:08:18.991 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:18.991 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:18.991 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:18.991 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.680418+0000", 2024-09-21T22:08:18.991 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000127792 2024-09-21T22:08:18.991 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:18.991 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:18.991 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:18.991 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.680465+0000", 2024-09-21T22:08:18.991 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7360000000000001e-05 2024-09-21T22:08:18.991 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:18.991 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:18.992 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:18.992 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.732923+0000", 2024-09-21T22:08:18.992 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.052458312999999999 2024-09-21T22:08:18.992 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:18.992 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:18.992 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:18.992 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.732997+0000", 2024-09-21T22:08:18.992 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3991000000000004e-05 2024-09-21T22:08:18.992 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:18.992 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:18.992 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:18.992 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:18.992 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:18.992 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4840.0:680 2.7 e63/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 63'61, mlcod=63'59)", 2024-09-21T22:08:18.993 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.745996+0000", 2024-09-21T22:08:18.993 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 54.237426757000001, 2024-09-21T22:08:18.993 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.061804132999999997, 2024-09-21T22:08:18.993 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:18.993 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:18.993 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:18.993 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:18.993 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:18.993 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:18.993 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.745996+0000", 2024-09-21T22:08:18.993 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:18.993 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:18.993 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:18.993 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:18.993 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.745996+0000", 2024-09-21T22:08:18.994 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:18.994 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:18.994 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:18.994 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:18.994 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.745998+0000", 2024-09-21T22:08:18.994 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.717e-06 2024-09-21T22:08:18.994 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:18.994 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:18.994 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:18.994 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.746523+0000", 2024-09-21T22:08:18.994 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00052525399999999996 2024-09-21T22:08:18.994 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:18.994 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:18.994 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:18.995 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.746554+0000", 2024-09-21T22:08:18.995 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0586000000000002e-05 2024-09-21T22:08:18.995 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:18.995 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:18.995 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:18.995 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.746577+0000", 2024-09-21T22:08:18.995 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3079e-05 2024-09-21T22:08:18.995 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:18.995 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:18.995 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:18.995 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.746689+0000", 2024-09-21T22:08:18.995 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011177 2024-09-21T22:08:18.995 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:18.995 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:18.995 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:18.996 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.746722+0000", 2024-09-21T22:08:18.996 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2870000000000002e-05 2024-09-21T22:08:18.996 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:18.996 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:18.996 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:18.996 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.807740+0000", 2024-09-21T22:08:18.996 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.061018612 2024-09-21T22:08:18.996 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:18.996 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:18.996 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:18.996 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.807800+0000", 2024-09-21T22:08:18.996 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0244999999999997e-05 2024-09-21T22:08:18.996 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:18.996 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:18.997 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:18.997 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:18.997 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:18.997 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4840.0:682 2.7 e63/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 63'63, mlcod=63'61)", 2024-09-21T22:08:18.997 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.824126+0000", 2024-09-21T22:08:18.997 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 54.159297187999996, 2024-09-21T22:08:18.997 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.038159327, 2024-09-21T22:08:18.997 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:18.997 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:18.997 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:18.997 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:18.997 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:18.997 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:18.997 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.824126+0000", 2024-09-21T22:08:18.998 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:18.998 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:18.998 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:18.998 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:18.998 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.824126+0000", 2024-09-21T22:08:18.998 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:18.998 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:18.998 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:18.998 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:18.998 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.824128+0000", 2024-09-21T22:08:18.998 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9659999999999999e-06 2024-09-21T22:08:18.998 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:18.998 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:18.998 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:18.999 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.824842+0000", 2024-09-21T22:08:18.999 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00071423999999999999 2024-09-21T22:08:18.999 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:18.999 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:18.999 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:18.999 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.824883+0000", 2024-09-21T22:08:18.999 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1131999999999999e-05 2024-09-21T22:08:18.999 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:18.999 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:18.999 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:18.999 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.824931+0000", 2024-09-21T22:08:18.999 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7327000000000003e-05 2024-09-21T22:08:18.999 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:18.999 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.000 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:19.000 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.825058+0000", 2024-09-21T22:08:19.000 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012790499999999999 2024-09-21T22:08:19.000 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.000 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.000 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:19.000 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.825087+0000", 2024-09-21T22:08:19.000 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8192e-05 2024-09-21T22:08:19.000 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.000 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.000 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:19.000 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.862225+0000", 2024-09-21T22:08:19.000 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.037138309000000001 2024-09-21T22:08:19.000 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.001 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.001 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:19.001 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.862285+0000", 2024-09-21T22:08:19.001 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0256e-05 2024-09-21T22:08:19.001 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:19.001 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:19.001 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:19.001 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.001 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.001 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5008.0:116 2.7 e71/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 71'148, mlcod=71'146)", 2024-09-21T22:08:19.001 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:37.867612+0000", 2024-09-21T22:08:19.001 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 41.115810693, 2024-09-21T22:08:19.001 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.045123155999999998, 2024-09-21T22:08:19.001 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:19.001 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:19.002 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:19.002 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:19.002 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.002 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:19.002 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.867612+0000", 2024-09-21T22:08:19.002 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:19.002 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.002 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.002 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:19.002 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.867612+0000", 2024-09-21T22:08:19.002 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:19.002 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.002 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.003 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:19.003 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.867613+0000", 2024-09-21T22:08:19.003 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1200000000000002e-07 2024-09-21T22:08:19.003 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.003 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.003 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:19.003 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.868135+0000", 2024-09-21T22:08:19.003 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00052132700000000003 2024-09-21T22:08:19.003 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.003 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.003 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:19.003 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.868150+0000", 2024-09-21T22:08:19.003 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5271e-05 2024-09-21T22:08:19.003 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.003 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.004 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:19.004 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.868156+0000", 2024-09-21T22:08:19.004 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4969999999999998e-06 2024-09-21T22:08:19.004 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.004 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.004 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:19.004 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.868213+0000", 2024-09-21T22:08:19.004 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7114999999999998e-05 2024-09-21T22:08:19.004 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.004 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.004 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:19.004 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.868238+0000", 2024-09-21T22:08:19.004 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4987999999999999e-05 2024-09-21T22:08:19.004 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.005 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.005 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:19.005 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.912677+0000", 2024-09-21T22:08:19.005 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.044439019000000003 2024-09-21T22:08:19.005 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.005 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.005 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:19.005 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.912736+0000", 2024-09-21T22:08:19.005 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8127000000000001e-05 2024-09-21T22:08:19.005 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:19.005 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:19.005 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:19.005 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.005 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.006 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5008.0:118 2.7 e71/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 71'150, mlcod=71'148)", 2024-09-21T22:08:19.006 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:37.924700+0000", 2024-09-21T22:08:19.006 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 41.058723501999999, 2024-09-21T22:08:19.006 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.034650561000000003, 2024-09-21T22:08:19.006 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:19.006 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:19.006 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:19.006 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:19.006 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.006 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:19.006 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.924700+0000", 2024-09-21T22:08:19.006 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:19.006 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.006 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.007 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:19.007 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.924700+0000", 2024-09-21T22:08:19.007 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:19.007 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.007 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.007 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:19.007 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.924702+0000", 2024-09-21T22:08:19.007 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9460000000000001e-06 2024-09-21T22:08:19.007 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.007 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.007 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:19.007 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.925132+0000", 2024-09-21T22:08:19.007 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00043021200000000002 2024-09-21T22:08:19.007 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.008 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:19.008 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.925156+0000", 2024-09-21T22:08:19.008 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4451999999999999e-05 2024-09-21T22:08:19.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.008 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:19.008 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.925168+0000", 2024-09-21T22:08:19.008 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2249000000000001e-05 2024-09-21T22:08:19.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.008 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:19.008 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.925241+0000", 2024-09-21T22:08:19.008 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2502000000000005e-05 2024-09-21T22:08:19.009 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.009 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.009 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:19.009 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.925267+0000", 2024-09-21T22:08:19.009 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6471000000000001e-05 2024-09-21T22:08:19.009 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.009 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.009 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:19.009 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.959217+0000", 2024-09-21T22:08:19.009 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.033949569999999998 2024-09-21T22:08:19.009 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.009 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.009 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:19.009 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.959350+0000", 2024-09-21T22:08:19.009 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013315899999999999 2024-09-21T22:08:19.010 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:19.010 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:19.010 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:19.010 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.010 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.010 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5008.0:120 2.7 e71/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 71'152, mlcod=71'150)", 2024-09-21T22:08:19.010 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:37.970837+0000", 2024-09-21T22:08:19.010 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 41.012586251000002, 2024-09-21T22:08:19.010 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.03044943, 2024-09-21T22:08:19.010 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:19.010 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:19.010 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:19.010 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:19.010 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.011 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:19.011 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.970837+0000", 2024-09-21T22:08:19.011 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:19.011 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.011 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.011 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:19.011 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.970837+0000", 2024-09-21T22:08:19.011 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:19.011 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.011 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.011 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:19.011 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.970838+0000", 2024-09-21T22:08:19.011 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4649999999999999e-06 2024-09-21T22:08:19.011 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.012 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:19.012 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.971265+0000", 2024-09-21T22:08:19.012 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00042665800000000001 2024-09-21T22:08:19.012 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.012 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:19.012 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.971285+0000", 2024-09-21T22:08:19.012 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9639999999999998e-05 2024-09-21T22:08:19.012 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.012 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:19.012 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.971294+0000", 2024-09-21T22:08:19.012 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7210000000000006e-06 2024-09-21T22:08:19.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.013 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.013 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:19.013 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.971354+0000", 2024-09-21T22:08:19.013 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9613e-05 2024-09-21T22:08:19.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.013 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.013 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:19.013 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.971381+0000", 2024-09-21T22:08:19.013 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7538999999999998e-05 2024-09-21T22:08:19.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.013 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.013 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:19.013 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.001199+0000", 2024-09-21T22:08:19.014 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029817537000000002 2024-09-21T22:08:19.014 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.014 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.014 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:19.014 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.001286+0000", 2024-09-21T22:08:19.014 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7256999999999995e-05 2024-09-21T22:08:19.014 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:19.014 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:19.014 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:19.014 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.014 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.014 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5008.0:122 2.7 e71/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 71'154, mlcod=71'152)", 2024-09-21T22:08:19.014 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:38.014709+0000", 2024-09-21T22:08:19.014 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 40.968714376000001, 2024-09-21T22:08:19.015 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029567408999999999, 2024-09-21T22:08:19.015 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:19.015 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:19.015 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:19.015 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:19.015 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.015 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:19.015 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.014709+0000", 2024-09-21T22:08:19.015 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:19.015 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.015 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.015 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:19.015 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.014709+0000", 2024-09-21T22:08:19.015 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:19.015 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.016 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.016 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:19.016 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.014709+0000", 2024-09-21T22:08:19.016 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9599999999999999e-07 2024-09-21T22:08:19.016 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.016 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.016 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:19.016 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.015201+0000", 2024-09-21T22:08:19.016 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00049175599999999999 2024-09-21T22:08:19.016 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.016 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.016 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:19.016 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.015215+0000", 2024-09-21T22:08:19.016 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3692999999999999e-05 2024-09-21T22:08:19.017 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.017 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.017 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:19.017 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.015225+0000", 2024-09-21T22:08:19.017 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.815e-06 2024-09-21T22:08:19.017 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.017 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.017 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:19.017 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.015287+0000", 2024-09-21T22:08:19.017 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1858000000000003e-05 2024-09-21T22:08:19.017 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.017 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.017 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:19.017 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.015315+0000", 2024-09-21T22:08:19.018 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8490999999999999e-05 2024-09-21T22:08:19.018 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.018 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.018 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:19.018 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.044182+0000", 2024-09-21T22:08:19.018 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.028866936999999999 2024-09-21T22:08:19.018 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.018 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.018 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:19.018 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.044276+0000", 2024-09-21T22:08:19.018 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4163e-05 2024-09-21T22:08:19.018 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:19.018 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:19.018 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:19.018 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.019 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.019 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5207.0:70 2.7 e79/16 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head v 79'184, mlcod=79'181)", 2024-09-21T22:08:19.019 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:52.938385+0000", 2024-09-21T22:08:19.019 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 26.045038470000001, 2024-09-21T22:08:19.019 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023928324000000001, 2024-09-21T22:08:19.019 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:19.019 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:19.019 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:19.019 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:19.019 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.019 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:19.019 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938385+0000", 2024-09-21T22:08:19.019 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:19.019 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.020 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.020 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:19.020 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938385+0000", 2024-09-21T22:08:19.020 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:19.020 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.020 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.020 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:19.020 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938387+0000", 2024-09-21T22:08:19.020 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1639999999999999e-06 2024-09-21T22:08:19.020 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.020 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.020 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:19.020 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938511+0000", 2024-09-21T22:08:19.020 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000124485 2024-09-21T22:08:19.020 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.021 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.021 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:19.021 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938549+0000", 2024-09-21T22:08:19.021 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7347e-05 2024-09-21T22:08:19.021 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.021 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.021 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:19.021 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938570+0000", 2024-09-21T22:08:19.021 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1013e-05 2024-09-21T22:08:19.021 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.021 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.021 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:19.021 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938718+0000", 2024-09-21T22:08:19.021 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000148789 2024-09-21T22:08:19.022 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.022 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.022 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:19.022 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938769+0000", 2024-09-21T22:08:19.022 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1002999999999999e-05 2024-09-21T22:08:19.022 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.022 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.022 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:19.022 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.962199+0000", 2024-09-21T22:08:19.022 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023429783999999999 2024-09-21T22:08:19.022 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.022 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.022 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:19.022 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.962313+0000", 2024-09-21T22:08:19.023 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000113739 2024-09-21T22:08:19.023 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:19.023 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:19.023 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:19.023 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.023 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.023 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5207.0:71 2.7 e79/16 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head v 79'185, mlcod=79'181)", 2024-09-21T22:08:19.023 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:52.951522+0000", 2024-09-21T22:08:19.023 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 26.031900577999998, 2024-09-21T22:08:19.023 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021108028000000001, 2024-09-21T22:08:19.023 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:19.023 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:19.023 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:19.023 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:19.023 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.024 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:19.024 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.951522+0000", 2024-09-21T22:08:19.024 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:19.024 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.024 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.024 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:19.024 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.951522+0000", 2024-09-21T22:08:19.024 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:19.024 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.024 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.024 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:19.024 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.951524+0000", 2024-09-21T22:08:19.024 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9199999999999998e-06 2024-09-21T22:08:19.024 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.024 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.025 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:19.025 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.951686+0000", 2024-09-21T22:08:19.025 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016202499999999999 2024-09-21T22:08:19.025 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.025 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.025 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:19.025 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.951713+0000", 2024-09-21T22:08:19.025 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7007000000000001e-05 2024-09-21T22:08:19.025 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.025 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.025 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:19.025 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.951734+0000", 2024-09-21T22:08:19.025 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0228999999999999e-05 2024-09-21T22:08:19.025 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.026 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.026 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:19.026 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.952740+0000", 2024-09-21T22:08:19.026 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001006631 2024-09-21T22:08:19.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.026 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.026 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:19.026 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.952778+0000", 2024-09-21T22:08:19.026 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7316e-05 2024-09-21T22:08:19.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.026 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.026 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:19.026 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.972505+0000", 2024-09-21T22:08:19.026 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019727776999999998 2024-09-21T22:08:19.027 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.027 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.027 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:19.027 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.972631+0000", 2024-09-21T22:08:19.027 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012512299999999999 2024-09-21T22:08:19.027 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:19.027 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:19.027 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:19.027 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.027 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.027 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5207.0:76 2.7 e79/16 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head v 79'190, mlcod=79'186)", 2024-09-21T22:08:19.027 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:53.001917+0000", 2024-09-21T22:08:19.027 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 25.981506129, 2024-09-21T22:08:19.027 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022783040000000001, 2024-09-21T22:08:19.028 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:19.028 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:19.028 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:19.028 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:19.028 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.028 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:19.028 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.001917+0000", 2024-09-21T22:08:19.028 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:19.028 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.028 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.028 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:19.028 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.001917+0000", 2024-09-21T22:08:19.028 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:19.028 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.028 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.029 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:19.029 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.001919+0000", 2024-09-21T22:08:19.029 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.703e-06 2024-09-21T22:08:19.029 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.029 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.029 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:19.029 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.002006+0000", 2024-09-21T22:08:19.029 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6978999999999995e-05 2024-09-21T22:08:19.029 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.029 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.029 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:19.029 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.002042+0000", 2024-09-21T22:08:19.029 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6847000000000001e-05 2024-09-21T22:08:19.029 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.030 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.030 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:19.030 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.002057+0000", 2024-09-21T22:08:19.030 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.419e-05 2024-09-21T22:08:19.030 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.030 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.030 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:19.030 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.004387+0000", 2024-09-21T22:08:19.030 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002330365 2024-09-21T22:08:19.030 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.030 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.030 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:19.030 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.004420+0000", 2024-09-21T22:08:19.030 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3170000000000003e-05 2024-09-21T22:08:19.031 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.031 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.031 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:19.031 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.024610+0000", 2024-09-21T22:08:19.031 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020190084000000001 2024-09-21T22:08:19.031 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.031 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.031 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:19.031 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.024700+0000", 2024-09-21T22:08:19.031 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9702000000000003e-05 2024-09-21T22:08:19.031 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:19.031 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:19.031 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:19.031 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.032 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.032 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5207.0:93 2.7 e79/16 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head v 79'193, mlcod=79'191)", 2024-09-21T22:08:19.032 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:53.149813+0000", 2024-09-21T22:08:19.032 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 25.833610069999999, 2024-09-21T22:08:19.032 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.060053738000000002, 2024-09-21T22:08:19.032 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:19.032 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:19.032 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:19.032 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:19.032 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.032 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:19.032 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.149813+0000", 2024-09-21T22:08:19.032 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:19.032 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.033 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.033 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:19.033 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.149813+0000", 2024-09-21T22:08:19.033 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:19.033 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.033 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.033 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:19.033 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.149815+0000", 2024-09-21T22:08:19.033 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9099999999999999e-06 2024-09-21T22:08:19.033 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.033 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.033 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:19.033 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.150828+0000", 2024-09-21T22:08:19.033 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0010129099999999999 2024-09-21T22:08:19.033 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.034 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.034 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:19.034 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.150860+0000", 2024-09-21T22:08:19.034 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2443000000000001e-05 2024-09-21T22:08:19.034 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.034 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.034 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:19.034 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.150874+0000", 2024-09-21T22:08:19.034 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3627e-05 2024-09-21T22:08:19.034 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.034 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.034 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:19.034 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.151010+0000", 2024-09-21T22:08:19.034 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013656599999999999 2024-09-21T22:08:19.035 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.035 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:19.035 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.151047+0000", 2024-09-21T22:08:19.035 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6495999999999998e-05 2024-09-21T22:08:19.035 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.035 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:19.035 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.209791+0000", 2024-09-21T22:08:19.035 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058744004000000002 2024-09-21T22:08:19.035 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.035 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:19.035 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.209867+0000", 2024-09-21T22:08:19.036 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5782000000000001e-05 2024-09-21T22:08:19.036 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:19.036 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:19.036 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:19.036 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.036 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.036 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5207.0:128 2.7 e83/16 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head v 83'198, mlcod=83'195)", 2024-09-21T22:08:19.036 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:08:01.058831+0000", 2024-09-21T22:08:19.036 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 17.924592435000001, 2024-09-21T22:08:19.036 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058096534999999998, 2024-09-21T22:08:19.036 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:19.036 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:19.036 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:19.036 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:19.037 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.037 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:19.037 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.058831+0000", 2024-09-21T22:08:19.037 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:19.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.037 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.037 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:19.037 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.058831+0000", 2024-09-21T22:08:19.037 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:19.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.037 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.037 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:19.037 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.058833+0000", 2024-09-21T22:08:19.037 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0660000000000002e-06 2024-09-21T22:08:19.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.038 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.038 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:19.038 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.059877+0000", 2024-09-21T22:08:19.038 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0010447 2024-09-21T22:08:19.038 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.038 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.038 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:19.038 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.059908+0000", 2024-09-21T22:08:19.038 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0800000000000003e-05 2024-09-21T22:08:19.038 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.038 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.038 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:19.039 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.059923+0000", 2024-09-21T22:08:19.039 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4497e-05 2024-09-21T22:08:19.039 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.039 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.039 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:19.039 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.060053+0000", 2024-09-21T22:08:19.039 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013002900000000001 2024-09-21T22:08:19.039 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.039 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.039 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:19.039 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.060091+0000", 2024-09-21T22:08:19.039 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8389999999999997e-05 2024-09-21T22:08:19.039 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.039 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.040 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:19.040 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.116840+0000", 2024-09-21T22:08:19.040 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.056749253999999999 2024-09-21T22:08:19.040 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.040 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.040 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:19.040 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.116927+0000", 2024-09-21T22:08:19.040 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6798999999999993e-05 2024-09-21T22:08:19.040 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:19.040 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:19.040 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:19.040 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.040 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.040 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.6 87/86 [PushOp(2:7e2189ff:ns::rbd_data.127f79bc213d.0000000000000001:head, version: 57'4, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:7e2189ff:ns::rbd_data.127f79bc213d.0000000000000001:head@57'4, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T22:08:19.041 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:08:07.095630+0000", 2024-09-21T22:08:19.041 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 11.887793479000001, 2024-09-21T22:08:19.041 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.063566191999999994, 2024-09-21T22:08:19.041 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:19.041 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:19.041 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:08:19.041 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:19.041 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.041 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:19.041 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.095630+0000", 2024-09-21T22:08:19.041 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:19.041 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.041 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.041 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:19.041 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.095630+0000", 2024-09-21T22:08:19.042 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:19.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.042 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.042 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:19.042 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.095631+0000", 2024-09-21T22:08:19.042 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3069999999999999e-06 2024-09-21T22:08:19.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.042 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.042 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:19.042 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.097075+0000", 2024-09-21T22:08:19.042 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001443831 2024-09-21T22:08:19.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.042 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.042 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:19.043 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.097153+0000", 2024-09-21T22:08:19.043 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8485999999999994e-05 2024-09-21T22:08:19.043 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.043 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.043 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:19.043 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.097208+0000", 2024-09-21T22:08:19.043 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4716000000000003e-05 2024-09-21T22:08:19.043 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.043 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.043 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:19.043 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.097459+0000", 2024-09-21T22:08:19.043 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025113799999999998 2024-09-21T22:08:19.043 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.043 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.043 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:19.044 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.097481+0000", 2024-09-21T22:08:19.044 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2344000000000002e-05 2024-09-21T22:08:19.044 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.044 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.044 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:19.044 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.159196+0000", 2024-09-21T22:08:19.044 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.061714369999999998 2024-09-21T22:08:19.044 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:19.044 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:19.044 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:19.044 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.044 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.044 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.6 87/86 [PushOp(2:7e2189ff:ns::rbd_data.127f79bc213d.0000000000000001:head, version: 57'4, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:7e2189ff:ns::rbd_data.127f79bc213d.0000000000000001:head@57'4, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:08:19.044 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:08:07.163314+0000", 2024-09-21T22:08:19.045 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 11.820109425, 2024-09-21T22:08:19.045 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.044796923000000002, 2024-09-21T22:08:19.045 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:19.045 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:19.045 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:08:19.045 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:19.045 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.045 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:19.045 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.163314+0000", 2024-09-21T22:08:19.045 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:19.045 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.045 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.045 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:19.045 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.163314+0000", 2024-09-21T22:08:19.046 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:19.046 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.046 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.046 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:19.046 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.163315+0000", 2024-09-21T22:08:19.046 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7120000000000001e-06 2024-09-21T22:08:19.046 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.046 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.046 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:19.046 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.164270+0000", 2024-09-21T22:08:19.046 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000955034 2024-09-21T22:08:19.046 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.046 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.046 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:19.047 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.164326+0000", 2024-09-21T22:08:19.047 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5627999999999997e-05 2024-09-21T22:08:19.047 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.047 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.047 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:19.047 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.164351+0000", 2024-09-21T22:08:19.047 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5219999999999999e-05 2024-09-21T22:08:19.047 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.047 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.047 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:19.047 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.164571+0000", 2024-09-21T22:08:19.047 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000219996 2024-09-21T22:08:19.047 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.047 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.047 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:19.048 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.164596+0000", 2024-09-21T22:08:19.048 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4564000000000001e-05 2024-09-21T22:08:19.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.048 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:19.048 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.208111+0000", 2024-09-21T22:08:19.048 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.043514769000000002 2024-09-21T22:08:19.048 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:19.048 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:19.048 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:19.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.048 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.6 87/86 [PushOp(2:7e2189ff:ns::rbd_data.127f79bc213d.0000000000000001:head, version: 57'4, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:7e2189ff:ns::rbd_data.127f79bc213d.0000000000000001:head@57'4, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:08:19.048 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:08:07.212243+0000", 2024-09-21T22:08:19.049 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 11.771179982, 2024-09-21T22:08:19.049 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.051665924000000002, 2024-09-21T22:08:19.049 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:19.049 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:19.049 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:08:19.049 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:19.049 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.049 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:19.049 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.212243+0000", 2024-09-21T22:08:19.049 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:19.049 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.049 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.049 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:19.049 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.212243+0000", 2024-09-21T22:08:19.050 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:19.050 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.050 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.050 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:19.050 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.212244+0000", 2024-09-21T22:08:19.050 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2839999999999999e-06 2024-09-21T22:08:19.050 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.050 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.050 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:19.050 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.212993+0000", 2024-09-21T22:08:19.050 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00074894100000000004 2024-09-21T22:08:19.050 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.050 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.050 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:19.051 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.213032+0000", 2024-09-21T22:08:19.051 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8948e-05 2024-09-21T22:08:19.051 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.051 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.051 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:19.051 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.213054+0000", 2024-09-21T22:08:19.051 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1937999999999999e-05 2024-09-21T22:08:19.051 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.051 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.051 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:19.051 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.213194+0000", 2024-09-21T22:08:19.051 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013942699999999999 2024-09-21T22:08:19.051 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.051 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.051 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:19.052 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.213210+0000", 2024-09-21T22:08:19.052 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5883999999999999e-05 2024-09-21T22:08:19.052 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.052 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.052 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:19.052 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.263909+0000", 2024-09-21T22:08:19.052 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.050699502 2024-09-21T22:08:19.052 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:19.052 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:19.052 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:19.052 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.052 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.052 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.6 87/86 [PushOp(2:7e2189ff:ns::rbd_data.127f79bc213d.0000000000000001:head, version: 57'4, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:7e2189ff:ns::rbd_data.127f79bc213d.0000000000000001:head@57'4, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 4194304, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:08:19.052 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:08:07.268604+0000", 2024-09-21T22:08:19.053 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 11.714818709999999, 2024-09-21T22:08:19.053 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.038386754000000002, 2024-09-21T22:08:19.053 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:19.053 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:19.053 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:08:19.053 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:19.053 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.053 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:19.053 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.268604+0000", 2024-09-21T22:08:19.053 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:19.053 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.053 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.053 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:19.053 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.268604+0000", 2024-09-21T22:08:19.054 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:19.054 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.054 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.054 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:19.054 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.268606+0000", 2024-09-21T22:08:19.054 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2020000000000001e-06 2024-09-21T22:08:19.054 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.054 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.054 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:19.054 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.269149+0000", 2024-09-21T22:08:19.054 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00054327900000000003 2024-09-21T22:08:19.054 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.054 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.054 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:19.055 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.269173+0000", 2024-09-21T22:08:19.055 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4046999999999999e-05 2024-09-21T22:08:19.055 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.055 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.055 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:19.055 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.269187+0000", 2024-09-21T22:08:19.055 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4185999999999999e-05 2024-09-21T22:08:19.055 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.055 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.055 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:19.055 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.269303+0000", 2024-09-21T22:08:19.055 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000115831 2024-09-21T22:08:19.055 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.055 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.056 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:19.056 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.269320+0000", 2024-09-21T22:08:19.056 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6878999999999999e-05 2024-09-21T22:08:19.056 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.056 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.056 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:19.056 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.306991+0000", 2024-09-21T22:08:19.056 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.037671330000000003 2024-09-21T22:08:19.056 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:19.056 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:19.056 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:19.056 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.056 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.056 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.6 87/86 [PushOp(2:762dd302:ns::rbd_data.127f79bc213d.0000000000000007:head, version: 57'5, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:762dd302:ns::rbd_data.127f79bc213d.0000000000000007:head@57'5, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T22:08:19.057 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:08:07.310599+0000", 2024-09-21T22:08:19.057 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 11.672824516, 2024-09-21T22:08:19.057 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025565551999999998, 2024-09-21T22:08:19.057 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:19.057 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:19.057 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:08:19.057 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:19.057 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.057 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:19.057 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.310599+0000", 2024-09-21T22:08:19.057 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:19.057 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.057 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.057 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:19.057 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.310599+0000", 2024-09-21T22:08:19.058 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:19.058 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.058 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.058 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:19.058 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.310600+0000", 2024-09-21T22:08:19.058 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0559999999999999e-06 2024-09-21T22:08:19.058 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.058 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.058 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:19.058 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.311158+0000", 2024-09-21T22:08:19.058 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00055862200000000005 2024-09-21T22:08:19.058 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.058 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.058 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:19.059 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.311180+0000", 2024-09-21T22:08:19.059 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1577999999999999e-05 2024-09-21T22:08:19.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.059 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.059 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:19.059 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.311198+0000", 2024-09-21T22:08:19.059 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8359000000000002e-05 2024-09-21T22:08:19.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.059 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.059 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:19.059 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.311291+0000", 2024-09-21T22:08:19.059 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2887999999999996e-05 2024-09-21T22:08:19.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.059 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.059 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:19.060 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.311300+0000", 2024-09-21T22:08:19.060 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8640000000000002e-06 2024-09-21T22:08:19.060 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.060 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.060 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:19.060 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.336164+0000", 2024-09-21T22:08:19.060 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024864185 2024-09-21T22:08:19.060 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:19.060 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:19.060 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:19.060 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.060 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.060 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.6 87/86 [PushOp(2:762dd302:ns::rbd_data.127f79bc213d.0000000000000007:head, version: 57'5, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:762dd302:ns::rbd_data.127f79bc213d.0000000000000007:head@57'5, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:08:19.060 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:08:07.338463+0000", 2024-09-21T22:08:19.061 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 11.644959905, 2024-09-21T22:08:19.061 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026221639000000001, 2024-09-21T22:08:19.061 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:19.061 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:19.061 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:08:19.061 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:19.061 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.061 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:19.061 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.338463+0000", 2024-09-21T22:08:19.061 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:19.061 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.061 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.061 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:19.061 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.338463+0000", 2024-09-21T22:08:19.062 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:19.062 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.062 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.062 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:19.062 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.338464+0000", 2024-09-21T22:08:19.062 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.035e-06 2024-09-21T22:08:19.062 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.062 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.062 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:19.062 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.339044+0000", 2024-09-21T22:08:19.062 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00057999699999999996 2024-09-21T22:08:19.062 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.062 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.062 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:19.063 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.339070+0000", 2024-09-21T22:08:19.063 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.565e-05 2024-09-21T22:08:19.063 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.063 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.063 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:19.063 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.339088+0000", 2024-09-21T22:08:19.063 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8189000000000002e-05 2024-09-21T22:08:19.063 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.063 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.063 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:19.063 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.339184+0000", 2024-09-21T22:08:19.063 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5475000000000003e-05 2024-09-21T22:08:19.063 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.063 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.064 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:19.064 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.339193+0000", 2024-09-21T22:08:19.064 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7790000000000007e-06 2024-09-21T22:08:19.064 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.064 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.064 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:19.064 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.364685+0000", 2024-09-21T22:08:19.064 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025491514 2024-09-21T22:08:19.064 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:19.064 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:19.064 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:19.064 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.064 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.064 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5398.0:204 2.6 e89/86 2:633e2f34:::rbd_data.126483d75f2d.0000000000000001:head v 89'32, mlcod=88'29)", 2024-09-21T22:08:19.064 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:08:10.148990+0000", 2024-09-21T22:08:19.065 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 8.8344333600000002, 2024-09-21T22:08:19.065 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.034897343999999997, 2024-09-21T22:08:19.065 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:19.065 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:19.065 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:19.065 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:19.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.065 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:19.065 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:10.148990+0000", 2024-09-21T22:08:19.065 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:19.065 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.065 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:19.065 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:10.148990+0000", 2024-09-21T22:08:19.066 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:19.066 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.066 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.066 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:19.066 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:10.148991+0000", 2024-09-21T22:08:19.066 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.61e-06 2024-09-21T22:08:19.066 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.066 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.066 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:19.066 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:10.149966+0000", 2024-09-21T22:08:19.066 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00097419800000000001 2024-09-21T22:08:19.066 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.066 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.066 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:19.067 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:10.149996+0000", 2024-09-21T22:08:19.067 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0762999999999997e-05 2024-09-21T22:08:19.067 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.067 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.067 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:19.067 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:10.150017+0000", 2024-09-21T22:08:19.067 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.109e-05 2024-09-21T22:08:19.067 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.067 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.067 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:19.067 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:10.150090+0000", 2024-09-21T22:08:19.067 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2835000000000003e-05 2024-09-21T22:08:19.067 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.067 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.068 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:19.068 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:10.150112+0000", 2024-09-21T22:08:19.068 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1746000000000002e-05 2024-09-21T22:08:19.068 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.068 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.068 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:19.068 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:10.183831+0000", 2024-09-21T22:08:19.068 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.033718571000000003 2024-09-21T22:08:19.068 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.068 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.068 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:19.068 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:10.183887+0000", 2024-09-21T22:08:19.068 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6530999999999998e-05 2024-09-21T22:08:19.068 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:19.068 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:19.069 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:19.069 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:19.069 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:19.069 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:08:19.070 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-21T22:08:19.032+0000 7f257bfff640 -1 JournalMetadata: 0x56172f19fd50 failed to locate client: INSPECT 2024-09-21T22:08:19.071 INFO:tasks.workunit.client.0.smithi157.stderr:+ queue_length=4 2024-09-21T22:08:19.071 INFO:tasks.workunit.client.0.smithi157.stderr:+ tries=3 2024-09-21T22:08:19.071 INFO:tasks.workunit.client.0.smithi157.stderr:+ '[' 3 -lt 10 ']' 2024-09-21T22:08:19.071 INFO:tasks.workunit.client.0.smithi157.stderr:+ '[' 4 -gt 0 ']' 2024-09-21T22:08:19.071 INFO:tasks.workunit.client.0.smithi157.stderr:+ rbd journal inspect --pool rbd --image testrbdnbd21553 --verbose 2024-09-21T22:08:19.071 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T22:08:19.072 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:08:19.065+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:08:19.120 INFO:tasks.workunit.client.0.smithi157.stdout:Entry: tag_id=1, commit_tid=1 2024-09-21T22:08:19.121 INFO:tasks.workunit.client.0.smithi157.stdout:{ 2024-09-21T22:08:19.121 INFO:tasks.workunit.client.0.smithi157.stdout: "event_type": "AioDiscard", 2024-09-21T22:08:19.121 INFO:tasks.workunit.client.0.smithi157.stdout: "offset": 0, 2024-09-21T22:08:19.121 INFO:tasks.workunit.client.0.smithi157.stdout: "length": 4096, 2024-09-21T22:08:19.121 INFO:tasks.workunit.client.0.smithi157.stdout: "discard_granularity_bytes": 4096, 2024-09-21T22:08:19.121 INFO:tasks.workunit.client.0.smithi157.stdout: "timestamp": "2024-09-21T22:08:15.344115+0000" 2024-09-21T22:08:19.121 INFO:tasks.workunit.client.0.smithi157.stdout:} 2024-09-21T22:08:19.121 INFO:tasks.workunit.client.0.smithi157.stdout:Entry: tag_id=1, commit_tid=2 2024-09-21T22:08:19.121 INFO:tasks.workunit.client.0.smithi157.stdout:{ 2024-09-21T22:08:19.121 INFO:tasks.workunit.client.0.smithi157.stdout: "event_type": "AioDiscard", 2024-09-21T22:08:19.121 INFO:tasks.workunit.client.0.smithi157.stdout: "offset": 0, 2024-09-21T22:08:19.121 INFO:tasks.workunit.client.0.smithi157.stdout: "length": 4096, 2024-09-21T22:08:19.122 INFO:tasks.workunit.client.0.smithi157.stdout: "discard_granularity_bytes": 4096, 2024-09-21T22:08:19.122 INFO:tasks.workunit.client.0.smithi157.stdout: "timestamp": "2024-09-21T22:08:15.360522+0000" 2024-09-21T22:08:19.122 INFO:tasks.workunit.client.0.smithi157.stdout:} 2024-09-21T22:08:19.122 INFO:tasks.workunit.client.0.smithi157.stdout:Entry: tag_id=1, commit_tid=3 2024-09-21T22:08:19.122 INFO:tasks.workunit.client.0.smithi157.stdout:{ 2024-09-21T22:08:19.122 INFO:tasks.workunit.client.0.smithi157.stdout: "event_type": "AioDiscard", 2024-09-21T22:08:19.122 INFO:tasks.workunit.client.0.smithi157.stdout: "offset": 4096, 2024-09-21T22:08:19.122 INFO:tasks.workunit.client.0.smithi157.stdout: "length": 4096, 2024-09-21T22:08:19.122 INFO:tasks.workunit.client.0.smithi157.stdout: "discard_granularity_bytes": 4096, 2024-09-21T22:08:19.122 INFO:tasks.workunit.client.0.smithi157.stdout: "timestamp": "2024-09-21T22:08:15.387911+0000" 2024-09-21T22:08:19.122 INFO:tasks.workunit.client.0.smithi157.stdout:} 2024-09-21T22:08:19.122 INFO:tasks.workunit.client.0.smithi157.stdout:Entry: tag_id=1, commit_tid=4 2024-09-21T22:08:19.123 INFO:tasks.workunit.client.0.smithi157.stdout:{ 2024-09-21T22:08:19.123 INFO:tasks.workunit.client.0.smithi157.stdout: "event_type": "AioDiscard", 2024-09-21T22:08:19.123 INFO:tasks.workunit.client.0.smithi157.stdout: "offset": 4096, 2024-09-21T22:08:19.123 INFO:tasks.workunit.client.0.smithi157.stdout: "length": 4096, 2024-09-21T22:08:19.123 INFO:tasks.workunit.client.0.smithi157.stdout: "discard_granularity_bytes": 4096, 2024-09-21T22:08:19.123 INFO:tasks.workunit.client.0.smithi157.stdout: "timestamp": "2024-09-21T22:08:15.404762+0000" 2024-09-21T22:08:19.123 INFO:tasks.workunit.client.0.smithi157.stdout:} 2024-09-21T22:08:19.123 INFO:tasks.workunit.client.0.smithi157.stdout:Summary: 2024-09-21T22:08:19.123 INFO:tasks.workunit.client.0.smithi157.stdout: 4 entries inspected, 0 errors 2024-09-21T22:08:19.131 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-21T22:08:19.128+0000 7fcd3a022640 -1 JournalMetadata: 0x55de01f02c00 failed to locate client: INSPECT 2024-09-21T22:08:19.135 INFO:tasks.workunit.client.0.smithi157.stderr:+ sleep 1 2024-09-21T22:08:19.171 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:08:19.167+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:08:19.248 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:08:19.248 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:08:19.248 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-09-21T22:08:19.248 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:08:19.262 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T22:08:19.272 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:08:19.269+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:08:19.373 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:08:19.367+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:08:19.434 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:08:19.435 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:08:19.435 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-21T22:08:19.435 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-21T22:08:19.435 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:08:19.448 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T22:08:19.490 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:08:19.487+0000 7f2075a06640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31720) UID: 0 2024-09-21T22:08:19.494 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-21T22:08:19.575 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:08:19.571+0000 7f2075a06640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31720) UID: 0 2024-09-21T22:08:19.617 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:08:19.617 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-21T22:08:19.618 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-21T22:08:19.618 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-21T22:08:19.618 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.618 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5031.0:3 2.2 2:53d5e6d0:::rbd_header.126483d75f2d:head [call rbd.get_size in=11b,call,call rbd.get_features in=15b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e65)", 2024-09-21T22:08:19.618 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:27.373436+0000", 2024-09-21T22:08:19.618 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 52.242099908, 2024-09-21T22:08:19.618 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.86524013700000002, 2024-09-21T22:08:19.618 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:19.618 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:19.618 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:08:19.618 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:19.618 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5031", 2024-09-21T22:08:19.619 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/3232705659", 2024-09-21T22:08:19.619 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3 2024-09-21T22:08:19.619 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.619 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:19.619 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.619 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:19.619 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:27.373436+0000", 2024-09-21T22:08:19.619 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:19.619 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.619 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.619 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:19.619 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:27.373436+0000", 2024-09-21T22:08:19.619 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:19.620 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.620 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.620 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:19.620 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:27.373439+0000", 2024-09-21T22:08:19.620 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5969999999999999e-06 2024-09-21T22:08:19.620 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.620 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.620 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:19.620 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:27.373450+0000", 2024-09-21T22:08:19.620 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1326e-05 2024-09-21T22:08:19.620 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.620 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.620 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:19.620 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:27.373471+0000", 2024-09-21T22:08:19.621 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0169999999999998e-05 2024-09-21T22:08:19.621 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.621 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.621 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:19.621 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:27.373486+0000", 2024-09-21T22:08:19.621 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5483000000000001e-05 2024-09-21T22:08:19.621 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.621 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.621 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:19.621 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:27.373602+0000", 2024-09-21T22:08:19.621 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000115603 2024-09-21T22:08:19.621 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.621 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.621 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-21T22:08:19.622 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:27.373617+0000", 2024-09-21T22:08:19.622 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4956999999999999e-05 2024-09-21T22:08:19.622 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.622 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.622 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:19.622 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.234578+0000", 2024-09-21T22:08:19.622 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.86096097900000002 2024-09-21T22:08:19.622 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.622 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.622 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for missing object", 2024-09-21T22:08:19.622 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.234771+0000", 2024-09-21T22:08:19.622 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019394700000000001 2024-09-21T22:08:19.622 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.623 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.623 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:19.623 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.237470+0000", 2024-09-21T22:08:19.623 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026981319999999998 2024-09-21T22:08:19.623 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.623 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.623 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:19.623 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.237791+0000", 2024-09-21T22:08:19.623 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00032138499999999999 2024-09-21T22:08:19.623 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.623 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.623 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:19.623 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.238677+0000", 2024-09-21T22:08:19.623 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00088555800000000005 2024-09-21T22:08:19.624 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:19.624 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:19.624 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:19.624 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.624 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.624 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.2 66/65 [PushOp(2:4481fad7:ns::rbd_data.127f79bc213d.0000000000000006:head, version: 57'9, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:4481fad7:ns::rbd_data.127f79bc213d.0000000000000006:head@57'9, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T22:08:19.624 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:28.247056+0000", 2024-09-21T22:08:19.624 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 51.368480245000001, 2024-09-21T22:08:19.624 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.045025012000000003, 2024-09-21T22:08:19.624 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:19.624 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:19.624 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:08:19.624 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:19.625 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.625 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:19.625 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.247056+0000", 2024-09-21T22:08:19.625 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:19.625 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.625 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.625 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:19.625 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.247056+0000", 2024-09-21T22:08:19.625 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:19.625 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.625 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.625 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:19.625 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.247057+0000", 2024-09-21T22:08:19.625 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2500000000000001e-06 2024-09-21T22:08:19.626 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.626 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.626 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:19.626 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.249384+0000", 2024-09-21T22:08:19.626 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0023265619999999999 2024-09-21T22:08:19.626 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.626 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.626 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:19.626 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.249429+0000", 2024-09-21T22:08:19.626 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5392999999999997e-05 2024-09-21T22:08:19.626 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.626 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.626 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:19.627 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.249476+0000", 2024-09-21T22:08:19.627 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6570000000000003e-05 2024-09-21T22:08:19.627 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.627 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.627 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:19.627 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.249611+0000", 2024-09-21T22:08:19.627 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000135143 2024-09-21T22:08:19.627 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.627 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.627 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:19.627 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.249626+0000", 2024-09-21T22:08:19.627 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.469e-05 2024-09-21T22:08:19.627 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.627 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.628 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:19.628 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.292081+0000", 2024-09-21T22:08:19.628 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.042455404000000002 2024-09-21T22:08:19.628 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:19.628 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:19.628 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:19.628 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.628 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.628 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5031.0:9 2.2 2:53d5e6d0:::rbd_header.126483d75f2d:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-21T22:08:19.628 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:28.250163+0000", 2024-09-21T22:08:19.628 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 51.365372850999997, 2024-09-21T22:08:19.628 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.042986430999999999, 2024-09-21T22:08:19.628 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:19.628 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:19.629 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:08:19.629 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:19.629 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5031", 2024-09-21T22:08:19.629 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/3232705659", 2024-09-21T22:08:19.629 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 9 2024-09-21T22:08:19.629 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.629 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:19.629 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.629 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:19.629 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.250163+0000", 2024-09-21T22:08:19.629 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:19.629 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.629 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.629 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:19.630 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.250163+0000", 2024-09-21T22:08:19.630 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:19.630 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.630 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.630 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:19.630 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.250166+0000", 2024-09-21T22:08:19.630 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7039999999999999e-06 2024-09-21T22:08:19.630 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.630 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.630 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:19.630 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.250175+0000", 2024-09-21T22:08:19.630 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7560000000000005e-06 2024-09-21T22:08:19.630 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.630 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.631 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:19.631 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.250199+0000", 2024-09-21T22:08:19.631 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4049999999999998e-05 2024-09-21T22:08:19.631 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.631 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.631 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:19.631 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.250218+0000", 2024-09-21T22:08:19.631 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9236e-05 2024-09-21T22:08:19.631 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.631 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.631 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:19.631 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.292193+0000", 2024-09-21T22:08:19.631 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.041974694 2024-09-21T22:08:19.631 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.632 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.632 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:19.632 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.292492+0000", 2024-09-21T22:08:19.632 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00029900100000000002 2024-09-21T22:08:19.632 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.632 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.632 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:19.632 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.293150+0000", 2024-09-21T22:08:19.632 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00065799000000000001 2024-09-21T22:08:19.632 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:19.632 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:19.632 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:19.632 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.632 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.632 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.2 66/65 [PushOp(2:4481fad7:ns::rbd_data.127f79bc213d.0000000000000006:head, version: 57'9, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:4481fad7:ns::rbd_data.127f79bc213d.0000000000000006:head@57'9, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:08:19.633 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:28.331166+0000", 2024-09-21T22:08:19.633 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 51.284370101, 2024-09-21T22:08:19.633 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.039995378999999998, 2024-09-21T22:08:19.633 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:19.633 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:19.633 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:08:19.633 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:19.633 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.633 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:19.633 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.331166+0000", 2024-09-21T22:08:19.633 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:19.633 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.633 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.633 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:19.634 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.331166+0000", 2024-09-21T22:08:19.634 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:19.634 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.634 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.634 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:19.634 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.331167+0000", 2024-09-21T22:08:19.634 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9199999999999999e-07 2024-09-21T22:08:19.634 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.634 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.634 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:19.634 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.332127+0000", 2024-09-21T22:08:19.634 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00095969800000000004 2024-09-21T22:08:19.634 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.634 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.635 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:19.635 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.332149+0000", 2024-09-21T22:08:19.635 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1974999999999999e-05 2024-09-21T22:08:19.635 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.635 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.635 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:19.635 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.332161+0000", 2024-09-21T22:08:19.635 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2276000000000001e-05 2024-09-21T22:08:19.635 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.635 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.635 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:19.635 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.332290+0000", 2024-09-21T22:08:19.635 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000129214 2024-09-21T22:08:19.635 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.635 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.636 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:19.636 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.332312+0000", 2024-09-21T22:08:19.636 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1308999999999999e-05 2024-09-21T22:08:19.636 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.636 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.636 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:19.636 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.371162+0000", 2024-09-21T22:08:19.636 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.038850015000000002 2024-09-21T22:08:19.636 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:19.636 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:19.636 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:19.636 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.636 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.636 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4811.0:3 2.3 2:c4c92e5a:::rbd_trash:head [call rbd.trash_list in=13b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-21T22:08:19.637 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:28.475306+0000", 2024-09-21T22:08:19.637 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 51.140230557000002, 2024-09-21T22:08:19.637 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.77996670099999998, 2024-09-21T22:08:19.637 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:19.637 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:19.637 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:08:19.637 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:19.637 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4811", 2024-09-21T22:08:19.637 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/2028736390", 2024-09-21T22:08:19.637 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3 2024-09-21T22:08:19.637 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.637 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:19.638 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.638 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:19.638 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.475306+0000", 2024-09-21T22:08:19.638 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:19.638 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.638 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.638 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:19.638 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.475306+0000", 2024-09-21T22:08:19.638 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:19.638 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.638 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.638 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:19.638 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.475308+0000", 2024-09-21T22:08:19.639 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8959999999999999e-06 2024-09-21T22:08:19.639 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.639 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.639 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:19.639 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.475314+0000", 2024-09-21T22:08:19.639 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2280000000000003e-06 2024-09-21T22:08:19.639 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.639 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.639 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:19.639 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.475329+0000", 2024-09-21T22:08:19.639 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5054e-05 2024-09-21T22:08:19.639 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.639 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.640 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:19.640 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.475338+0000", 2024-09-21T22:08:19.640 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3770000000000002e-06 2024-09-21T22:08:19.640 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.640 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.640 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:19.640 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.475484+0000", 2024-09-21T22:08:19.640 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014612300000000001 2024-09-21T22:08:19.640 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.640 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.640 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-21T22:08:19.640 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.475506+0000", 2024-09-21T22:08:19.640 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1319999999999999e-05 2024-09-21T22:08:19.641 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.641 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.641 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:19.641 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.254433+0000", 2024-09-21T22:08:19.641 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.778927708 2024-09-21T22:08:19.641 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.641 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.641 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:19.641 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.254979+0000", 2024-09-21T22:08:19.641 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00054573100000000004 2024-09-21T22:08:19.641 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.641 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.641 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:19.642 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.255272+0000", 2024-09-21T22:08:19.642 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000293264 2024-09-21T22:08:19.642 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:19.642 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:19.642 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:19.642 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.642 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.642 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.3 67/66 [PushOp(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head, version: 63'11, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head@63'11, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T22:08:19.642 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:29.262595+0000", 2024-09-21T22:08:19.642 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 50.352941373999997, 2024-09-21T22:08:19.642 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.036027272999999999, 2024-09-21T22:08:19.642 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:19.643 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:19.643 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:08:19.643 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:19.643 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.643 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:19.643 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.262595+0000", 2024-09-21T22:08:19.643 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:19.643 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.643 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.643 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:19.643 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.262595+0000", 2024-09-21T22:08:19.643 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:19.643 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.644 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.644 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:19.644 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.262596+0000", 2024-09-21T22:08:19.644 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4300000000000001e-06 2024-09-21T22:08:19.644 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.644 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.644 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:19.644 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.264505+0000", 2024-09-21T22:08:19.644 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019085650000000001 2024-09-21T22:08:19.644 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.644 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.644 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:19.645 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.264557+0000", 2024-09-21T22:08:19.645 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2040000000000003e-05 2024-09-21T22:08:19.645 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.645 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.645 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:19.645 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.264589+0000", 2024-09-21T22:08:19.645 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1636e-05 2024-09-21T22:08:19.645 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.645 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.645 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:19.645 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.264778+0000", 2024-09-21T22:08:19.645 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018935600000000001 2024-09-21T22:08:19.645 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.646 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.646 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:19.646 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.264803+0000", 2024-09-21T22:08:19.646 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4692999999999999e-05 2024-09-21T22:08:19.646 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.646 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.646 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:19.646 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.298622+0000", 2024-09-21T22:08:19.646 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.033819553000000002 2024-09-21T22:08:19.646 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:19.646 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:19.646 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:19.646 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.646 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.647 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.3 67/66 [PushOp(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head, version: 63'11, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head@63'11, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:08:19.647 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:29.301490+0000", 2024-09-21T22:08:19.647 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 50.314046587999997, 2024-09-21T22:08:19.647 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.041178293999999997, 2024-09-21T22:08:19.647 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:19.647 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:19.647 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:08:19.647 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:19.647 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.647 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:19.647 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.301490+0000", 2024-09-21T22:08:19.647 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:19.647 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.648 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.648 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:19.648 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.301490+0000", 2024-09-21T22:08:19.648 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:19.648 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.648 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.648 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:19.648 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.301491+0000", 2024-09-21T22:08:19.648 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.513e-06 2024-09-21T22:08:19.648 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.648 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.648 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:19.648 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.302815+0000", 2024-09-21T22:08:19.648 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001323722 2024-09-21T22:08:19.648 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.649 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.649 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:19.649 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.302863+0000", 2024-09-21T22:08:19.649 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8449999999999999e-05 2024-09-21T22:08:19.649 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.649 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.649 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:19.649 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.302895+0000", 2024-09-21T22:08:19.649 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1072999999999999e-05 2024-09-21T22:08:19.649 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.649 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.649 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:19.649 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.303014+0000", 2024-09-21T22:08:19.650 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011907400000000001 2024-09-21T22:08:19.650 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.650 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.650 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:19.650 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.303029+0000", 2024-09-21T22:08:19.650 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5322e-05 2024-09-21T22:08:19.650 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.650 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.650 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:19.650 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.342668+0000", 2024-09-21T22:08:19.650 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.039639140000000003 2024-09-21T22:08:19.650 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:19.650 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:19.650 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:19.651 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.651 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.651 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.3 67/66 [PushOp(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head, version: 63'11, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head@63'11, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:08:19.651 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:29.345626+0000", 2024-09-21T22:08:19.651 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 50.269909853999998, 2024-09-21T22:08:19.651 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.034820216000000001, 2024-09-21T22:08:19.651 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:19.651 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:19.651 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:08:19.651 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:19.651 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.651 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:19.651 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.345626+0000", 2024-09-21T22:08:19.651 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:19.652 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.652 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.652 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:19.652 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.345626+0000", 2024-09-21T22:08:19.652 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:19.652 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.652 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.652 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:19.652 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.345628+0000", 2024-09-21T22:08:19.652 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3799999999999999e-06 2024-09-21T22:08:19.652 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.652 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.652 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:19.652 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.347215+0000", 2024-09-21T22:08:19.652 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015868760000000001 2024-09-21T22:08:19.653 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.653 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.653 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:19.653 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.347283+0000", 2024-09-21T22:08:19.653 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8267000000000002e-05 2024-09-21T22:08:19.653 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.653 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.653 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:19.653 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.347336+0000", 2024-09-21T22:08:19.653 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3455999999999998e-05 2024-09-21T22:08:19.653 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.653 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.653 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:19.653 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.347463+0000", 2024-09-21T22:08:19.654 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012614399999999999 2024-09-21T22:08:19.654 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.654 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.654 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:19.654 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.347473+0000", 2024-09-21T22:08:19.654 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0462000000000001e-05 2024-09-21T22:08:19.654 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.654 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.654 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:19.654 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.380447+0000", 2024-09-21T22:08:19.654 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.032973631000000003 2024-09-21T22:08:19.654 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:19.654 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:19.654 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:19.655 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.655 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.655 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.3 67/66 [PushOp(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head, version: 63'19, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head@63'19, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T22:08:19.655 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:29.420098+0000", 2024-09-21T22:08:19.655 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 50.195438019000001, 2024-09-21T22:08:19.655 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.067283846999999994, 2024-09-21T22:08:19.655 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:19.655 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:19.655 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:08:19.655 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:19.655 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.655 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:19.655 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.420098+0000", 2024-09-21T22:08:19.656 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:19.656 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.656 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.656 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:19.656 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.420098+0000", 2024-09-21T22:08:19.656 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:19.656 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.656 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.656 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:19.656 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.420100+0000", 2024-09-21T22:08:19.656 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.716e-06 2024-09-21T22:08:19.656 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.656 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.656 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:19.657 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.421949+0000", 2024-09-21T22:08:19.657 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0018485190000000001 2024-09-21T22:08:19.657 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.657 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.657 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:19.657 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.421996+0000", 2024-09-21T22:08:19.657 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7089000000000003e-05 2024-09-21T22:08:19.657 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.657 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.657 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:19.657 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.422042+0000", 2024-09-21T22:08:19.657 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6010000000000002e-05 2024-09-21T22:08:19.657 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.657 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.658 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:19.658 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.422231+0000", 2024-09-21T22:08:19.658 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018960900000000001 2024-09-21T22:08:19.658 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.658 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.658 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:19.658 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.422254+0000", 2024-09-21T22:08:19.658 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2864e-05 2024-09-21T22:08:19.658 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.658 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.658 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:19.658 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.487382+0000", 2024-09-21T22:08:19.658 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.065128039999999998 2024-09-21T22:08:19.658 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:19.659 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:19.659 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:19.659 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.659 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.659 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.3 67/66 [PushOp(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head, version: 63'19, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head@63'19, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:08:19.659 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:29.491783+0000", 2024-09-21T22:08:19.659 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 50.123753674, 2024-09-21T22:08:19.659 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.04476596, 2024-09-21T22:08:19.659 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:19.659 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:19.659 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:08:19.659 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:19.659 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.659 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:19.659 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.491783+0000", 2024-09-21T22:08:19.660 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:19.660 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.660 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.660 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:19.660 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.491783+0000", 2024-09-21T22:08:19.660 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:19.660 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.660 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.660 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:19.660 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.491784+0000", 2024-09-21T22:08:19.660 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.17e-06 2024-09-21T22:08:19.660 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.660 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.660 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:19.661 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.493126+0000", 2024-09-21T22:08:19.661 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0013424489999999999 2024-09-21T22:08:19.661 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.661 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.661 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:19.661 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.493170+0000", 2024-09-21T22:08:19.661 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4153000000000002e-05 2024-09-21T22:08:19.661 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.661 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.661 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:19.661 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.493218+0000", 2024-09-21T22:08:19.661 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7392999999999998e-05 2024-09-21T22:08:19.661 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.661 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.662 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:19.662 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.493392+0000", 2024-09-21T22:08:19.662 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017392699999999999 2024-09-21T22:08:19.662 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.662 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.662 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:19.662 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.493416+0000", 2024-09-21T22:08:19.662 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4117e-05 2024-09-21T22:08:19.662 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.662 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.662 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:19.662 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.536549+0000", 2024-09-21T22:08:19.662 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.043132750999999997 2024-09-21T22:08:19.662 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:19.663 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:19.663 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:19.663 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.663 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.663 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.3 67/66 [PushOp(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head, version: 63'19, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head@63'19, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:08:19.663 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:29.539831+0000", 2024-09-21T22:08:19.663 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 50.075704913999999, 2024-09-21T22:08:19.663 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.036233222000000002, 2024-09-21T22:08:19.663 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:19.663 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:19.663 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:08:19.663 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:19.663 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.663 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:19.664 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.539831+0000", 2024-09-21T22:08:19.664 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:19.664 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.664 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.664 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:19.664 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.539831+0000", 2024-09-21T22:08:19.664 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:19.664 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.664 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.664 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:19.664 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.539833+0000", 2024-09-21T22:08:19.664 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5200000000000001e-06 2024-09-21T22:08:19.664 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.664 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.665 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:19.665 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.541130+0000", 2024-09-21T22:08:19.665 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0012972719999999999 2024-09-21T22:08:19.665 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.665 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.665 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:19.665 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.541179+0000", 2024-09-21T22:08:19.665 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8507000000000002e-05 2024-09-21T22:08:19.665 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.665 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.665 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:19.665 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.541226+0000", 2024-09-21T22:08:19.665 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7663000000000001e-05 2024-09-21T22:08:19.665 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.666 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.666 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:19.666 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.541361+0000", 2024-09-21T22:08:19.666 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000135065 2024-09-21T22:08:19.666 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.666 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.666 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:19.666 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.541374+0000", 2024-09-21T22:08:19.666 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2295e-05 2024-09-21T22:08:19.666 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.666 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.666 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:19.666 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.576065+0000", 2024-09-21T22:08:19.666 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.034690899999999997 2024-09-21T22:08:19.667 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:19.667 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:19.667 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:19.667 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.667 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.667 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.3 67/66 [PushOp(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head, version: 63'19, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head@63'19, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 4194304, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:08:19.667 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:29.579455+0000", 2024-09-21T22:08:19.667 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 50.036081606000003, 2024-09-21T22:08:19.667 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.034295594999999998, 2024-09-21T22:08:19.667 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:19.667 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:19.667 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:08:19.667 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:19.667 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.668 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:19.668 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.579455+0000", 2024-09-21T22:08:19.668 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:19.668 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.668 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.668 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:19.668 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.579455+0000", 2024-09-21T22:08:19.668 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:19.668 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.668 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.668 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:19.668 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.579456+0000", 2024-09-21T22:08:19.668 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.404e-06 2024-09-21T22:08:19.669 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.669 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.669 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:19.669 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.580763+0000", 2024-09-21T22:08:19.669 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001307325 2024-09-21T22:08:19.669 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.669 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.669 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:19.669 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.580811+0000", 2024-09-21T22:08:19.669 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7843000000000003e-05 2024-09-21T22:08:19.669 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.669 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.669 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:19.669 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.580860+0000", 2024-09-21T22:08:19.670 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9036999999999998e-05 2024-09-21T22:08:19.670 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.670 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.670 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:19.670 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.580988+0000", 2024-09-21T22:08:19.670 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000127529 2024-09-21T22:08:19.670 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.670 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.670 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:19.670 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.580998+0000", 2024-09-21T22:08:19.670 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0617e-05 2024-09-21T22:08:19.670 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.670 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.671 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:19.671 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.613750+0000", 2024-09-21T22:08:19.671 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.032751839999999997 2024-09-21T22:08:19.671 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:19.671 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:19.671 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:19.671 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.671 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.671 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5008.0:125 2.3 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-21T22:08:19.671 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:38.066298+0000", 2024-09-21T22:08:19.671 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 41.549237941999998, 2024-09-21T22:08:19.671 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.062817136999999995, 2024-09-21T22:08:19.671 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:19.671 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:19.672 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:19.672 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:19.672 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5008", 2024-09-21T22:08:19.672 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/1400114666", 2024-09-21T22:08:19.672 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 125 2024-09-21T22:08:19.672 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.672 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:19.672 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.672 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:19.672 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.066298+0000", 2024-09-21T22:08:19.672 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:19.672 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.672 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.672 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:19.672 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.066298+0000", 2024-09-21T22:08:19.673 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:19.673 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.673 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.673 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:19.673 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.066300+0000", 2024-09-21T22:08:19.673 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.474e-06 2024-09-21T22:08:19.673 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.673 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.673 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:19.673 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.067443+0000", 2024-09-21T22:08:19.673 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011428009999999999 2024-09-21T22:08:19.673 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.673 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.673 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:19.674 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.067477+0000", 2024-09-21T22:08:19.674 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4059e-05 2024-09-21T22:08:19.674 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.674 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.674 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:19.674 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.067492+0000", 2024-09-21T22:08:19.674 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5054e-05 2024-09-21T22:08:19.674 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.674 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.674 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:19.674 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.067638+0000", 2024-09-21T22:08:19.674 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014583900000000001 2024-09-21T22:08:19.674 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.674 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.675 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:19.675 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.067882+0000", 2024-09-21T22:08:19.675 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024415500000000002 2024-09-21T22:08:19.675 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.675 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.675 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-09-21T22:08:19.675 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.068247+0000", 2024-09-21T22:08:19.675 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00036570800000000002 2024-09-21T22:08:19.675 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.675 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.675 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:19.675 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.119971+0000", 2024-09-21T22:08:19.675 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.051723866 2024-09-21T22:08:19.675 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.676 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.676 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:19.676 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.128849+0000", 2024-09-21T22:08:19.676 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0088779489999999996 2024-09-21T22:08:19.676 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.676 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.676 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:19.676 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.129000+0000", 2024-09-21T22:08:19.676 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015062499999999999 2024-09-21T22:08:19.676 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.676 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.676 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:19.676 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.129116+0000", 2024-09-21T22:08:19.676 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011560700000000001 2024-09-21T22:08:19.677 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:19.677 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:19.677 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:19.677 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.677 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.677 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5008.0:127 2.3 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-21T22:08:19.677 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:38.140308+0000", 2024-09-21T22:08:19.677 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 41.475227840000002, 2024-09-21T22:08:19.677 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.055290512999999999, 2024-09-21T22:08:19.677 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:19.677 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:19.677 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:19.677 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:19.678 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5008", 2024-09-21T22:08:19.678 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/1400114666", 2024-09-21T22:08:19.678 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 127 2024-09-21T22:08:19.678 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.678 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:19.678 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.678 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:19.678 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.140308+0000", 2024-09-21T22:08:19.678 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:19.678 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.678 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.678 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:19.678 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.140308+0000", 2024-09-21T22:08:19.678 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:19.679 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.679 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.679 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:19.679 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.140310+0000", 2024-09-21T22:08:19.679 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6509999999999999e-06 2024-09-21T22:08:19.679 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.679 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.679 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:19.679 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.141504+0000", 2024-09-21T22:08:19.679 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011935750000000001 2024-09-21T22:08:19.679 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.679 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.679 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:19.679 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.141522+0000", 2024-09-21T22:08:19.680 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8349e-05 2024-09-21T22:08:19.680 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.680 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.680 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:19.680 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.141531+0000", 2024-09-21T22:08:19.680 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3239999999999992e-06 2024-09-21T22:08:19.680 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.680 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.680 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:19.680 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.141661+0000", 2024-09-21T22:08:19.680 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013001000000000001 2024-09-21T22:08:19.680 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.680 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.681 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:19.681 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.141907+0000", 2024-09-21T22:08:19.681 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024547499999999999 2024-09-21T22:08:19.681 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.681 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.681 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-09-21T22:08:19.681 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.142274+0000", 2024-09-21T22:08:19.681 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00036716400000000001 2024-09-21T22:08:19.681 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.681 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.681 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:19.681 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.189864+0000", 2024-09-21T22:08:19.681 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047589617000000001 2024-09-21T22:08:19.681 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.682 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.682 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:19.682 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.195406+0000", 2024-09-21T22:08:19.682 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0055425860000000004 2024-09-21T22:08:19.682 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.682 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.682 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:19.682 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.195507+0000", 2024-09-21T22:08:19.682 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000100862 2024-09-21T22:08:19.682 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.682 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.682 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:19.682 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.195599+0000", 2024-09-21T22:08:19.682 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1899999999999998e-05 2024-09-21T22:08:19.683 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:19.683 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:19.683 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:19.683 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.683 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.683 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5008.0:129 2.3 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head [write 2228224~917504 in=917504b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-21T22:08:19.683 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:38.209297+0000", 2024-09-21T22:08:19.683 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 41.406238959, 2024-09-21T22:08:19.683 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.060441054000000001, 2024-09-21T22:08:19.683 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:19.683 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:19.683 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:19.683 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:19.684 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5008", 2024-09-21T22:08:19.684 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/1400114666", 2024-09-21T22:08:19.684 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 129 2024-09-21T22:08:19.684 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.684 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:19.684 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.684 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:19.684 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.209297+0000", 2024-09-21T22:08:19.684 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:19.684 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.684 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.684 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:19.684 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.209297+0000", 2024-09-21T22:08:19.684 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:19.685 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.685 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.685 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:19.685 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.209299+0000", 2024-09-21T22:08:19.685 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.274e-06 2024-09-21T22:08:19.685 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.685 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.685 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:19.685 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.210077+0000", 2024-09-21T22:08:19.685 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00077849799999999997 2024-09-21T22:08:19.685 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.685 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.685 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:19.685 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.210094+0000", 2024-09-21T22:08:19.685 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6772000000000001e-05 2024-09-21T22:08:19.686 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.686 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.686 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:19.686 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.210103+0000", 2024-09-21T22:08:19.686 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6780000000000006e-06 2024-09-21T22:08:19.686 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.686 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.686 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:19.686 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.210254+0000", 2024-09-21T22:08:19.686 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015141699999999999 2024-09-21T22:08:19.686 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.686 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.686 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:19.687 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.210501+0000", 2024-09-21T22:08:19.687 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000246498 2024-09-21T22:08:19.687 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.687 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.687 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-09-21T22:08:19.687 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.210872+0000", 2024-09-21T22:08:19.687 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00037147500000000002 2024-09-21T22:08:19.687 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.687 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.687 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:19.687 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.269019+0000", 2024-09-21T22:08:19.687 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058147114 2024-09-21T22:08:19.687 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.687 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.688 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:19.688 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.269638+0000", 2024-09-21T22:08:19.688 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00061896499999999997 2024-09-21T22:08:19.688 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.688 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.688 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:19.688 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.269693+0000", 2024-09-21T22:08:19.688 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4786e-05 2024-09-21T22:08:19.688 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.688 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.688 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:19.688 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.269738+0000", 2024-09-21T22:08:19.688 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5577000000000001e-05 2024-09-21T22:08:19.688 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:19.689 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:19.689 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:19.689 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.689 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.689 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5008.0:131 2.3 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-21T22:08:19.689 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:38.279879+0000", 2024-09-21T22:08:19.689 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 41.335657269000002, 2024-09-21T22:08:19.689 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.054184296999999999, 2024-09-21T22:08:19.689 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:19.689 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:19.689 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:19.689 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:19.689 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5008", 2024-09-21T22:08:19.689 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/1400114666", 2024-09-21T22:08:19.690 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 131 2024-09-21T22:08:19.690 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.690 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:19.690 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.690 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:19.690 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.279879+0000", 2024-09-21T22:08:19.690 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:19.690 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.690 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.690 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:19.690 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.279879+0000", 2024-09-21T22:08:19.690 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:19.690 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.691 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.691 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:19.691 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.279880+0000", 2024-09-21T22:08:19.691 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.136e-06 2024-09-21T22:08:19.691 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.691 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.691 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:19.691 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.280636+0000", 2024-09-21T22:08:19.691 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00075625900000000001 2024-09-21T22:08:19.691 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.691 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.691 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:19.691 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.280648+0000", 2024-09-21T22:08:19.691 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2026e-05 2024-09-21T22:08:19.692 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.692 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.692 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:19.692 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.280654+0000", 2024-09-21T22:08:19.692 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8839999999999999e-06 2024-09-21T22:08:19.692 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.692 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.692 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:19.692 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.280739+0000", 2024-09-21T22:08:19.692 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4196000000000005e-05 2024-09-21T22:08:19.692 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.692 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.692 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:19.692 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.280808+0000", 2024-09-21T22:08:19.692 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9685999999999997e-05 2024-09-21T22:08:19.693 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.693 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.693 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-09-21T22:08:19.693 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.280900+0000", 2024-09-21T22:08:19.693 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1407000000000001e-05 2024-09-21T22:08:19.693 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.693 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.693 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:19.693 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.307730+0000", 2024-09-21T22:08:19.693 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026830568999999999 2024-09-21T22:08:19.693 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.693 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.693 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:19.693 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.333901+0000", 2024-09-21T22:08:19.694 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026170787000000001 2024-09-21T22:08:19.694 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.694 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.694 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:19.694 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.333996+0000", 2024-09-21T22:08:19.694 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4723999999999996e-05 2024-09-21T22:08:19.694 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.694 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.694 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:19.694 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.334063+0000", 2024-09-21T22:08:19.694 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7623e-05 2024-09-21T22:08:19.694 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:19.694 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:19.694 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:19.695 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.695 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.695 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.1 87/86 [PushOp(2:99eff0b8:ns::rbd_data.127f79bc213d.0000000000000002:head, version: 57'3, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:99eff0b8:ns::rbd_data.127f79bc213d.0000000000000002:head@57'3, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T22:08:19.695 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:08:07.091918+0000", 2024-09-21T22:08:19.695 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 12.523618826, 2024-09-21T22:08:19.695 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.071321166000000005, 2024-09-21T22:08:19.695 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:19.695 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:19.695 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:08:19.695 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:19.695 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.695 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:19.695 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.091918+0000", 2024-09-21T22:08:19.696 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:19.696 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.696 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.696 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:19.696 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.091918+0000", 2024-09-21T22:08:19.696 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:19.696 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.696 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.696 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:19.696 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.091919+0000", 2024-09-21T22:08:19.696 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.57e-06 2024-09-21T22:08:19.696 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.696 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.697 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:19.697 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.093627+0000", 2024-09-21T22:08:19.697 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001708162 2024-09-21T22:08:19.697 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.697 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.697 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:19.697 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.093675+0000", 2024-09-21T22:08:19.697 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.761e-05 2024-09-21T22:08:19.697 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.697 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.697 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:19.697 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.093707+0000", 2024-09-21T22:08:19.697 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1847000000000002e-05 2024-09-21T22:08:19.697 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.698 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.698 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:19.698 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.093899+0000", 2024-09-21T22:08:19.698 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000192449 2024-09-21T22:08:19.698 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.698 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.698 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:19.698 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.093922+0000", 2024-09-21T22:08:19.698 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2976000000000002e-05 2024-09-21T22:08:19.698 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.698 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.698 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:19.698 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.163239+0000", 2024-09-21T22:08:19.698 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.069316552000000003 2024-09-21T22:08:19.699 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:19.699 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:19.699 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:19.699 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.699 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.699 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.1 87/86 [PushOp(2:99eff0b8:ns::rbd_data.127f79bc213d.0000000000000002:head, version: 57'3, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:99eff0b8:ns::rbd_data.127f79bc213d.0000000000000002:head@57'3, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:08:19.699 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:08:07.167681+0000", 2024-09-21T22:08:19.699 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 12.447855239999999, 2024-09-21T22:08:19.699 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.043679153999999998, 2024-09-21T22:08:19.699 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:19.699 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:19.699 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:08:19.699 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:19.699 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.700 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:19.700 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.167681+0000", 2024-09-21T22:08:19.700 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:19.700 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.700 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.700 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:19.700 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.167681+0000", 2024-09-21T22:08:19.700 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:19.700 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.700 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.700 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:19.700 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.167683+0000", 2024-09-21T22:08:19.700 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.59e-06 2024-09-21T22:08:19.700 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.701 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.701 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:19.701 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.168793+0000", 2024-09-21T22:08:19.701 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011100229999999999 2024-09-21T22:08:19.701 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.701 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.701 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:19.701 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.168830+0000", 2024-09-21T22:08:19.701 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7580999999999998e-05 2024-09-21T22:08:19.701 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.701 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.701 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:19.701 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.168852+0000", 2024-09-21T22:08:19.701 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1766000000000001e-05 2024-09-21T22:08:19.702 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.702 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.702 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:19.702 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.168987+0000", 2024-09-21T22:08:19.702 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000135203 2024-09-21T22:08:19.702 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.702 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.702 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:19.702 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.169003+0000", 2024-09-21T22:08:19.702 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.543e-05 2024-09-21T22:08:19.702 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.702 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.702 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:19.702 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.211360+0000", 2024-09-21T22:08:19.703 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.042357561000000002 2024-09-21T22:08:19.703 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:19.703 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:19.703 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:19.703 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.703 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.703 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.1 87/86 [PushOp(2:99eff0b8:ns::rbd_data.127f79bc213d.0000000000000002:head, version: 57'3, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:99eff0b8:ns::rbd_data.127f79bc213d.0000000000000002:head@57'3, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:08:19.703 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:08:07.215689+0000", 2024-09-21T22:08:19.703 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 12.399847085999999, 2024-09-21T22:08:19.703 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.052501277999999998, 2024-09-21T22:08:19.703 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:19.703 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:19.703 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:08:19.703 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:19.704 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.704 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:19.704 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.215689+0000", 2024-09-21T22:08:19.704 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:19.704 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.704 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.704 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:19.704 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.215689+0000", 2024-09-21T22:08:19.704 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:19.704 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.704 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.704 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:19.704 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.215690+0000", 2024-09-21T22:08:19.704 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1319999999999999e-06 2024-09-21T22:08:19.705 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.705 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.705 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:19.705 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.216711+0000", 2024-09-21T22:08:19.705 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0010210600000000001 2024-09-21T22:08:19.705 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.705 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.705 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:19.705 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.216748+0000", 2024-09-21T22:08:19.705 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6776000000000002e-05 2024-09-21T22:08:19.705 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.705 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.705 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:19.705 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.216769+0000", 2024-09-21T22:08:19.706 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1189999999999999e-05 2024-09-21T22:08:19.706 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.706 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.706 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:19.706 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.216932+0000", 2024-09-21T22:08:19.706 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016283700000000001 2024-09-21T22:08:19.706 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.706 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.706 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:19.706 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.216957+0000", 2024-09-21T22:08:19.706 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4896999999999999e-05 2024-09-21T22:08:19.706 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.706 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.706 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:19.707 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.268191+0000", 2024-09-21T22:08:19.707 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.051233385999999999 2024-09-21T22:08:19.707 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:19.707 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:19.707 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:19.707 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.707 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.707 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.1 87/86 [PushOp(2:922055e2:ns::rbd_data.127f79bc213d.0000000000000009:head, version: 57'4, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:922055e2:ns::rbd_data.127f79bc213d.0000000000000009:head@57'4, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T22:08:19.707 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:08:07.302602+0000", 2024-09-21T22:08:19.707 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 12.312933951, 2024-09-21T22:08:19.707 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.038952326000000002, 2024-09-21T22:08:19.707 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:19.707 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:19.707 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:08:19.708 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:19.708 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.708 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:19.708 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.302602+0000", 2024-09-21T22:08:19.708 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:19.708 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.708 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.708 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:19.708 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.302602+0000", 2024-09-21T22:08:19.708 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:19.708 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.708 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.708 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:19.708 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.302604+0000", 2024-09-21T22:08:19.709 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9769999999999999e-06 2024-09-21T22:08:19.709 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.709 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.709 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:19.709 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.303971+0000", 2024-09-21T22:08:19.709 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001366719 2024-09-21T22:08:19.709 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.709 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.709 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:19.709 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.304022+0000", 2024-09-21T22:08:19.709 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1307999999999997e-05 2024-09-21T22:08:19.709 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.709 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.709 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:19.709 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.304065+0000", 2024-09-21T22:08:19.710 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.214e-05 2024-09-21T22:08:19.710 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.710 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.710 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:19.710 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.304200+0000", 2024-09-21T22:08:19.710 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000135038 2024-09-21T22:08:19.710 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.710 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.710 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:19.710 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.304217+0000", 2024-09-21T22:08:19.710 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7856e-05 2024-09-21T22:08:19.710 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:19.710 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:19.710 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:19.711 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.341555+0000", 2024-09-21T22:08:19.711 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.037337288000000003 2024-09-21T22:08:19.711 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:19.711 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:19.711 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:19.711 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:19.711 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:19.711 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:08:19.712 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:08:19.707+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:08:19.713 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T22:08:19.814 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:08:19.811+0000 7f2075a06640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31720) UID: 0 2024-09-21T22:08:19.834 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:08:19.834 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:08:19.834 INFO:teuthology.orchestra.run.smithi039.stdout: "num_ops": 0 2024-09-21T22:08:19.834 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:08:19.848 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T22:08:19.915 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:08:19.911+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:08:19.972 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:08:19.973 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:08:19.973 INFO:teuthology.orchestra.run.smithi039.stdout: "complaint_time": 30, 2024-09-21T22:08:19.973 INFO:teuthology.orchestra.run.smithi039.stdout: "num_blocked_ops": 0 2024-09-21T22:08:19.973 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:08:19.987 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T22:08:20.016 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:08:20.011+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:08:20.117 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:08:20.111+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:08:20.117 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:08:20.117 INFO:teuthology.orchestra.run.smithi039.stdout: "size": 20, 2024-09-21T22:08:20.117 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 600, 2024-09-21T22:08:20.117 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [ 2024-09-21T22:08:20.117 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.118 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:694 2.5 e63/37 2:b1d55633:::rbd_data.126483d75f2d.000000000000000b:head v 63'26, mlcod=63'24)", 2024-09-21T22:08:20.118 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.163199+0000", 2024-09-21T22:08:20.118 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 54.949674321000003, 2024-09-21T22:08:20.118 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.034676672999999998, 2024-09-21T22:08:20.118 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:20.118 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:20.118 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:20.118 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:20.118 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.118 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:20.118 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.163199+0000", 2024-09-21T22:08:20.118 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:20.118 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.119 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.119 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:20.119 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.163199+0000", 2024-09-21T22:08:20.119 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:20.119 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.119 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.119 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:20.119 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.163200+0000", 2024-09-21T22:08:20.119 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.703e-06 2024-09-21T22:08:20.119 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.119 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.119 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:20.119 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.164013+0000", 2024-09-21T22:08:20.119 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000812447 2024-09-21T22:08:20.120 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.120 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.120 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:20.120 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.164042+0000", 2024-09-21T22:08:20.120 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9006e-05 2024-09-21T22:08:20.120 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.120 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.120 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:20.120 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.164064+0000", 2024-09-21T22:08:20.120 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1719999999999999e-05 2024-09-21T22:08:20.120 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.120 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.120 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:20.120 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.164143+0000", 2024-09-21T22:08:20.121 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.9283999999999995e-05 2024-09-21T22:08:20.121 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.121 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.121 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:20.121 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.164165+0000", 2024-09-21T22:08:20.121 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.2379e-05 2024-09-21T22:08:20.121 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.121 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.121 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:20.121 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.197827+0000", 2024-09-21T22:08:20.121 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.033661782000000001 2024-09-21T22:08:20.121 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.121 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.121 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:20.121 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.197875+0000", 2024-09-21T22:08:20.122 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8352000000000001e-05 2024-09-21T22:08:20.122 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:20.122 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:20.122 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:20.122 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.122 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.122 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:696 2.5 e63/37 2:b1d55633:::rbd_data.126483d75f2d.000000000000000b:head v 63'28, mlcod=63'26)", 2024-09-21T22:08:20.122 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.218913+0000", 2024-09-21T22:08:20.122 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 54.893959875, 2024-09-21T22:08:20.122 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.056187599999999997, 2024-09-21T22:08:20.122 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:20.122 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:20.122 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:20.122 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:20.123 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.123 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:20.123 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.218913+0000", 2024-09-21T22:08:20.123 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:20.123 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.123 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.123 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:20.123 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.218913+0000", 2024-09-21T22:08:20.123 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:20.123 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.123 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.123 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:20.123 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.218914+0000", 2024-09-21T22:08:20.123 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.1930000000000001e-06 2024-09-21T22:08:20.124 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.124 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.124 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:20.124 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.219721+0000", 2024-09-21T22:08:20.124 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00080701500000000001 2024-09-21T22:08:20.124 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.124 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.124 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:20.124 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.219743+0000", 2024-09-21T22:08:20.124 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1364000000000001e-05 2024-09-21T22:08:20.124 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.124 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.124 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:20.124 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.219759+0000", 2024-09-21T22:08:20.125 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.611e-05 2024-09-21T22:08:20.125 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.125 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.125 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:20.125 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.219908+0000", 2024-09-21T22:08:20.125 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000149252 2024-09-21T22:08:20.125 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.125 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.125 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:20.125 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.219951+0000", 2024-09-21T22:08:20.125 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.2518999999999997e-05 2024-09-21T22:08:20.125 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.125 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.125 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:20.126 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.275031+0000", 2024-09-21T22:08:20.126 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.055080241000000002 2024-09-21T22:08:20.126 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.126 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.126 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:20.126 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.275101+0000", 2024-09-21T22:08:20.126 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.9906000000000005e-05 2024-09-21T22:08:20.126 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:20.126 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:20.126 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:20.126 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.126 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.126 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:698 2.5 e63/37 2:b1d55633:::rbd_data.126483d75f2d.000000000000000b:head v 63'30, mlcod=63'28)", 2024-09-21T22:08:20.126 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.289319+0000", 2024-09-21T22:08:20.127 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 54.823553666000002, 2024-09-21T22:08:20.127 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.053930944000000001, 2024-09-21T22:08:20.127 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:20.127 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:20.127 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:20.127 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:20.127 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.127 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:20.127 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.289319+0000", 2024-09-21T22:08:20.127 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:20.127 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.127 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.127 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:20.127 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.289319+0000", 2024-09-21T22:08:20.127 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:20.128 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.128 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.128 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:20.128 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.289321+0000", 2024-09-21T22:08:20.128 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.221e-06 2024-09-21T22:08:20.128 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.128 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.128 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:20.128 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.290091+0000", 2024-09-21T22:08:20.128 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00077069300000000003 2024-09-21T22:08:20.128 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.128 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.128 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:20.128 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.290109+0000", 2024-09-21T22:08:20.129 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7218999999999999e-05 2024-09-21T22:08:20.129 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.129 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.129 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:20.129 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.290122+0000", 2024-09-21T22:08:20.129 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3678e-05 2024-09-21T22:08:20.129 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.129 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.129 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:20.129 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.290274+0000", 2024-09-21T22:08:20.129 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00015205899999999999 2024-09-21T22:08:20.129 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.129 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.129 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:20.130 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.290313+0000", 2024-09-21T22:08:20.130 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.8636000000000001e-05 2024-09-21T22:08:20.130 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.130 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.130 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:20.130 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.343179+0000", 2024-09-21T22:08:20.130 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.052866199000000003 2024-09-21T22:08:20.130 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.130 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.130 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:20.130 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.343250+0000", 2024-09-21T22:08:20.130 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.1239e-05 2024-09-21T22:08:20.130 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:20.130 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:20.131 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:20.131 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.131 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.131 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:702 2.5 e63/37 2:a70ae99f:::rbd_data.126483d75f2d.000000000000000c:head v 63'34, mlcod=63'32)", 2024-09-21T22:08:20.131 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.392532+0000", 2024-09-21T22:08:20.131 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 54.720341077999997, 2024-09-21T22:08:20.131 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.035703990999999997, 2024-09-21T22:08:20.131 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:20.131 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:20.131 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:20.131 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:20.131 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.131 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:20.131 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.392532+0000", 2024-09-21T22:08:20.132 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:20.132 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.132 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.132 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:20.132 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.392532+0000", 2024-09-21T22:08:20.132 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:20.132 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.132 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.132 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:20.132 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.392534+0000", 2024-09-21T22:08:20.132 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.64e-06 2024-09-21T22:08:20.132 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.132 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.132 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:20.133 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.393321+0000", 2024-09-21T22:08:20.133 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00078705699999999997 2024-09-21T22:08:20.133 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.133 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.133 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:20.133 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.393347+0000", 2024-09-21T22:08:20.133 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.6735999999999999e-05 2024-09-21T22:08:20.133 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.133 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.133 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:20.133 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.393361+0000", 2024-09-21T22:08:20.133 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3346000000000001e-05 2024-09-21T22:08:20.133 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.133 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.133 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:20.134 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.393469+0000", 2024-09-21T22:08:20.134 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000108039 2024-09-21T22:08:20.134 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.134 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.134 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:20.134 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.393512+0000", 2024-09-21T22:08:20.134 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.2862999999999999e-05 2024-09-21T22:08:20.134 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.134 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.134 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:20.134 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.428170+0000", 2024-09-21T22:08:20.134 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.034658858000000001 2024-09-21T22:08:20.134 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.134 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.135 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:20.135 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.428236+0000", 2024-09-21T22:08:20.135 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.5451999999999996e-05 2024-09-21T22:08:20.135 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:20.135 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:20.135 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:20.135 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.135 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.135 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:708 2.1 e63/53 2:837ddd83:::rbd_data.126483d75f2d.000000000000000d:head v 63'6, mlcod=57'4)", 2024-09-21T22:08:20.135 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.517007+0000", 2024-09-21T22:08:20.135 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 54.595866151999999, 2024-09-21T22:08:20.135 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.032495584000000001, 2024-09-21T22:08:20.135 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:20.135 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:20.136 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:20.136 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:20.136 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.136 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:20.136 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.517007+0000", 2024-09-21T22:08:20.136 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:20.136 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.136 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.136 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:20.136 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.517007+0000", 2024-09-21T22:08:20.136 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:20.136 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.136 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.136 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:20.137 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.517008+0000", 2024-09-21T22:08:20.137 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.604e-06 2024-09-21T22:08:20.137 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.137 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.137 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:20.137 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.517603+0000", 2024-09-21T22:08:20.137 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00059461900000000001 2024-09-21T22:08:20.137 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.137 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.137 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:20.137 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.517632+0000", 2024-09-21T22:08:20.137 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8663e-05 2024-09-21T22:08:20.137 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.137 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.137 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:20.138 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.517647+0000", 2024-09-21T22:08:20.138 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4833e-05 2024-09-21T22:08:20.138 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.138 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.138 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:20.138 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.517714+0000", 2024-09-21T22:08:20.138 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.7669000000000006e-05 2024-09-21T22:08:20.138 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.138 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.138 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:20.138 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.517737+0000", 2024-09-21T22:08:20.138 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.2469000000000001e-05 2024-09-21T22:08:20.138 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.139 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.139 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:20.139 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.549370+0000", 2024-09-21T22:08:20.139 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.031633221000000003 2024-09-21T22:08:20.139 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.139 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.139 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:20.139 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.549502+0000", 2024-09-21T22:08:20.139 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013250600000000001 2024-09-21T22:08:20.139 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:20.139 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:20.139 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:20.139 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.139 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.139 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:712 2.1 e63/53 2:837ddd83:::rbd_data.126483d75f2d.000000000000000d:head v 63'10, mlcod=63'8)", 2024-09-21T22:08:20.140 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.604029+0000", 2024-09-21T22:08:20.140 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 54.508844213000003, 2024-09-21T22:08:20.140 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.037982371000000001, 2024-09-21T22:08:20.140 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:20.140 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:20.140 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:20.140 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:20.140 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.140 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:20.140 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.604029+0000", 2024-09-21T22:08:20.140 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:20.140 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.140 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.140 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:20.141 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.604029+0000", 2024-09-21T22:08:20.141 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:20.141 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.141 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.141 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:20.141 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.604031+0000", 2024-09-21T22:08:20.141 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8870000000000001e-06 2024-09-21T22:08:20.141 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.141 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.141 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:20.141 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.604395+0000", 2024-09-21T22:08:20.141 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00036422000000000001 2024-09-21T22:08:20.141 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.142 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.142 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:20.142 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.604424+0000", 2024-09-21T22:08:20.142 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8609e-05 2024-09-21T22:08:20.142 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.142 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.142 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:20.142 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.604437+0000", 2024-09-21T22:08:20.142 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3086e-05 2024-09-21T22:08:20.142 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.142 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.142 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:20.142 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.604509+0000", 2024-09-21T22:08:20.142 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.2448999999999997e-05 2024-09-21T22:08:20.143 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.143 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.143 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:20.143 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.604531+0000", 2024-09-21T22:08:20.143 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1602000000000001e-05 2024-09-21T22:08:20.143 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.143 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.143 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:20.143 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.641885+0000", 2024-09-21T22:08:20.143 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.037354545000000003 2024-09-21T22:08:20.143 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.143 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.143 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:20.143 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.642011+0000", 2024-09-21T22:08:20.144 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00012597300000000001 2024-09-21T22:08:20.144 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:20.144 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:20.144 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:20.144 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.144 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.144 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:720 2.1 e63/53 2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head v 63'18, mlcod=63'16)", 2024-09-21T22:08:20.144 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.805111+0000", 2024-09-21T22:08:20.144 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 54.307762152999999, 2024-09-21T22:08:20.144 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.046082864000000001, 2024-09-21T22:08:20.144 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:20.144 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:20.144 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:20.144 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:20.144 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.145 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:20.145 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.805111+0000", 2024-09-21T22:08:20.145 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:20.145 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.145 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.145 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:20.145 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.805111+0000", 2024-09-21T22:08:20.145 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:20.145 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.145 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.145 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:20.145 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.805112+0000", 2024-09-21T22:08:20.145 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.02e-06 2024-09-21T22:08:20.146 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.146 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.146 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:20.146 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.805429+0000", 2024-09-21T22:08:20.146 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00031740799999999998 2024-09-21T22:08:20.146 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.146 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.146 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:20.146 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.805459+0000", 2024-09-21T22:08:20.146 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.955e-05 2024-09-21T22:08:20.146 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.146 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.146 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:20.146 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.805472+0000", 2024-09-21T22:08:20.147 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3533000000000001e-05 2024-09-21T22:08:20.147 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.147 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.147 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:20.147 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.805581+0000", 2024-09-21T22:08:20.147 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000108194 2024-09-21T22:08:20.147 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.147 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.147 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:20.147 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.805615+0000", 2024-09-21T22:08:20.147 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.4310000000000002e-05 2024-09-21T22:08:20.147 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.147 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.147 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:20.148 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.851146+0000", 2024-09-21T22:08:20.148 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.045531092000000002 2024-09-21T22:08:20.148 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.148 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.148 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:20.148 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.851194+0000", 2024-09-21T22:08:20.148 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.7756999999999997e-05 2024-09-21T22:08:20.148 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:20.148 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:20.148 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:20.148 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.148 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.148 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:723 2.1 e63/53 2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head v 63'20, mlcod=63'18)", 2024-09-21T22:08:20.148 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.862341+0000", 2024-09-21T22:08:20.149 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 54.250532219, 2024-09-21T22:08:20.149 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.036454671000000001, 2024-09-21T22:08:20.149 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:20.149 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:20.149 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:20.149 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:20.149 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.149 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:20.149 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.862341+0000", 2024-09-21T22:08:20.149 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:20.149 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.149 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.149 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:20.149 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.862341+0000", 2024-09-21T22:08:20.149 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:20.150 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.150 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.150 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:20.150 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.862342+0000", 2024-09-21T22:08:20.150 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.0299999999999997e-07 2024-09-21T22:08:20.150 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.150 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.150 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:20.150 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.862721+0000", 2024-09-21T22:08:20.150 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00037889800000000002 2024-09-21T22:08:20.150 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.150 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.150 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:20.150 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.862738+0000", 2024-09-21T22:08:20.151 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6957999999999999e-05 2024-09-21T22:08:20.151 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.151 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.151 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:20.151 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.862747+0000", 2024-09-21T22:08:20.151 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.0729999999999994e-06 2024-09-21T22:08:20.151 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.151 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.151 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:20.151 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.862830+0000", 2024-09-21T22:08:20.151 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.3113000000000002e-05 2024-09-21T22:08:20.151 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.151 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.151 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:20.152 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.862860+0000", 2024-09-21T22:08:20.152 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9980000000000001e-05 2024-09-21T22:08:20.152 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.152 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.152 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:20.152 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.898705+0000", 2024-09-21T22:08:20.152 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.035845063000000003 2024-09-21T22:08:20.152 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.152 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.152 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:20.152 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.898795+0000", 2024-09-21T22:08:20.152 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.0682999999999999e-05 2024-09-21T22:08:20.152 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:20.152 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:20.152 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:20.153 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.153 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.153 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:136 2.5 e72/37 2:b1d55633:::rbd_data.126483d75f2d.000000000000000b:head v 72'44, mlcod=72'42)", 2024-09-21T22:08:20.153 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.386273+0000", 2024-09-21T22:08:20.153 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 41.726600103999999, 2024-09-21T22:08:20.153 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.033806744999999999, 2024-09-21T22:08:20.153 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:20.153 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:20.153 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:20.153 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:20.153 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.153 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:20.153 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.386273+0000", 2024-09-21T22:08:20.153 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:20.153 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.154 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.154 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:20.154 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.386273+0000", 2024-09-21T22:08:20.154 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:20.154 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.154 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.154 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:20.154 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.386274+0000", 2024-09-21T22:08:20.154 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.8299999999999996e-07 2024-09-21T22:08:20.154 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.154 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.154 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:20.154 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.387108+0000", 2024-09-21T22:08:20.154 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00083467800000000005 2024-09-21T22:08:20.154 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.155 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.155 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:20.155 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.387121+0000", 2024-09-21T22:08:20.155 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2823000000000001e-05 2024-09-21T22:08:20.155 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.155 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.155 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:20.155 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.387128+0000", 2024-09-21T22:08:20.155 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.404e-06 2024-09-21T22:08:20.155 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.155 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.155 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:20.155 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.387179+0000", 2024-09-21T22:08:20.155 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.1888000000000002e-05 2024-09-21T22:08:20.156 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.156 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.156 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:20.156 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.387201+0000", 2024-09-21T22:08:20.156 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.2019999999999999e-05 2024-09-21T22:08:20.156 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.156 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.156 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:20.156 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.419993+0000", 2024-09-21T22:08:20.156 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.032791182000000002 2024-09-21T22:08:20.156 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.156 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.156 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:20.156 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.420080+0000", 2024-09-21T22:08:20.157 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.7066999999999998e-05 2024-09-21T22:08:20.157 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:20.157 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:20.157 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:20.157 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.157 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.157 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:138 2.5 e72/37 2:b1d55633:::rbd_data.126483d75f2d.000000000000000b:head v 72'46, mlcod=72'44)", 2024-09-21T22:08:20.157 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.430104+0000", 2024-09-21T22:08:20.157 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 41.682768756000002, 2024-09-21T22:08:20.157 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.034087458000000001, 2024-09-21T22:08:20.157 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:20.157 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:20.157 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:20.157 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:20.157 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.158 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:20.158 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.430104+0000", 2024-09-21T22:08:20.158 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:20.158 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.158 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.158 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:20.158 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.430104+0000", 2024-09-21T22:08:20.158 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:20.158 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.158 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.158 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:20.158 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.430105+0000", 2024-09-21T22:08:20.158 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.1680000000000001e-06 2024-09-21T22:08:20.158 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.158 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.159 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:20.159 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.430864+0000", 2024-09-21T22:08:20.159 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00075819400000000001 2024-09-21T22:08:20.159 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.159 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.159 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:20.159 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.430881+0000", 2024-09-21T22:08:20.159 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7000999999999999e-05 2024-09-21T22:08:20.159 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.159 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.159 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:20.159 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.430889+0000", 2024-09-21T22:08:20.159 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.9540000000000004e-06 2024-09-21T22:08:20.159 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.160 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.160 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:20.160 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.430959+0000", 2024-09-21T22:08:20.160 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.0858999999999994e-05 2024-09-21T22:08:20.160 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.160 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.160 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:20.160 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.430996+0000", 2024-09-21T22:08:20.160 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.6443999999999999e-05 2024-09-21T22:08:20.160 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.160 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.160 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:20.160 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.464087+0000", 2024-09-21T22:08:20.160 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.033090799999999997 2024-09-21T22:08:20.160 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.161 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.161 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:20.161 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.464192+0000", 2024-09-21T22:08:20.161 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00010503800000000001 2024-09-21T22:08:20.161 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:20.161 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:20.161 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:20.161 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.161 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.161 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:140 2.5 e72/37 2:b1d55633:::rbd_data.126483d75f2d.000000000000000b:head v 72'48, mlcod=72'46)", 2024-09-21T22:08:20.161 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.475737+0000", 2024-09-21T22:08:20.161 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 41.637135807999996, 2024-09-21T22:08:20.161 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.037632025, 2024-09-21T22:08:20.161 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:20.162 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:20.162 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:20.162 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:20.162 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.162 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:20.162 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.475737+0000", 2024-09-21T22:08:20.162 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:20.162 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.162 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.162 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:20.162 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.475737+0000", 2024-09-21T22:08:20.162 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:20.162 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.162 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.163 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:20.163 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.475739+0000", 2024-09-21T22:08:20.163 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.666e-06 2024-09-21T22:08:20.163 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.163 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.163 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:20.163 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.476538+0000", 2024-09-21T22:08:20.163 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00079927399999999999 2024-09-21T22:08:20.163 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.163 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.163 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:20.163 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.476556+0000", 2024-09-21T22:08:20.164 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7666999999999999e-05 2024-09-21T22:08:20.164 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.164 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.165 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:20.165 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.476565+0000", 2024-09-21T22:08:20.166 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.0650000000000005e-06 2024-09-21T22:08:20.166 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.166 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.166 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:20.166 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.476651+0000", 2024-09-21T22:08:20.166 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.5916999999999997e-05 2024-09-21T22:08:20.166 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.166 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.166 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:20.166 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.476696+0000", 2024-09-21T22:08:20.166 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.5537000000000001e-05 2024-09-21T22:08:20.166 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.166 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.167 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:20.167 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.513300+0000", 2024-09-21T22:08:20.167 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.036603498999999998 2024-09-21T22:08:20.167 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.167 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.167 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:20.167 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.513369+0000", 2024-09-21T22:08:20.167 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.9400000000000006e-05 2024-09-21T22:08:20.167 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:20.167 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:20.167 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:20.167 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.167 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.167 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:145 2.5 e72/37 2:a70ae99f:::rbd_data.126483d75f2d.000000000000000c:head v 72'52, mlcod=72'50)", 2024-09-21T22:08:20.168 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.578171+0000", 2024-09-21T22:08:20.168 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 41.534701957999999, 2024-09-21T22:08:20.168 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.037181537000000001, 2024-09-21T22:08:20.168 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:20.168 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:20.168 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:20.168 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:20.168 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.168 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:20.168 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.578171+0000", 2024-09-21T22:08:20.168 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:20.168 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.168 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.168 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:20.168 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.578171+0000", 2024-09-21T22:08:20.169 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:20.169 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.169 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.169 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:20.169 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.578172+0000", 2024-09-21T22:08:20.169 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.017e-06 2024-09-21T22:08:20.169 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.169 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.169 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:20.169 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.579000+0000", 2024-09-21T22:08:20.169 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00082799499999999999 2024-09-21T22:08:20.169 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.169 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.169 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:20.170 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.579020+0000", 2024-09-21T22:08:20.170 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.0165000000000001e-05 2024-09-21T22:08:20.170 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.170 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.170 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:20.170 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.579036+0000", 2024-09-21T22:08:20.170 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5767e-05 2024-09-21T22:08:20.170 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.170 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.170 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:20.170 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.579106+0000", 2024-09-21T22:08:20.170 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.0091000000000004e-05 2024-09-21T22:08:20.170 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.171 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.171 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:20.171 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.579131+0000", 2024-09-21T22:08:20.171 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4978e-05 2024-09-21T22:08:20.171 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.171 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.171 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:20.171 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.615249+0000", 2024-09-21T22:08:20.171 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.036118355999999997 2024-09-21T22:08:20.171 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.171 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.171 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:20.171 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.615353+0000", 2024-09-21T22:08:20.171 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000103168 2024-09-21T22:08:20.171 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:20.172 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:20.172 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:20.172 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.172 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.172 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:147 2.5 e72/37 2:a70ae99f:::rbd_data.126483d75f2d.000000000000000c:head v 72'54, mlcod=72'52)", 2024-09-21T22:08:20.172 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.627203+0000", 2024-09-21T22:08:20.172 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 41.485669846, 2024-09-21T22:08:20.172 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.038203972000000003, 2024-09-21T22:08:20.172 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:20.172 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:20.172 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:20.172 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:20.172 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.172 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:20.172 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.627203+0000", 2024-09-21T22:08:20.173 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:20.173 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.173 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.173 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:20.173 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.627203+0000", 2024-09-21T22:08:20.173 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:20.173 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.173 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.173 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:20.173 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.627204+0000", 2024-09-21T22:08:20.173 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3069999999999999e-06 2024-09-21T22:08:20.173 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.173 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.173 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:20.173 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.628114+0000", 2024-09-21T22:08:20.174 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00090942099999999999 2024-09-21T22:08:20.174 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.174 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.174 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:20.174 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.628140+0000", 2024-09-21T22:08:20.174 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5882000000000001e-05 2024-09-21T22:08:20.174 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.174 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.174 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:20.174 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.628162+0000", 2024-09-21T22:08:20.174 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.2293000000000002e-05 2024-09-21T22:08:20.174 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.174 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.175 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:20.175 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.628250+0000", 2024-09-21T22:08:20.175 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.7973000000000006e-05 2024-09-21T22:08:20.175 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.175 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.175 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:20.175 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.628272+0000", 2024-09-21T22:08:20.175 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1682999999999999e-05 2024-09-21T22:08:20.175 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.175 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.175 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:20.175 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.665322+0000", 2024-09-21T22:08:20.175 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.037050650999999997 2024-09-21T22:08:20.175 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.175 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.176 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:20.176 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.665407+0000", 2024-09-21T22:08:20.176 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.4761999999999999e-05 2024-09-21T22:08:20.176 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:20.176 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:20.176 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:20.176 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.176 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.176 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:149 2.5 e72/37 2:a70ae99f:::rbd_data.126483d75f2d.000000000000000c:head v 72'56, mlcod=72'54)", 2024-09-21T22:08:20.176 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.690377+0000", 2024-09-21T22:08:20.176 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 41.422496103, 2024-09-21T22:08:20.176 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.049254799000000002, 2024-09-21T22:08:20.176 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:20.176 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:20.177 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:20.177 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:20.177 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.177 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:20.177 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.690377+0000", 2024-09-21T22:08:20.177 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:20.177 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.177 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.177 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:20.177 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.690377+0000", 2024-09-21T22:08:20.177 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:20.177 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.177 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.177 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:20.177 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.690378+0000", 2024-09-21T22:08:20.178 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.344e-06 2024-09-21T22:08:20.178 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.178 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.178 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:20.178 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.691302+0000", 2024-09-21T22:08:20.178 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00092322400000000005 2024-09-21T22:08:20.178 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.178 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.178 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:20.178 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.691321+0000", 2024-09-21T22:08:20.178 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9928999999999998e-05 2024-09-21T22:08:20.178 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.178 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.178 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:20.179 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.691345+0000", 2024-09-21T22:08:20.179 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.372e-05 2024-09-21T22:08:20.179 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.179 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.179 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:20.179 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.691442+0000", 2024-09-21T22:08:20.179 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.7270999999999998e-05 2024-09-21T22:08:20.179 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.179 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.179 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:20.179 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.691512+0000", 2024-09-21T22:08:20.179 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.9183000000000005e-05 2024-09-21T22:08:20.179 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.179 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.180 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:20.180 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.739545+0000", 2024-09-21T22:08:20.180 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.048033657 2024-09-21T22:08:20.180 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.180 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.180 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:20.180 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.739632+0000", 2024-09-21T22:08:20.180 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.6471000000000006e-05 2024-09-21T22:08:20.180 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:20.180 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:20.180 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:20.180 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.180 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.180 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:156 2.1 e72/53 2:837ddd83:::rbd_data.126483d75f2d.000000000000000d:head v 72'28, mlcod=72'26)", 2024-09-21T22:08:20.180 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.824856+0000", 2024-09-21T22:08:20.181 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 41.288016665000001, 2024-09-21T22:08:20.181 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.034320985999999998, 2024-09-21T22:08:20.181 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:20.181 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:20.181 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:20.181 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:20.181 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.181 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:20.181 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.824856+0000", 2024-09-21T22:08:20.181 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:20.181 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.181 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.181 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:20.181 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.824856+0000", 2024-09-21T22:08:20.182 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:20.182 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.182 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.182 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:20.182 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.824858+0000", 2024-09-21T22:08:20.182 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2979999999999999e-06 2024-09-21T22:08:20.182 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.182 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.182 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:20.182 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.825174+0000", 2024-09-21T22:08:20.182 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000316335 2024-09-21T22:08:20.182 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.182 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.182 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:20.183 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.825193+0000", 2024-09-21T22:08:20.183 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9436000000000002e-05 2024-09-21T22:08:20.183 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.183 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.183 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:20.183 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.825202+0000", 2024-09-21T22:08:20.183 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.8359999999999998e-06 2024-09-21T22:08:20.183 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.183 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.183 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:20.183 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.825286+0000", 2024-09-21T22:08:20.183 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.3683999999999994e-05 2024-09-21T22:08:20.183 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.183 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.183 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:20.184 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.825314+0000", 2024-09-21T22:08:20.184 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.7653000000000001e-05 2024-09-21T22:08:20.184 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.184 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.184 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:20.184 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.859122+0000", 2024-09-21T22:08:20.184 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.033808624000000002 2024-09-21T22:08:20.184 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.184 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.184 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:20.184 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.859177+0000", 2024-09-21T22:08:20.184 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.5120000000000001e-05 2024-09-21T22:08:20.184 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:20.184 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:20.185 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:20.185 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.185 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.185 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:158 2.1 e72/53 2:837ddd83:::rbd_data.126483d75f2d.000000000000000d:head v 72'30, mlcod=72'28)", 2024-09-21T22:08:20.185 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.869277+0000", 2024-09-21T22:08:20.185 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 41.243596468, 2024-09-21T22:08:20.185 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.040508390999999998, 2024-09-21T22:08:20.185 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:20.185 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:20.185 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:20.185 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:20.185 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.185 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:20.185 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.869277+0000", 2024-09-21T22:08:20.185 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:20.186 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.186 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.186 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:20.186 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.869277+0000", 2024-09-21T22:08:20.186 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:20.186 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.186 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.186 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:20.186 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.869278+0000", 2024-09-21T22:08:20.186 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5269999999999999e-06 2024-09-21T22:08:20.186 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.186 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.186 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:20.186 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.869902+0000", 2024-09-21T22:08:20.187 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00062399000000000005 2024-09-21T22:08:20.187 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.187 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.187 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:20.187 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.869924+0000", 2024-09-21T22:08:20.187 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1875999999999999e-05 2024-09-21T22:08:20.187 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.187 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.187 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:20.187 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.869937+0000", 2024-09-21T22:08:20.187 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2685e-05 2024-09-21T22:08:20.187 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.187 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.187 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:20.187 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.870058+0000", 2024-09-21T22:08:20.188 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000120862 2024-09-21T22:08:20.188 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.188 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.188 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:20.188 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.870093+0000", 2024-09-21T22:08:20.188 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.5896000000000003e-05 2024-09-21T22:08:20.188 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.188 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.188 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:20.188 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.909728+0000", 2024-09-21T22:08:20.188 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.039634461000000003 2024-09-21T22:08:20.188 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.188 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.188 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:20.189 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.909785+0000", 2024-09-21T22:08:20.189 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.7093999999999999e-05 2024-09-21T22:08:20.189 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:20.189 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:20.189 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:20.189 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.189 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.189 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:161 2.1 e72/53 2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head v 72'32, mlcod=72'30)", 2024-09-21T22:08:20.189 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.924455+0000", 2024-09-21T22:08:20.189 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 41.188418106, 2024-09-21T22:08:20.189 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.038366863000000001, 2024-09-21T22:08:20.189 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:20.189 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:20.189 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:20.190 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:20.190 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.190 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:20.190 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.924455+0000", 2024-09-21T22:08:20.190 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:20.190 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.190 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.190 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:20.190 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.924455+0000", 2024-09-21T22:08:20.190 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:20.190 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.190 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.190 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:20.190 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.924456+0000", 2024-09-21T22:08:20.191 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.003e-06 2024-09-21T22:08:20.191 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.191 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.191 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:20.191 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.924811+0000", 2024-09-21T22:08:20.191 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00035519499999999999 2024-09-21T22:08:20.191 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.191 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.191 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:20.191 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.924826+0000", 2024-09-21T22:08:20.191 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5296999999999999e-05 2024-09-21T22:08:20.191 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.192 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.192 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:20.192 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.924834+0000", 2024-09-21T22:08:20.192 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.5379999999999997e-06 2024-09-21T22:08:20.192 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.192 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.192 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:20.192 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.924910+0000", 2024-09-21T22:08:20.192 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.6488000000000004e-05 2024-09-21T22:08:20.192 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.192 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.192 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:20.192 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.924934+0000", 2024-09-21T22:08:20.192 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3598999999999999e-05 2024-09-21T22:08:20.192 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.193 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.193 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:20.193 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.962739+0000", 2024-09-21T22:08:20.193 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.037805370999999997 2024-09-21T22:08:20.193 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.193 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.193 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:20.193 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.962822+0000", 2024-09-21T22:08:20.193 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.2372000000000003e-05 2024-09-21T22:08:20.193 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:20.193 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:20.193 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:20.193 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.193 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.194 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:163 2.1 e72/53 2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head v 72'34, mlcod=72'32)", 2024-09-21T22:08:20.194 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.971683+0000", 2024-09-21T22:08:20.194 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 41.141189566000001, 2024-09-21T22:08:20.194 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.043171320999999999, 2024-09-21T22:08:20.194 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:20.194 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:20.194 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:20.194 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:20.194 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.194 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:20.194 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.971683+0000", 2024-09-21T22:08:20.194 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:20.194 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.194 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.194 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:20.195 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.971683+0000", 2024-09-21T22:08:20.195 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:20.195 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.195 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.195 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:20.195 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.971684+0000", 2024-09-21T22:08:20.195 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.6600000000000005e-07 2024-09-21T22:08:20.195 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.195 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.195 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:20.195 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.971945+0000", 2024-09-21T22:08:20.195 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00026101899999999998 2024-09-21T22:08:20.195 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.195 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.196 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:20.196 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.971959+0000", 2024-09-21T22:08:20.196 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4054e-05 2024-09-21T22:08:20.196 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.196 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.196 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:20.196 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.971969+0000", 2024-09-21T22:08:20.196 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.499e-06 2024-09-21T22:08:20.196 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.196 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.196 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:20.196 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.972037+0000", 2024-09-21T22:08:20.196 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.8448000000000006e-05 2024-09-21T22:08:20.196 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.196 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.197 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:20.197 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.972057+0000", 2024-09-21T22:08:20.197 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9817999999999999e-05 2024-09-21T22:08:20.197 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.197 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.197 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:20.197 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.014770+0000", 2024-09-21T22:08:20.197 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.042712464999999998 2024-09-21T22:08:20.197 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.197 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.197 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:20.197 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.014855+0000", 2024-09-21T22:08:20.197 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.5152000000000001e-05 2024-09-21T22:08:20.197 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:20.197 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:20.198 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:20.198 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.198 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.198 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:165 2.1 e72/53 2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head v 72'36, mlcod=72'34)", 2024-09-21T22:08:20.198 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:39.028631+0000", 2024-09-21T22:08:20.198 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 41.084242089, 2024-09-21T22:08:20.198 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.033430561999999997, 2024-09-21T22:08:20.198 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:20.198 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:20.198 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:20.198 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:20.198 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.198 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:20.198 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.028631+0000", 2024-09-21T22:08:20.199 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:20.199 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.199 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.199 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:20.199 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.028631+0000", 2024-09-21T22:08:20.199 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:20.199 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.199 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.199 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:20.199 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.028633+0000", 2024-09-21T22:08:20.199 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5799999999999999e-06 2024-09-21T22:08:20.199 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.199 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.199 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:20.199 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.029209+0000", 2024-09-21T22:08:20.200 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00057670799999999999 2024-09-21T22:08:20.200 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.200 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.200 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:20.200 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.029238+0000", 2024-09-21T22:08:20.200 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8943000000000001e-05 2024-09-21T22:08:20.200 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.200 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.200 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:20.200 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.029252+0000", 2024-09-21T22:08:20.200 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3672999999999999e-05 2024-09-21T22:08:20.200 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.200 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.200 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:20.201 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.029328+0000", 2024-09-21T22:08:20.201 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.6497999999999998e-05 2024-09-21T22:08:20.201 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.201 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.201 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:20.201 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.029353+0000", 2024-09-21T22:08:20.201 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5060999999999999e-05 2024-09-21T22:08:20.201 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.201 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.201 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:20.201 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.061993+0000", 2024-09-21T22:08:20.201 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.032639286000000003 2024-09-21T22:08:20.201 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.201 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.201 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:20.202 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.062062+0000", 2024-09-21T22:08:20.202 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.8813000000000007e-05 2024-09-21T22:08:20.202 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:20.202 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:20.202 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:20.202 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.202 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.202 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:167 2.1 e72/53 2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head v 72'38, mlcod=72'36)", 2024-09-21T22:08:20.202 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:39.077324+0000", 2024-09-21T22:08:20.202 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 41.035549275000001, 2024-09-21T22:08:20.202 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.036031961000000001, 2024-09-21T22:08:20.202 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:20.202 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:20.202 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:20.203 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:20.203 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.203 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:20.203 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.077324+0000", 2024-09-21T22:08:20.203 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:20.203 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.203 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.203 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:20.203 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.077324+0000", 2024-09-21T22:08:20.203 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:20.203 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.203 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.203 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:20.203 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.077326+0000", 2024-09-21T22:08:20.203 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.75e-06 2024-09-21T22:08:20.204 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.204 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.204 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:20.204 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.077719+0000", 2024-09-21T22:08:20.204 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00039322599999999999 2024-09-21T22:08:20.204 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.204 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.204 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:20.204 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.077749+0000", 2024-09-21T22:08:20.204 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9895000000000001e-05 2024-09-21T22:08:20.204 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.204 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.204 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:20.204 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.077763+0000", 2024-09-21T22:08:20.204 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4517e-05 2024-09-21T22:08:20.205 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.205 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.205 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:20.205 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.077843+0000", 2024-09-21T22:08:20.205 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.9994999999999996e-05 2024-09-21T22:08:20.205 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.205 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.205 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:20.205 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.077867+0000", 2024-09-21T22:08:20.205 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3640000000000001e-05 2024-09-21T22:08:20.205 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.205 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.205 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:20.205 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.113216+0000", 2024-09-21T22:08:20.205 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.035349168 2024-09-21T22:08:20.206 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.206 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.206 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:20.206 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.113356+0000", 2024-09-21T22:08:20.206 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013977 2024-09-21T22:08:20.206 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:20.206 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:20.206 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:20.206 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:20.206 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:20.206 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:08:20.207 INFO:tasks.workunit.client.0.smithi157.stderr:++ rbd journal inspect --pool rbd --image testrbdnbd21553 2024-09-21T22:08:20.207 INFO:tasks.workunit.client.0.smithi157.stderr:++ awk '/entries inspected/ {print $1}' 2024-09-21T22:08:20.208 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T22:08:20.217 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:08:20.211+0000 7f2075a06640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31720) UID: 0 2024-09-21T22:08:20.229 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-21T22:08:20.224+0000 7ff0d6774640 -1 JournalMetadata: 0x55f8fc9e0730 failed to locate client: INSPECT 2024-09-21T22:08:20.234 INFO:tasks.workunit.client.0.smithi157.stderr:+ queue_length=4 2024-09-21T22:08:20.234 INFO:tasks.workunit.client.0.smithi157.stderr:+ tries=4 2024-09-21T22:08:20.235 INFO:tasks.workunit.client.0.smithi157.stderr:+ '[' 4 -lt 10 ']' 2024-09-21T22:08:20.235 INFO:tasks.workunit.client.0.smithi157.stderr:+ '[' 4 -gt 0 ']' 2024-09-21T22:08:20.235 INFO:tasks.workunit.client.0.smithi157.stderr:+ rbd journal inspect --pool rbd --image testrbdnbd21553 --verbose 2024-09-21T22:08:20.316 INFO:tasks.workunit.client.0.smithi157.stdout:Entry: tag_id=1, commit_tid=1 2024-09-21T22:08:20.316 INFO:tasks.workunit.client.0.smithi157.stdout:{ 2024-09-21T22:08:20.316 INFO:tasks.workunit.client.0.smithi157.stdout: "event_type": "AioDiscard", 2024-09-21T22:08:20.316 INFO:tasks.workunit.client.0.smithi157.stdout: "offset": 0, 2024-09-21T22:08:20.316 INFO:tasks.workunit.client.0.smithi157.stdout: "length": 4096, 2024-09-21T22:08:20.316 INFO:tasks.workunit.client.0.smithi157.stdout: "discard_granularity_bytes": 4096, 2024-09-21T22:08:20.316 INFO:tasks.workunit.client.0.smithi157.stdout: "timestamp": "2024-09-21T22:08:15.344115+0000" 2024-09-21T22:08:20.316 INFO:tasks.workunit.client.0.smithi157.stdout:} 2024-09-21T22:08:20.316 INFO:tasks.workunit.client.0.smithi157.stdout:Entry: tag_id=1, commit_tid=2 2024-09-21T22:08:20.316 INFO:tasks.workunit.client.0.smithi157.stdout:{ 2024-09-21T22:08:20.317 INFO:tasks.workunit.client.0.smithi157.stdout: "event_type": "AioDiscard", 2024-09-21T22:08:20.317 INFO:tasks.workunit.client.0.smithi157.stdout: "offset": 0, 2024-09-21T22:08:20.317 INFO:tasks.workunit.client.0.smithi157.stdout: "length": 4096, 2024-09-21T22:08:20.317 INFO:tasks.workunit.client.0.smithi157.stdout: "discard_granularity_bytes": 4096, 2024-09-21T22:08:20.317 INFO:tasks.workunit.client.0.smithi157.stdout: "timestamp": "2024-09-21T22:08:15.360522+0000" 2024-09-21T22:08:20.317 INFO:tasks.workunit.client.0.smithi157.stdout:} 2024-09-21T22:08:20.317 INFO:tasks.workunit.client.0.smithi157.stdout:Entry: tag_id=1, commit_tid=3 2024-09-21T22:08:20.317 INFO:tasks.workunit.client.0.smithi157.stdout:{ 2024-09-21T22:08:20.317 INFO:tasks.workunit.client.0.smithi157.stdout: "event_type": "AioDiscard", 2024-09-21T22:08:20.317 INFO:tasks.workunit.client.0.smithi157.stdout: "offset": 4096, 2024-09-21T22:08:20.317 INFO:tasks.workunit.client.0.smithi157.stdout: "length": 4096, 2024-09-21T22:08:20.317 INFO:tasks.workunit.client.0.smithi157.stdout: "discard_granularity_bytes": 4096, 2024-09-21T22:08:20.317 INFO:tasks.workunit.client.0.smithi157.stdout: "timestamp": "2024-09-21T22:08:15.387911+0000" 2024-09-21T22:08:20.317 INFO:tasks.workunit.client.0.smithi157.stdout:} 2024-09-21T22:08:20.317 INFO:tasks.workunit.client.0.smithi157.stdout:Entry: tag_id=1, commit_tid=4 2024-09-21T22:08:20.318 INFO:tasks.workunit.client.0.smithi157.stdout:{ 2024-09-21T22:08:20.318 INFO:tasks.workunit.client.0.smithi157.stdout: "event_type": "AioDiscard", 2024-09-21T22:08:20.318 INFO:tasks.workunit.client.0.smithi157.stdout: "offset": 4096, 2024-09-21T22:08:20.318 INFO:tasks.workunit.client.0.smithi157.stdout: "length": 4096, 2024-09-21T22:08:20.319 INFO:tasks.workunit.client.0.smithi157.stdout: "discard_granularity_bytes": 4096, 2024-09-21T22:08:20.319 INFO:tasks.workunit.client.0.smithi157.stdout: "timestamp": "2024-09-21T22:08:15.404762+0000" 2024-09-21T22:08:20.319 INFO:tasks.workunit.client.0.smithi157.stdout:} 2024-09-21T22:08:20.319 INFO:tasks.workunit.client.0.smithi157.stdout:Summary: 2024-09-21T22:08:20.319 INFO:tasks.workunit.client.0.smithi157.stdout: 4 entries inspected, 0 errors 2024-09-21T22:08:20.319 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:08:20.315+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:08:20.323 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-21T22:08:20.320+0000 7fc955803640 -1 JournalMetadata: 0x556b0efd9d30 failed to locate client: INSPECT 2024-09-21T22:08:20.330 INFO:tasks.workunit.client.0.smithi157.stderr:+ sleep 1 2024-09-21T22:08:20.340 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:08:20.340 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:08:20.340 INFO:teuthology.orchestra.run.smithi039.stdout: "num_ops": 0 2024-09-21T22:08:20.341 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:08:20.355 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T22:08:20.420 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:08:20.415+0000 7f2075a06640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31720) UID: 0 2024-09-21T22:08:20.480 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:08:20.480 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:08:20.480 INFO:teuthology.orchestra.run.smithi039.stdout: "complaint_time": 30, 2024-09-21T22:08:20.480 INFO:teuthology.orchestra.run.smithi039.stdout: "num_blocked_ops": 0 2024-09-21T22:08:20.480 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:08:20.495 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T22:08:20.506 INFO:teuthology.orchestra.run.smithi039.stderr:noscrub is unset 2024-09-21T22:08:20.524 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:08:20.519+0000 7f2075a06640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31720) UID: 0 2024-09-21T22:08:20.623 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:08:20.617+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:08:20.635 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:08:20.635 INFO:teuthology.orchestra.run.smithi039.stdout: "size": 20, 2024-09-21T22:08:20.635 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 600, 2024-09-21T22:08:20.635 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [ 2024-09-21T22:08:20.635 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.636 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.3 55/54 [PushOp(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head, version: 52'2, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head@52'2, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T22:08:20.636 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.084379+0000", 2024-09-21T22:08:20.636 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 68.548031743999999, 2024-09-21T22:08:20.636 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.051020674000000002, 2024-09-21T22:08:20.636 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:20.636 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:20.636 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:08:20.636 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:20.636 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.636 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:20.636 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.084379+0000", 2024-09-21T22:08:20.636 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:20.636 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.636 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.637 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:20.637 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.084379+0000", 2024-09-21T22:08:20.637 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:20.637 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.637 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.637 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:20.637 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.084381+0000", 2024-09-21T22:08:20.637 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.73e-06 2024-09-21T22:08:20.637 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.637 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.637 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:20.637 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.085494+0000", 2024-09-21T22:08:20.637 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011131559999999999 2024-09-21T22:08:20.637 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.638 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.638 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:20.638 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.085542+0000", 2024-09-21T22:08:20.638 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8334999999999997e-05 2024-09-21T22:08:20.638 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.638 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.638 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:20.638 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.085591+0000", 2024-09-21T22:08:20.638 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.9165000000000001e-05 2024-09-21T22:08:20.638 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.638 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.638 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:20.638 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.085777+0000", 2024-09-21T22:08:20.638 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000186152 2024-09-21T22:08:20.639 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.639 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.639 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:20.639 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.085796+0000", 2024-09-21T22:08:20.639 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8612999999999999e-05 2024-09-21T22:08:20.639 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.639 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.639 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:20.639 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.135399+0000", 2024-09-21T22:08:20.639 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.049603522999999997 2024-09-21T22:08:20.639 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:20.639 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:20.639 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:20.639 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.640 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.640 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.3 55/54 [PushOp(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head, version: 52'2, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head@52'2, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:08:20.640 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.139554+0000", 2024-09-21T22:08:20.640 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 68.492856641000003, 2024-09-21T22:08:20.640 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.056215389999999997, 2024-09-21T22:08:20.640 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:20.640 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:20.640 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:08:20.640 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:20.640 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.640 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:20.641 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.139554+0000", 2024-09-21T22:08:20.641 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:20.641 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.641 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.641 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:20.641 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.139554+0000", 2024-09-21T22:08:20.641 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:20.641 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.641 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.641 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:20.641 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.139556+0000", 2024-09-21T22:08:20.641 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7999999999999999e-06 2024-09-21T22:08:20.641 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.641 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.642 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:20.642 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.140671+0000", 2024-09-21T22:08:20.642 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001115612 2024-09-21T22:08:20.642 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.642 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.642 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:20.642 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.140720+0000", 2024-09-21T22:08:20.642 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8631999999999998e-05 2024-09-21T22:08:20.642 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.642 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.642 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:20.642 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.140752+0000", 2024-09-21T22:08:20.642 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.2199000000000001e-05 2024-09-21T22:08:20.642 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.643 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.643 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:20.643 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.140900+0000", 2024-09-21T22:08:20.643 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00014786000000000001 2024-09-21T22:08:20.643 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.643 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.643 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:20.643 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.140917+0000", 2024-09-21T22:08:20.643 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6753999999999999e-05 2024-09-21T22:08:20.643 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.643 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.643 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:20.643 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.195769+0000", 2024-09-21T22:08:20.643 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.054852533000000002 2024-09-21T22:08:20.643 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:20.644 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:20.644 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:20.644 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.644 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.644 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.3 55/54 [PushOp(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head, version: 52'2, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head@52'2, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:08:20.644 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.199187+0000", 2024-09-21T22:08:20.644 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 68.433223096999996, 2024-09-21T22:08:20.644 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.051543975999999998, 2024-09-21T22:08:20.644 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:20.644 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:20.644 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:08:20.644 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:20.644 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.644 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:20.645 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.199187+0000", 2024-09-21T22:08:20.645 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:20.645 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.645 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.645 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:20.645 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.199187+0000", 2024-09-21T22:08:20.645 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:20.645 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.645 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.645 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:20.645 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.199189+0000", 2024-09-21T22:08:20.645 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7969999999999999e-06 2024-09-21T22:08:20.645 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.645 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.645 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:20.646 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.200355+0000", 2024-09-21T22:08:20.646 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011660139999999999 2024-09-21T22:08:20.646 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.646 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.646 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:20.646 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.200405+0000", 2024-09-21T22:08:20.646 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.0124000000000003e-05 2024-09-21T22:08:20.646 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.646 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.646 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:20.646 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.200457+0000", 2024-09-21T22:08:20.646 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.1177999999999997e-05 2024-09-21T22:08:20.646 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.647 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.647 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:20.647 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.200635+0000", 2024-09-21T22:08:20.647 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00017841299999999999 2024-09-21T22:08:20.647 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.647 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.647 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:20.647 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.200653+0000", 2024-09-21T22:08:20.647 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7944999999999998e-05 2024-09-21T22:08:20.647 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.647 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.647 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:20.647 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.250731+0000", 2024-09-21T22:08:20.647 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.050078505000000002 2024-09-21T22:08:20.647 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:20.648 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:20.648 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:20.648 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.648 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.648 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.3 55/54 [PushOp(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head, version: 52'3, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head@52'3, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:08:20.648 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.343754+0000", 2024-09-21T22:08:20.648 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 68.288656386, 2024-09-21T22:08:20.648 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.063560607000000005, 2024-09-21T22:08:20.648 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:20.648 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:20.648 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:08:20.648 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:20.648 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.648 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:20.649 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.343754+0000", 2024-09-21T22:08:20.649 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:20.649 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.649 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.649 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:20.649 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.343754+0000", 2024-09-21T22:08:20.649 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:20.649 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.649 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.649 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:20.649 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.343756+0000", 2024-09-21T22:08:20.649 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7069999999999999e-06 2024-09-21T22:08:20.649 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.649 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.649 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:20.650 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.344864+0000", 2024-09-21T22:08:20.650 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011085159999999999 2024-09-21T22:08:20.650 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.650 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.650 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:20.650 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.344914+0000", 2024-09-21T22:08:20.650 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.9462000000000002e-05 2024-09-21T22:08:20.650 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.650 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.650 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:20.650 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.344961+0000", 2024-09-21T22:08:20.650 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.7633999999999998e-05 2024-09-21T22:08:20.650 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.650 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.651 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:20.651 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.345171+0000", 2024-09-21T22:08:20.651 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000209652 2024-09-21T22:08:20.651 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.651 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.651 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:20.651 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.345193+0000", 2024-09-21T22:08:20.651 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1985999999999999e-05 2024-09-21T22:08:20.653 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.653 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.653 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:20.653 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.407315+0000", 2024-09-21T22:08:20.653 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.06212165 2024-09-21T22:08:20.653 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:20.653 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:20.653 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:20.653 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.654 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.654 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.3 55/54 [PushOp(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head, version: 52'3, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head@52'3, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:08:20.654 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.410551+0000", 2024-09-21T22:08:20.654 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 68.221859254999998, 2024-09-21T22:08:20.654 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.054517522999999998, 2024-09-21T22:08:20.654 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:20.654 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:20.654 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:08:20.654 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:20.654 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.654 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:20.654 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.410551+0000", 2024-09-21T22:08:20.654 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:20.655 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.655 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.655 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:20.655 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.410551+0000", 2024-09-21T22:08:20.655 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:20.655 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.655 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.655 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:20.655 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.410553+0000", 2024-09-21T22:08:20.655 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5200000000000001e-06 2024-09-21T22:08:20.655 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.655 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.655 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:20.655 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.411747+0000", 2024-09-21T22:08:20.655 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011946089999999999 2024-09-21T22:08:20.656 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.656 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.656 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:20.656 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.411796+0000", 2024-09-21T22:08:20.656 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8717999999999997e-05 2024-09-21T22:08:20.656 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.656 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.656 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:20.656 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.411844+0000", 2024-09-21T22:08:20.656 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8291999999999998e-05 2024-09-21T22:08:20.656 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.656 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.656 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:20.656 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.412066+0000", 2024-09-21T22:08:20.657 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00022115899999999999 2024-09-21T22:08:20.657 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.657 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.657 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:20.657 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.412081+0000", 2024-09-21T22:08:20.657 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5654e-05 2024-09-21T22:08:20.657 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.657 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.657 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:20.657 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.465069+0000", 2024-09-21T22:08:20.657 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.052987570999999997 2024-09-21T22:08:20.657 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:20.657 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:20.657 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:20.657 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.658 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.658 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.3 55/54 [PushOp(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head, version: 52'3, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head@52'3, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 4194304, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:08:20.658 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.468339+0000", 2024-09-21T22:08:20.658 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 68.164071590999995, 2024-09-21T22:08:20.658 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.064010024999999998, 2024-09-21T22:08:20.658 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:20.658 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:20.658 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:08:20.658 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:20.658 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.658 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:20.658 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.468339+0000", 2024-09-21T22:08:20.658 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:20.658 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.659 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.659 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:20.659 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.468339+0000", 2024-09-21T22:08:20.659 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:20.659 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.659 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.659 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:20.659 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.468340+0000", 2024-09-21T22:08:20.659 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4330000000000001e-06 2024-09-21T22:08:20.659 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.659 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.659 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:20.659 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.469484+0000", 2024-09-21T22:08:20.659 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001143535 2024-09-21T22:08:20.660 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.660 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.660 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:20.660 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.469533+0000", 2024-09-21T22:08:20.660 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8980999999999997e-05 2024-09-21T22:08:20.660 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.660 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.660 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:20.660 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.469581+0000", 2024-09-21T22:08:20.660 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8551999999999999e-05 2024-09-21T22:08:20.660 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.660 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.660 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:20.660 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.469804+0000", 2024-09-21T22:08:20.660 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00022287699999999999 2024-09-21T22:08:20.661 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.661 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.661 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:20.661 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.469829+0000", 2024-09-21T22:08:20.661 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4766e-05 2024-09-21T22:08:20.661 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.661 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.661 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:20.661 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.532349+0000", 2024-09-21T22:08:20.661 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.062519880999999999 2024-09-21T22:08:20.661 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:20.661 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:20.661 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:20.661 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.661 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.662 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4840.0:668 2.3 2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:08:20.662 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.409114+0000", 2024-09-21T22:08:20.662 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 56.223296384999998, 2024-09-21T22:08:20.662 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.048703758999999999, 2024-09-21T22:08:20.662 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:20.662 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:20.662 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:20.662 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-21T22:08:20.662 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4840", 2024-09-21T22:08:20.662 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:08:20.662 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 668 2024-09-21T22:08:20.662 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.662 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:20.662 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.663 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:20.663 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.409114+0000", 2024-09-21T22:08:20.663 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:20.663 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.663 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.663 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:20.663 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.409114+0000", 2024-09-21T22:08:20.663 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:20.663 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.663 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.663 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:20.663 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.409115+0000", 2024-09-21T22:08:20.663 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.114e-06 2024-09-21T22:08:20.663 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.664 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.664 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:20.664 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.410308+0000", 2024-09-21T22:08:20.664 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011930949999999999 2024-09-21T22:08:20.664 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.664 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.664 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:20.664 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.410325+0000", 2024-09-21T22:08:20.664 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7088e-05 2024-09-21T22:08:20.664 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.664 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.664 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:20.664 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.410334+0000", 2024-09-21T22:08:20.664 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.2549999999999997e-06 2024-09-21T22:08:20.664 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.665 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.665 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:20.665 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.410393+0000", 2024-09-21T22:08:20.665 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.9327999999999998e-05 2024-09-21T22:08:20.665 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.665 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.665 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:20.665 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.410482+0000", 2024-09-21T22:08:20.665 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.9019999999999998e-05 2024-09-21T22:08:20.665 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.665 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.665 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-21T22:08:20.665 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.410617+0000", 2024-09-21T22:08:20.665 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000135232 2024-09-21T22:08:20.666 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.666 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.666 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:20.666 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.457051+0000", 2024-09-21T22:08:20.666 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.04643415 2024-09-21T22:08:20.666 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.666 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.666 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-21T22:08:20.666 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.457675+0000", 2024-09-21T22:08:20.666 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00062332700000000002 2024-09-21T22:08:20.666 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.666 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.666 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:20.666 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.457735+0000", 2024-09-21T22:08:20.667 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.029e-05 2024-09-21T22:08:20.667 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.667 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.667 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:20.667 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.457818+0000", 2024-09-21T22:08:20.667 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.2860000000000003e-05 2024-09-21T22:08:20.667 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:20.667 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:20.667 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:20.667 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.667 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.667 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4840.0:670 2.3 2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:08:20.667 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.466980+0000", 2024-09-21T22:08:20.667 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 56.165430317000002, 2024-09-21T22:08:20.667 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.050913930000000003, 2024-09-21T22:08:20.668 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:20.668 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:20.668 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:20.668 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-21T22:08:20.668 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4840", 2024-09-21T22:08:20.668 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:08:20.668 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 670 2024-09-21T22:08:20.668 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.668 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:20.668 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.668 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:20.668 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.466980+0000", 2024-09-21T22:08:20.668 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:20.668 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.668 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.669 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:20.669 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.466980+0000", 2024-09-21T22:08:20.669 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:20.669 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.669 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.669 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:20.669 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.466981+0000", 2024-09-21T22:08:20.669 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0839999999999999e-06 2024-09-21T22:08:20.669 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.669 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.669 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:20.669 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.467815+0000", 2024-09-21T22:08:20.669 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00083330399999999999 2024-09-21T22:08:20.669 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.670 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.670 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:20.670 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.467832+0000", 2024-09-21T22:08:20.670 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7601e-05 2024-09-21T22:08:20.670 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.670 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.670 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:20.670 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.467840+0000", 2024-09-21T22:08:20.670 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.2220000000000002e-06 2024-09-21T22:08:20.670 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.670 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.670 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:20.670 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.467974+0000", 2024-09-21T22:08:20.670 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000133411 2024-09-21T22:08:20.671 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.671 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.671 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:20.671 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.468223+0000", 2024-09-21T22:08:20.671 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00024906400000000001 2024-09-21T22:08:20.671 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.671 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.671 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-21T22:08:20.671 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.468616+0000", 2024-09-21T22:08:20.671 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000393433 2024-09-21T22:08:20.671 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.671 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.671 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:20.671 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.517016+0000", 2024-09-21T22:08:20.671 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.048399708 2024-09-21T22:08:20.672 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.672 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.672 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-21T22:08:20.672 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.517693+0000", 2024-09-21T22:08:20.672 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00067681000000000002 2024-09-21T22:08:20.672 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.672 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.672 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:20.672 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.517789+0000", 2024-09-21T22:08:20.672 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.5778000000000004e-05 2024-09-21T22:08:20.672 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.672 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.672 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:20.673 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.517894+0000", 2024-09-21T22:08:20.673 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000105515 2024-09-21T22:08:20.673 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:20.673 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:20.673 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:20.673 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.673 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.673 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4840.0:672 2.3 2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:08:20.673 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.527127+0000", 2024-09-21T22:08:20.673 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 56.105283855000003, 2024-09-21T22:08:20.673 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.047345602000000001, 2024-09-21T22:08:20.673 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:20.673 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:20.673 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:20.674 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-21T22:08:20.674 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4840", 2024-09-21T22:08:20.674 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:08:20.674 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 672 2024-09-21T22:08:20.674 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.674 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:20.674 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.674 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:20.674 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.527127+0000", 2024-09-21T22:08:20.674 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:20.674 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.674 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.674 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:20.674 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.527127+0000", 2024-09-21T22:08:20.675 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:20.675 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.675 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.675 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:20.675 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.527128+0000", 2024-09-21T22:08:20.675 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2810000000000001e-06 2024-09-21T22:08:20.675 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.675 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.675 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:20.675 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.527962+0000", 2024-09-21T22:08:20.675 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00083445199999999996 2024-09-21T22:08:20.675 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.675 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.675 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:20.675 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.527981+0000", 2024-09-21T22:08:20.676 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9009e-05 2024-09-21T22:08:20.676 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.676 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.676 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:20.676 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.527990+0000", 2024-09-21T22:08:20.676 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.8619999999999992e-06 2024-09-21T22:08:20.676 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.676 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.676 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:20.676 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.528061+0000", 2024-09-21T22:08:20.676 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.0226000000000002e-05 2024-09-21T22:08:20.676 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.676 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.676 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:20.677 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.528156+0000", 2024-09-21T22:08:20.677 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.5724999999999996e-05 2024-09-21T22:08:20.677 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.677 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.677 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-21T22:08:20.677 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.528303+0000", 2024-09-21T22:08:20.677 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000146374 2024-09-21T22:08:20.677 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.677 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.677 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:20.677 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.573772+0000", 2024-09-21T22:08:20.677 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.045469745999999998 2024-09-21T22:08:20.677 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.677 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.678 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-21T22:08:20.678 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.574299+0000", 2024-09-21T22:08:20.678 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00052649200000000004 2024-09-21T22:08:20.678 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.678 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.678 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:20.678 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.574366+0000", 2024-09-21T22:08:20.678 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.6872000000000006e-05 2024-09-21T22:08:20.678 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.678 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.678 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:20.678 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.574472+0000", 2024-09-21T22:08:20.678 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000106563 2024-09-21T22:08:20.678 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:20.679 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:20.679 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:20.679 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.679 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.679 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4840.0:686 2.3 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:08:20.679 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.943561+0000", 2024-09-21T22:08:20.679 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 55.688849578999999, 2024-09-21T22:08:20.679 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.049021493999999999, 2024-09-21T22:08:20.679 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:20.679 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:20.679 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:20.679 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-21T22:08:20.679 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4840", 2024-09-21T22:08:20.679 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:08:20.680 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 686 2024-09-21T22:08:20.680 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.680 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:20.680 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.680 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:20.680 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.943561+0000", 2024-09-21T22:08:20.680 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:20.680 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.680 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.680 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:20.680 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.943561+0000", 2024-09-21T22:08:20.680 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:20.680 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.680 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.680 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:20.681 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.943563+0000", 2024-09-21T22:08:20.681 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9420000000000002e-06 2024-09-21T22:08:20.681 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.681 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.681 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:20.681 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.944546+0000", 2024-09-21T22:08:20.681 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00098357099999999992 2024-09-21T22:08:20.681 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.681 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.681 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:20.681 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.944565+0000", 2024-09-21T22:08:20.681 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8621e-05 2024-09-21T22:08:20.681 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.681 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.682 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:20.682 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.944575+0000", 2024-09-21T22:08:20.682 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.4639999999999995e-06 2024-09-21T22:08:20.682 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.682 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.682 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:20.682 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.944647+0000", 2024-09-21T22:08:20.682 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.2786000000000004e-05 2024-09-21T22:08:20.682 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.682 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.682 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:20.682 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.944746+0000", 2024-09-21T22:08:20.682 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.8196999999999995e-05 2024-09-21T22:08:20.682 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.683 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.683 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-21T22:08:20.683 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.944890+0000", 2024-09-21T22:08:20.683 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000144804 2024-09-21T22:08:20.683 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.683 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.683 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:20.683 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.992404+0000", 2024-09-21T22:08:20.683 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.047513211999999999 2024-09-21T22:08:20.683 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.683 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.683 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-21T22:08:20.683 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.992480+0000", 2024-09-21T22:08:20.683 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.6491999999999999e-05 2024-09-21T22:08:20.683 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.684 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.684 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:20.684 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.992521+0000", 2024-09-21T22:08:20.684 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.0608000000000002e-05 2024-09-21T22:08:20.684 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.684 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.684 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:20.684 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.992582+0000", 2024-09-21T22:08:20.684 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.1797000000000005e-05 2024-09-21T22:08:20.684 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:20.684 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:20.684 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:20.684 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.684 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.685 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4840.0:720 2.1 2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:08:20.685 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.803827+0000", 2024-09-21T22:08:20.685 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 54.828583303000002, 2024-09-21T22:08:20.685 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.049890510999999998, 2024-09-21T22:08:20.685 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:20.685 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:20.685 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:20.685 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-21T22:08:20.685 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4840", 2024-09-21T22:08:20.685 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:08:20.685 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 720 2024-09-21T22:08:20.685 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.685 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:20.685 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.685 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:20.686 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.803827+0000", 2024-09-21T22:08:20.686 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:20.686 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.686 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.686 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:20.686 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.803827+0000", 2024-09-21T22:08:20.686 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:20.686 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.686 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.686 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:20.686 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.803828+0000", 2024-09-21T22:08:20.686 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.569999999999999e-07 2024-09-21T22:08:20.686 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.687 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.687 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:20.687 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.804689+0000", 2024-09-21T22:08:20.687 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00086121799999999997 2024-09-21T22:08:20.687 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.687 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.687 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:20.687 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.804706+0000", 2024-09-21T22:08:20.687 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6266999999999999e-05 2024-09-21T22:08:20.687 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.687 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.687 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:20.687 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.804713+0000", 2024-09-21T22:08:20.687 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.1010000000000003e-06 2024-09-21T22:08:20.687 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.688 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.688 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:20.688 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.804781+0000", 2024-09-21T22:08:20.688 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.7871000000000001e-05 2024-09-21T22:08:20.688 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.688 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.688 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:20.688 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.804859+0000", 2024-09-21T22:08:20.688 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.7891000000000004e-05 2024-09-21T22:08:20.688 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.688 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.688 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 0", 2024-09-21T22:08:20.688 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.804971+0000", 2024-09-21T22:08:20.688 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00011275100000000001 2024-09-21T22:08:20.689 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.689 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.689 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:20.689 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.852704+0000", 2024-09-21T22:08:20.689 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.047732361000000001 2024-09-21T22:08:20.689 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.689 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.689 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-21T22:08:20.689 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.853446+0000", 2024-09-21T22:08:20.689 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00074258799999999997 2024-09-21T22:08:20.689 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.689 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.689 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:20.689 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.853608+0000", 2024-09-21T22:08:20.689 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00016190900000000001 2024-09-21T22:08:20.690 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.690 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.690 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:20.690 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.853718+0000", 2024-09-21T22:08:20.690 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000109597 2024-09-21T22:08:20.690 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:20.690 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:20.690 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:20.690 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.690 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.690 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4840.0:723 2.1 2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:08:20.690 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.860543+0000", 2024-09-21T22:08:20.690 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 54.771867868999998, 2024-09-21T22:08:20.690 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.062354957000000003, 2024-09-21T22:08:20.691 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:20.691 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:20.691 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:20.691 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-21T22:08:20.691 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4840", 2024-09-21T22:08:20.691 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:08:20.691 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 723 2024-09-21T22:08:20.691 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.691 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:20.691 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.691 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:20.691 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.860543+0000", 2024-09-21T22:08:20.691 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:20.691 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.691 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.692 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:20.692 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.860543+0000", 2024-09-21T22:08:20.692 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:20.692 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.692 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.692 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:20.692 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.860544+0000", 2024-09-21T22:08:20.692 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.1119999999999999e-06 2024-09-21T22:08:20.692 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.692 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.692 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:20.692 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.861400+0000", 2024-09-21T22:08:20.692 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000856393 2024-09-21T22:08:20.692 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.693 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.693 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:20.693 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.861419+0000", 2024-09-21T22:08:20.693 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8992e-05 2024-09-21T22:08:20.693 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.693 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.693 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:20.693 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.861428+0000", 2024-09-21T22:08:20.693 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.551e-06 2024-09-21T22:08:20.693 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.693 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.693 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:20.693 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.861555+0000", 2024-09-21T22:08:20.693 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000127286 2024-09-21T22:08:20.693 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.694 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.694 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:20.694 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.861816+0000", 2024-09-21T22:08:20.694 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00026100899999999998 2024-09-21T22:08:20.694 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.694 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.694 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 0", 2024-09-21T22:08:20.694 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.862189+0000", 2024-09-21T22:08:20.694 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00037261899999999998 2024-09-21T22:08:20.694 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.694 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.694 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:20.694 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.921514+0000", 2024-09-21T22:08:20.694 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.059325210000000003 2024-09-21T22:08:20.695 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.695 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.695 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-21T22:08:20.695 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.922721+0000", 2024-09-21T22:08:20.695 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0012066819999999999 2024-09-21T22:08:20.695 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.695 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.695 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:20.695 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.922797+0000", 2024-09-21T22:08:20.695 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.6162999999999996e-05 2024-09-21T22:08:20.695 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.695 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.695 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:20.695 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.922898+0000", 2024-09-21T22:08:20.695 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00010094000000000001 2024-09-21T22:08:20.696 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:20.696 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:20.696 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:20.696 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.696 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.696 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:725 2.0 e63/16 2:08b4f96a:::rbd_data.126483d75f2d.000000000000000f:head v 63'33, mlcod=63'31)", 2024-09-21T22:08:20.696 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.936778+0000", 2024-09-21T22:08:20.696 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 54.695632127000003, 2024-09-21T22:08:20.696 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.047671937999999997, 2024-09-21T22:08:20.696 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:20.696 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:20.696 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:20.696 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:20.696 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.696 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:20.697 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.936778+0000", 2024-09-21T22:08:20.697 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:20.697 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.697 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.697 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:20.697 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.936778+0000", 2024-09-21T22:08:20.697 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:20.697 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.697 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.697 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:20.697 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.936780+0000", 2024-09-21T22:08:20.697 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7799999999999999e-06 2024-09-21T22:08:20.697 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.697 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.697 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:20.698 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.938475+0000", 2024-09-21T22:08:20.698 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0016951749999999999 2024-09-21T22:08:20.698 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.698 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.698 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:20.698 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.938506+0000", 2024-09-21T22:08:20.698 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.0865999999999999e-05 2024-09-21T22:08:20.698 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.698 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.698 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:20.698 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.938520+0000", 2024-09-21T22:08:20.698 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3825999999999999e-05 2024-09-21T22:08:20.698 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.699 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.699 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:20.699 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.938624+0000", 2024-09-21T22:08:20.699 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00010391900000000001 2024-09-21T22:08:20.699 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.699 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.699 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:20.699 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.938651+0000", 2024-09-21T22:08:20.699 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.7481999999999999e-05 2024-09-21T22:08:20.699 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.699 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.699 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:20.699 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.984369+0000", 2024-09-21T22:08:20.699 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.045717595 2024-09-21T22:08:20.699 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.700 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.700 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:20.700 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.984450+0000", 2024-09-21T22:08:20.700 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.1295e-05 2024-09-21T22:08:20.700 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:20.700 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:20.700 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:20.700 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.700 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.700 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:727 2.0 e63/16 2:08b4f96a:::rbd_data.126483d75f2d.000000000000000f:head v 63'35, mlcod=63'33)", 2024-09-21T22:08:20.700 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.998090+0000", 2024-09-21T22:08:20.700 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 54.634320950999999, 2024-09-21T22:08:20.700 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.052205704999999998, 2024-09-21T22:08:20.700 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:20.701 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:20.701 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:20.701 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:20.701 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.701 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:20.701 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.998090+0000", 2024-09-21T22:08:20.701 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:20.701 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.701 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.701 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:20.701 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.998090+0000", 2024-09-21T22:08:20.701 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:20.701 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.701 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.702 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:20.702 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.998091+0000", 2024-09-21T22:08:20.702 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6929999999999999e-06 2024-09-21T22:08:20.702 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.702 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.702 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:20.702 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.998940+0000", 2024-09-21T22:08:20.702 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00084875200000000004 2024-09-21T22:08:20.702 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.702 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.702 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:20.702 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.998970+0000", 2024-09-21T22:08:20.702 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9709999999999998e-05 2024-09-21T22:08:20.702 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.702 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.703 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:20.703 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.998984+0000", 2024-09-21T22:08:20.703 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3739e-05 2024-09-21T22:08:20.703 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.703 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.703 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:20.703 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.999097+0000", 2024-09-21T22:08:20.703 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000113484 2024-09-21T22:08:20.703 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.703 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.703 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:20.703 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.999135+0000", 2024-09-21T22:08:20.703 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.8405999999999998e-05 2024-09-21T22:08:20.703 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.704 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.704 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:20.704 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.050245+0000", 2024-09-21T22:08:20.704 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.051109935000000002 2024-09-21T22:08:20.704 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.704 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.704 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:20.704 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.050295+0000", 2024-09-21T22:08:20.704 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.9985999999999999e-05 2024-09-21T22:08:20.704 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:20.704 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:20.704 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:20.704 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.704 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.704 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:729 2.0 e63/16 2:08b4f96a:::rbd_data.126483d75f2d.000000000000000f:head v 63'37, mlcod=63'35)", 2024-09-21T22:08:20.705 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:26.062424+0000", 2024-09-21T22:08:20.705 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 54.569986329000002, 2024-09-21T22:08:20.705 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.047312186999999999, 2024-09-21T22:08:20.705 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:20.705 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:20.705 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:20.705 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:20.705 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.705 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:20.705 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.062424+0000", 2024-09-21T22:08:20.705 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:20.705 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.705 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.705 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:20.705 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.062424+0000", 2024-09-21T22:08:20.706 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:20.706 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.706 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.706 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:20.706 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.062426+0000", 2024-09-21T22:08:20.706 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7129999999999999e-06 2024-09-21T22:08:20.706 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.706 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.706 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:20.706 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.063211+0000", 2024-09-21T22:08:20.706 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00078488500000000003 2024-09-21T22:08:20.706 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.706 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.706 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:20.707 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.063241+0000", 2024-09-21T22:08:20.707 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.0056000000000002e-05 2024-09-21T22:08:20.707 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.707 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.707 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:20.707 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.063255+0000", 2024-09-21T22:08:20.707 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4022999999999999e-05 2024-09-21T22:08:20.707 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.707 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.707 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:20.707 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.063384+0000", 2024-09-21T22:08:20.707 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00012920600000000001 2024-09-21T22:08:20.707 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.707 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.708 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:20.708 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.063421+0000", 2024-09-21T22:08:20.708 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.7242e-05 2024-09-21T22:08:20.708 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.708 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.708 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:20.708 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.109666+0000", 2024-09-21T22:08:20.708 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.046244446000000002 2024-09-21T22:08:20.708 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.708 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.708 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:20.708 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.109736+0000", 2024-09-21T22:08:20.708 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.0616000000000003e-05 2024-09-21T22:08:20.708 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:20.708 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:20.709 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:20.709 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.709 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.709 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:731 2.0 e63/16 2:08b4f96a:::rbd_data.126483d75f2d.000000000000000f:head v 63'39, mlcod=63'37)", 2024-09-21T22:08:20.709 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:26.135946+0000", 2024-09-21T22:08:20.709 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 54.496464822, 2024-09-21T22:08:20.709 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.054290359000000003, 2024-09-21T22:08:20.709 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:20.709 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:20.709 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:20.709 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:20.709 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.709 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:20.709 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.135946+0000", 2024-09-21T22:08:20.710 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:20.710 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.710 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.710 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:20.710 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.135946+0000", 2024-09-21T22:08:20.710 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:20.710 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.710 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.710 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:20.710 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.135948+0000", 2024-09-21T22:08:20.710 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7999999999999999e-06 2024-09-21T22:08:20.710 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.710 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.710 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:20.710 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.136730+0000", 2024-09-21T22:08:20.711 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00078271899999999997 2024-09-21T22:08:20.711 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.711 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.711 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:20.711 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.136760+0000", 2024-09-21T22:08:20.711 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9876e-05 2024-09-21T22:08:20.711 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.711 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.711 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:20.711 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.136774+0000", 2024-09-21T22:08:20.711 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3478999999999999e-05 2024-09-21T22:08:20.711 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.711 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.711 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:20.711 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.136944+0000", 2024-09-21T22:08:20.712 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000170626 2024-09-21T22:08:20.712 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.712 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.712 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:20.712 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.136985+0000", 2024-09-21T22:08:20.712 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.0377999999999999e-05 2024-09-21T22:08:20.712 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.712 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.712 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:20.712 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.190137+0000", 2024-09-21T22:08:20.712 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.053152438000000003 2024-09-21T22:08:20.712 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.712 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.712 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:20.713 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.190236+0000", 2024-09-21T22:08:20.713 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.9042999999999993e-05 2024-09-21T22:08:20.713 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:20.713 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:20.713 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:20.713 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.713 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.713 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:125 2.3 e71/66 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head v 71'31, mlcod=71'29)", 2024-09-21T22:08:20.713 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.066855+0000", 2024-09-21T22:08:20.713 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 42.565555387000003, 2024-09-21T22:08:20.713 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.059856059000000003, 2024-09-21T22:08:20.713 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:20.713 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:20.713 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:20.714 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:20.714 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.714 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:20.714 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.066855+0000", 2024-09-21T22:08:20.714 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:20.714 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.714 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.714 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:20.714 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.066855+0000", 2024-09-21T22:08:20.714 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:20.714 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.714 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.714 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:20.714 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.066857+0000", 2024-09-21T22:08:20.715 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.807e-06 2024-09-21T22:08:20.715 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.715 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.715 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:20.715 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.068365+0000", 2024-09-21T22:08:20.715 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0015077910000000001 2024-09-21T22:08:20.715 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.715 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.715 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:20.715 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.068395+0000", 2024-09-21T22:08:20.715 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.0326000000000001e-05 2024-09-21T22:08:20.715 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.715 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.715 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:20.716 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.068415+0000", 2024-09-21T22:08:20.716 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9871999999999999e-05 2024-09-21T22:08:20.716 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.716 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.716 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:20.716 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.068580+0000", 2024-09-21T22:08:20.716 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000165317 2024-09-21T22:08:20.716 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.716 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.716 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:20.716 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.068618+0000", 2024-09-21T22:08:20.716 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.7753000000000003e-05 2024-09-21T22:08:20.716 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.716 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.716 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:20.717 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.126623+0000", 2024-09-21T22:08:20.717 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.058004642000000002 2024-09-21T22:08:20.717 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.717 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.717 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:20.717 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.126711+0000", 2024-09-21T22:08:20.717 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.8551e-05 2024-09-21T22:08:20.717 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:20.717 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:20.717 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:20.717 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.717 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.717 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:127 2.3 e71/66 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head v 71'33, mlcod=71'31)", 2024-09-21T22:08:20.717 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.140864+0000", 2024-09-21T22:08:20.718 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 42.491546358999997, 2024-09-21T22:08:20.718 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.052512380999999997, 2024-09-21T22:08:20.718 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:20.718 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:20.718 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:20.718 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:20.718 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.718 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:20.718 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.140864+0000", 2024-09-21T22:08:20.718 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:20.718 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.718 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.718 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:20.718 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.140864+0000", 2024-09-21T22:08:20.719 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:20.719 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.719 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.719 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:20.719 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.140866+0000", 2024-09-21T22:08:20.719 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.694e-06 2024-09-21T22:08:20.719 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.719 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.719 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:20.719 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.141712+0000", 2024-09-21T22:08:20.719 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00084581499999999998 2024-09-21T22:08:20.719 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.719 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.719 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:20.719 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.141742+0000", 2024-09-21T22:08:20.720 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9865999999999998e-05 2024-09-21T22:08:20.720 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.720 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.720 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:20.720 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.141761+0000", 2024-09-21T22:08:20.720 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9473000000000001e-05 2024-09-21T22:08:20.720 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.720 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.720 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:20.720 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.141919+0000", 2024-09-21T22:08:20.720 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00015758300000000001 2024-09-21T22:08:20.720 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.720 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.720 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:20.721 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.141955+0000", 2024-09-21T22:08:20.721 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.6613000000000003e-05 2024-09-21T22:08:20.721 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.721 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.721 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:20.721 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.193333+0000", 2024-09-21T22:08:20.721 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.051377260000000001 2024-09-21T22:08:20.721 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.721 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.721 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:20.721 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.193377+0000", 2024-09-21T22:08:20.721 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.4076999999999998e-05 2024-09-21T22:08:20.721 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:20.721 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:20.721 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:20.722 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.722 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.722 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:129 2.3 e71/66 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head v 71'35, mlcod=71'33)", 2024-09-21T22:08:20.722 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.209458+0000", 2024-09-21T22:08:20.722 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 42.422952610000003, 2024-09-21T22:08:20.722 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.056595578000000001, 2024-09-21T22:08:20.722 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:20.722 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:20.722 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:20.722 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:20.722 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.722 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:20.722 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.209458+0000", 2024-09-21T22:08:20.722 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:20.722 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.723 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.723 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:20.723 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.209458+0000", 2024-09-21T22:08:20.723 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:20.723 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.723 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.723 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:20.723 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.209460+0000", 2024-09-21T22:08:20.723 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7600000000000001e-06 2024-09-21T22:08:20.723 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.723 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.723 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:20.723 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.210317+0000", 2024-09-21T22:08:20.723 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00085771700000000003 2024-09-21T22:08:20.724 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.724 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.724 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:20.724 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.210346+0000", 2024-09-21T22:08:20.724 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8334e-05 2024-09-21T22:08:20.724 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.724 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.724 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:20.724 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.210365+0000", 2024-09-21T22:08:20.724 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9439000000000001e-05 2024-09-21T22:08:20.724 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.724 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.724 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:20.725 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.210544+0000", 2024-09-21T22:08:20.725 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00017845599999999999 2024-09-21T22:08:20.725 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.725 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.725 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:20.725 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.210584+0000", 2024-09-21T22:08:20.725 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.0423000000000002e-05 2024-09-21T22:08:20.725 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.725 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.725 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:20.725 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.265970+0000", 2024-09-21T22:08:20.725 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.055385765000000003 2024-09-21T22:08:20.725 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.725 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.725 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:20.726 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.266054+0000", 2024-09-21T22:08:20.726 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.3683999999999994e-05 2024-09-21T22:08:20.726 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:20.726 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:20.726 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:20.726 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.726 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.726 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:131 2.3 e71/66 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head v 71'37, mlcod=71'35)", 2024-09-21T22:08:20.726 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.279550+0000", 2024-09-21T22:08:20.726 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 42.352860772, 2024-09-21T22:08:20.726 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.052290419999999997, 2024-09-21T22:08:20.726 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:20.726 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:20.727 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:20.727 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:20.727 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.727 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:20.727 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.279550+0000", 2024-09-21T22:08:20.727 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:20.727 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.727 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.727 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:20.727 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.279550+0000", 2024-09-21T22:08:20.727 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:20.727 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.727 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.727 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:20.728 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.279552+0000", 2024-09-21T22:08:20.728 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9439999999999999e-06 2024-09-21T22:08:20.728 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.728 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.728 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:20.728 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.280296+0000", 2024-09-21T22:08:20.728 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00074417000000000005 2024-09-21T22:08:20.728 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.728 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.728 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:20.728 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.280324+0000", 2024-09-21T22:08:20.728 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8045999999999999e-05 2024-09-21T22:08:20.728 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.728 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.729 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:20.729 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.280345+0000", 2024-09-21T22:08:20.729 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.0849999999999999e-05 2024-09-21T22:08:20.729 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.729 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.729 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:20.729 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.280497+0000", 2024-09-21T22:08:20.729 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00015252500000000001 2024-09-21T22:08:20.729 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.729 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.729 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:20.729 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.280531+0000", 2024-09-21T22:08:20.729 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.4034e-05 2024-09-21T22:08:20.729 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.729 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.730 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:20.730 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.331769+0000", 2024-09-21T22:08:20.730 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.051237999999999999 2024-09-21T22:08:20.730 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:20.730 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:20.730 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:20.730 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.331840+0000", 2024-09-21T22:08:20.730 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.0851000000000003e-05 2024-09-21T22:08:20.730 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:20.730 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:20.730 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:20.730 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:20.730 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:20.730 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:08:20.732 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T22:08:20.732 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:08:20.727+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:08:20.761 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T22:08:20.833 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:08:20.829+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:08:20.869 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:08:20.869 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:08:20.870 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-09-21T22:08:20.870 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:08:20.884 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T22:08:20.931 INFO:teuthology.orchestra.run.smithi039.stdout:osd.0: {} 2024-09-21T22:08:20.931 INFO:teuthology.orchestra.run.smithi039.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T22:08:20.933 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:08:20.927+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:08:20.947 INFO:teuthology.orchestra.run.smithi039.stdout:osd.1: {} 2024-09-21T22:08:20.947 INFO:teuthology.orchestra.run.smithi039.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T22:08:20.964 INFO:teuthology.orchestra.run.smithi039.stdout:osd.2: {} 2024-09-21T22:08:20.964 INFO:teuthology.orchestra.run.smithi039.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T22:08:20.983 INFO:teuthology.orchestra.run.smithi039.stdout:osd.3: {} 2024-09-21T22:08:20.984 INFO:teuthology.orchestra.run.smithi039.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T22:08:21.003 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:21.003 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:21.003 INFO:teuthology.orchestra.run.smithi039.stdout:osd.4: {} 2024-09-21T22:08:21.004 INFO:teuthology.orchestra.run.smithi039.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T22:08:21.018 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T22:08:21.024 INFO:teuthology.orchestra.run.smithi039.stdout:osd.5: {} 2024-09-21T22:08:21.024 INFO:teuthology.orchestra.run.smithi039.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T22:08:21.034 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:08:21.029+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:08:21.135 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:08:21.131+0000 7f2075a06640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31720) UID: 0 2024-09-21T22:08:21.187 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:21.187 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:21.201 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T22:08:21.242 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:08:21.239+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:08:21.332 INFO:tasks.workunit.client.0.smithi157.stderr:++ rbd journal inspect --pool rbd --image testrbdnbd21553 2024-09-21T22:08:21.332 INFO:tasks.workunit.client.0.smithi157.stderr:++ awk '/entries inspected/ {print $1}' 2024-09-21T22:08:21.337 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:08:21.331+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:08:21.342 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T22:08:21.367 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:21.367 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:21.382 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T22:08:21.417 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-21T22:08:21.412+0000 7fb604c41640 -1 JournalMetadata: 0x55e6f30deeb0 failed to locate client: INSPECT 2024-09-21T22:08:21.422 INFO:tasks.workunit.client.0.smithi157.stderr:+ queue_length=0 2024-09-21T22:08:21.423 INFO:tasks.workunit.client.0.smithi157.stderr:+ tries=5 2024-09-21T22:08:21.423 INFO:tasks.workunit.client.0.smithi157.stderr:+ '[' 5 -lt 10 ']' 2024-09-21T22:08:21.423 INFO:tasks.workunit.client.0.smithi157.stderr:+ '[' 0 -gt 0 ']' 2024-09-21T22:08:21.424 INFO:tasks.workunit.client.0.smithi157.stderr:+ '[' 0 -eq 0 ']' 2024-09-21T22:08:21.424 INFO:tasks.workunit.client.0.smithi157.stderr:+ unmap_device /dev/nbd2 24419 2024-09-21T22:08:21.424 INFO:tasks.workunit.client.0.smithi157.stderr:+ local args=/dev/nbd2 2024-09-21T22:08:21.424 INFO:tasks.workunit.client.0.smithi157.stderr:+ local pid=24419 2024-09-21T22:08:21.424 INFO:tasks.workunit.client.0.smithi157.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd2 2024-09-21T22:08:21.424 INFO:tasks.workunit.client.0.smithi157.stderr:+ local cmd 2024-09-21T22:08:21.424 INFO:tasks.workunit.client.0.smithi157.stderr:++ id -u 2024-09-21T22:08:21.424 INFO:tasks.workunit.client.0.smithi157.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T22:08:21.424 INFO:tasks.workunit.client.0.smithi157.stderr:++ which rbd 2024-09-21T22:08:21.425 INFO:tasks.workunit.client.0.smithi157.stderr:+ cmd=/usr/bin/rbd 2024-09-21T22:08:21.425 INFO:tasks.workunit.client.0.smithi157.stderr:+ shift 2024-09-21T22:08:21.425 INFO:tasks.workunit.client.0.smithi157.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd2 2024-09-21T22:08:21.437 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:08:21.433+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:08:21.479 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:21.480 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:21.539 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:08:21.535+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:08:21.543 INFO:tasks.workunit.client.0.smithi157.stderr:+ rbd device --device-type nbd list 2024-09-21T22:08:21.543 INFO:tasks.workunit.client.0.smithi157.stderr:+ expect_false grep '^24419\b' 2024-09-21T22:08:21.544 INFO:tasks.workunit.client.0.smithi157.stderr:+ grep '^24419\b' 2024-09-21T22:08:21.560 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:21.561 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:21.575 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T22:08:21.607 INFO:tasks.workunit.client.0.smithi157.stderr:+ return 0 2024-09-21T22:08:21.607 INFO:tasks.workunit.client.0.smithi157.stderr:+ ps -C rbd-nbd 2024-09-21T22:08:21.608 INFO:tasks.workunit.client.0.smithi157.stderr:+ expect_false grep '^ *24419\b' 2024-09-21T22:08:21.608 INFO:tasks.workunit.client.0.smithi157.stderr:+ grep '^ *24419\b' 2024-09-21T22:08:21.615 INFO:tasks.workunit.client.0.smithi157.stderr:+ return 0 2024-09-21T22:08:21.615 INFO:tasks.workunit.client.0.smithi157.stderr:+ sleep 0.5 2024-09-21T22:08:21.640 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:08:21.637+0000 7fe38af2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 34631) UID: 0 2024-09-21T22:08:21.738 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:21.738 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:21.741 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:08:21.735+0000 7f2075a06640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31720) UID: 0 2024-09-21T22:08:21.753 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T22:08:21.842 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:08:21.837+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:08:21.915 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:21.915 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:21.930 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T22:08:21.943 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:08:21.937+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:08:22.044 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:08:22.041+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:08:22.051 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:22.051 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:22.065 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T22:08:22.116 INFO:tasks.workunit.client.0.smithi157.stderr:+ DEV= 2024-09-21T22:08:22.116 INFO:tasks.workunit.client.0.smithi157.stderr:+ rbd feature disable rbd/testrbdnbd21553 journaling 2024-09-21T22:08:22.145 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:08:22.141+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:08:22.195 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:22.195 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:22.210 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T22:08:22.246 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:08:22.243+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:08:22.269 INFO:tasks.workunit.client.0.smithi157.stderr:+ rbd config image rm rbd/testrbdnbd21553 rbd_discard_granularity_bytes 2024-09-21T22:08:22.347 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:08:22.341+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:08:22.349 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:22.349 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:22.364 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T22:08:22.365 INFO:tasks.workunit.client.0.smithi157.stderr:++ _sudo rbd device --device-type nbd map rbd/testrbdnbd21553 2024-09-21T22:08:22.365 INFO:tasks.workunit.client.0.smithi157.stderr:++ local cmd 2024-09-21T22:08:22.366 INFO:tasks.workunit.client.0.smithi157.stderr:+++ id -u 2024-09-21T22:08:22.366 INFO:tasks.workunit.client.0.smithi157.stderr:++ '[' 1000 -eq 0 ']' 2024-09-21T22:08:22.366 INFO:tasks.workunit.client.0.smithi157.stderr:+++ which rbd 2024-09-21T22:08:22.368 INFO:tasks.workunit.client.0.smithi157.stderr:++ cmd=/usr/bin/rbd 2024-09-21T22:08:22.368 INFO:tasks.workunit.client.0.smithi157.stderr:++ shift 2024-09-21T22:08:22.368 INFO:tasks.workunit.client.0.smithi157.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map rbd/testrbdnbd21553 2024-09-21T22:08:22.448 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:08:22.443+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:08:22.509 INFO:tasks.workunit.client.0.smithi157.stderr:+ DEV=/dev/nbd2 2024-09-21T22:08:22.509 INFO:tasks.workunit.client.0.smithi157.stderr:+ get_pid rbd 2024-09-21T22:08:22.509 INFO:tasks.workunit.client.0.smithi157.stderr:+ local pool=rbd 2024-09-21T22:08:22.509 INFO:tasks.workunit.client.0.smithi157.stderr:+ local ns= 2024-09-21T22:08:22.510 INFO:tasks.workunit.client.0.smithi157.stderr:++ rbd device --device-type nbd --format xml list 2024-09-21T22:08:22.510 INFO:tasks.workunit.client.0.smithi157.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd21553'\''][device='\''/dev/nbd2'\'']/id' 2024-09-21T22:08:22.527 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-21T22:08:22.528 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:22.529 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:22.544 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T22:08:22.592 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:08:22.589+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:08:22.608 INFO:tasks.workunit.client.0.smithi157.stderr:+ PID=24924 2024-09-21T22:08:22.608 INFO:tasks.workunit.client.0.smithi157.stderr:+ test -n 24924 2024-09-21T22:08:22.608 INFO:tasks.workunit.client.0.smithi157.stderr:+ ps -p 24924 -C rbd-nbd 2024-09-21T22:08:22.618 INFO:tasks.workunit.client.0.smithi157.stdout: PID TTY TIME CMD 2024-09-21T22:08:22.618 INFO:tasks.workunit.client.0.smithi157.stdout: 24924 ? 00:00:00 rbd-nbd 2024-09-21T22:08:22.618 INFO:tasks.workunit.client.0.smithi157.stderr:+ rbd feature enable rbd/testrbdnbd21553 journaling 2024-09-21T22:08:22.649 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:08:22.643+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:08:22.714 INFO:tasks.workunit.client.0.smithi157.stderr:+ _sudo blkdiscard --offset 0 --length 4096 /dev/nbd2 2024-09-21T22:08:22.714 INFO:tasks.workunit.client.0.smithi157.stderr:+ local cmd 2024-09-21T22:08:22.714 INFO:tasks.workunit.client.0.smithi157.stderr:++ id -u 2024-09-21T22:08:22.715 INFO:tasks.workunit.client.0.smithi157.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T22:08:22.716 INFO:tasks.workunit.client.0.smithi157.stderr:++ which blkdiscard 2024-09-21T22:08:22.717 INFO:tasks.workunit.client.0.smithi157.stderr:+ cmd=/usr/sbin/blkdiscard 2024-09-21T22:08:22.717 INFO:tasks.workunit.client.0.smithi157.stderr:+ shift 2024-09-21T22:08:22.718 INFO:tasks.workunit.client.0.smithi157.stderr:+ sudo -nE /usr/sbin/blkdiscard --offset 0 --length 4096 /dev/nbd2 2024-09-21T22:08:22.728 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:22.728 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:22.744 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T22:08:22.756 INFO:tasks.workunit.client.0.smithi157.stderr:+ rbd feature disable rbd/testrbdnbd21553 journaling 2024-09-21T22:08:22.788 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:08:22.785+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:08:22.851 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:08:22.847+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:08:22.864 INFO:tasks.workunit.client.0.smithi157.stderr:+ unmap_device /dev/nbd2 24924 2024-09-21T22:08:22.864 INFO:tasks.workunit.client.0.smithi157.stderr:+ local args=/dev/nbd2 2024-09-21T22:08:22.864 INFO:tasks.workunit.client.0.smithi157.stderr:+ local pid=24924 2024-09-21T22:08:22.864 INFO:tasks.workunit.client.0.smithi157.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd2 2024-09-21T22:08:22.864 INFO:tasks.workunit.client.0.smithi157.stderr:+ local cmd 2024-09-21T22:08:22.865 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:22.865 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:22.865 INFO:tasks.workunit.client.0.smithi157.stderr:++ id -u 2024-09-21T22:08:22.866 INFO:tasks.workunit.client.0.smithi157.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T22:08:22.866 INFO:tasks.workunit.client.0.smithi157.stderr:++ which rbd 2024-09-21T22:08:22.868 INFO:tasks.workunit.client.0.smithi157.stderr:+ cmd=/usr/bin/rbd 2024-09-21T22:08:22.868 INFO:tasks.workunit.client.0.smithi157.stderr:+ shift 2024-09-21T22:08:22.868 INFO:tasks.workunit.client.0.smithi157.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd2 2024-09-21T22:08:22.878 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T22:08:22.905 INFO:teuthology.orchestra.run.smithi039.stderr:nodeep-scrub is unset 2024-09-21T22:08:22.951 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:08:22.947+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:08:23.004 INFO:tasks.workunit.client.0.smithi157.stderr:+ rbd device --device-type nbd list 2024-09-21T22:08:23.004 INFO:tasks.workunit.client.0.smithi157.stderr:+ expect_false grep '^24924\b' 2024-09-21T22:08:23.004 INFO:tasks.workunit.client.0.smithi157.stderr:+ grep '^24924\b' 2024-09-21T22:08:23.030 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:23.030 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:23.042 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T22:08:23.044 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T22:08:23.070 INFO:tasks.workunit.client.0.smithi157.stderr:+ return 0 2024-09-21T22:08:23.070 INFO:tasks.workunit.client.0.smithi157.stderr:+ ps -C rbd-nbd 2024-09-21T22:08:23.070 INFO:tasks.workunit.client.0.smithi157.stderr:+ expect_false grep '^ *24924\b' 2024-09-21T22:08:23.070 INFO:tasks.workunit.client.0.smithi157.stderr:+ grep '^ *24924\b' 2024-09-21T22:08:23.076 INFO:tasks.workunit.client.0.smithi157.stderr:+ return 0 2024-09-21T22:08:23.076 INFO:tasks.workunit.client.0.smithi157.stderr:+ sleep 0.5 2024-09-21T22:08:23.088 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:08:23.085+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:08:23.154 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:08:23.151+0000 7f2075a06640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31720) UID: 0 2024-09-21T22:08:23.182 INFO:teuthology.orchestra.run.smithi039.stdout:osd.0: {} 2024-09-21T22:08:23.183 INFO:teuthology.orchestra.run.smithi039.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T22:08:23.199 INFO:teuthology.orchestra.run.smithi039.stdout:osd.1: {} 2024-09-21T22:08:23.200 INFO:teuthology.orchestra.run.smithi039.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T22:08:23.211 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:08:23.211 INFO:teuthology.orchestra.run.smithi039.stdout: "size": 20, 2024-09-21T22:08:23.212 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 600, 2024-09-21T22:08:23.212 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [ 2024-09-21T22:08:23.212 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.212 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:694 2.5 e63/37 2:b1d55633:::rbd_data.126483d75f2d.000000000000000b:head v 63'26, mlcod=63'24)", 2024-09-21T22:08:23.212 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.163199+0000", 2024-09-21T22:08:23.212 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 58.046332530000001, 2024-09-21T22:08:23.212 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.034676672999999998, 2024-09-21T22:08:23.212 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:23.212 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:23.212 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:23.212 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:23.212 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.212 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:23.213 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.163199+0000", 2024-09-21T22:08:23.213 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:23.213 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.213 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.213 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:23.213 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.163199+0000", 2024-09-21T22:08:23.213 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:23.213 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.213 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.213 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:23.213 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.163200+0000", 2024-09-21T22:08:23.213 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.703e-06 2024-09-21T22:08:23.213 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.213 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.214 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:23.214 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.164013+0000", 2024-09-21T22:08:23.214 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000812447 2024-09-21T22:08:23.214 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.214 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.214 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:23.214 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.164042+0000", 2024-09-21T22:08:23.214 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9006e-05 2024-09-21T22:08:23.214 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.214 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.214 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:23.214 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.164064+0000", 2024-09-21T22:08:23.214 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1719999999999999e-05 2024-09-21T22:08:23.214 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.215 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.215 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:23.215 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.164143+0000", 2024-09-21T22:08:23.215 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.9283999999999995e-05 2024-09-21T22:08:23.215 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.215 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.215 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:23.215 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.164165+0000", 2024-09-21T22:08:23.215 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.2379e-05 2024-09-21T22:08:23.215 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.215 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.216 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:23.217 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.197827+0000", 2024-09-21T22:08:23.217 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.033661782000000001 2024-09-21T22:08:23.217 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.217 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.217 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:23.217 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.197875+0000", 2024-09-21T22:08:23.218 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8352000000000001e-05 2024-09-21T22:08:23.218 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:23.218 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:23.218 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:23.218 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.218 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.218 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:696 2.5 e63/37 2:b1d55633:::rbd_data.126483d75f2d.000000000000000b:head v 63'28, mlcod=63'26)", 2024-09-21T22:08:23.218 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.218913+0000", 2024-09-21T22:08:23.218 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 57.990618083999998, 2024-09-21T22:08:23.218 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.056187599999999997, 2024-09-21T22:08:23.218 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:23.218 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:23.218 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:23.219 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:23.219 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.219 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:23.219 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.218913+0000", 2024-09-21T22:08:23.219 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:23.219 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.219 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.219 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:23.219 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.218913+0000", 2024-09-21T22:08:23.219 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:23.219 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.219 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.219 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:23.219 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.218914+0000", 2024-09-21T22:08:23.220 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.1930000000000001e-06 2024-09-21T22:08:23.220 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.220 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.220 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:23.220 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.219721+0000", 2024-09-21T22:08:23.220 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00080701500000000001 2024-09-21T22:08:23.220 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.220 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.220 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:23.220 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.219743+0000", 2024-09-21T22:08:23.220 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1364000000000001e-05 2024-09-21T22:08:23.220 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.220 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.220 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:23.221 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.219759+0000", 2024-09-21T22:08:23.221 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.611e-05 2024-09-21T22:08:23.221 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.221 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.221 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:23.221 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.219908+0000", 2024-09-21T22:08:23.221 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000149252 2024-09-21T22:08:23.221 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.221 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.221 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:23.221 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.219951+0000", 2024-09-21T22:08:23.221 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.2518999999999997e-05 2024-09-21T22:08:23.221 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.221 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.222 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:23.222 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.275031+0000", 2024-09-21T22:08:23.222 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.055080241000000002 2024-09-21T22:08:23.222 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.222 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.222 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:23.222 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.275101+0000", 2024-09-21T22:08:23.222 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.9906000000000005e-05 2024-09-21T22:08:23.222 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:23.222 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:23.222 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:23.222 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.222 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.223 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:698 2.5 e63/37 2:b1d55633:::rbd_data.126483d75f2d.000000000000000b:head v 63'30, mlcod=63'28)", 2024-09-21T22:08:23.223 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.289319+0000", 2024-09-21T22:08:23.223 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 57.920211875, 2024-09-21T22:08:23.223 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.053930944000000001, 2024-09-21T22:08:23.223 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:23.223 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:23.223 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:23.223 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:23.223 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.223 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:23.224 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.289319+0000", 2024-09-21T22:08:23.224 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:23.224 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.224 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.224 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:23.224 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.289319+0000", 2024-09-21T22:08:23.224 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:23.224 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.224 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.224 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:23.224 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.289321+0000", 2024-09-21T22:08:23.224 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.221e-06 2024-09-21T22:08:23.224 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.224 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.225 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:23.225 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.290091+0000", 2024-09-21T22:08:23.225 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00077069300000000003 2024-09-21T22:08:23.225 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.225 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.225 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:23.225 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.290109+0000", 2024-09-21T22:08:23.225 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7218999999999999e-05 2024-09-21T22:08:23.225 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.225 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.225 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:23.225 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.290122+0000", 2024-09-21T22:08:23.225 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3678e-05 2024-09-21T22:08:23.225 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.226 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.226 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:23.226 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.290274+0000", 2024-09-21T22:08:23.226 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00015205899999999999 2024-09-21T22:08:23.226 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.226 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.226 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:23.226 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.290313+0000", 2024-09-21T22:08:23.226 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.8636000000000001e-05 2024-09-21T22:08:23.226 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.226 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.226 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:23.226 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.343179+0000", 2024-09-21T22:08:23.226 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.052866199000000003 2024-09-21T22:08:23.227 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.227 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.227 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:23.227 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.343250+0000", 2024-09-21T22:08:23.227 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.1239e-05 2024-09-21T22:08:23.227 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:23.227 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:23.227 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:23.227 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.227 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.227 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:702 2.5 e63/37 2:a70ae99f:::rbd_data.126483d75f2d.000000000000000c:head v 63'34, mlcod=63'32)", 2024-09-21T22:08:23.227 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.392532+0000", 2024-09-21T22:08:23.227 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 57.816999287000002, 2024-09-21T22:08:23.227 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.035703990999999997, 2024-09-21T22:08:23.227 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:23.228 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:23.228 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:23.228 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:23.229 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.229 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:23.229 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.392532+0000", 2024-09-21T22:08:23.230 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:23.230 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.230 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.230 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:23.230 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.392532+0000", 2024-09-21T22:08:23.230 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:23.230 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.230 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.230 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:23.230 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.392534+0000", 2024-09-21T22:08:23.230 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.64e-06 2024-09-21T22:08:23.230 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.230 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.231 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:23.231 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.393321+0000", 2024-09-21T22:08:23.231 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00078705699999999997 2024-09-21T22:08:23.231 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.231 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.231 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:23.231 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.393347+0000", 2024-09-21T22:08:23.231 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.6735999999999999e-05 2024-09-21T22:08:23.231 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.231 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.231 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:23.231 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.393361+0000", 2024-09-21T22:08:23.231 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3346000000000001e-05 2024-09-21T22:08:23.231 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.232 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.232 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:23.232 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.393469+0000", 2024-09-21T22:08:23.232 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000108039 2024-09-21T22:08:23.232 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.232 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.232 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:23.232 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.393512+0000", 2024-09-21T22:08:23.232 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.2862999999999999e-05 2024-09-21T22:08:23.232 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.232 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.232 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:23.232 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.428170+0000", 2024-09-21T22:08:23.232 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.034658858000000001 2024-09-21T22:08:23.232 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.233 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.233 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:23.233 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.428236+0000", 2024-09-21T22:08:23.233 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.5451999999999996e-05 2024-09-21T22:08:23.233 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:23.233 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:23.233 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:23.233 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.233 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.233 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:708 2.1 e63/53 2:837ddd83:::rbd_data.126483d75f2d.000000000000000d:head v 63'6, mlcod=57'4)", 2024-09-21T22:08:23.233 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.517007+0000", 2024-09-21T22:08:23.233 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 57.692524360999997, 2024-09-21T22:08:23.233 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.032495584000000001, 2024-09-21T22:08:23.233 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:23.234 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:23.234 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:23.234 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:23.234 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.234 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:23.234 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.517007+0000", 2024-09-21T22:08:23.234 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:23.234 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.234 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.234 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:23.234 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.517007+0000", 2024-09-21T22:08:23.234 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:23.234 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.235 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.235 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:23.235 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.517008+0000", 2024-09-21T22:08:23.235 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.604e-06 2024-09-21T22:08:23.235 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.235 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.235 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:23.235 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.517603+0000", 2024-09-21T22:08:23.235 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00059461900000000001 2024-09-21T22:08:23.235 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.235 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.235 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:23.235 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.517632+0000", 2024-09-21T22:08:23.235 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8663e-05 2024-09-21T22:08:23.236 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.236 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.236 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:23.236 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.517647+0000", 2024-09-21T22:08:23.236 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4833e-05 2024-09-21T22:08:23.236 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.236 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.236 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:23.236 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.517714+0000", 2024-09-21T22:08:23.236 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.7669000000000006e-05 2024-09-21T22:08:23.236 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.236 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.236 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:23.236 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.517737+0000", 2024-09-21T22:08:23.237 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.2469000000000001e-05 2024-09-21T22:08:23.237 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.237 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.237 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:23.237 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.549370+0000", 2024-09-21T22:08:23.237 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.031633221000000003 2024-09-21T22:08:23.237 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.237 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.237 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:23.237 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.549502+0000", 2024-09-21T22:08:23.237 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013250600000000001 2024-09-21T22:08:23.237 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:23.237 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:23.237 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:23.238 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.238 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.238 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:712 2.1 e63/53 2:837ddd83:::rbd_data.126483d75f2d.000000000000000d:head v 63'10, mlcod=63'8)", 2024-09-21T22:08:23.238 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.604029+0000", 2024-09-21T22:08:23.238 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 57.605502422000001, 2024-09-21T22:08:23.238 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.037982371000000001, 2024-09-21T22:08:23.238 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:23.238 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:23.238 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:23.238 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:23.238 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.238 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:23.238 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.604029+0000", 2024-09-21T22:08:23.238 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:23.239 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.239 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.239 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:23.239 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.604029+0000", 2024-09-21T22:08:23.239 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:23.239 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.239 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.239 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:23.239 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.604031+0000", 2024-09-21T22:08:23.239 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8870000000000001e-06 2024-09-21T22:08:23.239 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.239 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.239 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:23.239 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.604395+0000", 2024-09-21T22:08:23.240 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00036422000000000001 2024-09-21T22:08:23.240 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.240 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.240 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:23.240 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.604424+0000", 2024-09-21T22:08:23.240 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8609e-05 2024-09-21T22:08:23.240 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.240 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.240 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:23.240 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.604437+0000", 2024-09-21T22:08:23.240 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3086e-05 2024-09-21T22:08:23.240 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.240 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.240 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:23.241 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.604509+0000", 2024-09-21T22:08:23.241 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.2448999999999997e-05 2024-09-21T22:08:23.241 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.241 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.241 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:23.241 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.604531+0000", 2024-09-21T22:08:23.241 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1602000000000001e-05 2024-09-21T22:08:23.241 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.241 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.241 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:23.241 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.641885+0000", 2024-09-21T22:08:23.241 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.037354545000000003 2024-09-21T22:08:23.241 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.241 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.242 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:23.242 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.642011+0000", 2024-09-21T22:08:23.242 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00012597300000000001 2024-09-21T22:08:23.242 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:23.242 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:23.242 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:23.242 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.242 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.242 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:720 2.1 e63/53 2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head v 63'18, mlcod=63'16)", 2024-09-21T22:08:23.242 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.805111+0000", 2024-09-21T22:08:23.242 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 57.404420362000003, 2024-09-21T22:08:23.243 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.046082864000000001, 2024-09-21T22:08:23.243 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:23.243 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:23.243 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:23.243 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:23.243 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.243 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:23.243 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.805111+0000", 2024-09-21T22:08:23.243 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:23.243 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.243 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.243 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:23.243 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.805111+0000", 2024-09-21T22:08:23.244 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:23.244 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.244 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.244 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:23.244 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.805112+0000", 2024-09-21T22:08:23.244 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.02e-06 2024-09-21T22:08:23.244 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.244 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.244 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:23.244 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.805429+0000", 2024-09-21T22:08:23.244 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00031740799999999998 2024-09-21T22:08:23.244 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.244 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.244 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:23.245 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.805459+0000", 2024-09-21T22:08:23.245 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.955e-05 2024-09-21T22:08:23.245 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.245 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.245 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:23.245 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.805472+0000", 2024-09-21T22:08:23.245 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3533000000000001e-05 2024-09-21T22:08:23.245 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.245 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.245 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:23.245 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.805581+0000", 2024-09-21T22:08:23.245 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000108194 2024-09-21T22:08:23.245 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.245 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.245 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:23.246 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.805615+0000", 2024-09-21T22:08:23.246 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.4310000000000002e-05 2024-09-21T22:08:23.246 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.246 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.246 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:23.246 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.851146+0000", 2024-09-21T22:08:23.246 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.045531092000000002 2024-09-21T22:08:23.246 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.246 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.246 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:23.246 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.851194+0000", 2024-09-21T22:08:23.246 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.7756999999999997e-05 2024-09-21T22:08:23.246 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:23.247 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:23.247 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:23.247 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.247 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.247 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:723 2.1 e63/53 2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head v 63'20, mlcod=63'18)", 2024-09-21T22:08:23.247 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.862341+0000", 2024-09-21T22:08:23.247 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 57.347190427999998, 2024-09-21T22:08:23.247 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.036454671000000001, 2024-09-21T22:08:23.247 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:23.247 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:23.247 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:23.247 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:23.247 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.247 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:23.248 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.862341+0000", 2024-09-21T22:08:23.248 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:23.248 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.248 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.248 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:23.248 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.862341+0000", 2024-09-21T22:08:23.248 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:23.248 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.248 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.248 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:23.248 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.862342+0000", 2024-09-21T22:08:23.248 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.0299999999999997e-07 2024-09-21T22:08:23.248 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.248 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.248 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:23.249 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.862721+0000", 2024-09-21T22:08:23.249 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00037889800000000002 2024-09-21T22:08:23.249 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.249 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.249 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:23.249 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.862738+0000", 2024-09-21T22:08:23.249 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6957999999999999e-05 2024-09-21T22:08:23.249 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.249 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.249 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:23.249 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.862747+0000", 2024-09-21T22:08:23.249 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.0729999999999994e-06 2024-09-21T22:08:23.249 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.249 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.250 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:23.250 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.862830+0000", 2024-09-21T22:08:23.250 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.3113000000000002e-05 2024-09-21T22:08:23.250 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.250 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.250 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:23.250 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.862860+0000", 2024-09-21T22:08:23.250 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9980000000000001e-05 2024-09-21T22:08:23.250 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.250 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.250 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:23.251 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.898705+0000", 2024-09-21T22:08:23.251 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.035845063000000003 2024-09-21T22:08:23.251 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.251 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.251 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:23.251 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.898795+0000", 2024-09-21T22:08:23.251 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.0682999999999999e-05 2024-09-21T22:08:23.251 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:23.251 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:23.251 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:23.251 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.251 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.251 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:136 2.5 e72/37 2:b1d55633:::rbd_data.126483d75f2d.000000000000000b:head v 72'44, mlcod=72'42)", 2024-09-21T22:08:23.251 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.386273+0000", 2024-09-21T22:08:23.252 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 44.823258312999997, 2024-09-21T22:08:23.252 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.033806744999999999, 2024-09-21T22:08:23.252 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:23.252 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:23.252 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:23.252 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:23.252 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.252 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:23.252 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.386273+0000", 2024-09-21T22:08:23.252 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:23.252 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.252 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.252 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:23.252 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.386273+0000", 2024-09-21T22:08:23.253 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:23.253 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.253 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.253 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:23.253 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.386274+0000", 2024-09-21T22:08:23.253 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.8299999999999996e-07 2024-09-21T22:08:23.253 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.253 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.253 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:23.253 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.387108+0000", 2024-09-21T22:08:23.253 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00083467800000000005 2024-09-21T22:08:23.253 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.253 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.253 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:23.253 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.387121+0000", 2024-09-21T22:08:23.254 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2823000000000001e-05 2024-09-21T22:08:23.254 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.254 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.254 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:23.254 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.387128+0000", 2024-09-21T22:08:23.254 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.404e-06 2024-09-21T22:08:23.254 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.254 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.254 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:23.254 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.387179+0000", 2024-09-21T22:08:23.254 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.1888000000000002e-05 2024-09-21T22:08:23.254 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.254 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.254 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:23.255 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.387201+0000", 2024-09-21T22:08:23.255 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.2019999999999999e-05 2024-09-21T22:08:23.255 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.255 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.255 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:23.255 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.419993+0000", 2024-09-21T22:08:23.255 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.032791182000000002 2024-09-21T22:08:23.255 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.255 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.255 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:23.255 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.420080+0000", 2024-09-21T22:08:23.255 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.7066999999999998e-05 2024-09-21T22:08:23.255 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:23.255 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:23.256 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:23.256 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.256 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.256 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:138 2.5 e72/37 2:b1d55633:::rbd_data.126483d75f2d.000000000000000b:head v 72'46, mlcod=72'44)", 2024-09-21T22:08:23.256 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.430104+0000", 2024-09-21T22:08:23.256 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 44.779426964999999, 2024-09-21T22:08:23.256 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.034087458000000001, 2024-09-21T22:08:23.256 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:23.256 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:23.256 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:23.256 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:23.256 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.256 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:23.256 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.430104+0000", 2024-09-21T22:08:23.257 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:23.257 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.257 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.257 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:23.257 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.430104+0000", 2024-09-21T22:08:23.257 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:23.257 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.257 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.257 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:23.257 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.430105+0000", 2024-09-21T22:08:23.257 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.1680000000000001e-06 2024-09-21T22:08:23.257 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.257 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.257 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:23.258 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.430864+0000", 2024-09-21T22:08:23.258 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00075819400000000001 2024-09-21T22:08:23.258 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.258 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.258 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:23.258 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.430881+0000", 2024-09-21T22:08:23.258 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7000999999999999e-05 2024-09-21T22:08:23.258 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.258 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.258 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:23.258 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.430889+0000", 2024-09-21T22:08:23.258 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.9540000000000004e-06 2024-09-21T22:08:23.258 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.258 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.259 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:23.259 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.430959+0000", 2024-09-21T22:08:23.259 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.0858999999999994e-05 2024-09-21T22:08:23.259 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.259 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.259 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:23.259 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.430996+0000", 2024-09-21T22:08:23.259 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.6443999999999999e-05 2024-09-21T22:08:23.259 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.259 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.259 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:23.259 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.464087+0000", 2024-09-21T22:08:23.259 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.033090799999999997 2024-09-21T22:08:23.259 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.260 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.260 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:23.260 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.464192+0000", 2024-09-21T22:08:23.260 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00010503800000000001 2024-09-21T22:08:23.260 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:23.260 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:23.260 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:23.260 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.260 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.260 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:140 2.5 e72/37 2:b1d55633:::rbd_data.126483d75f2d.000000000000000b:head v 72'48, mlcod=72'46)", 2024-09-21T22:08:23.260 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.475737+0000", 2024-09-21T22:08:23.261 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 44.733794017000001, 2024-09-21T22:08:23.261 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.037632025, 2024-09-21T22:08:23.261 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:23.261 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:23.261 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:23.261 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:23.261 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.261 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:23.261 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.475737+0000", 2024-09-21T22:08:23.261 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:23.261 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.261 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.261 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:23.261 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.475737+0000", 2024-09-21T22:08:23.262 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:23.262 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.262 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.262 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:23.262 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.475739+0000", 2024-09-21T22:08:23.262 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.666e-06 2024-09-21T22:08:23.262 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.262 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.262 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:23.262 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.476538+0000", 2024-09-21T22:08:23.262 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00079927399999999999 2024-09-21T22:08:23.262 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.262 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.262 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:23.263 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.476556+0000", 2024-09-21T22:08:23.263 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7666999999999999e-05 2024-09-21T22:08:23.263 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.263 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.263 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:23.263 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.476565+0000", 2024-09-21T22:08:23.263 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.0650000000000005e-06 2024-09-21T22:08:23.263 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.263 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.263 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:23.263 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.476651+0000", 2024-09-21T22:08:23.263 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.5916999999999997e-05 2024-09-21T22:08:23.263 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.263 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.264 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:23.264 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.476696+0000", 2024-09-21T22:08:23.264 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.5537000000000001e-05 2024-09-21T22:08:23.264 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.264 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.264 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:23.264 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.513300+0000", 2024-09-21T22:08:23.264 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.036603498999999998 2024-09-21T22:08:23.264 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.264 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.264 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:23.264 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.513369+0000", 2024-09-21T22:08:23.264 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.9400000000000006e-05 2024-09-21T22:08:23.264 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:23.265 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:23.265 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:23.265 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.265 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.265 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:145 2.5 e72/37 2:a70ae99f:::rbd_data.126483d75f2d.000000000000000c:head v 72'52, mlcod=72'50)", 2024-09-21T22:08:23.265 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.578171+0000", 2024-09-21T22:08:23.265 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 44.631360166999997, 2024-09-21T22:08:23.265 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.037181537000000001, 2024-09-21T22:08:23.265 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:23.265 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:23.265 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:23.265 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:23.265 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.265 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:23.266 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.578171+0000", 2024-09-21T22:08:23.266 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:23.266 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.266 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.266 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:23.266 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.578171+0000", 2024-09-21T22:08:23.266 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:23.266 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.266 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.266 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:23.267 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.578172+0000", 2024-09-21T22:08:23.267 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.017e-06 2024-09-21T22:08:23.267 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.267 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.267 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:23.267 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.579000+0000", 2024-09-21T22:08:23.267 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00082799499999999999 2024-09-21T22:08:23.267 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.267 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.267 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:23.267 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.579020+0000", 2024-09-21T22:08:23.267 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.0165000000000001e-05 2024-09-21T22:08:23.267 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.267 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.268 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:23.268 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.579036+0000", 2024-09-21T22:08:23.268 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5767e-05 2024-09-21T22:08:23.268 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.268 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.268 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:23.268 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.579106+0000", 2024-09-21T22:08:23.268 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.0091000000000004e-05 2024-09-21T22:08:23.268 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.268 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.268 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:23.268 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.579131+0000", 2024-09-21T22:08:23.268 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4978e-05 2024-09-21T22:08:23.268 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.268 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.269 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:23.269 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.615249+0000", 2024-09-21T22:08:23.269 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.036118355999999997 2024-09-21T22:08:23.269 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.269 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.269 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:23.269 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.615353+0000", 2024-09-21T22:08:23.269 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000103168 2024-09-21T22:08:23.269 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:23.269 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:23.269 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:23.269 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.269 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.269 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:147 2.5 e72/37 2:a70ae99f:::rbd_data.126483d75f2d.000000000000000c:head v 72'54, mlcod=72'52)", 2024-09-21T22:08:23.270 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.627203+0000", 2024-09-21T22:08:23.270 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 44.582328054999998, 2024-09-21T22:08:23.270 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.038203972000000003, 2024-09-21T22:08:23.270 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:23.270 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:23.270 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:23.270 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:23.270 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.270 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:23.270 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.627203+0000", 2024-09-21T22:08:23.270 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:23.270 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.270 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.270 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:23.271 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.627203+0000", 2024-09-21T22:08:23.271 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:23.271 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.271 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.271 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:23.271 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.627204+0000", 2024-09-21T22:08:23.271 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3069999999999999e-06 2024-09-21T22:08:23.271 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.271 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.271 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:23.271 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.628114+0000", 2024-09-21T22:08:23.271 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00090942099999999999 2024-09-21T22:08:23.271 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.271 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.272 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:23.272 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.628140+0000", 2024-09-21T22:08:23.272 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5882000000000001e-05 2024-09-21T22:08:23.272 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.272 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.272 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:23.272 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.628162+0000", 2024-09-21T22:08:23.272 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.2293000000000002e-05 2024-09-21T22:08:23.272 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.272 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.272 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:23.272 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.628250+0000", 2024-09-21T22:08:23.272 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.7973000000000006e-05 2024-09-21T22:08:23.272 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.273 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.273 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:23.273 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.628272+0000", 2024-09-21T22:08:23.273 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1682999999999999e-05 2024-09-21T22:08:23.273 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.273 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.273 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:23.273 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.665322+0000", 2024-09-21T22:08:23.273 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.037050650999999997 2024-09-21T22:08:23.273 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.273 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.273 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:23.273 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.665407+0000", 2024-09-21T22:08:23.273 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.4761999999999999e-05 2024-09-21T22:08:23.274 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:23.274 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:23.274 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:23.274 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.274 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.274 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:149 2.5 e72/37 2:a70ae99f:::rbd_data.126483d75f2d.000000000000000c:head v 72'56, mlcod=72'54)", 2024-09-21T22:08:23.274 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.690377+0000", 2024-09-21T22:08:23.274 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 44.519154311999998, 2024-09-21T22:08:23.274 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.049254799000000002, 2024-09-21T22:08:23.274 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:23.274 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:23.274 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:23.274 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:23.274 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.275 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:23.275 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.690377+0000", 2024-09-21T22:08:23.275 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:23.275 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.275 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.275 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:23.275 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.690377+0000", 2024-09-21T22:08:23.275 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:23.275 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.275 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.275 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:23.275 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.690378+0000", 2024-09-21T22:08:23.275 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.344e-06 2024-09-21T22:08:23.275 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.276 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.276 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:23.276 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.691302+0000", 2024-09-21T22:08:23.276 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00092322400000000005 2024-09-21T22:08:23.276 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.276 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.276 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:23.276 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.691321+0000", 2024-09-21T22:08:23.276 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9928999999999998e-05 2024-09-21T22:08:23.276 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.276 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.276 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:23.276 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.691345+0000", 2024-09-21T22:08:23.276 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.372e-05 2024-09-21T22:08:23.277 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.277 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.277 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:23.277 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.691442+0000", 2024-09-21T22:08:23.277 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.7270999999999998e-05 2024-09-21T22:08:23.277 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.277 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.277 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:23.277 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.691512+0000", 2024-09-21T22:08:23.277 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.9183000000000005e-05 2024-09-21T22:08:23.277 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.277 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.277 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:23.277 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.739545+0000", 2024-09-21T22:08:23.278 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.048033657 2024-09-21T22:08:23.278 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.278 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.278 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:23.278 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.739632+0000", 2024-09-21T22:08:23.278 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.6471000000000006e-05 2024-09-21T22:08:23.278 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:23.278 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:23.278 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:23.278 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.278 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.278 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:156 2.1 e72/53 2:837ddd83:::rbd_data.126483d75f2d.000000000000000d:head v 72'28, mlcod=72'26)", 2024-09-21T22:08:23.278 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.824856+0000", 2024-09-21T22:08:23.278 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 44.384674873999998, 2024-09-21T22:08:23.279 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.034320985999999998, 2024-09-21T22:08:23.279 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:23.279 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:23.279 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:23.279 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:23.279 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.279 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:23.279 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.824856+0000", 2024-09-21T22:08:23.279 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:23.279 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.279 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.279 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:23.279 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.824856+0000", 2024-09-21T22:08:23.279 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:23.280 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.280 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.280 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:23.280 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.824858+0000", 2024-09-21T22:08:23.280 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2979999999999999e-06 2024-09-21T22:08:23.280 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.280 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.280 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:23.280 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.825174+0000", 2024-09-21T22:08:23.280 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000316335 2024-09-21T22:08:23.280 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.280 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.280 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:23.280 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.825193+0000", 2024-09-21T22:08:23.281 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9436000000000002e-05 2024-09-21T22:08:23.281 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.281 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.281 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:23.281 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.825202+0000", 2024-09-21T22:08:23.281 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.8359999999999998e-06 2024-09-21T22:08:23.281 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.281 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.282 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:23.282 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.825286+0000", 2024-09-21T22:08:23.282 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.3683999999999994e-05 2024-09-21T22:08:23.282 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.282 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.282 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:23.282 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.825314+0000", 2024-09-21T22:08:23.282 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.7653000000000001e-05 2024-09-21T22:08:23.282 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.282 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.282 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:23.282 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.859122+0000", 2024-09-21T22:08:23.282 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.033808624000000002 2024-09-21T22:08:23.282 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.283 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.283 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:23.283 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.859177+0000", 2024-09-21T22:08:23.283 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.5120000000000001e-05 2024-09-21T22:08:23.283 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:23.283 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:23.283 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:23.283 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.283 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.283 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:158 2.1 e72/53 2:837ddd83:::rbd_data.126483d75f2d.000000000000000d:head v 72'30, mlcod=72'28)", 2024-09-21T22:08:23.283 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.869277+0000", 2024-09-21T22:08:23.283 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 44.340254676999997, 2024-09-21T22:08:23.283 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.040508390999999998, 2024-09-21T22:08:23.283 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:23.284 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:23.284 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:23.284 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:23.284 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.284 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:23.284 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.869277+0000", 2024-09-21T22:08:23.284 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:23.284 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.284 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.284 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:23.284 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.869277+0000", 2024-09-21T22:08:23.284 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:23.284 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.284 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.285 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:23.285 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.869278+0000", 2024-09-21T22:08:23.285 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5269999999999999e-06 2024-09-21T22:08:23.285 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.285 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.285 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:23.285 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.869902+0000", 2024-09-21T22:08:23.285 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00062399000000000005 2024-09-21T22:08:23.285 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.285 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.285 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:23.285 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.869924+0000", 2024-09-21T22:08:23.285 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1875999999999999e-05 2024-09-21T22:08:23.285 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.285 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.286 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:23.286 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.869937+0000", 2024-09-21T22:08:23.286 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2685e-05 2024-09-21T22:08:23.286 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.286 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.286 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:23.286 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.870058+0000", 2024-09-21T22:08:23.286 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000120862 2024-09-21T22:08:23.286 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.286 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.286 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:23.286 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.870093+0000", 2024-09-21T22:08:23.286 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.5896000000000003e-05 2024-09-21T22:08:23.287 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.287 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.287 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:23.287 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.909728+0000", 2024-09-21T22:08:23.287 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.039634461000000003 2024-09-21T22:08:23.287 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.287 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.287 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:23.287 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.909785+0000", 2024-09-21T22:08:23.287 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.7093999999999999e-05 2024-09-21T22:08:23.287 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:23.287 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:23.287 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:23.287 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.288 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.288 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:161 2.1 e72/53 2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head v 72'32, mlcod=72'30)", 2024-09-21T22:08:23.288 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.924455+0000", 2024-09-21T22:08:23.288 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 44.285076314999998, 2024-09-21T22:08:23.288 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.038366863000000001, 2024-09-21T22:08:23.288 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:23.288 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:23.288 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:23.288 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:23.288 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.288 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:23.288 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.924455+0000", 2024-09-21T22:08:23.288 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:23.288 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.289 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.289 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:23.289 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.924455+0000", 2024-09-21T22:08:23.289 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:23.289 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.289 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.289 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:23.289 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.924456+0000", 2024-09-21T22:08:23.289 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.003e-06 2024-09-21T22:08:23.289 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.289 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.289 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:23.289 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.924811+0000", 2024-09-21T22:08:23.289 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00035519499999999999 2024-09-21T22:08:23.290 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.290 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.290 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:23.290 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.924826+0000", 2024-09-21T22:08:23.290 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5296999999999999e-05 2024-09-21T22:08:23.290 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.290 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.290 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:23.290 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.924834+0000", 2024-09-21T22:08:23.290 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.5379999999999997e-06 2024-09-21T22:08:23.290 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.290 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.290 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:23.290 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.924910+0000", 2024-09-21T22:08:23.291 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.6488000000000004e-05 2024-09-21T22:08:23.291 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.291 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.291 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:23.291 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.924934+0000", 2024-09-21T22:08:23.291 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3598999999999999e-05 2024-09-21T22:08:23.291 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.291 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.291 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:23.291 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.962739+0000", 2024-09-21T22:08:23.291 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.037805370999999997 2024-09-21T22:08:23.291 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.291 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.291 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:23.292 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.962822+0000", 2024-09-21T22:08:23.292 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.2372000000000003e-05 2024-09-21T22:08:23.292 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:23.292 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:23.292 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:23.292 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.292 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.292 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:163 2.1 e72/53 2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head v 72'34, mlcod=72'32)", 2024-09-21T22:08:23.292 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.971683+0000", 2024-09-21T22:08:23.292 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 44.237847774999999, 2024-09-21T22:08:23.292 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.043171320999999999, 2024-09-21T22:08:23.292 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:23.292 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:23.292 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:23.292 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:23.293 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.293 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:23.293 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.971683+0000", 2024-09-21T22:08:23.293 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:23.293 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.293 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.293 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:23.293 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.971683+0000", 2024-09-21T22:08:23.293 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:23.293 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.293 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.293 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:23.293 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.971684+0000", 2024-09-21T22:08:23.293 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.6600000000000005e-07 2024-09-21T22:08:23.294 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.294 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.294 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:23.294 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.971945+0000", 2024-09-21T22:08:23.294 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00026101899999999998 2024-09-21T22:08:23.294 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.294 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.294 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:23.294 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.971959+0000", 2024-09-21T22:08:23.294 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4054e-05 2024-09-21T22:08:23.294 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.294 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.294 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:23.295 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.971969+0000", 2024-09-21T22:08:23.295 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.499e-06 2024-09-21T22:08:23.295 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.295 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.295 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:23.295 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.972037+0000", 2024-09-21T22:08:23.295 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.8448000000000006e-05 2024-09-21T22:08:23.295 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.295 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.295 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:23.295 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.972057+0000", 2024-09-21T22:08:23.295 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9817999999999999e-05 2024-09-21T22:08:23.295 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.295 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.295 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:23.296 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.014770+0000", 2024-09-21T22:08:23.296 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.042712464999999998 2024-09-21T22:08:23.296 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.296 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.296 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:23.296 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.014855+0000", 2024-09-21T22:08:23.296 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.5152000000000001e-05 2024-09-21T22:08:23.296 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:23.296 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:23.296 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:23.296 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.296 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.296 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:165 2.1 e72/53 2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head v 72'36, mlcod=72'34)", 2024-09-21T22:08:23.296 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:39.028631+0000", 2024-09-21T22:08:23.297 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 44.180900297999997, 2024-09-21T22:08:23.297 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.033430561999999997, 2024-09-21T22:08:23.297 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:23.297 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:23.297 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:23.297 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:23.297 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.297 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:23.297 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.028631+0000", 2024-09-21T22:08:23.297 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:23.297 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.297 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.297 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:23.297 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.028631+0000", 2024-09-21T22:08:23.298 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:23.298 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.298 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.298 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:23.298 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.028633+0000", 2024-09-21T22:08:23.298 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5799999999999999e-06 2024-09-21T22:08:23.298 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.298 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.298 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:23.298 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.029209+0000", 2024-09-21T22:08:23.298 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00057670799999999999 2024-09-21T22:08:23.298 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.298 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.298 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:23.299 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.029238+0000", 2024-09-21T22:08:23.299 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8943000000000001e-05 2024-09-21T22:08:23.299 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.299 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.299 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:23.299 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.029252+0000", 2024-09-21T22:08:23.299 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3672999999999999e-05 2024-09-21T22:08:23.299 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.299 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.299 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:23.299 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.029328+0000", 2024-09-21T22:08:23.299 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.6497999999999998e-05 2024-09-21T22:08:23.299 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.299 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.300 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:23.300 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.029353+0000", 2024-09-21T22:08:23.300 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5060999999999999e-05 2024-09-21T22:08:23.300 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.300 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.300 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:23.300 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.061993+0000", 2024-09-21T22:08:23.300 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.032639286000000003 2024-09-21T22:08:23.300 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.300 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.300 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:23.300 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.062062+0000", 2024-09-21T22:08:23.300 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.8813000000000007e-05 2024-09-21T22:08:23.301 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:23.301 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:23.301 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:23.301 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.301 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.301 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:167 2.1 e72/53 2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head v 72'38, mlcod=72'36)", 2024-09-21T22:08:23.301 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:39.077324+0000", 2024-09-21T22:08:23.301 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 44.132207483999998, 2024-09-21T22:08:23.301 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.036031961000000001, 2024-09-21T22:08:23.301 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:23.302 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:23.302 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:23.302 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:23.302 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.302 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:23.302 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.077324+0000", 2024-09-21T22:08:23.302 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:23.302 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.302 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.302 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:23.302 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.077324+0000", 2024-09-21T22:08:23.302 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:23.302 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.302 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.303 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:23.303 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.077326+0000", 2024-09-21T22:08:23.303 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.75e-06 2024-09-21T22:08:23.303 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.303 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.303 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:23.303 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.077719+0000", 2024-09-21T22:08:23.303 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00039322599999999999 2024-09-21T22:08:23.303 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.303 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.303 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:23.303 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.077749+0000", 2024-09-21T22:08:23.303 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9895000000000001e-05 2024-09-21T22:08:23.303 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.304 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.304 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:23.304 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.077763+0000", 2024-09-21T22:08:23.304 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4517e-05 2024-09-21T22:08:23.304 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.304 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.304 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:23.304 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.077843+0000", 2024-09-21T22:08:23.304 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.9994999999999996e-05 2024-09-21T22:08:23.304 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.304 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.304 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:23.304 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.077867+0000", 2024-09-21T22:08:23.304 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3640000000000001e-05 2024-09-21T22:08:23.304 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.305 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.305 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:23.305 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.113216+0000", 2024-09-21T22:08:23.305 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.035349168 2024-09-21T22:08:23.305 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.305 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.305 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:23.305 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.113356+0000", 2024-09-21T22:08:23.305 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013977 2024-09-21T22:08:23.305 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:23.305 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:23.305 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:23.305 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:23.305 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:23.305 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:08:23.306 INFO:teuthology.orchestra.run.smithi039.stdout:osd.2: {} 2024-09-21T22:08:23.307 INFO:teuthology.orchestra.run.smithi039.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T22:08:23.308 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T22:08:23.308 INFO:teuthology.orchestra.run.smithi039.stdout:osd.3: {} 2024-09-21T22:08:23.308 INFO:teuthology.orchestra.run.smithi039.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T22:08:23.308 INFO:teuthology.orchestra.run.smithi039.stdout:osd.4: {} 2024-09-21T22:08:23.309 INFO:teuthology.orchestra.run.smithi039.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T22:08:23.309 INFO:teuthology.orchestra.run.smithi039.stdout:osd.5: {} 2024-09-21T22:08:23.310 INFO:teuthology.orchestra.run.smithi039.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T22:08:23.310 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:08:23.301+0000 7fe38af2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 34631) UID: 0 2024-09-21T22:08:23.408 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:08:23.405+0000 7fe38af2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 34631) UID: 0 2024-09-21T22:08:23.477 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:08:23.477 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:08:23.478 INFO:teuthology.orchestra.run.smithi039.stdout: "num_ops": 0 2024-09-21T22:08:23.478 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:08:23.491 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T22:08:23.509 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:08:23.507+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:08:23.577 INFO:tasks.workunit.client.0.smithi157.stderr:+ DEV= 2024-09-21T22:08:23.578 INFO:tasks.workunit.client.0.smithi157.stderr:++ ceph-conf --show-config-value librados_thread_count 2024-09-21T22:08:23.603 INFO:tasks.workunit.client.0.smithi157.stderr:+ EXPECTED=2 2024-09-21T22:08:23.603 INFO:tasks.workunit.client.0.smithi157.stderr:++ _sudo rbd device --device-type nbd map rbd/testrbdnbd21553 2024-09-21T22:08:23.604 INFO:tasks.workunit.client.0.smithi157.stderr:++ local cmd 2024-09-21T22:08:23.604 INFO:tasks.workunit.client.0.smithi157.stderr:+++ id -u 2024-09-21T22:08:23.606 INFO:tasks.workunit.client.0.smithi157.stderr:++ '[' 1000 -eq 0 ']' 2024-09-21T22:08:23.606 INFO:tasks.workunit.client.0.smithi157.stderr:+++ which rbd 2024-09-21T22:08:23.607 INFO:tasks.workunit.client.0.smithi157.stderr:++ cmd=/usr/bin/rbd 2024-09-21T22:08:23.608 INFO:tasks.workunit.client.0.smithi157.stderr:++ shift 2024-09-21T22:08:23.608 INFO:tasks.workunit.client.0.smithi157.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map rbd/testrbdnbd21553 2024-09-21T22:08:23.610 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:08:23.607+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:08:23.644 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:08:23.644 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:08:23.645 INFO:teuthology.orchestra.run.smithi039.stdout: "complaint_time": 30, 2024-09-21T22:08:23.645 INFO:teuthology.orchestra.run.smithi039.stdout: "num_blocked_ops": 0 2024-09-21T22:08:23.645 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:08:23.657 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T22:08:23.710 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:08:23.707+0000 7f2075a06640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31720) UID: 0 2024-09-21T22:08:23.762 INFO:tasks.workunit.client.0.smithi157.stderr:+ DEV=/dev/nbd2 2024-09-21T22:08:23.762 INFO:tasks.workunit.client.0.smithi157.stderr:+ get_pid rbd 2024-09-21T22:08:23.762 INFO:tasks.workunit.client.0.smithi157.stderr:+ local pool=rbd 2024-09-21T22:08:23.762 INFO:tasks.workunit.client.0.smithi157.stderr:+ local ns= 2024-09-21T22:08:23.763 INFO:tasks.workunit.client.0.smithi157.stderr:++ rbd device --device-type nbd --format xml list 2024-09-21T22:08:23.763 INFO:tasks.workunit.client.0.smithi157.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd21553'\''][device='\''/dev/nbd2'\'']/id' 2024-09-21T22:08:23.773 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:08:23.773 INFO:teuthology.orchestra.run.smithi039.stdout: "size": 20, 2024-09-21T22:08:23.773 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 600, 2024-09-21T22:08:23.773 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [ 2024-09-21T22:08:23.773 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.773 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.3 55/54 [PushOp(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head, version: 52'2, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head@52'2, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T22:08:23.773 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.084379+0000", 2024-09-21T22:08:23.773 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 71.684771855999998, 2024-09-21T22:08:23.774 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.051020674000000002, 2024-09-21T22:08:23.774 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:23.774 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:23.774 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:08:23.774 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:23.774 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.774 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:23.774 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.084379+0000", 2024-09-21T22:08:23.774 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:23.774 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.774 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.774 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:23.774 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.084379+0000", 2024-09-21T22:08:23.775 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:23.775 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.775 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.775 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:23.775 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.084381+0000", 2024-09-21T22:08:23.775 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.73e-06 2024-09-21T22:08:23.775 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.775 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.775 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:23.775 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.085494+0000", 2024-09-21T22:08:23.775 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011131559999999999 2024-09-21T22:08:23.775 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.775 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.775 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:23.776 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.085542+0000", 2024-09-21T22:08:23.776 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8334999999999997e-05 2024-09-21T22:08:23.776 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.776 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.776 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:23.776 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.085591+0000", 2024-09-21T22:08:23.776 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.9165000000000001e-05 2024-09-21T22:08:23.776 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.776 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.776 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:23.776 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.085777+0000", 2024-09-21T22:08:23.776 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000186152 2024-09-21T22:08:23.776 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.777 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.777 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:23.777 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.085796+0000", 2024-09-21T22:08:23.777 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8612999999999999e-05 2024-09-21T22:08:23.777 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.777 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.777 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:23.777 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.135399+0000", 2024-09-21T22:08:23.777 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.049603522999999997 2024-09-21T22:08:23.777 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:23.777 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:23.777 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:23.778 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.778 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.778 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.3 55/54 [PushOp(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head, version: 52'2, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head@52'2, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:08:23.778 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.139554+0000", 2024-09-21T22:08:23.778 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 71.629596753000001, 2024-09-21T22:08:23.778 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.056215389999999997, 2024-09-21T22:08:23.778 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:23.778 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:23.778 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:08:23.778 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:23.778 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.779 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:23.779 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.139554+0000", 2024-09-21T22:08:23.779 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:23.779 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.779 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.779 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:23.779 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.139554+0000", 2024-09-21T22:08:23.779 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:23.779 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.779 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.779 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:23.779 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.139556+0000", 2024-09-21T22:08:23.779 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7999999999999999e-06 2024-09-21T22:08:23.780 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.780 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.780 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:23.780 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.140671+0000", 2024-09-21T22:08:23.780 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001115612 2024-09-21T22:08:23.780 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.780 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.780 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:23.780 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.140720+0000", 2024-09-21T22:08:23.780 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8631999999999998e-05 2024-09-21T22:08:23.780 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.780 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.781 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:23.781 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.140752+0000", 2024-09-21T22:08:23.781 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.2199000000000001e-05 2024-09-21T22:08:23.781 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.781 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.781 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:23.781 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.140900+0000", 2024-09-21T22:08:23.781 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00014786000000000001 2024-09-21T22:08:23.781 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.781 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.781 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:23.781 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.140917+0000", 2024-09-21T22:08:23.782 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6753999999999999e-05 2024-09-21T22:08:23.782 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.782 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.782 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:23.782 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.195769+0000", 2024-09-21T22:08:23.782 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.054852533000000002 2024-09-21T22:08:23.782 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:23.782 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:23.782 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:23.782 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.782 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.783 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.3 55/54 [PushOp(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head, version: 52'2, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head@52'2, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:08:23.783 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.199187+0000", 2024-09-21T22:08:23.783 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 71.569963208999994, 2024-09-21T22:08:23.783 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.051543975999999998, 2024-09-21T22:08:23.783 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:23.783 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:23.783 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:08:23.783 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:23.783 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.783 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:23.783 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.199187+0000", 2024-09-21T22:08:23.783 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:23.784 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.784 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.784 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:23.784 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.199187+0000", 2024-09-21T22:08:23.784 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:23.784 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.784 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.784 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:23.784 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.199189+0000", 2024-09-21T22:08:23.784 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7969999999999999e-06 2024-09-21T22:08:23.784 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.784 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.784 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:23.785 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.200355+0000", 2024-09-21T22:08:23.785 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011660139999999999 2024-09-21T22:08:23.785 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.785 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.785 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:23.785 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.200405+0000", 2024-09-21T22:08:23.785 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.0124000000000003e-05 2024-09-21T22:08:23.785 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.785 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.785 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:23.785 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.200457+0000", 2024-09-21T22:08:23.785 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.1177999999999997e-05 2024-09-21T22:08:23.785 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.786 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.786 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:23.786 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.200635+0000", 2024-09-21T22:08:23.786 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00017841299999999999 2024-09-21T22:08:23.786 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.786 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.786 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:23.786 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.200653+0000", 2024-09-21T22:08:23.786 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7944999999999998e-05 2024-09-21T22:08:23.786 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.786 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.786 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:23.786 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.250731+0000", 2024-09-21T22:08:23.787 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.050078505000000002 2024-09-21T22:08:23.787 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:23.787 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:23.787 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:23.787 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.787 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.787 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.3 55/54 [PushOp(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head, version: 52'3, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head@52'3, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:08:23.787 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.343754+0000", 2024-09-21T22:08:23.787 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 71.425396497999998, 2024-09-21T22:08:23.787 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.063560607000000005, 2024-09-21T22:08:23.787 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:23.787 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:23.787 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:08:23.787 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:23.788 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.788 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:23.788 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.343754+0000", 2024-09-21T22:08:23.788 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:23.788 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.788 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.788 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:23.788 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.343754+0000", 2024-09-21T22:08:23.788 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:23.788 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.788 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.788 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:23.788 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.343756+0000", 2024-09-21T22:08:23.788 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7069999999999999e-06 2024-09-21T22:08:23.789 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.789 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.789 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:23.789 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.344864+0000", 2024-09-21T22:08:23.789 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011085159999999999 2024-09-21T22:08:23.789 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.789 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.789 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:23.789 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.344914+0000", 2024-09-21T22:08:23.789 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.9462000000000002e-05 2024-09-21T22:08:23.789 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.789 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.789 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:23.789 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.344961+0000", 2024-09-21T22:08:23.789 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.7633999999999998e-05 2024-09-21T22:08:23.790 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.790 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.790 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:23.790 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.345171+0000", 2024-09-21T22:08:23.790 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000209652 2024-09-21T22:08:23.790 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.790 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.790 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:23.790 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.345193+0000", 2024-09-21T22:08:23.790 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1985999999999999e-05 2024-09-21T22:08:23.790 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.790 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.790 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:23.790 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.407315+0000", 2024-09-21T22:08:23.791 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.06212165 2024-09-21T22:08:23.791 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:23.791 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:23.791 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:23.791 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.791 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.791 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.3 55/54 [PushOp(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head, version: 52'3, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head@52'3, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:08:23.791 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.410551+0000", 2024-09-21T22:08:23.791 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 71.358599366999997, 2024-09-21T22:08:23.791 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.054517522999999998, 2024-09-21T22:08:23.791 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:23.791 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:23.791 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:08:23.792 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:23.792 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.792 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:23.792 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.410551+0000", 2024-09-21T22:08:23.792 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:23.792 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.792 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.792 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:23.792 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.410551+0000", 2024-09-21T22:08:23.792 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:23.792 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.792 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.792 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:23.792 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.410553+0000", 2024-09-21T22:08:23.792 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5200000000000001e-06 2024-09-21T22:08:23.793 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.793 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.793 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:23.793 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.411747+0000", 2024-09-21T22:08:23.793 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011946089999999999 2024-09-21T22:08:23.793 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.793 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.793 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:23.793 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.411796+0000", 2024-09-21T22:08:23.793 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8717999999999997e-05 2024-09-21T22:08:23.793 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.793 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.793 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:23.793 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.411844+0000", 2024-09-21T22:08:23.794 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8291999999999998e-05 2024-09-21T22:08:23.794 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.794 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.794 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:23.794 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.412066+0000", 2024-09-21T22:08:23.794 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00022115899999999999 2024-09-21T22:08:23.794 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.794 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.794 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:23.794 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.412081+0000", 2024-09-21T22:08:23.794 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5654e-05 2024-09-21T22:08:23.794 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.794 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.795 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:23.795 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.465069+0000", 2024-09-21T22:08:23.795 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.052987570999999997 2024-09-21T22:08:23.795 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:23.795 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:23.795 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:23.795 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.795 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.795 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.3 55/54 [PushOp(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head, version: 52'3, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head@52'3, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 4194304, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:08:23.795 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.468339+0000", 2024-09-21T22:08:23.795 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 71.300811702999994, 2024-09-21T22:08:23.795 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.064010024999999998, 2024-09-21T22:08:23.795 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:23.795 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:23.796 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:08:23.796 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:23.796 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.796 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:23.796 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.468339+0000", 2024-09-21T22:08:23.796 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:23.796 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.796 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.796 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:23.796 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.468339+0000", 2024-09-21T22:08:23.797 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:23.797 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.797 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.797 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:23.797 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.468340+0000", 2024-09-21T22:08:23.797 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4330000000000001e-06 2024-09-21T22:08:23.797 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.798 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.798 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:23.798 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.469484+0000", 2024-09-21T22:08:23.798 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001143535 2024-09-21T22:08:23.798 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.798 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.799 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:23.799 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.469533+0000", 2024-09-21T22:08:23.799 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8980999999999997e-05 2024-09-21T22:08:23.799 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.799 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.800 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:23.800 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.469581+0000", 2024-09-21T22:08:23.800 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8551999999999999e-05 2024-09-21T22:08:23.800 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.800 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.800 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:23.800 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.469804+0000", 2024-09-21T22:08:23.801 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00022287699999999999 2024-09-21T22:08:23.801 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.801 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.801 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:23.801 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.469829+0000", 2024-09-21T22:08:23.801 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4766e-05 2024-09-21T22:08:23.801 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.801 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.801 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:23.802 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.532349+0000", 2024-09-21T22:08:23.802 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.062519880999999999 2024-09-21T22:08:23.802 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:23.802 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:23.802 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:23.802 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.802 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.803 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4840.0:668 2.3 2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:08:23.803 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.409114+0000", 2024-09-21T22:08:23.803 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 59.360036497000003, 2024-09-21T22:08:23.803 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.048703758999999999, 2024-09-21T22:08:23.803 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:23.803 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:23.804 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:23.804 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-21T22:08:23.804 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4840", 2024-09-21T22:08:23.804 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:08:23.804 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 668 2024-09-21T22:08:23.805 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.805 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:23.805 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.805 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:23.805 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.409114+0000", 2024-09-21T22:08:23.805 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:23.805 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.805 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.805 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:23.805 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.409114+0000", 2024-09-21T22:08:23.805 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:23.806 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.806 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.806 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:23.806 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.409115+0000", 2024-09-21T22:08:23.806 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.114e-06 2024-09-21T22:08:23.806 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.806 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.806 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:23.806 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.410308+0000", 2024-09-21T22:08:23.806 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011930949999999999 2024-09-21T22:08:23.806 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.806 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.807 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:23.807 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.410325+0000", 2024-09-21T22:08:23.807 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7088e-05 2024-09-21T22:08:23.807 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.807 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.807 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:23.807 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.410334+0000", 2024-09-21T22:08:23.807 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.2549999999999997e-06 2024-09-21T22:08:23.807 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.807 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.807 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:23.807 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.410393+0000", 2024-09-21T22:08:23.807 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.9327999999999998e-05 2024-09-21T22:08:23.808 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.808 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.808 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:23.808 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.410482+0000", 2024-09-21T22:08:23.808 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.9019999999999998e-05 2024-09-21T22:08:23.808 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.808 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.808 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-21T22:08:23.808 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.410617+0000", 2024-09-21T22:08:23.808 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000135232 2024-09-21T22:08:23.808 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.809 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.809 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:23.809 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.457051+0000", 2024-09-21T22:08:23.809 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.04643415 2024-09-21T22:08:23.809 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.809 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.809 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-21T22:08:23.809 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.457675+0000", 2024-09-21T22:08:23.809 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00062332700000000002 2024-09-21T22:08:23.810 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.810 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.810 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:23.810 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.457735+0000", 2024-09-21T22:08:23.810 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.029e-05 2024-09-21T22:08:23.810 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.810 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.810 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:23.810 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.457818+0000", 2024-09-21T22:08:23.810 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.2860000000000003e-05 2024-09-21T22:08:23.810 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:23.811 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:23.811 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:23.811 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.811 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.811 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4840.0:670 2.3 2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:08:23.811 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.466980+0000", 2024-09-21T22:08:23.811 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 59.302170429, 2024-09-21T22:08:23.811 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.050913930000000003, 2024-09-21T22:08:23.811 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:23.811 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:23.811 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:23.811 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-21T22:08:23.811 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4840", 2024-09-21T22:08:23.812 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:08:23.812 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 670 2024-09-21T22:08:23.812 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.812 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:23.812 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.812 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:23.812 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.466980+0000", 2024-09-21T22:08:23.812 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:23.812 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.812 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.812 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:23.812 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.466980+0000", 2024-09-21T22:08:23.813 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:23.813 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.813 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.813 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:23.813 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.466981+0000", 2024-09-21T22:08:23.813 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0839999999999999e-06 2024-09-21T22:08:23.813 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.813 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.813 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:23.813 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.467815+0000", 2024-09-21T22:08:23.814 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00083330399999999999 2024-09-21T22:08:23.814 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.814 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.814 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:23.814 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.467832+0000", 2024-09-21T22:08:23.814 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7601e-05 2024-09-21T22:08:23.814 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.814 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.814 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:23.814 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.467840+0000", 2024-09-21T22:08:23.814 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.2220000000000002e-06 2024-09-21T22:08:23.814 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.814 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.815 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:23.815 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.467974+0000", 2024-09-21T22:08:23.815 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000133411 2024-09-21T22:08:23.815 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.815 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.815 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:23.815 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.468223+0000", 2024-09-21T22:08:23.815 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00024906400000000001 2024-09-21T22:08:23.815 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.815 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.815 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-21T22:08:23.815 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.468616+0000", 2024-09-21T22:08:23.815 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000393433 2024-09-21T22:08:23.815 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.816 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.816 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:23.816 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.517016+0000", 2024-09-21T22:08:23.816 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.048399708 2024-09-21T22:08:23.816 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.816 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.816 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-21T22:08:23.816 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.517693+0000", 2024-09-21T22:08:23.816 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00067681000000000002 2024-09-21T22:08:23.816 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.816 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.816 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:23.816 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.517789+0000", 2024-09-21T22:08:23.816 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.5778000000000004e-05 2024-09-21T22:08:23.817 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.817 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.817 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:23.817 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.517894+0000", 2024-09-21T22:08:23.817 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000105515 2024-09-21T22:08:23.817 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:23.817 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:23.817 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:23.817 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.817 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.817 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4840.0:672 2.3 2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:08:23.817 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.527127+0000", 2024-09-21T22:08:23.817 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 59.242023967000002, 2024-09-21T22:08:23.818 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.047345602000000001, 2024-09-21T22:08:23.818 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:23.818 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:23.818 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:23.818 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-21T22:08:23.818 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4840", 2024-09-21T22:08:23.818 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:08:23.818 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 672 2024-09-21T22:08:23.818 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.818 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:23.818 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.818 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:23.818 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.527127+0000", 2024-09-21T22:08:23.819 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:23.819 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.819 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.819 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:23.819 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.527127+0000", 2024-09-21T22:08:23.819 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:23.819 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.819 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.819 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:23.819 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.527128+0000", 2024-09-21T22:08:23.819 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2810000000000001e-06 2024-09-21T22:08:23.819 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.819 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.820 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:23.820 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.527962+0000", 2024-09-21T22:08:23.820 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00083445199999999996 2024-09-21T22:08:23.820 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.820 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.820 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:23.820 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.527981+0000", 2024-09-21T22:08:23.820 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9009e-05 2024-09-21T22:08:23.820 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.820 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.820 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:23.821 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.527990+0000", 2024-09-21T22:08:23.821 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.8619999999999992e-06 2024-09-21T22:08:23.821 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.821 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.821 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:23.821 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.528061+0000", 2024-09-21T22:08:23.821 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.0226000000000002e-05 2024-09-21T22:08:23.821 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.821 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.821 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:23.821 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.528156+0000", 2024-09-21T22:08:23.821 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.5724999999999996e-05 2024-09-21T22:08:23.822 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.822 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.822 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-21T22:08:23.822 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.528303+0000", 2024-09-21T22:08:23.822 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000146374 2024-09-21T22:08:23.822 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.822 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.822 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:23.822 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.573772+0000", 2024-09-21T22:08:23.822 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.045469745999999998 2024-09-21T22:08:23.822 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.822 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.822 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-21T22:08:23.822 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.574299+0000", 2024-09-21T22:08:23.823 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00052649200000000004 2024-09-21T22:08:23.823 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.823 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.823 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:23.823 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.574366+0000", 2024-09-21T22:08:23.823 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.6872000000000006e-05 2024-09-21T22:08:23.823 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.823 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.823 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:23.823 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.574472+0000", 2024-09-21T22:08:23.823 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000106563 2024-09-21T22:08:23.823 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:23.823 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:23.823 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:23.824 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.824 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.824 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4840.0:686 2.3 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:08:23.824 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.943561+0000", 2024-09-21T22:08:23.824 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 58.825589690999998, 2024-09-21T22:08:23.824 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.049021493999999999, 2024-09-21T22:08:23.824 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:23.824 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:23.824 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:23.824 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-21T22:08:23.824 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4840", 2024-09-21T22:08:23.824 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:08:23.824 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 686 2024-09-21T22:08:23.825 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.825 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:23.825 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.825 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:23.825 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.943561+0000", 2024-09-21T22:08:23.825 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:23.825 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.825 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.825 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:23.825 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.943561+0000", 2024-09-21T22:08:23.825 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:23.825 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.826 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.826 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:23.826 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.943563+0000", 2024-09-21T22:08:23.826 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9420000000000002e-06 2024-09-21T22:08:23.826 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.826 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.826 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:23.826 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.944546+0000", 2024-09-21T22:08:23.826 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00098357099999999992 2024-09-21T22:08:23.827 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.827 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.827 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:23.827 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.944565+0000", 2024-09-21T22:08:23.827 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8621e-05 2024-09-21T22:08:23.827 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.828 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.828 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:23.828 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.944575+0000", 2024-09-21T22:08:23.828 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.4639999999999995e-06 2024-09-21T22:08:23.828 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.828 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.828 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:23.829 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.944647+0000", 2024-09-21T22:08:23.829 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.2786000000000004e-05 2024-09-21T22:08:23.829 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.829 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.829 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:23.829 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.944746+0000", 2024-09-21T22:08:23.829 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.8196999999999995e-05 2024-09-21T22:08:23.829 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.829 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.829 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-21T22:08:23.829 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.944890+0000", 2024-09-21T22:08:23.829 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000144804 2024-09-21T22:08:23.830 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.830 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.830 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:23.830 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.992404+0000", 2024-09-21T22:08:23.830 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.047513211999999999 2024-09-21T22:08:23.831 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.831 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.831 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-21T22:08:23.831 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.992480+0000", 2024-09-21T22:08:23.831 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.6491999999999999e-05 2024-09-21T22:08:23.831 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.831 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.831 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:23.831 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.992521+0000", 2024-09-21T22:08:23.831 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.0608000000000002e-05 2024-09-21T22:08:23.831 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.831 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.831 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:23.831 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.992582+0000", 2024-09-21T22:08:23.832 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.1797000000000005e-05 2024-09-21T22:08:23.832 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:23.832 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:23.832 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:23.832 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.832 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.832 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4840.0:720 2.1 2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:08:23.832 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.803827+0000", 2024-09-21T22:08:23.832 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 57.965323415, 2024-09-21T22:08:23.832 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.049890510999999998, 2024-09-21T22:08:23.832 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:23.832 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:23.832 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:23.832 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-21T22:08:23.833 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4840", 2024-09-21T22:08:23.833 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:08:23.833 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 720 2024-09-21T22:08:23.833 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.833 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:23.833 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.833 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:23.833 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.803827+0000", 2024-09-21T22:08:23.833 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:23.833 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.833 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.833 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:23.833 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.803827+0000", 2024-09-21T22:08:23.834 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:23.834 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.834 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.834 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:23.834 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.803828+0000", 2024-09-21T22:08:23.834 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.569999999999999e-07 2024-09-21T22:08:23.834 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.834 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.834 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:23.834 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.804689+0000", 2024-09-21T22:08:23.834 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00086121799999999997 2024-09-21T22:08:23.834 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.834 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.835 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:23.835 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.804706+0000", 2024-09-21T22:08:23.835 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6266999999999999e-05 2024-09-21T22:08:23.835 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.835 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.835 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:23.835 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.804713+0000", 2024-09-21T22:08:23.835 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.1010000000000003e-06 2024-09-21T22:08:23.835 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.835 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.835 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:23.835 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.804781+0000", 2024-09-21T22:08:23.835 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.7871000000000001e-05 2024-09-21T22:08:23.835 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.836 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.836 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:23.836 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.804859+0000", 2024-09-21T22:08:23.836 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.7891000000000004e-05 2024-09-21T22:08:23.836 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.836 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.836 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 0", 2024-09-21T22:08:23.836 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.804971+0000", 2024-09-21T22:08:23.836 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00011275100000000001 2024-09-21T22:08:23.836 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.836 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.836 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:23.836 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.852704+0000", 2024-09-21T22:08:23.836 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.047732361000000001 2024-09-21T22:08:23.837 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.837 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.837 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-21T22:08:23.837 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.853446+0000", 2024-09-21T22:08:23.837 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00074258799999999997 2024-09-21T22:08:23.837 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.837 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.837 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:23.837 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.853608+0000", 2024-09-21T22:08:23.837 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00016190900000000001 2024-09-21T22:08:23.837 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.837 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.837 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:23.837 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.853718+0000", 2024-09-21T22:08:23.838 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000109597 2024-09-21T22:08:23.838 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:23.838 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:23.838 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:23.838 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.838 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.838 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4840.0:723 2.1 2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:08:23.838 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.860543+0000", 2024-09-21T22:08:23.838 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 57.908607981000003, 2024-09-21T22:08:23.838 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.062354957000000003, 2024-09-21T22:08:23.838 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:23.838 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:23.838 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:23.839 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-21T22:08:23.839 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4840", 2024-09-21T22:08:23.839 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:08:23.839 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 723 2024-09-21T22:08:23.839 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.839 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:23.839 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.839 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:23.839 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.860543+0000", 2024-09-21T22:08:23.839 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:23.839 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.839 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.839 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:23.840 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.860543+0000", 2024-09-21T22:08:23.840 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:23.840 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.840 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.840 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:23.840 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.860544+0000", 2024-09-21T22:08:23.840 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.1119999999999999e-06 2024-09-21T22:08:23.840 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.840 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.840 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:23.840 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.861400+0000", 2024-09-21T22:08:23.840 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000856393 2024-09-21T22:08:23.840 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.841 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.841 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:23.841 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.861419+0000", 2024-09-21T22:08:23.842 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8992e-05 2024-09-21T22:08:23.842 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.842 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.842 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:23.842 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.861428+0000", 2024-09-21T22:08:23.842 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.551e-06 2024-09-21T22:08:23.842 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.842 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.842 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:23.842 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.861555+0000", 2024-09-21T22:08:23.842 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000127286 2024-09-21T22:08:23.842 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.843 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.843 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:23.843 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.861816+0000", 2024-09-21T22:08:23.843 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00026100899999999998 2024-09-21T22:08:23.843 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.843 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.843 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 0", 2024-09-21T22:08:23.843 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.862189+0000", 2024-09-21T22:08:23.843 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00037261899999999998 2024-09-21T22:08:23.843 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.843 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.843 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:23.843 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.921514+0000", 2024-09-21T22:08:23.844 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.059325210000000003 2024-09-21T22:08:23.844 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.844 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.844 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-21T22:08:23.844 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.922721+0000", 2024-09-21T22:08:23.844 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0012066819999999999 2024-09-21T22:08:23.844 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.844 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.844 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:23.844 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.922797+0000", 2024-09-21T22:08:23.844 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.6162999999999996e-05 2024-09-21T22:08:23.844 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.845 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.845 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:23.845 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.922898+0000", 2024-09-21T22:08:23.845 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00010094000000000001 2024-09-21T22:08:23.845 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:23.845 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:23.845 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:23.845 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.845 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.845 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:725 2.0 e63/16 2:08b4f96a:::rbd_data.126483d75f2d.000000000000000f:head v 63'33, mlcod=63'31)", 2024-09-21T22:08:23.845 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.936778+0000", 2024-09-21T22:08:23.845 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 57.832372239000001, 2024-09-21T22:08:23.845 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.047671937999999997, 2024-09-21T22:08:23.846 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:23.846 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:23.846 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:23.846 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:23.846 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.846 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:23.846 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.936778+0000", 2024-09-21T22:08:23.846 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:23.846 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.846 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.846 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:23.846 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.936778+0000", 2024-09-21T22:08:23.847 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:23.847 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.847 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.847 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:23.847 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.936780+0000", 2024-09-21T22:08:23.847 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7799999999999999e-06 2024-09-21T22:08:23.847 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.847 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.847 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:23.847 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.938475+0000", 2024-09-21T22:08:23.847 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0016951749999999999 2024-09-21T22:08:23.847 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.847 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.847 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:23.848 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.938506+0000", 2024-09-21T22:08:23.848 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.0865999999999999e-05 2024-09-21T22:08:23.848 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.848 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.848 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:23.848 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.938520+0000", 2024-09-21T22:08:23.848 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3825999999999999e-05 2024-09-21T22:08:23.848 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.848 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.848 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:23.848 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.938624+0000", 2024-09-21T22:08:23.848 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00010391900000000001 2024-09-21T22:08:23.848 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.849 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.849 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:23.849 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.938651+0000", 2024-09-21T22:08:23.849 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.7481999999999999e-05 2024-09-21T22:08:23.849 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.849 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.849 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:23.849 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.984369+0000", 2024-09-21T22:08:23.849 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.045717595 2024-09-21T22:08:23.849 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.849 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.849 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:23.849 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.984450+0000", 2024-09-21T22:08:23.850 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.1295e-05 2024-09-21T22:08:23.850 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:23.850 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:23.850 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:23.850 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.850 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.850 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:727 2.0 e63/16 2:08b4f96a:::rbd_data.126483d75f2d.000000000000000f:head v 63'35, mlcod=63'33)", 2024-09-21T22:08:23.850 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.998090+0000", 2024-09-21T22:08:23.850 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 57.771061062999998, 2024-09-21T22:08:23.851 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.052205704999999998, 2024-09-21T22:08:23.851 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:23.851 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:23.851 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:23.851 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:23.851 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.851 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:23.851 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.998090+0000", 2024-09-21T22:08:23.851 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:23.851 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.852 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.852 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:23.852 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.998090+0000", 2024-09-21T22:08:23.852 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:23.852 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.852 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.853 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:23.853 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.998091+0000", 2024-09-21T22:08:23.853 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6929999999999999e-06 2024-09-21T22:08:23.853 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.853 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.853 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:23.853 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.998940+0000", 2024-09-21T22:08:23.853 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00084875200000000004 2024-09-21T22:08:23.853 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.853 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.853 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:23.854 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.998970+0000", 2024-09-21T22:08:23.854 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9709999999999998e-05 2024-09-21T22:08:23.854 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.854 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.854 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:23.854 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.998984+0000", 2024-09-21T22:08:23.854 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3739e-05 2024-09-21T22:08:23.854 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.854 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.855 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:23.855 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.999097+0000", 2024-09-21T22:08:23.855 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000113484 2024-09-21T22:08:23.855 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.855 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.855 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:23.855 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.999135+0000", 2024-09-21T22:08:23.855 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.8405999999999998e-05 2024-09-21T22:08:23.855 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.855 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.855 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:23.855 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.050245+0000", 2024-09-21T22:08:23.855 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.051109935000000002 2024-09-21T22:08:23.855 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.855 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.856 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:23.856 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.050295+0000", 2024-09-21T22:08:23.856 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.9985999999999999e-05 2024-09-21T22:08:23.856 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:23.856 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:23.856 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:23.856 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.856 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.856 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:729 2.0 e63/16 2:08b4f96a:::rbd_data.126483d75f2d.000000000000000f:head v 63'37, mlcod=63'35)", 2024-09-21T22:08:23.856 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:26.062424+0000", 2024-09-21T22:08:23.856 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 57.706726441000001, 2024-09-21T22:08:23.856 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.047312186999999999, 2024-09-21T22:08:23.856 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:23.857 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:23.857 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:23.857 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:23.857 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.857 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:23.857 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.062424+0000", 2024-09-21T22:08:23.857 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:23.857 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.857 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.857 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:23.857 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.062424+0000", 2024-09-21T22:08:23.857 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:23.857 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.857 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.858 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:23.858 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.062426+0000", 2024-09-21T22:08:23.858 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7129999999999999e-06 2024-09-21T22:08:23.858 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.858 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.858 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:23.858 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.063211+0000", 2024-09-21T22:08:23.858 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00078488500000000003 2024-09-21T22:08:23.858 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.858 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.858 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:23.858 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.063241+0000", 2024-09-21T22:08:23.858 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.0056000000000002e-05 2024-09-21T22:08:23.858 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.859 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.859 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:23.859 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.063255+0000", 2024-09-21T22:08:23.859 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4022999999999999e-05 2024-09-21T22:08:23.859 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.859 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.859 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:23.859 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.063384+0000", 2024-09-21T22:08:23.859 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00012920600000000001 2024-09-21T22:08:23.859 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.859 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.859 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:23.859 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.063421+0000", 2024-09-21T22:08:23.859 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.7242e-05 2024-09-21T22:08:23.860 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.860 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.860 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:23.860 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.109666+0000", 2024-09-21T22:08:23.860 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.046244446000000002 2024-09-21T22:08:23.860 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.860 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.860 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:23.860 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.109736+0000", 2024-09-21T22:08:23.860 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.0616000000000003e-05 2024-09-21T22:08:23.860 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:23.860 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:23.860 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:23.860 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.861 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.861 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:731 2.0 e63/16 2:08b4f96a:::rbd_data.126483d75f2d.000000000000000f:head v 63'39, mlcod=63'37)", 2024-09-21T22:08:23.861 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:26.135946+0000", 2024-09-21T22:08:23.861 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 57.633204933999998, 2024-09-21T22:08:23.861 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.054290359000000003, 2024-09-21T22:08:23.861 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:23.861 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:23.861 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:23.861 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:23.861 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.861 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:23.861 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.135946+0000", 2024-09-21T22:08:23.861 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:23.861 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.862 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.862 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:23.862 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.135946+0000", 2024-09-21T22:08:23.862 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:23.862 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.862 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.862 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:23.862 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.135948+0000", 2024-09-21T22:08:23.862 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7999999999999999e-06 2024-09-21T22:08:23.862 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.862 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.862 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:23.862 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.136730+0000", 2024-09-21T22:08:23.862 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00078271899999999997 2024-09-21T22:08:23.863 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.863 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.863 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:23.863 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.136760+0000", 2024-09-21T22:08:23.863 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9876e-05 2024-09-21T22:08:23.863 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.863 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.863 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:23.863 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.136774+0000", 2024-09-21T22:08:23.863 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3478999999999999e-05 2024-09-21T22:08:23.863 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.863 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.863 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:23.863 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.136944+0000", 2024-09-21T22:08:23.864 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000170626 2024-09-21T22:08:23.864 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.864 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.864 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:23.864 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.136985+0000", 2024-09-21T22:08:23.864 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.0377999999999999e-05 2024-09-21T22:08:23.864 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.864 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.864 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:23.864 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.190137+0000", 2024-09-21T22:08:23.864 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.053152438000000003 2024-09-21T22:08:23.864 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.864 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.864 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:23.865 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.190236+0000", 2024-09-21T22:08:23.865 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.9042999999999993e-05 2024-09-21T22:08:23.865 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:23.865 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:23.865 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:23.865 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.865 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.865 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:125 2.3 e71/66 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head v 71'31, mlcod=71'29)", 2024-09-21T22:08:23.865 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.066855+0000", 2024-09-21T22:08:23.865 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 45.702295499000002, 2024-09-21T22:08:23.866 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.059856059000000003, 2024-09-21T22:08:23.866 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:23.866 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:23.866 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:23.866 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:23.866 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.866 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:23.866 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.066855+0000", 2024-09-21T22:08:23.866 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:23.866 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.866 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.866 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:23.866 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.066855+0000", 2024-09-21T22:08:23.867 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:23.867 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.867 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.867 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:23.867 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.066857+0000", 2024-09-21T22:08:23.867 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.807e-06 2024-09-21T22:08:23.867 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.867 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.867 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:23.867 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.068365+0000", 2024-09-21T22:08:23.867 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0015077910000000001 2024-09-21T22:08:23.867 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.867 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.867 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:23.868 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.068395+0000", 2024-09-21T22:08:23.868 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.0326000000000001e-05 2024-09-21T22:08:23.868 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.868 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.868 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:23.868 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.068415+0000", 2024-09-21T22:08:23.868 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9871999999999999e-05 2024-09-21T22:08:23.868 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.868 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.868 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:23.868 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.068580+0000", 2024-09-21T22:08:23.868 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000165317 2024-09-21T22:08:23.868 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.869 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.869 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:23.869 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.068618+0000", 2024-09-21T22:08:23.869 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.7753000000000003e-05 2024-09-21T22:08:23.869 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.869 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.869 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:23.869 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.126623+0000", 2024-09-21T22:08:23.869 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.058004642000000002 2024-09-21T22:08:23.869 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.869 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.869 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:23.869 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.126711+0000", 2024-09-21T22:08:23.869 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.8551e-05 2024-09-21T22:08:23.870 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:23.870 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:23.870 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:23.870 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.870 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.870 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:127 2.3 e71/66 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head v 71'33, mlcod=71'31)", 2024-09-21T22:08:23.870 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.140864+0000", 2024-09-21T22:08:23.870 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 45.628286471000003, 2024-09-21T22:08:23.870 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.052512380999999997, 2024-09-21T22:08:23.870 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:23.871 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:23.871 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:23.871 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:23.871 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.871 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:23.871 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.140864+0000", 2024-09-21T22:08:23.871 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:23.871 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.871 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.871 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:23.871 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.140864+0000", 2024-09-21T22:08:23.872 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:23.872 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.872 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.872 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:23.872 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.140866+0000", 2024-09-21T22:08:23.872 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.694e-06 2024-09-21T22:08:23.872 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.872 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.872 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:23.872 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.141712+0000", 2024-09-21T22:08:23.872 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00084581499999999998 2024-09-21T22:08:23.872 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.873 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.873 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:23.873 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.141742+0000", 2024-09-21T22:08:23.873 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9865999999999998e-05 2024-09-21T22:08:23.873 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.873 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.873 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:23.873 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.141761+0000", 2024-09-21T22:08:23.873 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9473000000000001e-05 2024-09-21T22:08:23.873 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.873 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.873 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:23.873 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.141919+0000", 2024-09-21T22:08:23.874 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00015758300000000001 2024-09-21T22:08:23.874 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.874 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.874 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:23.874 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.141955+0000", 2024-09-21T22:08:23.874 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.6613000000000003e-05 2024-09-21T22:08:23.874 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.874 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.874 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:23.874 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.193333+0000", 2024-09-21T22:08:23.874 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.051377260000000001 2024-09-21T22:08:23.874 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.875 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.875 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:23.875 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.193377+0000", 2024-09-21T22:08:23.875 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.4076999999999998e-05 2024-09-21T22:08:23.875 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:23.875 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:23.875 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:23.875 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.875 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.875 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:129 2.3 e71/66 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head v 71'35, mlcod=71'33)", 2024-09-21T22:08:23.875 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.209458+0000", 2024-09-21T22:08:23.875 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 45.559692722000001, 2024-09-21T22:08:23.875 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.056595578000000001, 2024-09-21T22:08:23.876 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:23.876 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:23.876 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:23.876 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:23.876 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.876 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:23.876 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.209458+0000", 2024-09-21T22:08:23.876 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:23.876 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.876 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.876 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:23.876 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.209458+0000", 2024-09-21T22:08:23.876 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:23.876 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.877 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.877 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:23.877 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.209460+0000", 2024-09-21T22:08:23.877 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7600000000000001e-06 2024-09-21T22:08:23.877 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.877 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.877 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:23.877 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.210317+0000", 2024-09-21T22:08:23.877 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00085771700000000003 2024-09-21T22:08:23.877 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.877 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.877 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:23.877 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.210346+0000", 2024-09-21T22:08:23.877 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8334e-05 2024-09-21T22:08:23.877 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.878 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.878 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:23.878 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.210365+0000", 2024-09-21T22:08:23.878 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9439000000000001e-05 2024-09-21T22:08:23.878 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.878 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.878 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:23.878 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.210544+0000", 2024-09-21T22:08:23.878 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00017845599999999999 2024-09-21T22:08:23.878 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.878 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.878 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:23.878 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.210584+0000", 2024-09-21T22:08:23.879 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.0423000000000002e-05 2024-09-21T22:08:23.879 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.879 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.879 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:23.879 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.265970+0000", 2024-09-21T22:08:23.879 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.055385765000000003 2024-09-21T22:08:23.879 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.879 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.879 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:23.879 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.266054+0000", 2024-09-21T22:08:23.879 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.3683999999999994e-05 2024-09-21T22:08:23.879 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:23.879 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:23.879 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:23.879 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.880 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.880 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:131 2.3 e71/66 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head v 71'37, mlcod=71'35)", 2024-09-21T22:08:23.880 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.279550+0000", 2024-09-21T22:08:23.880 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 45.489600883999998, 2024-09-21T22:08:23.880 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.052290419999999997, 2024-09-21T22:08:23.880 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:23.880 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:23.880 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:23.880 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:23.880 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.881 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:23.881 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.279550+0000", 2024-09-21T22:08:23.881 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:23.881 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.881 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.881 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:23.881 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.279550+0000", 2024-09-21T22:08:23.881 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:23.881 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.881 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.881 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:23.881 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.279552+0000", 2024-09-21T22:08:23.882 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9439999999999999e-06 2024-09-21T22:08:23.882 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.882 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.882 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:23.882 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.280296+0000", 2024-09-21T22:08:23.882 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00074417000000000005 2024-09-21T22:08:23.882 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.882 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.882 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:23.882 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.280324+0000", 2024-09-21T22:08:23.882 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8045999999999999e-05 2024-09-21T22:08:23.882 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.882 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.883 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:23.883 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.280345+0000", 2024-09-21T22:08:23.883 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.0849999999999999e-05 2024-09-21T22:08:23.883 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.883 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.883 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:23.884 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.280497+0000", 2024-09-21T22:08:23.884 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00015252500000000001 2024-09-21T22:08:23.884 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.884 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.884 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:23.884 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.280531+0000", 2024-09-21T22:08:23.884 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.4034e-05 2024-09-21T22:08:23.884 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.885 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.885 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:23.885 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.331769+0000", 2024-09-21T22:08:23.885 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.051237999999999999 2024-09-21T22:08:23.885 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:23.885 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:23.885 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:23.885 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.331840+0000", 2024-09-21T22:08:23.886 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.0851000000000003e-05 2024-09-21T22:08:23.886 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:23.886 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:23.886 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:23.886 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:23.886 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:23.886 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:08:23.888 INFO:tasks.workunit.client.0.smithi157.stderr:+ PID=25035 2024-09-21T22:08:23.889 INFO:tasks.workunit.client.0.smithi157.stderr:+ test -n 25035 2024-09-21T22:08:23.889 INFO:tasks.workunit.client.0.smithi157.stderr:+ ps -p 25035 -C rbd-nbd 2024-09-21T22:08:23.889 INFO:tasks.workunit.client.0.smithi157.stdout: PID TTY TIME CMD 2024-09-21T22:08:23.889 INFO:tasks.workunit.client.0.smithi157.stdout: 25035 ? 00:00:00 rbd-nbd 2024-09-21T22:08:23.890 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T22:08:23.891 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:08:23.885+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:08:23.892 INFO:tasks.workunit.client.0.smithi157.stderr:++ ps -p 25035 -T 2024-09-21T22:08:23.892 INFO:tasks.workunit.client.0.smithi157.stderr:++ grep -c io_context_pool 2024-09-21T22:08:23.892 INFO:tasks.workunit.client.0.smithi157.stderr:+ ACTUAL=2 2024-09-21T22:08:23.892 INFO:tasks.workunit.client.0.smithi157.stderr:+ '[' 2 -eq 2 ']' 2024-09-21T22:08:23.892 INFO:tasks.workunit.client.0.smithi157.stderr:+ unmap_device /dev/nbd2 25035 2024-09-21T22:08:23.893 INFO:tasks.workunit.client.0.smithi157.stderr:+ local args=/dev/nbd2 2024-09-21T22:08:23.893 INFO:tasks.workunit.client.0.smithi157.stderr:+ local pid=25035 2024-09-21T22:08:23.893 INFO:tasks.workunit.client.0.smithi157.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd2 2024-09-21T22:08:23.893 INFO:tasks.workunit.client.0.smithi157.stderr:+ local cmd 2024-09-21T22:08:23.893 INFO:tasks.workunit.client.0.smithi157.stderr:++ id -u 2024-09-21T22:08:23.893 INFO:tasks.workunit.client.0.smithi157.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T22:08:23.893 INFO:tasks.workunit.client.0.smithi157.stderr:++ which rbd 2024-09-21T22:08:23.893 INFO:tasks.workunit.client.0.smithi157.stderr:+ cmd=/usr/bin/rbd 2024-09-21T22:08:23.893 INFO:tasks.workunit.client.0.smithi157.stderr:+ shift 2024-09-21T22:08:23.894 INFO:tasks.workunit.client.0.smithi157.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd2 2024-09-21T22:08:23.990 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:08:23.985+0000 7fe38af2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 34631) UID: 0 2024-09-21T22:08:23.993 INFO:tasks.workunit.client.0.smithi157.stderr:+ rbd device --device-type nbd list 2024-09-21T22:08:23.993 INFO:tasks.workunit.client.0.smithi157.stderr:+ expect_false grep '^25035\b' 2024-09-21T22:08:23.993 INFO:tasks.workunit.client.0.smithi157.stderr:+ grep '^25035\b' 2024-09-21T22:08:24.029 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:08:24.029 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:08:24.029 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-09-21T22:08:24.029 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:08:24.044 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T22:08:24.052 INFO:tasks.workunit.client.0.smithi157.stderr:+ return 0 2024-09-21T22:08:24.053 INFO:tasks.workunit.client.0.smithi157.stderr:+ ps -C rbd-nbd 2024-09-21T22:08:24.053 INFO:tasks.workunit.client.0.smithi157.stderr:+ expect_false grep '^ *25035\b' 2024-09-21T22:08:24.053 INFO:tasks.workunit.client.0.smithi157.stderr:+ grep '^ *25035\b' 2024-09-21T22:08:24.061 INFO:tasks.workunit.client.0.smithi157.stderr:+ return 0 2024-09-21T22:08:24.061 INFO:tasks.workunit.client.0.smithi157.stderr:+ sleep 0.5 2024-09-21T22:08:24.094 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:08:24.091+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:08:24.171 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:08:24.171 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:08:24.171 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-21T22:08:24.171 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-21T22:08:24.171 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:08:24.185 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T22:08:24.192 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:08:24.189+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:08:24.293 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:08:24.289+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:08:24.362 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:08:24.362 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-21T22:08:24.362 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-21T22:08:24.362 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-21T22:08:24.362 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.362 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:08:24.362 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.036355+0000", 2024-09-21T22:08:24.362 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 142.32280039, 2024-09-21T22:08:24.362 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.062441934999999997, 2024-09-21T22:08:24.362 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:24.362 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:24.363 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:24.363 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:24.363 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:08:24.363 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:08:24.363 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1 2024-09-21T22:08:24.363 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.363 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:24.363 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.363 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:24.363 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036355+0000", 2024-09-21T22:08:24.363 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:24.363 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.363 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.363 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:24.364 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036355+0000", 2024-09-21T22:08:24.364 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:24.364 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.364 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.364 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:24.364 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036357+0000", 2024-09-21T22:08:24.364 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.446e-06 2024-09-21T22:08:24.364 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.364 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.364 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:24.364 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036373+0000", 2024-09-21T22:08:24.364 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5447e-05 2024-09-21T22:08:24.364 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.364 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.364 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:24.365 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036399+0000", 2024-09-21T22:08:24.365 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6423000000000001e-05 2024-09-21T22:08:24.365 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.365 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.365 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:24.365 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036420+0000", 2024-09-21T22:08:24.365 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1129999999999999e-05 2024-09-21T22:08:24.365 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.365 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.365 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:24.365 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036557+0000", 2024-09-21T22:08:24.365 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013686899999999999 2024-09-21T22:08:24.365 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.365 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.366 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-21T22:08:24.366 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036585+0000", 2024-09-21T22:08:24.366 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8362999999999999e-05 2024-09-21T22:08:24.366 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.366 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.366 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:24.366 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.095068+0000", 2024-09-21T22:08:24.366 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058482972000000001 2024-09-21T22:08:24.366 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.366 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.366 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:24.366 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.095516+0000", 2024-09-21T22:08:24.366 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00044802999999999998 2024-09-21T22:08:24.366 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.366 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.367 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:08:24.367 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.095869+0000", 2024-09-21T22:08:24.367 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000352104 2024-09-21T22:08:24.367 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.367 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.367 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:24.367 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.097055+0000", 2024-09-21T22:08:24.367 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011865599999999999 2024-09-21T22:08:24.367 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.367 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.367 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:24.367 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.098637+0000", 2024-09-21T22:08:24.367 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001581458 2024-09-21T22:08:24.367 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.368 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.368 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:24.368 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.098728+0000", 2024-09-21T22:08:24.368 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1043000000000002e-05 2024-09-21T22:08:24.368 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.368 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.368 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:24.368 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.098797+0000", 2024-09-21T22:08:24.368 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9090000000000004e-05 2024-09-21T22:08:24.368 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:24.368 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:24.368 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:24.368 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.368 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.368 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4840.0:680 2.7 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:08:24.369 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.744506+0000", 2024-09-21T22:08:24.369 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 59.614649536000002, 2024-09-21T22:08:24.369 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.063865055000000004, 2024-09-21T22:08:24.369 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:24.369 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:24.369 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:24.369 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:24.369 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4840", 2024-09-21T22:08:24.369 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:08:24.369 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 680 2024-09-21T22:08:24.369 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.369 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:24.369 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.369 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:24.370 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.744506+0000", 2024-09-21T22:08:24.370 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:24.370 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.370 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.370 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:24.370 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.744506+0000", 2024-09-21T22:08:24.370 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:24.370 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.370 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.370 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:24.370 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.744508+0000", 2024-09-21T22:08:24.370 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.103e-06 2024-09-21T22:08:24.371 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.371 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.371 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:24.371 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.745371+0000", 2024-09-21T22:08:24.371 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00086322199999999997 2024-09-21T22:08:24.371 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.371 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.371 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:24.371 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.745401+0000", 2024-09-21T22:08:24.371 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9955999999999999e-05 2024-09-21T22:08:24.371 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.371 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.371 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:24.371 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.745416+0000", 2024-09-21T22:08:24.371 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4873e-05 2024-09-21T22:08:24.372 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.372 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.372 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:24.372 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.745509+0000", 2024-09-21T22:08:24.372 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3171000000000006e-05 2024-09-21T22:08:24.372 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.372 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.372 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:24.372 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.745632+0000", 2024-09-21T22:08:24.372 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012272699999999999 2024-09-21T22:08:24.372 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.372 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.372 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-09-21T22:08:24.372 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.745789+0000", 2024-09-21T22:08:24.373 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015781399999999999 2024-09-21T22:08:24.373 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.373 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.373 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:24.373 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.785494+0000", 2024-09-21T22:08:24.373 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.039704401 2024-09-21T22:08:24.373 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.373 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.373 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:24.373 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.808090+0000", 2024-09-21T22:08:24.373 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022596207 2024-09-21T22:08:24.373 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.373 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.373 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:24.373 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.808246+0000", 2024-09-21T22:08:24.374 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015567200000000001 2024-09-21T22:08:24.374 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.374 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.374 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:24.374 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.808371+0000", 2024-09-21T22:08:24.374 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012490900000000001 2024-09-21T22:08:24.374 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:24.374 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:24.374 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:24.374 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.374 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.374 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4840.0:682 2.7 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:08:24.374 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.822415+0000", 2024-09-21T22:08:24.374 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 59.536739973000003, 2024-09-21T22:08:24.375 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.065044633000000004, 2024-09-21T22:08:24.375 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:24.375 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:24.375 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:24.375 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:24.375 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4840", 2024-09-21T22:08:24.375 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:08:24.375 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 682 2024-09-21T22:08:24.375 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.375 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:24.375 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.375 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:24.375 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.822415+0000", 2024-09-21T22:08:24.375 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:24.376 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.376 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.376 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:24.376 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.822415+0000", 2024-09-21T22:08:24.376 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:24.376 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.376 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.376 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:24.376 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.822417+0000", 2024-09-21T22:08:24.376 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8029999999999999e-06 2024-09-21T22:08:24.376 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.376 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.376 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:24.376 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.823308+0000", 2024-09-21T22:08:24.377 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00089146500000000003 2024-09-21T22:08:24.377 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.377 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.377 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:24.377 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.823339+0000", 2024-09-21T22:08:24.377 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0493000000000001e-05 2024-09-21T22:08:24.377 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.377 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.377 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:24.377 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.823354+0000", 2024-09-21T22:08:24.377 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5472999999999999e-05 2024-09-21T22:08:24.377 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.377 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.377 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:24.378 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.823476+0000", 2024-09-21T22:08:24.378 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000121209 2024-09-21T22:08:24.378 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.378 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.378 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:24.378 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.823650+0000", 2024-09-21T22:08:24.378 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017394599999999999 2024-09-21T22:08:24.378 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.378 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.378 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-09-21T22:08:24.378 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.823873+0000", 2024-09-21T22:08:24.378 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022314899999999999 2024-09-21T22:08:24.378 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.378 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.378 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:24.379 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.884396+0000", 2024-09-21T22:08:24.379 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.060523674999999999 2024-09-21T22:08:24.379 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.379 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.379 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:24.379 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.887238+0000", 2024-09-21T22:08:24.379 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028420450000000001 2024-09-21T22:08:24.379 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.379 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.379 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:24.379 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.887369+0000", 2024-09-21T22:08:24.379 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013010699999999999 2024-09-21T22:08:24.379 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.379 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.380 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:24.380 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.887460+0000", 2024-09-21T22:08:24.380 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1267999999999994e-05 2024-09-21T22:08:24.380 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:24.380 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:24.380 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:24.380 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.380 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.380 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4840.0:692 2.5 2:b1d55633:::rbd_data.126483d75f2d.000000000000000b:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:08:24.380 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:25.091048+0000", 2024-09-21T22:08:24.380 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 59.268107438999998, 2024-09-21T22:08:24.380 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.060917092999999999, 2024-09-21T22:08:24.380 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:24.380 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:24.380 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:24.381 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:24.381 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4840", 2024-09-21T22:08:24.381 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:08:24.381 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 692 2024-09-21T22:08:24.381 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.381 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:24.381 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.381 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:24.381 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.091048+0000", 2024-09-21T22:08:24.381 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:24.381 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.381 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.381 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:24.381 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.091048+0000", 2024-09-21T22:08:24.382 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:24.382 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.382 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.382 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:24.382 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.091050+0000", 2024-09-21T22:08:24.382 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.03e-06 2024-09-21T22:08:24.382 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.382 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.382 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:24.382 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.091989+0000", 2024-09-21T22:08:24.382 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00093932100000000001 2024-09-21T22:08:24.382 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.382 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.382 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:24.383 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.092023+0000", 2024-09-21T22:08:24.383 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4310000000000002e-05 2024-09-21T22:08:24.383 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.383 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.383 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:24.383 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.092040+0000", 2024-09-21T22:08:24.383 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6426000000000002e-05 2024-09-21T22:08:24.383 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.383 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.383 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:24.383 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.092175+0000", 2024-09-21T22:08:24.383 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013500299999999999 2024-09-21T22:08:24.383 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.383 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.383 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:24.384 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.092307+0000", 2024-09-21T22:08:24.384 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000132562 2024-09-21T22:08:24.384 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.384 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.384 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:08:24.384 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.092499+0000", 2024-09-21T22:08:24.384 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000191972 2024-09-21T22:08:24.384 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.384 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.384 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:24.384 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.151065+0000", 2024-09-21T22:08:24.384 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058565514999999999 2024-09-21T22:08:24.384 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.384 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.384 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:24.385 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.151844+0000", 2024-09-21T22:08:24.385 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00077930800000000004 2024-09-21T22:08:24.385 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.385 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.385 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:24.385 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.151916+0000", 2024-09-21T22:08:24.385 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1921999999999993e-05 2024-09-21T22:08:24.385 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.385 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.385 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:24.385 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.151965+0000", 2024-09-21T22:08:24.385 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8724000000000003e-05 2024-09-21T22:08:24.385 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:24.385 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:24.385 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:24.386 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.386 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.386 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4840.0:696 2.5 2:b1d55633:::rbd_data.126483d75f2d.000000000000000b:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:08:24.386 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:25.218641+0000", 2024-09-21T22:08:24.386 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 59.140514171, 2024-09-21T22:08:24.386 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058363669999999999, 2024-09-21T22:08:24.386 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:24.386 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:24.386 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:24.386 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:24.386 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4840", 2024-09-21T22:08:24.386 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:08:24.386 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 696 2024-09-21T22:08:24.386 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.387 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:24.387 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.387 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:24.387 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.218641+0000", 2024-09-21T22:08:24.387 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:24.387 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.387 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.387 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:24.387 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.218641+0000", 2024-09-21T22:08:24.387 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:24.387 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.387 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.387 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:24.387 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.218642+0000", 2024-09-21T22:08:24.387 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.266e-06 2024-09-21T22:08:24.388 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.388 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.388 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:24.388 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.219432+0000", 2024-09-21T22:08:24.388 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00078986300000000002 2024-09-21T22:08:24.388 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.388 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.388 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:24.388 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.219450+0000", 2024-09-21T22:08:24.388 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7526000000000001e-05 2024-09-21T22:08:24.388 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.388 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.388 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:24.388 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.219459+0000", 2024-09-21T22:08:24.389 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3000000000000007e-06 2024-09-21T22:08:24.389 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.389 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.389 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:24.389 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.219655+0000", 2024-09-21T22:08:24.389 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019599099999999999 2024-09-21T22:08:24.389 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.389 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.389 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:24.389 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.219917+0000", 2024-09-21T22:08:24.389 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026198200000000001 2024-09-21T22:08:24.389 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.389 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.389 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:08:24.389 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.220198+0000", 2024-09-21T22:08:24.390 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00028154500000000002 2024-09-21T22:08:24.390 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.390 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.390 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:24.390 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.268856+0000", 2024-09-21T22:08:24.390 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.048657687999999998 2024-09-21T22:08:24.390 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.390 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.390 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:24.390 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.276903+0000", 2024-09-21T22:08:24.390 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0080472249999999999 2024-09-21T22:08:24.390 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.390 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.390 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:24.391 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.276958+0000", 2024-09-21T22:08:24.391 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4184999999999997e-05 2024-09-21T22:08:24.391 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.391 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.391 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:24.391 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.277005+0000", 2024-09-21T22:08:24.391 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7098999999999997e-05 2024-09-21T22:08:24.391 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:24.391 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:24.391 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:24.391 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.391 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.391 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4840.0:729 2.0 2:08b4f96a:::rbd_data.126483d75f2d.000000000000000f:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:08:24.391 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:26.062012+0000", 2024-09-21T22:08:24.392 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 58.297142938, 2024-09-21T22:08:24.392 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.060656823999999998, 2024-09-21T22:08:24.392 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:24.392 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:24.392 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:24.392 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:24.392 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4840", 2024-09-21T22:08:24.392 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:08:24.392 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 729 2024-09-21T22:08:24.392 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.392 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:24.392 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.392 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:24.392 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.062012+0000", 2024-09-21T22:08:24.393 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:24.393 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.393 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.393 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:24.393 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.062012+0000", 2024-09-21T22:08:24.393 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:24.393 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.393 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.393 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:24.393 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.062013+0000", 2024-09-21T22:08:24.393 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.243e-06 2024-09-21T22:08:24.393 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.393 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.393 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:24.393 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.062827+0000", 2024-09-21T22:08:24.394 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00081374799999999999 2024-09-21T22:08:24.394 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.394 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.394 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:24.394 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.062845+0000", 2024-09-21T22:08:24.394 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.808e-05 2024-09-21T22:08:24.394 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.394 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.394 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:24.394 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.062854+0000", 2024-09-21T22:08:24.394 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1039999999999996e-06 2024-09-21T22:08:24.394 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.394 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.394 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:24.395 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.062970+0000", 2024-09-21T22:08:24.395 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011601 2024-09-21T22:08:24.395 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.395 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.395 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:24.395 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.063230+0000", 2024-09-21T22:08:24.395 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025918799999999999 2024-09-21T22:08:24.395 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.395 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.395 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-09-21T22:08:24.395 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.063641+0000", 2024-09-21T22:08:24.395 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00041152799999999998 2024-09-21T22:08:24.395 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.395 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.395 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:24.396 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.122015+0000", 2024-09-21T22:08:24.396 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058374246999999997 2024-09-21T22:08:24.396 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.396 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.396 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:24.396 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.122451+0000", 2024-09-21T22:08:24.396 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000435771 2024-09-21T22:08:24.396 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.396 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.396 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:24.396 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.122576+0000", 2024-09-21T22:08:24.396 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012469099999999999 2024-09-21T22:08:24.396 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.396 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.397 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:24.397 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.122669+0000", 2024-09-21T22:08:24.397 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3214000000000006e-05 2024-09-21T22:08:24.397 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:24.397 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:24.397 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:24.397 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.397 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.397 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4840.0:731 2.0 2:08b4f96a:::rbd_data.126483d75f2d.000000000000000f:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:08:24.397 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:26.135603+0000", 2024-09-21T22:08:24.397 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 58.223552437999999, 2024-09-21T22:08:24.397 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.056838474999999999, 2024-09-21T22:08:24.397 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:24.397 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:24.397 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:24.398 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:24.398 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4840", 2024-09-21T22:08:24.398 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:08:24.398 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 731 2024-09-21T22:08:24.398 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.398 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:24.398 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.398 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:24.398 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.135603+0000", 2024-09-21T22:08:24.398 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:24.398 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.398 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.398 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:24.398 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.135603+0000", 2024-09-21T22:08:24.399 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:24.399 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.399 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.399 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:24.399 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.135605+0000", 2024-09-21T22:08:24.399 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9700000000000002e-06 2024-09-21T22:08:24.399 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.399 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.399 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:24.399 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.136554+0000", 2024-09-21T22:08:24.399 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00094886700000000005 2024-09-21T22:08:24.399 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.399 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.399 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:24.400 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.136582+0000", 2024-09-21T22:08:24.400 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8393000000000001e-05 2024-09-21T22:08:24.400 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.400 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.400 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:24.400 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.136596+0000", 2024-09-21T22:08:24.400 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.43e-05 2024-09-21T22:08:24.400 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.400 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.400 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:24.400 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.136691+0000", 2024-09-21T22:08:24.400 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4843e-05 2024-09-21T22:08:24.400 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.400 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.400 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:24.401 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.136873+0000", 2024-09-21T22:08:24.401 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018237099999999999 2024-09-21T22:08:24.401 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.401 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.401 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-09-21T22:08:24.401 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.137140+0000", 2024-09-21T22:08:24.401 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026642700000000002 2024-09-21T22:08:24.401 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.401 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.401 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:24.401 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.180580+0000", 2024-09-21T22:08:24.401 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.043440078 2024-09-21T22:08:24.401 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.401 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.402 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:24.402 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.192175+0000", 2024-09-21T22:08:24.402 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011594959 2024-09-21T22:08:24.402 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.402 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.402 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:24.402 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.192353+0000", 2024-09-21T22:08:24.402 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017850200000000001 2024-09-21T22:08:24.402 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.402 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.402 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:24.402 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.192441+0000", 2024-09-21T22:08:24.402 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7764999999999997e-05 2024-09-21T22:08:24.402 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:24.403 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:24.403 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:24.403 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.403 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.403 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5207.0:34 2.7 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head [write 262144~131072] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e78)", 2024-09-21T22:08:24.403 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:52.587708+0000", 2024-09-21T22:08:24.403 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 31.771447417000001, 2024-09-21T22:08:24.403 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.056690213000000003, 2024-09-21T22:08:24.403 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:24.403 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:24.403 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:24.403 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:24.403 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5207", 2024-09-21T22:08:24.403 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/357731919", 2024-09-21T22:08:24.403 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 34 2024-09-21T22:08:24.404 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.404 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:24.404 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.404 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:24.404 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.587708+0000", 2024-09-21T22:08:24.404 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:24.404 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.404 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.404 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:24.404 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.587708+0000", 2024-09-21T22:08:24.404 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:24.404 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.404 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.404 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:24.404 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.587709+0000", 2024-09-21T22:08:24.405 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.72e-06 2024-09-21T22:08:24.405 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.405 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.405 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:24.405 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588017+0000", 2024-09-21T22:08:24.405 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00030767499999999998 2024-09-21T22:08:24.405 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.405 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.405 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:24.405 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588037+0000", 2024-09-21T22:08:24.405 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9865999999999999e-05 2024-09-21T22:08:24.405 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.405 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.405 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:24.406 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588048+0000", 2024-09-21T22:08:24.406 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1384e-05 2024-09-21T22:08:24.406 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.406 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.406 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:24.406 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.614263+0000", 2024-09-21T22:08:24.406 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026214232000000001 2024-09-21T22:08:24.406 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.406 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.406 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:24.406 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.614397+0000", 2024-09-21T22:08:24.406 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013396299999999999 2024-09-21T22:08:24.406 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.406 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.407 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-09-21T22:08:24.407 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.614670+0000", 2024-09-21T22:08:24.407 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027295500000000002 2024-09-21T22:08:24.407 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.407 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.407 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:24.407 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.644046+0000", 2024-09-21T22:08:24.407 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029376973000000001 2024-09-21T22:08:24.407 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.407 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.407 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:24.407 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.644208+0000", 2024-09-21T22:08:24.407 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016169900000000001 2024-09-21T22:08:24.407 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.408 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.408 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:24.408 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.644335+0000", 2024-09-21T22:08:24.408 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012656000000000001 2024-09-21T22:08:24.408 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.408 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.408 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:24.408 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.644398+0000", 2024-09-21T22:08:24.408 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3186000000000001e-05 2024-09-21T22:08:24.408 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:24.408 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:24.408 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:24.408 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.408 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.409 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5207.0:35 2.7 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head [write 393216~131072] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e78)", 2024-09-21T22:08:24.409 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:52.588070+0000", 2024-09-21T22:08:24.409 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 31.771085519, 2024-09-21T22:08:24.409 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.076411232999999995, 2024-09-21T22:08:24.409 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:24.409 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:24.409 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:24.409 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:24.409 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5207", 2024-09-21T22:08:24.409 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/357731919", 2024-09-21T22:08:24.409 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 35 2024-09-21T22:08:24.409 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.409 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:24.409 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.409 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:24.410 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588070+0000", 2024-09-21T22:08:24.410 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:24.410 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.410 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.410 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:24.410 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588070+0000", 2024-09-21T22:08:24.410 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:24.410 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.410 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.410 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:24.410 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588071+0000", 2024-09-21T22:08:24.410 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1299999999999998e-07 2024-09-21T22:08:24.410 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.410 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.411 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:24.411 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588327+0000", 2024-09-21T22:08:24.411 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025630199999999997 2024-09-21T22:08:24.411 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.411 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.411 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:24.411 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588343+0000", 2024-09-21T22:08:24.411 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6140000000000001e-05 2024-09-21T22:08:24.411 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.411 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.411 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:24.411 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588353+0000", 2024-09-21T22:08:24.411 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0043e-05 2024-09-21T22:08:24.411 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.411 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.412 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:24.412 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.623978+0000", 2024-09-21T22:08:24.412 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.035625281000000002 2024-09-21T22:08:24.412 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.412 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.412 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:24.412 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.624104+0000", 2024-09-21T22:08:24.412 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012591300000000001 2024-09-21T22:08:24.412 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.412 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.412 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-09-21T22:08:24.412 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.624296+0000", 2024-09-21T22:08:24.412 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019226200000000001 2024-09-21T22:08:24.412 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.413 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.413 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:24.413 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.654488+0000", 2024-09-21T22:08:24.413 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.030191551 2024-09-21T22:08:24.413 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.413 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.413 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:24.413 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.664329+0000", 2024-09-21T22:08:24.413 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0098405119999999992 2024-09-21T22:08:24.413 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.413 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.413 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:24.413 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.664414+0000", 2024-09-21T22:08:24.413 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5385999999999999e-05 2024-09-21T22:08:24.413 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.414 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.414 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:24.414 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.664481+0000", 2024-09-21T22:08:24.414 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6929999999999998e-05 2024-09-21T22:08:24.414 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:24.414 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:24.414 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:24.414 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.414 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.414 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5207.0:36 2.7 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head [write 524288~131072] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e78)", 2024-09-21T22:08:24.414 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:52.588370+0000", 2024-09-21T22:08:24.414 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 31.770784741, 2024-09-21T22:08:24.414 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.076552001999999994, 2024-09-21T22:08:24.414 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:24.415 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:24.415 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:24.415 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:24.415 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5207", 2024-09-21T22:08:24.415 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/357731919", 2024-09-21T22:08:24.415 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 36 2024-09-21T22:08:24.415 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.415 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:24.415 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.415 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:24.415 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588370+0000", 2024-09-21T22:08:24.415 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:24.415 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.415 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.416 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:24.416 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588370+0000", 2024-09-21T22:08:24.416 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:24.416 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.416 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.416 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:24.416 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588372+0000", 2024-09-21T22:08:24.416 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.153e-06 2024-09-21T22:08:24.416 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.416 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.416 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:24.416 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588716+0000", 2024-09-21T22:08:24.416 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00034479200000000002 2024-09-21T22:08:24.416 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.417 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.417 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:24.417 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588752+0000", 2024-09-21T22:08:24.417 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5865999999999998e-05 2024-09-21T22:08:24.417 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.417 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.417 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:24.417 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588777+0000", 2024-09-21T22:08:24.417 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4583000000000002e-05 2024-09-21T22:08:24.417 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.417 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.417 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:24.417 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.633886+0000", 2024-09-21T22:08:24.417 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.045108941999999999 2024-09-21T22:08:24.417 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.418 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.418 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:24.418 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.634059+0000", 2024-09-21T22:08:24.418 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000173046 2024-09-21T22:08:24.418 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.418 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.418 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-09-21T22:08:24.418 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.634314+0000", 2024-09-21T22:08:24.418 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025482400000000001 2024-09-21T22:08:24.418 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.418 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.418 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:24.418 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.664541+0000", 2024-09-21T22:08:24.418 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.030227222000000002 2024-09-21T22:08:24.419 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.419 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.419 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:24.419 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.664653+0000", 2024-09-21T22:08:24.419 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011192900000000001 2024-09-21T22:08:24.419 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.419 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.419 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:24.419 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.664798+0000", 2024-09-21T22:08:24.419 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014499900000000001 2024-09-21T22:08:24.419 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.419 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.419 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:24.419 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.664922+0000", 2024-09-21T22:08:24.420 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012464600000000001 2024-09-21T22:08:24.420 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:24.420 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:24.420 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:24.420 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.420 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.420 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5207.0:37 2.7 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head [write 655360~131072] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e78)", 2024-09-21T22:08:24.420 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:52.588813+0000", 2024-09-21T22:08:24.420 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 31.770341727000002, 2024-09-21T22:08:24.420 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.087074796999999995, 2024-09-21T22:08:24.420 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:24.420 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:24.420 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:24.420 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:24.421 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5207", 2024-09-21T22:08:24.421 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/357731919", 2024-09-21T22:08:24.421 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 37 2024-09-21T22:08:24.421 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.421 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:24.421 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.421 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:24.421 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588813+0000", 2024-09-21T22:08:24.421 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:24.421 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.421 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.421 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:24.421 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588813+0000", 2024-09-21T22:08:24.421 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:24.422 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.422 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.422 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:24.422 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588815+0000", 2024-09-21T22:08:24.422 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.643e-06 2024-09-21T22:08:24.422 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.422 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.422 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:24.422 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.589071+0000", 2024-09-21T22:08:24.422 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025580200000000002 2024-09-21T22:08:24.422 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.422 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.422 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:24.422 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.589098+0000", 2024-09-21T22:08:24.422 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7336000000000001e-05 2024-09-21T22:08:24.423 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.423 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.423 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:24.423 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.589121+0000", 2024-09-21T22:08:24.423 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2373000000000001e-05 2024-09-21T22:08:24.423 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.423 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.423 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:24.423 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.644491+0000", 2024-09-21T22:08:24.423 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.055369994999999998 2024-09-21T22:08:24.423 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.423 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.423 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:24.423 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.644707+0000", 2024-09-21T22:08:24.424 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000216488 2024-09-21T22:08:24.424 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.424 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.424 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-09-21T22:08:24.424 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.644908+0000", 2024-09-21T22:08:24.424 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000201196 2024-09-21T22:08:24.424 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.424 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.424 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:24.424 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.675419+0000", 2024-09-21T22:08:24.424 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.03051071 2024-09-21T22:08:24.424 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.424 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.424 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:24.425 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.675591+0000", 2024-09-21T22:08:24.425 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000172545 2024-09-21T22:08:24.425 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.425 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.425 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:24.425 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.675761+0000", 2024-09-21T22:08:24.425 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016942300000000001 2024-09-21T22:08:24.425 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.425 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.425 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:24.425 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.675888+0000", 2024-09-21T22:08:24.425 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000127286 2024-09-21T22:08:24.425 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:24.425 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:24.425 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:24.426 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.426 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.426 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5207.0:38 2.7 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head [write 786432~131072] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e78)", 2024-09-21T22:08:24.426 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:52.589156+0000", 2024-09-21T22:08:24.426 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 31.769998823000002, 2024-09-21T22:08:24.426 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.087126880000000004, 2024-09-21T22:08:24.426 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:24.426 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:24.426 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:24.426 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:24.426 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5207", 2024-09-21T22:08:24.426 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/357731919", 2024-09-21T22:08:24.426 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 38 2024-09-21T22:08:24.426 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.427 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:24.427 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.427 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:24.427 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.589156+0000", 2024-09-21T22:08:24.427 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:24.427 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.427 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.427 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:24.427 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.589156+0000", 2024-09-21T22:08:24.427 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:24.427 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.427 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.427 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:24.427 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.589158+0000", 2024-09-21T22:08:24.427 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.717e-06 2024-09-21T22:08:24.428 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.428 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.428 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:24.428 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.589417+0000", 2024-09-21T22:08:24.428 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025860500000000002 2024-09-21T22:08:24.428 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.428 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.428 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:24.428 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.589447+0000", 2024-09-21T22:08:24.428 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9916e-05 2024-09-21T22:08:24.428 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.428 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.428 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:24.428 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.589467+0000", 2024-09-21T22:08:24.428 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0894e-05 2024-09-21T22:08:24.429 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.429 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.429 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:24.429 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.654576+0000", 2024-09-21T22:08:24.429 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.065108579999999999 2024-09-21T22:08:24.429 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.429 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.429 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:24.429 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.654694+0000", 2024-09-21T22:08:24.429 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000118083 2024-09-21T22:08:24.429 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.429 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.429 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-09-21T22:08:24.429 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.654880+0000", 2024-09-21T22:08:24.430 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000186069 2024-09-21T22:08:24.430 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.430 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.430 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:24.430 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.675453+0000", 2024-09-21T22:08:24.430 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020572498000000002 2024-09-21T22:08:24.430 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.430 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.430 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:24.430 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.676055+0000", 2024-09-21T22:08:24.430 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00060247000000000002 2024-09-21T22:08:24.430 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.430 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.430 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:24.430 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.676178+0000", 2024-09-21T22:08:24.431 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000122395 2024-09-21T22:08:24.431 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.431 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.431 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:24.431 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.676283+0000", 2024-09-21T22:08:24.431 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000105653 2024-09-21T22:08:24.431 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:24.431 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:24.431 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:24.431 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.431 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.431 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5207.0:39 2.7 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head [write 917504~131072] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e78)", 2024-09-21T22:08:24.431 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:52.589499+0000", 2024-09-21T22:08:24.431 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 31.769656215000001, 2024-09-21T22:08:24.432 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.087365371999999997, 2024-09-21T22:08:24.432 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:24.432 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:24.432 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:24.432 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:24.432 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5207", 2024-09-21T22:08:24.432 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/357731919", 2024-09-21T22:08:24.432 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 39 2024-09-21T22:08:24.432 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.432 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:24.432 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.432 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:24.432 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.589499+0000", 2024-09-21T22:08:24.432 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:24.432 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.433 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.433 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:24.433 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.589499+0000", 2024-09-21T22:08:24.433 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:24.433 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.433 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.433 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:24.433 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.589500+0000", 2024-09-21T22:08:24.433 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.547e-06 2024-09-21T22:08:24.433 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.433 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.433 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:24.433 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.589752+0000", 2024-09-21T22:08:24.433 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025112799999999998 2024-09-21T22:08:24.434 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.434 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.434 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:24.434 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.589783+0000", 2024-09-21T22:08:24.434 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1557000000000003e-05 2024-09-21T22:08:24.434 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.434 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.434 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:24.434 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.589832+0000", 2024-09-21T22:08:24.434 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9175999999999998e-05 2024-09-21T22:08:24.434 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.434 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.434 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:24.434 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.665028+0000", 2024-09-21T22:08:24.435 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.075195293999999996 2024-09-21T22:08:24.435 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.435 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.435 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:24.435 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.665169+0000", 2024-09-21T22:08:24.435 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000141299 2024-09-21T22:08:24.435 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.435 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.435 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-09-21T22:08:24.435 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.665464+0000", 2024-09-21T22:08:24.435 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00029459899999999999 2024-09-21T22:08:24.435 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.435 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.435 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:24.435 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.676551+0000", 2024-09-21T22:08:24.436 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011087107000000001 2024-09-21T22:08:24.436 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.436 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.436 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:24.436 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.676700+0000", 2024-09-21T22:08:24.436 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000149853 2024-09-21T22:08:24.436 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.436 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.436 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:24.436 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.676802+0000", 2024-09-21T22:08:24.436 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010194199999999999 2024-09-21T22:08:24.436 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.436 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.436 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:24.436 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.676864+0000", 2024-09-21T22:08:24.437 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1870000000000002e-05 2024-09-21T22:08:24.437 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:24.437 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:24.437 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:24.437 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.437 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.437 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5207.0:74 2.7 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head [write 524288~131072] snapc 5=[5,3] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-21T22:08:24.437 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:52.938124+0000", 2024-09-21T22:08:24.437 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 31.421031003, 2024-09-21T22:08:24.437 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.071435100000000001, 2024-09-21T22:08:24.437 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:24.437 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:24.437 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:24.437 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:24.438 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5207", 2024-09-21T22:08:24.438 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/357731919", 2024-09-21T22:08:24.438 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 74 2024-09-21T22:08:24.438 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.438 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:24.438 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.438 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:24.438 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938124+0000", 2024-09-21T22:08:24.438 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:24.438 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.438 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.438 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:24.438 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938124+0000", 2024-09-21T22:08:24.438 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:24.439 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.439 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.439 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:24.439 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938125+0000", 2024-09-21T22:08:24.439 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5600000000000005e-07 2024-09-21T22:08:24.439 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.439 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.439 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:24.439 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938218+0000", 2024-09-21T22:08:24.439 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3239999999999995e-05 2024-09-21T22:08:24.439 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.439 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.439 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:24.439 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938231+0000", 2024-09-21T22:08:24.439 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.272e-05 2024-09-21T22:08:24.440 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.440 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.440 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:24.440 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938239+0000", 2024-09-21T22:08:24.440 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5059999999999995e-06 2024-09-21T22:08:24.440 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.440 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.440 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:24.440 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.983312+0000", 2024-09-21T22:08:24.440 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.045072252 2024-09-21T22:08:24.440 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.440 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.440 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:24.440 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.983458+0000", 2024-09-21T22:08:24.440 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014628899999999999 2024-09-21T22:08:24.441 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.441 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.441 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-09-21T22:08:24.441 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.983682+0000", 2024-09-21T22:08:24.441 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022402899999999999 2024-09-21T22:08:24.441 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.441 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.441 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:24.441 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.993679+0000", 2024-09-21T22:08:24.441 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0099972740000000004 2024-09-21T22:08:24.441 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.441 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.441 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:24.441 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.009436+0000", 2024-09-21T22:08:24.442 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015756867000000001 2024-09-21T22:08:24.442 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.442 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.442 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:24.442 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.009506+0000", 2024-09-21T22:08:24.442 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0277999999999994e-05 2024-09-21T22:08:24.442 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.442 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.442 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:24.442 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.009559+0000", 2024-09-21T22:08:24.442 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2889000000000001e-05 2024-09-21T22:08:24.442 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:24.442 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:24.442 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:24.443 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.443 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.443 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5207.0:75 2.7 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head [write 655360~131072] snapc 5=[5,3] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-21T22:08:24.443 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:52.938253+0000", 2024-09-21T22:08:24.443 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 31.420901947000001, 2024-09-21T22:08:24.443 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082064041000000004, 2024-09-21T22:08:24.443 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:24.443 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:24.443 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:24.443 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:24.443 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5207", 2024-09-21T22:08:24.443 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/357731919", 2024-09-21T22:08:24.443 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 75 2024-09-21T22:08:24.443 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.443 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:24.444 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.444 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:24.444 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938253+0000", 2024-09-21T22:08:24.444 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:24.444 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.444 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.444 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:24.444 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938253+0000", 2024-09-21T22:08:24.444 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:24.444 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.444 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.444 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:24.444 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938254+0000", 2024-09-21T22:08:24.444 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8599999999999998e-07 2024-09-21T22:08:24.445 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.445 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.445 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:24.445 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938354+0000", 2024-09-21T22:08:24.445 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9696000000000002e-05 2024-09-21T22:08:24.445 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.445 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.445 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:24.445 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938381+0000", 2024-09-21T22:08:24.445 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.705e-05 2024-09-21T22:08:24.445 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.445 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.445 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:24.445 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938397+0000", 2024-09-21T22:08:24.445 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5996999999999999e-05 2024-09-21T22:08:24.446 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.446 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.446 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:24.446 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.993739+0000", 2024-09-21T22:08:24.446 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.055342823999999999 2024-09-21T22:08:24.446 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.446 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.446 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:24.446 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.993835+0000", 2024-09-21T22:08:24.446 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5919000000000001e-05 2024-09-21T22:08:24.446 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.446 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.446 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-09-21T22:08:24.446 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.993993+0000", 2024-09-21T22:08:24.447 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015729900000000001 2024-09-21T22:08:24.447 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.447 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.447 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:24.447 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.009579+0000", 2024-09-21T22:08:24.447 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015586723 2024-09-21T22:08:24.447 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.447 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.447 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:24.447 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.020134+0000", 2024-09-21T22:08:24.447 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010554908 2024-09-21T22:08:24.447 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.447 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.447 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:24.447 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.020241+0000", 2024-09-21T22:08:24.448 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000106246 2024-09-21T22:08:24.448 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.448 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.448 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:24.448 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.020317+0000", 2024-09-21T22:08:24.448 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6693000000000006e-05 2024-09-21T22:08:24.448 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:24.448 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:24.448 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:24.448 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.448 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.448 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5207.0:76 2.7 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head [write 786432~131072] snapc 5=[5,3] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-21T22:08:24.448 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:52.938410+0000", 2024-09-21T22:08:24.448 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 31.420744778, 2024-09-21T22:08:24.448 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.086926370000000003, 2024-09-21T22:08:24.449 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:24.449 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:24.449 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:24.449 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:24.449 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5207", 2024-09-21T22:08:24.449 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/357731919", 2024-09-21T22:08:24.449 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 76 2024-09-21T22:08:24.449 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.449 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:24.449 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.449 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:24.449 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938410+0000", 2024-09-21T22:08:24.449 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:24.449 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.450 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.450 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:24.450 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938410+0000", 2024-09-21T22:08:24.450 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:24.450 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.450 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.450 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:24.450 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938411+0000", 2024-09-21T22:08:24.450 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0299999999999998e-07 2024-09-21T22:08:24.450 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.450 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.450 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:24.450 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938502+0000", 2024-09-21T22:08:24.450 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0562000000000005e-05 2024-09-21T22:08:24.450 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.451 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.451 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:24.451 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938514+0000", 2024-09-21T22:08:24.451 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2327000000000001e-05 2024-09-21T22:08:24.451 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.451 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.451 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:24.451 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938522+0000", 2024-09-21T22:08:24.451 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1170000000000004e-06 2024-09-21T22:08:24.451 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.451 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.451 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:24.451 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.001427+0000", 2024-09-21T22:08:24.451 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.062904871000000001 2024-09-21T22:08:24.451 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.452 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.452 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:24.452 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.001519+0000", 2024-09-21T22:08:24.452 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1798000000000004e-05 2024-09-21T22:08:24.452 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.452 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.452 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-09-21T22:08:24.452 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.001671+0000", 2024-09-21T22:08:24.452 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015195900000000001 2024-09-21T22:08:24.452 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.452 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.452 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:24.452 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.020388+0000", 2024-09-21T22:08:24.452 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018717428000000001 2024-09-21T22:08:24.452 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.453 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.453 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:24.453 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.025041+0000", 2024-09-21T22:08:24.453 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0046524039999999997 2024-09-21T22:08:24.453 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.453 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.454 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:24.454 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.025213+0000", 2024-09-21T22:08:24.454 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017286499999999999 2024-09-21T22:08:24.454 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.454 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.454 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:24.454 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.025337+0000", 2024-09-21T22:08:24.454 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012333600000000001 2024-09-21T22:08:24.454 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:24.454 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:24.454 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:24.454 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.454 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.454 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5207.0:77 2.7 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head [write 917504~131072] snapc 5=[5,3] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-21T22:08:24.455 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:52.938534+0000", 2024-09-21T22:08:24.455 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 31.420620963000001, 2024-09-21T22:08:24.455 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.087709436000000002, 2024-09-21T22:08:24.455 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:24.455 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:24.455 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:24.455 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:24.455 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5207", 2024-09-21T22:08:24.455 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/357731919", 2024-09-21T22:08:24.455 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 77 2024-09-21T22:08:24.455 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.455 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:24.455 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.455 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:24.456 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938534+0000", 2024-09-21T22:08:24.456 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:24.456 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.456 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.456 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:24.456 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938534+0000", 2024-09-21T22:08:24.456 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:24.456 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.456 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.456 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:24.456 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938535+0000", 2024-09-21T22:08:24.456 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1399999999999996e-07 2024-09-21T22:08:24.456 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.456 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.456 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:24.457 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938623+0000", 2024-09-21T22:08:24.457 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8226000000000005e-05 2024-09-21T22:08:24.457 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.457 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.457 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:24.457 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938645+0000", 2024-09-21T22:08:24.457 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1948999999999999e-05 2024-09-21T22:08:24.457 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.457 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.457 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:24.457 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938660+0000", 2024-09-21T22:08:24.457 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4776999999999999e-05 2024-09-21T22:08:24.457 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.457 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.457 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:24.458 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.009669+0000", 2024-09-21T22:08:24.458 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.071009116999999997 2024-09-21T22:08:24.458 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.458 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.458 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:24.458 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.009949+0000", 2024-09-21T22:08:24.458 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027962899999999999 2024-09-21T22:08:24.458 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.458 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.458 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-09-21T22:08:24.458 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.010355+0000", 2024-09-21T22:08:24.458 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000406334 2024-09-21T22:08:24.458 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.458 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.459 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:24.459 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.020875+0000", 2024-09-21T22:08:24.459 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010520144 2024-09-21T22:08:24.459 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.459 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.459 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:24.459 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.026086+0000", 2024-09-21T22:08:24.459 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0052107569999999999 2024-09-21T22:08:24.459 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.459 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.459 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:24.459 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.026178+0000", 2024-09-21T22:08:24.459 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2113000000000004e-05 2024-09-21T22:08:24.459 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.459 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.460 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:24.460 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.026244+0000", 2024-09-21T22:08:24.460 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5675999999999999e-05 2024-09-21T22:08:24.460 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:24.460 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:24.460 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:24.460 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.460 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.460 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5207.0:93 2.7 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head [write 131072~917504] snapc 5=[5,3] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-21T22:08:24.460 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:53.147757+0000", 2024-09-21T22:08:24.460 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 31.211397784999999, 2024-09-21T22:08:24.460 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.062493043999999998, 2024-09-21T22:08:24.460 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:24.460 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:24.460 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:24.461 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:24.461 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5207", 2024-09-21T22:08:24.461 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/357731919", 2024-09-21T22:08:24.461 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 93 2024-09-21T22:08:24.461 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.461 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:24.461 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.461 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:24.461 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.147757+0000", 2024-09-21T22:08:24.461 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:24.461 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.461 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.461 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:24.461 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.147757+0000", 2024-09-21T22:08:24.461 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:24.462 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.462 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.462 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:24.462 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.147759+0000", 2024-09-21T22:08:24.462 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0159999999999998e-06 2024-09-21T22:08:24.462 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.462 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.462 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:24.462 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.149009+0000", 2024-09-21T22:08:24.462 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001250082 2024-09-21T22:08:24.462 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.462 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.462 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:24.462 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.149042+0000", 2024-09-21T22:08:24.463 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2617000000000003e-05 2024-09-21T22:08:24.463 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.463 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.463 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:24.463 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.149058+0000", 2024-09-21T22:08:24.463 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6113e-05 2024-09-21T22:08:24.463 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.463 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.463 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:24.463 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.149237+0000", 2024-09-21T22:08:24.463 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017834900000000001 2024-09-21T22:08:24.463 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.463 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.463 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:24.463 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.149378+0000", 2024-09-21T22:08:24.464 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014099199999999999 2024-09-21T22:08:24.464 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.464 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.464 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-09-21T22:08:24.464 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.149582+0000", 2024-09-21T22:08:24.464 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020414900000000001 2024-09-21T22:08:24.464 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.464 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.464 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:24.464 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.208648+0000", 2024-09-21T22:08:24.464 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.059066397999999999 2024-09-21T22:08:24.464 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.464 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.464 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:24.464 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.210125+0000", 2024-09-21T22:08:24.465 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001477119 2024-09-21T22:08:24.465 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.465 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.465 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:24.465 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.210200+0000", 2024-09-21T22:08:24.465 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4973000000000002e-05 2024-09-21T22:08:24.465 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.465 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.465 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:24.465 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.210250+0000", 2024-09-21T22:08:24.465 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0235999999999998e-05 2024-09-21T22:08:24.465 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:24.465 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:24.465 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:24.466 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.466 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.466 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5207.0:128 2.7 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head [write 131072~917504] snapc 6=[6,5,3] ondisk+write+known_if_redirected+supports_pool_eio e83)", 2024-09-21T22:08:24.466 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:08:01.056777+0000", 2024-09-21T22:08:24.466 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 23.302378314999999, 2024-09-21T22:08:24.466 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.062294929999999998, 2024-09-21T22:08:24.466 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:24.466 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:24.466 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:24.466 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:24.466 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5207", 2024-09-21T22:08:24.466 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/357731919", 2024-09-21T22:08:24.466 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 128 2024-09-21T22:08:24.466 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.467 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:24.467 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.467 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:24.467 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.056777+0000", 2024-09-21T22:08:24.467 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:24.467 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.467 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.467 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:24.467 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.056777+0000", 2024-09-21T22:08:24.467 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:24.467 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.467 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.467 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:24.467 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.056779+0000", 2024-09-21T22:08:24.467 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4669999999999998e-06 2024-09-21T22:08:24.468 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.468 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.468 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:24.468 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.058094+0000", 2024-09-21T22:08:24.468 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001314602 2024-09-21T22:08:24.468 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.468 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.468 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:24.468 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.058126+0000", 2024-09-21T22:08:24.468 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1625999999999998e-05 2024-09-21T22:08:24.468 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.468 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.468 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:24.468 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.058141+0000", 2024-09-21T22:08:24.469 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5274e-05 2024-09-21T22:08:24.469 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.469 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.469 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:24.469 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.058267+0000", 2024-09-21T22:08:24.469 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000126682 2024-09-21T22:08:24.469 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.469 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.469 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:24.469 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.058420+0000", 2024-09-21T22:08:24.469 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015212599999999999 2024-09-21T22:08:24.469 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.469 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.470 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-09-21T22:08:24.470 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.058644+0000", 2024-09-21T22:08:24.470 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000224749 2024-09-21T22:08:24.470 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.470 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.470 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:24.470 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.117883+0000", 2024-09-21T22:08:24.470 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.059238617 2024-09-21T22:08:24.470 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.470 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.470 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:24.470 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.118942+0000", 2024-09-21T22:08:24.470 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0010588780000000001 2024-09-21T22:08:24.470 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.471 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.471 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:24.471 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.119011+0000", 2024-09-21T22:08:24.471 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8802999999999998e-05 2024-09-21T22:08:24.471 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.471 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.471 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:24.471 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.119072+0000", 2024-09-21T22:08:24.471 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1105999999999994e-05 2024-09-21T22:08:24.471 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:24.471 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:24.471 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:24.471 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.471 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.471 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.3 87/86 [PushOp(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head, version: 71'29, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head@71'29, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T22:08:24.472 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:08:07.107088+0000", 2024-09-21T22:08:24.472 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 17.252067573000001, 2024-09-21T22:08:24.472 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.070273375999999999, 2024-09-21T22:08:24.472 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:24.472 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:24.472 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:08:24.472 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:24.472 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.472 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:24.472 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.107088+0000", 2024-09-21T22:08:24.472 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:24.472 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.472 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.472 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:24.472 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.107088+0000", 2024-09-21T22:08:24.473 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:24.473 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.473 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.473 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:24.473 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.107089+0000", 2024-09-21T22:08:24.473 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6559999999999999e-06 2024-09-21T22:08:24.473 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.473 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.473 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:24.473 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.108876+0000", 2024-09-21T22:08:24.473 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001786489 2024-09-21T22:08:24.473 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.473 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.473 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:24.473 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.108925+0000", 2024-09-21T22:08:24.474 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8843e-05 2024-09-21T22:08:24.474 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.474 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.474 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:24.474 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.108979+0000", 2024-09-21T22:08:24.474 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4945999999999999e-05 2024-09-21T22:08:24.474 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.474 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.474 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:24.474 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.109230+0000", 2024-09-21T22:08:24.474 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025086599999999998 2024-09-21T22:08:24.474 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.474 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.474 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:24.474 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.109269+0000", 2024-09-21T22:08:24.475 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8216000000000001e-05 2024-09-21T22:08:24.475 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.475 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.475 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:24.475 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.177361+0000", 2024-09-21T22:08:24.475 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.068092360000000005 2024-09-21T22:08:24.475 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:24.475 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:24.475 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:24.475 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:24.475 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:24.475 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:08:24.477 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T22:08:24.477 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:08:24.473+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:08:24.562 INFO:tasks.workunit.client.0.smithi157.stderr:+ EXPECTED=7 2024-09-21T22:08:24.563 INFO:tasks.workunit.client.0.smithi157.stderr:++ _sudo rbd device --device-type nbd --rbd-op-threads 7 map rbd/testrbdnbd21553 2024-09-21T22:08:24.563 INFO:tasks.workunit.client.0.smithi157.stderr:++ local cmd 2024-09-21T22:08:24.563 INFO:tasks.workunit.client.0.smithi157.stderr:+++ id -u 2024-09-21T22:08:24.565 INFO:tasks.workunit.client.0.smithi157.stderr:++ '[' 1000 -eq 0 ']' 2024-09-21T22:08:24.566 INFO:tasks.workunit.client.0.smithi157.stderr:+++ which rbd 2024-09-21T22:08:24.567 INFO:tasks.workunit.client.0.smithi157.stderr:++ cmd=/usr/bin/rbd 2024-09-21T22:08:24.567 INFO:tasks.workunit.client.0.smithi157.stderr:++ shift 2024-09-21T22:08:24.568 INFO:tasks.workunit.client.0.smithi157.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd --rbd-op-threads 7 map rbd/testrbdnbd21553 2024-09-21T22:08:24.577 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:08:24.571+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:08:24.607 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:08:24.607 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:08:24.608 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-09-21T22:08:24.608 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:08:24.622 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T22:08:24.677 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:08:24.673+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:08:24.714 INFO:tasks.workunit.client.0.smithi157.stderr:+ DEV=/dev/nbd2 2024-09-21T22:08:24.714 INFO:tasks.workunit.client.0.smithi157.stderr:+ get_pid rbd 2024-09-21T22:08:24.714 INFO:tasks.workunit.client.0.smithi157.stderr:+ local pool=rbd 2024-09-21T22:08:24.714 INFO:tasks.workunit.client.0.smithi157.stderr:+ local ns= 2024-09-21T22:08:24.714 INFO:tasks.workunit.client.0.smithi157.stderr:++ rbd device --device-type nbd --format xml list 2024-09-21T22:08:24.714 INFO:tasks.workunit.client.0.smithi157.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd21553'\''][device='\''/dev/nbd2'\'']/id' 2024-09-21T22:08:24.778 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:08:24.775+0000 7f2075a06640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31720) UID: 0 2024-09-21T22:08:24.802 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:08:24.802 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:08:24.802 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-21T22:08:24.802 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-21T22:08:24.803 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:08:24.815 INFO:tasks.workunit.client.0.smithi157.stderr:+ PID=25100 2024-09-21T22:08:24.815 INFO:tasks.workunit.client.0.smithi157.stderr:+ test -n 25100 2024-09-21T22:08:24.815 INFO:tasks.workunit.client.0.smithi157.stderr:+ ps -p 25100 -C rbd-nbd 2024-09-21T22:08:24.818 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T22:08:24.823 INFO:tasks.workunit.client.0.smithi157.stdout: PID TTY TIME CMD 2024-09-21T22:08:24.823 INFO:tasks.workunit.client.0.smithi157.stdout: 25100 ? 00:00:00 rbd-nbd 2024-09-21T22:08:24.824 INFO:tasks.workunit.client.0.smithi157.stderr:++ ps -p 25100 -T 2024-09-21T22:08:24.824 INFO:tasks.workunit.client.0.smithi157.stderr:++ grep -c io_context_pool 2024-09-21T22:08:24.831 INFO:tasks.workunit.client.0.smithi157.stderr:+ ACTUAL=7 2024-09-21T22:08:24.831 INFO:tasks.workunit.client.0.smithi157.stderr:+ '[' 7 -eq 7 ']' 2024-09-21T22:08:24.831 INFO:tasks.workunit.client.0.smithi157.stderr:+ unmap_device /dev/nbd2 25100 2024-09-21T22:08:24.831 INFO:tasks.workunit.client.0.smithi157.stderr:+ local args=/dev/nbd2 2024-09-21T22:08:24.831 INFO:tasks.workunit.client.0.smithi157.stderr:+ local pid=25100 2024-09-21T22:08:24.831 INFO:tasks.workunit.client.0.smithi157.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd2 2024-09-21T22:08:24.831 INFO:tasks.workunit.client.0.smithi157.stderr:+ local cmd 2024-09-21T22:08:24.831 INFO:tasks.workunit.client.0.smithi157.stderr:++ id -u 2024-09-21T22:08:24.832 INFO:tasks.workunit.client.0.smithi157.stderr:+ '[' 1000 -eq 0 ']' 2024-09-21T22:08:24.832 INFO:tasks.workunit.client.0.smithi157.stderr:++ which rbd 2024-09-21T22:08:24.833 INFO:tasks.workunit.client.0.smithi157.stderr:+ cmd=/usr/bin/rbd 2024-09-21T22:08:24.833 INFO:tasks.workunit.client.0.smithi157.stderr:+ shift 2024-09-21T22:08:24.833 INFO:tasks.workunit.client.0.smithi157.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd2 2024-09-21T22:08:24.879 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:08:24.873+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:08:24.928 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-21T22:08:24.947 INFO:tasks.workunit.client.0.smithi157.stderr:+ rbd device --device-type nbd list 2024-09-21T22:08:24.947 INFO:tasks.workunit.client.0.smithi157.stderr:+ expect_false grep '^25100\b' 2024-09-21T22:08:24.947 INFO:tasks.workunit.client.0.smithi157.stderr:+ grep '^25100\b' 2024-09-21T22:08:24.980 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:08:24.975+0000 7f2075a06640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31720) UID: 0 2024-09-21T22:08:24.994 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:08:24.994 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-21T22:08:24.994 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-21T22:08:24.994 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-21T22:08:24.994 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.994 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4840.0:676 2.7 e63/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 63'57, mlcod=63'55)", 2024-09-21T22:08:24.995 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.627276+0000", 2024-09-21T22:08:24.995 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 60.365407116, 2024-09-21T22:08:24.995 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.037500787000000001, 2024-09-21T22:08:24.995 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:24.995 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:24.995 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:24.995 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:24.995 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.995 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:24.995 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.627276+0000", 2024-09-21T22:08:24.995 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:24.995 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.995 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.995 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:24.996 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.627276+0000", 2024-09-21T22:08:24.996 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:24.996 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.996 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.996 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:24.996 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.627277+0000", 2024-09-21T22:08:24.996 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.068e-06 2024-09-21T22:08:24.996 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.996 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.996 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:24.996 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.628148+0000", 2024-09-21T22:08:24.996 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00087069999999999997 2024-09-21T22:08:24.996 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.996 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.996 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:24.997 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.628170+0000", 2024-09-21T22:08:24.997 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2147999999999999e-05 2024-09-21T22:08:24.997 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.997 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.997 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:24.997 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.628179+0000", 2024-09-21T22:08:24.997 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2450000000000004e-06 2024-09-21T22:08:24.997 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.997 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.997 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:24.997 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.628279+0000", 2024-09-21T22:08:24.997 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9298999999999999e-05 2024-09-21T22:08:24.997 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.998 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.998 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:24.998 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.628305+0000", 2024-09-21T22:08:24.998 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6316999999999999e-05 2024-09-21T22:08:24.998 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.998 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.998 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:24.998 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.664714+0000", 2024-09-21T22:08:24.998 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.036408923000000003 2024-09-21T22:08:24.998 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.998 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.998 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:24.998 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.664777+0000", 2024-09-21T22:08:24.998 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3087000000000001e-05 2024-09-21T22:08:24.999 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:24.999 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:24.999 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:24.999 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:24.999 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:24.999 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4840.0:678 2.7 e63/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 63'59, mlcod=63'57)", 2024-09-21T22:08:24.999 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.679689+0000", 2024-09-21T22:08:24.999 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 60.312994615999997, 2024-09-21T22:08:24.999 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053308406000000003, 2024-09-21T22:08:24.999 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:24.999 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:24.999 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:24.999 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:24.999 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.000 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:25.000 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.679689+0000", 2024-09-21T22:08:25.000 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:25.000 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.000 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.000 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:25.000 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.679689+0000", 2024-09-21T22:08:25.000 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:25.000 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.000 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.000 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:25.000 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.679691+0000", 2024-09-21T22:08:25.000 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9400000000000001e-06 2024-09-21T22:08:25.000 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.001 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.001 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:25.001 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.680245+0000", 2024-09-21T22:08:25.001 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00055382999999999995 2024-09-21T22:08:25.001 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.001 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.001 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:25.001 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.680276+0000", 2024-09-21T22:08:25.001 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1123e-05 2024-09-21T22:08:25.001 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.001 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.001 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:25.001 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.680290+0000", 2024-09-21T22:08:25.001 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4056999999999999e-05 2024-09-21T22:08:25.002 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.002 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.002 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:25.002 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.680418+0000", 2024-09-21T22:08:25.002 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000127792 2024-09-21T22:08:25.002 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.002 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.002 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:25.002 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.680465+0000", 2024-09-21T22:08:25.002 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7360000000000001e-05 2024-09-21T22:08:25.002 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.002 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.002 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:25.002 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.732923+0000", 2024-09-21T22:08:25.003 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.052458312999999999 2024-09-21T22:08:25.003 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.003 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.003 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:25.003 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.732997+0000", 2024-09-21T22:08:25.003 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3991000000000004e-05 2024-09-21T22:08:25.003 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:25.003 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:25.003 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:25.003 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.003 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.003 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4840.0:680 2.7 e63/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 63'61, mlcod=63'59)", 2024-09-21T22:08:25.003 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.745996+0000", 2024-09-21T22:08:25.003 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 60.246687102999999, 2024-09-21T22:08:25.003 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.061804132999999997, 2024-09-21T22:08:25.004 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:25.004 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:25.004 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:25.004 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:25.004 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.004 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:25.004 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.745996+0000", 2024-09-21T22:08:25.004 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:25.004 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.004 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.004 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:25.004 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.745996+0000", 2024-09-21T22:08:25.004 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:25.004 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.005 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.005 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:25.005 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.745998+0000", 2024-09-21T22:08:25.005 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.717e-06 2024-09-21T22:08:25.005 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.005 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.005 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:25.005 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.746523+0000", 2024-09-21T22:08:25.005 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00052525399999999996 2024-09-21T22:08:25.005 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.005 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.005 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:25.005 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.746554+0000", 2024-09-21T22:08:25.005 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0586000000000002e-05 2024-09-21T22:08:25.005 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.006 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.006 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:25.006 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.746577+0000", 2024-09-21T22:08:25.006 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3079e-05 2024-09-21T22:08:25.006 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.006 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.006 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:25.006 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.746689+0000", 2024-09-21T22:08:25.006 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011177 2024-09-21T22:08:25.006 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.006 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.006 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:25.006 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.746722+0000", 2024-09-21T22:08:25.006 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2870000000000002e-05 2024-09-21T22:08:25.007 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.007 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.007 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:25.007 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.807740+0000", 2024-09-21T22:08:25.007 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.061018612 2024-09-21T22:08:25.007 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.007 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.007 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:25.007 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.807800+0000", 2024-09-21T22:08:25.007 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0244999999999997e-05 2024-09-21T22:08:25.007 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:25.007 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:25.007 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:25.007 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.007 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.008 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4840.0:682 2.7 e63/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 63'63, mlcod=63'61)", 2024-09-21T22:08:25.008 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.824126+0000", 2024-09-21T22:08:25.008 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 60.168557534000001, 2024-09-21T22:08:25.008 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.038159327, 2024-09-21T22:08:25.008 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:25.008 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:25.008 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:25.008 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:25.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.008 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:25.008 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.824126+0000", 2024-09-21T22:08:25.008 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:25.008 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.008 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.008 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:25.009 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.824126+0000", 2024-09-21T22:08:25.009 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:25.009 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.009 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.009 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:25.009 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.824128+0000", 2024-09-21T22:08:25.009 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9659999999999999e-06 2024-09-21T22:08:25.009 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.009 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.009 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:25.009 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.824842+0000", 2024-09-21T22:08:25.009 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00071423999999999999 2024-09-21T22:08:25.009 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.009 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.010 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:25.010 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.824883+0000", 2024-09-21T22:08:25.010 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1131999999999999e-05 2024-09-21T22:08:25.010 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.010 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.010 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:25.010 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.824931+0000", 2024-09-21T22:08:25.010 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7327000000000003e-05 2024-09-21T22:08:25.010 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.010 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.010 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:25.010 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.825058+0000", 2024-09-21T22:08:25.010 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012790499999999999 2024-09-21T22:08:25.010 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.010 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.011 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:25.011 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.825087+0000", 2024-09-21T22:08:25.011 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8192e-05 2024-09-21T22:08:25.011 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.011 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.011 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:25.011 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.862225+0000", 2024-09-21T22:08:25.011 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.037138309000000001 2024-09-21T22:08:25.011 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.011 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.011 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:25.011 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.862285+0000", 2024-09-21T22:08:25.011 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0256e-05 2024-09-21T22:08:25.011 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:25.012 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:25.012 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:25.012 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.012 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5008.0:116 2.7 e71/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 71'148, mlcod=71'146)", 2024-09-21T22:08:25.012 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:37.867612+0000", 2024-09-21T22:08:25.012 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 47.125071038999998, 2024-09-21T22:08:25.012 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.045123155999999998, 2024-09-21T22:08:25.012 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:25.012 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:25.012 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:25.012 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:25.012 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.012 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:25.012 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.867612+0000", 2024-09-21T22:08:25.013 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:25.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.013 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.013 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:25.013 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.867612+0000", 2024-09-21T22:08:25.013 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:25.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.013 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.013 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:25.013 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.867613+0000", 2024-09-21T22:08:25.013 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1200000000000002e-07 2024-09-21T22:08:25.013 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.013 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.013 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:25.013 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.868135+0000", 2024-09-21T22:08:25.014 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00052132700000000003 2024-09-21T22:08:25.014 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.014 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.014 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:25.014 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.868150+0000", 2024-09-21T22:08:25.014 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5271e-05 2024-09-21T22:08:25.014 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.014 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.014 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:25.014 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.868156+0000", 2024-09-21T22:08:25.014 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4969999999999998e-06 2024-09-21T22:08:25.014 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.014 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.014 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:25.015 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.868213+0000", 2024-09-21T22:08:25.015 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7114999999999998e-05 2024-09-21T22:08:25.015 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.015 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.015 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:25.015 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.868238+0000", 2024-09-21T22:08:25.015 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4987999999999999e-05 2024-09-21T22:08:25.015 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.015 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.015 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:25.015 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.912677+0000", 2024-09-21T22:08:25.015 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.044439019000000003 2024-09-21T22:08:25.015 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.015 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.016 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:25.016 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.912736+0000", 2024-09-21T22:08:25.016 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8127000000000001e-05 2024-09-21T22:08:25.016 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:25.016 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:25.016 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:25.016 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.016 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.016 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5008.0:118 2.7 e71/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 71'150, mlcod=71'148)", 2024-09-21T22:08:25.016 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:37.924700+0000", 2024-09-21T22:08:25.016 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 47.067983847999997, 2024-09-21T22:08:25.016 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.034650561000000003, 2024-09-21T22:08:25.016 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:25.016 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:25.017 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:25.017 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:25.017 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.017 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:25.017 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.924700+0000", 2024-09-21T22:08:25.017 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:25.017 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.017 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.017 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:25.017 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.924700+0000", 2024-09-21T22:08:25.017 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:25.017 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.017 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.017 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:25.017 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.924702+0000", 2024-09-21T22:08:25.018 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9460000000000001e-06 2024-09-21T22:08:25.018 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.018 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.018 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:25.018 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.925132+0000", 2024-09-21T22:08:25.018 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00043021200000000002 2024-09-21T22:08:25.018 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.018 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.018 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:25.018 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.925156+0000", 2024-09-21T22:08:25.018 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4451999999999999e-05 2024-09-21T22:08:25.018 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.018 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.018 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:25.019 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.925168+0000", 2024-09-21T22:08:25.019 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2249000000000001e-05 2024-09-21T22:08:25.019 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.019 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.019 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:25.019 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.925241+0000", 2024-09-21T22:08:25.019 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2502000000000005e-05 2024-09-21T22:08:25.019 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.019 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.019 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:25.019 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.925267+0000", 2024-09-21T22:08:25.019 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6471000000000001e-05 2024-09-21T22:08:25.019 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.019 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.019 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:25.020 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.959217+0000", 2024-09-21T22:08:25.020 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.033949569999999998 2024-09-21T22:08:25.020 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.020 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.020 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:25.020 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.959350+0000", 2024-09-21T22:08:25.020 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013315899999999999 2024-09-21T22:08:25.020 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:25.020 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:25.020 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:25.020 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.020 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.020 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5008.0:120 2.7 e71/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 71'152, mlcod=71'150)", 2024-09-21T22:08:25.020 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:37.970837+0000", 2024-09-21T22:08:25.020 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 47.021846597, 2024-09-21T22:08:25.021 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.03044943, 2024-09-21T22:08:25.021 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:25.021 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:25.021 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:25.021 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:25.021 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.021 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:25.021 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.970837+0000", 2024-09-21T22:08:25.021 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:25.021 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.021 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.021 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:25.021 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.970837+0000", 2024-09-21T22:08:25.021 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:25.021 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.022 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.022 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:25.022 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.970838+0000", 2024-09-21T22:08:25.022 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4649999999999999e-06 2024-09-21T22:08:25.022 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.022 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.022 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:25.022 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.971265+0000", 2024-09-21T22:08:25.022 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00042665800000000001 2024-09-21T22:08:25.022 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.022 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.022 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:25.022 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.971285+0000", 2024-09-21T22:08:25.022 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9639999999999998e-05 2024-09-21T22:08:25.023 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.023 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.023 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:25.023 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.971294+0000", 2024-09-21T22:08:25.023 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7210000000000006e-06 2024-09-21T22:08:25.023 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.023 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.023 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:25.023 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.971354+0000", 2024-09-21T22:08:25.023 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9613e-05 2024-09-21T22:08:25.023 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.023 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.023 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:25.023 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.971381+0000", 2024-09-21T22:08:25.024 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7538999999999998e-05 2024-09-21T22:08:25.024 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.024 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.024 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:25.024 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.001199+0000", 2024-09-21T22:08:25.024 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029817537000000002 2024-09-21T22:08:25.024 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.024 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.024 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:25.024 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.001286+0000", 2024-09-21T22:08:25.024 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7256999999999995e-05 2024-09-21T22:08:25.024 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:25.024 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:25.024 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:25.024 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.025 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.025 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5008.0:122 2.7 e71/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 71'154, mlcod=71'152)", 2024-09-21T22:08:25.025 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:38.014709+0000", 2024-09-21T22:08:25.025 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 46.977974721999999, 2024-09-21T22:08:25.025 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029567408999999999, 2024-09-21T22:08:25.025 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:25.025 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:25.025 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:25.025 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:25.025 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.025 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:25.025 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.014709+0000", 2024-09-21T22:08:25.025 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:25.025 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.025 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.026 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:25.026 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.014709+0000", 2024-09-21T22:08:25.026 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:25.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.026 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.026 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:25.026 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.014709+0000", 2024-09-21T22:08:25.026 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9599999999999999e-07 2024-09-21T22:08:25.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.026 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.026 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:25.026 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.015201+0000", 2024-09-21T22:08:25.026 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00049175599999999999 2024-09-21T22:08:25.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.027 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.027 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:25.027 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.015215+0000", 2024-09-21T22:08:25.027 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3692999999999999e-05 2024-09-21T22:08:25.027 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.027 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.027 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:25.027 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.015225+0000", 2024-09-21T22:08:25.027 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.815e-06 2024-09-21T22:08:25.027 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.027 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.027 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:25.027 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.015287+0000", 2024-09-21T22:08:25.028 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1858000000000003e-05 2024-09-21T22:08:25.028 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.028 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.028 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:25.028 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.015315+0000", 2024-09-21T22:08:25.028 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8490999999999999e-05 2024-09-21T22:08:25.028 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.028 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.028 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:25.028 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.044182+0000", 2024-09-21T22:08:25.028 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.028866936999999999 2024-09-21T22:08:25.028 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.028 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.028 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:25.029 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.044276+0000", 2024-09-21T22:08:25.029 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4163e-05 2024-09-21T22:08:25.029 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:25.029 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:25.029 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:25.029 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.029 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.029 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5207.0:70 2.7 e79/16 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head v 79'184, mlcod=79'181)", 2024-09-21T22:08:25.029 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:52.938385+0000", 2024-09-21T22:08:25.029 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 32.054298815999999, 2024-09-21T22:08:25.029 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023928324000000001, 2024-09-21T22:08:25.029 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:25.029 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:25.030 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:25.030 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:25.030 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.030 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:25.030 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938385+0000", 2024-09-21T22:08:25.030 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:25.030 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.030 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.030 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:25.030 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938385+0000", 2024-09-21T22:08:25.030 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:25.030 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.030 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.031 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:25.031 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938387+0000", 2024-09-21T22:08:25.031 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1639999999999999e-06 2024-09-21T22:08:25.031 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.031 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.031 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:25.031 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938511+0000", 2024-09-21T22:08:25.031 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000124485 2024-09-21T22:08:25.031 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.031 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.031 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:25.031 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938549+0000", 2024-09-21T22:08:25.031 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7347e-05 2024-09-21T22:08:25.031 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.032 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.032 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:25.032 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938570+0000", 2024-09-21T22:08:25.032 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1013e-05 2024-09-21T22:08:25.032 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.032 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.032 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:25.032 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938718+0000", 2024-09-21T22:08:25.032 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000148789 2024-09-21T22:08:25.032 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.032 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.032 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:25.032 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938769+0000", 2024-09-21T22:08:25.032 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1002999999999999e-05 2024-09-21T22:08:25.033 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.033 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.033 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:25.033 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.962199+0000", 2024-09-21T22:08:25.033 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023429783999999999 2024-09-21T22:08:25.033 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.033 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.033 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:25.033 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.962313+0000", 2024-09-21T22:08:25.033 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000113739 2024-09-21T22:08:25.033 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:25.033 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:25.033 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:25.033 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.034 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.034 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5207.0:71 2.7 e79/16 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head v 79'185, mlcod=79'181)", 2024-09-21T22:08:25.034 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:52.951522+0000", 2024-09-21T22:08:25.034 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 32.041160924000003, 2024-09-21T22:08:25.034 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021108028000000001, 2024-09-21T22:08:25.034 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:25.034 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:25.034 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:25.034 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:25.034 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.034 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:25.034 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.951522+0000", 2024-09-21T22:08:25.034 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:25.034 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.035 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:25.035 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.951522+0000", 2024-09-21T22:08:25.035 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:25.035 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.035 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:25.035 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.951524+0000", 2024-09-21T22:08:25.035 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9199999999999998e-06 2024-09-21T22:08:25.035 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.035 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.035 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:25.035 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.951686+0000", 2024-09-21T22:08:25.035 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016202499999999999 2024-09-21T22:08:25.036 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.036 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.036 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:25.036 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.951713+0000", 2024-09-21T22:08:25.036 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7007000000000001e-05 2024-09-21T22:08:25.036 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.036 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.036 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:25.036 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.951734+0000", 2024-09-21T22:08:25.036 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0228999999999999e-05 2024-09-21T22:08:25.036 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.036 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.036 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:25.036 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.952740+0000", 2024-09-21T22:08:25.037 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001006631 2024-09-21T22:08:25.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.037 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.037 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:25.037 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.952778+0000", 2024-09-21T22:08:25.037 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7316e-05 2024-09-21T22:08:25.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.037 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.037 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:25.037 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.972505+0000", 2024-09-21T22:08:25.037 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019727776999999998 2024-09-21T22:08:25.037 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.037 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.037 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:25.038 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.972631+0000", 2024-09-21T22:08:25.038 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012512299999999999 2024-09-21T22:08:25.038 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:25.038 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:25.038 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:25.038 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.038 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.038 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5207.0:76 2.7 e79/16 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head v 79'190, mlcod=79'186)", 2024-09-21T22:08:25.038 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:53.001917+0000", 2024-09-21T22:08:25.038 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 31.990766475000001, 2024-09-21T22:08:25.038 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022783040000000001, 2024-09-21T22:08:25.038 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:25.038 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:25.038 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:25.039 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:25.039 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.039 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:25.039 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.001917+0000", 2024-09-21T22:08:25.039 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:25.039 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.039 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.039 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:25.039 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.001917+0000", 2024-09-21T22:08:25.039 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:25.039 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.039 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.039 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:25.039 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.001919+0000", 2024-09-21T22:08:25.039 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.703e-06 2024-09-21T22:08:25.040 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.040 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.040 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:25.040 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.002006+0000", 2024-09-21T22:08:25.040 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6978999999999995e-05 2024-09-21T22:08:25.040 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.040 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.040 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:25.040 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.002042+0000", 2024-09-21T22:08:25.040 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6847000000000001e-05 2024-09-21T22:08:25.040 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.040 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.040 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:25.040 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.002057+0000", 2024-09-21T22:08:25.041 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.419e-05 2024-09-21T22:08:25.041 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.041 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.041 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:25.041 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.004387+0000", 2024-09-21T22:08:25.041 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002330365 2024-09-21T22:08:25.041 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.041 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.041 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:25.041 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.004420+0000", 2024-09-21T22:08:25.041 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3170000000000003e-05 2024-09-21T22:08:25.041 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.041 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.041 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:25.042 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.024610+0000", 2024-09-21T22:08:25.042 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020190084000000001 2024-09-21T22:08:25.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.042 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.042 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:25.042 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.024700+0000", 2024-09-21T22:08:25.042 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9702000000000003e-05 2024-09-21T22:08:25.042 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:25.042 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:25.042 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:25.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.042 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.042 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5207.0:93 2.7 e79/16 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head v 79'193, mlcod=79'191)", 2024-09-21T22:08:25.042 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:53.149813+0000", 2024-09-21T22:08:25.042 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 31.842870416, 2024-09-21T22:08:25.043 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.060053738000000002, 2024-09-21T22:08:25.043 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:25.043 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:25.043 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:25.043 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:25.043 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.043 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:25.043 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.149813+0000", 2024-09-21T22:08:25.043 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:25.043 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.043 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.043 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:25.043 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.149813+0000", 2024-09-21T22:08:25.043 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:25.044 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.044 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.044 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:25.044 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.149815+0000", 2024-09-21T22:08:25.044 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9099999999999999e-06 2024-09-21T22:08:25.044 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.044 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.044 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:25.044 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.150828+0000", 2024-09-21T22:08:25.044 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0010129099999999999 2024-09-21T22:08:25.044 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.044 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.044 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:25.044 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.150860+0000", 2024-09-21T22:08:25.044 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2443000000000001e-05 2024-09-21T22:08:25.045 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.045 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.045 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:25.045 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.150874+0000", 2024-09-21T22:08:25.045 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3627e-05 2024-09-21T22:08:25.045 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.045 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.045 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:25.045 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.151010+0000", 2024-09-21T22:08:25.045 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013656599999999999 2024-09-21T22:08:25.045 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.045 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.045 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:25.045 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.151047+0000", 2024-09-21T22:08:25.046 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6495999999999998e-05 2024-09-21T22:08:25.046 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.046 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.046 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:25.046 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.209791+0000", 2024-09-21T22:08:25.046 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058744004000000002 2024-09-21T22:08:25.046 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.046 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.046 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:25.046 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.209867+0000", 2024-09-21T22:08:25.046 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5782000000000001e-05 2024-09-21T22:08:25.046 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:25.046 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:25.046 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:25.047 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.047 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.047 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5207.0:128 2.7 e83/16 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head v 83'198, mlcod=83'195)", 2024-09-21T22:08:25.047 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:08:01.058831+0000", 2024-09-21T22:08:25.047 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 23.933852780999999, 2024-09-21T22:08:25.047 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058096534999999998, 2024-09-21T22:08:25.047 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:25.047 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:25.047 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:25.047 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:25.047 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.047 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:25.047 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.058831+0000", 2024-09-21T22:08:25.047 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:25.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.048 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:25.048 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.058831+0000", 2024-09-21T22:08:25.048 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:25.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.048 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:25.048 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.058833+0000", 2024-09-21T22:08:25.048 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0660000000000002e-06 2024-09-21T22:08:25.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.048 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.048 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:25.048 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.059877+0000", 2024-09-21T22:08:25.049 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0010447 2024-09-21T22:08:25.049 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.049 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.049 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:25.049 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.059908+0000", 2024-09-21T22:08:25.049 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0800000000000003e-05 2024-09-21T22:08:25.049 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.049 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.049 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:25.049 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.059923+0000", 2024-09-21T22:08:25.049 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4497e-05 2024-09-21T22:08:25.049 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.049 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.049 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:25.050 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.060053+0000", 2024-09-21T22:08:25.050 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013002900000000001 2024-09-21T22:08:25.050 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.050 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.050 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:25.050 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.060091+0000", 2024-09-21T22:08:25.050 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8389999999999997e-05 2024-09-21T22:08:25.050 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.050 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.050 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:25.050 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.116840+0000", 2024-09-21T22:08:25.050 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.056749253999999999 2024-09-21T22:08:25.050 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.050 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.051 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:25.051 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.116927+0000", 2024-09-21T22:08:25.051 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6798999999999993e-05 2024-09-21T22:08:25.051 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:25.051 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:25.051 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:25.051 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.051 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.051 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.6 87/86 [PushOp(2:7e2189ff:ns::rbd_data.127f79bc213d.0000000000000001:head, version: 57'4, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:7e2189ff:ns::rbd_data.127f79bc213d.0000000000000001:head@57'4, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T22:08:25.051 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:08:07.095630+0000", 2024-09-21T22:08:25.051 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 17.897053825, 2024-09-21T22:08:25.051 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.063566191999999994, 2024-09-21T22:08:25.051 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:25.051 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:25.052 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:08:25.052 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:25.052 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.052 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:25.052 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.095630+0000", 2024-09-21T22:08:25.052 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:25.052 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.052 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.052 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:25.052 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.095630+0000", 2024-09-21T22:08:25.052 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:25.052 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.052 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.052 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:25.052 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.095631+0000", 2024-09-21T22:08:25.053 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3069999999999999e-06 2024-09-21T22:08:25.053 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.053 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.053 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:25.053 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.097075+0000", 2024-09-21T22:08:25.053 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001443831 2024-09-21T22:08:25.053 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.053 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.053 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:25.053 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.097153+0000", 2024-09-21T22:08:25.053 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8485999999999994e-05 2024-09-21T22:08:25.053 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.053 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.053 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:25.054 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.097208+0000", 2024-09-21T22:08:25.054 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4716000000000003e-05 2024-09-21T22:08:25.054 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.054 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.054 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:25.054 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.097459+0000", 2024-09-21T22:08:25.054 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025113799999999998 2024-09-21T22:08:25.054 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.054 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.054 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:25.054 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.097481+0000", 2024-09-21T22:08:25.054 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2344000000000002e-05 2024-09-21T22:08:25.054 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.054 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.055 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:25.055 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.159196+0000", 2024-09-21T22:08:25.055 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.061714369999999998 2024-09-21T22:08:25.055 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:25.055 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:25.055 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:25.055 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.055 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.055 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.6 87/86 [PushOp(2:7e2189ff:ns::rbd_data.127f79bc213d.0000000000000001:head, version: 57'4, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:7e2189ff:ns::rbd_data.127f79bc213d.0000000000000001:head@57'4, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:08:25.055 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:08:07.163314+0000", 2024-09-21T22:08:25.055 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 17.829369771, 2024-09-21T22:08:25.055 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.044796923000000002, 2024-09-21T22:08:25.055 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:25.055 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:25.056 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:08:25.056 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:25.056 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.056 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:25.056 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.163314+0000", 2024-09-21T22:08:25.056 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:25.056 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.056 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.056 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:25.056 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.163314+0000", 2024-09-21T22:08:25.056 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:25.056 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.056 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.056 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:25.056 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.163315+0000", 2024-09-21T22:08:25.057 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7120000000000001e-06 2024-09-21T22:08:25.057 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.057 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.057 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:25.057 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.164270+0000", 2024-09-21T22:08:25.057 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000955034 2024-09-21T22:08:25.057 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.057 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.057 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:25.057 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.164326+0000", 2024-09-21T22:08:25.057 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5627999999999997e-05 2024-09-21T22:08:25.057 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.057 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.057 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:25.057 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.164351+0000", 2024-09-21T22:08:25.058 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5219999999999999e-05 2024-09-21T22:08:25.058 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.058 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.058 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:25.058 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.164571+0000", 2024-09-21T22:08:25.058 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000219996 2024-09-21T22:08:25.058 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.058 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.058 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:25.058 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.164596+0000", 2024-09-21T22:08:25.058 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4564000000000001e-05 2024-09-21T22:08:25.058 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.058 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.058 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:25.058 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.208111+0000", 2024-09-21T22:08:25.059 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.043514769000000002 2024-09-21T22:08:25.059 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:25.059 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:25.059 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:25.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.059 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.059 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.6 87/86 [PushOp(2:7e2189ff:ns::rbd_data.127f79bc213d.0000000000000001:head, version: 57'4, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:7e2189ff:ns::rbd_data.127f79bc213d.0000000000000001:head@57'4, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:08:25.059 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:08:07.212243+0000", 2024-09-21T22:08:25.059 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 17.780440328000001, 2024-09-21T22:08:25.059 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.051665924000000002, 2024-09-21T22:08:25.059 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:25.059 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:25.059 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:08:25.059 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:25.060 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.060 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:25.060 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.212243+0000", 2024-09-21T22:08:25.060 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:25.060 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.060 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.060 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:25.060 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.212243+0000", 2024-09-21T22:08:25.060 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:25.060 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.060 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.060 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:25.060 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.212244+0000", 2024-09-21T22:08:25.060 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2839999999999999e-06 2024-09-21T22:08:25.061 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.061 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.061 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:25.061 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.212993+0000", 2024-09-21T22:08:25.061 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00074894100000000004 2024-09-21T22:08:25.061 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.061 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.061 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:25.061 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.213032+0000", 2024-09-21T22:08:25.061 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8948e-05 2024-09-21T22:08:25.061 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.061 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.061 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:25.061 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.213054+0000", 2024-09-21T22:08:25.061 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1937999999999999e-05 2024-09-21T22:08:25.061 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.062 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.062 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:25.062 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.213194+0000", 2024-09-21T22:08:25.062 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013942699999999999 2024-09-21T22:08:25.062 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.062 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.062 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:25.062 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.213210+0000", 2024-09-21T22:08:25.062 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5883999999999999e-05 2024-09-21T22:08:25.062 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.062 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.062 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:25.062 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.263909+0000", 2024-09-21T22:08:25.062 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.050699502 2024-09-21T22:08:25.062 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:25.063 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:25.063 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:25.063 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.063 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.063 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.6 87/86 [PushOp(2:7e2189ff:ns::rbd_data.127f79bc213d.0000000000000001:head, version: 57'4, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:7e2189ff:ns::rbd_data.127f79bc213d.0000000000000001:head@57'4, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 4194304, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:08:25.063 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:08:07.268604+0000", 2024-09-21T22:08:25.063 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 17.724079056000001, 2024-09-21T22:08:25.063 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.038386754000000002, 2024-09-21T22:08:25.063 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:25.063 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:25.063 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:08:25.063 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:25.063 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.063 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:25.063 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.268604+0000", 2024-09-21T22:08:25.064 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:25.064 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.064 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.064 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:25.064 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.268604+0000", 2024-09-21T22:08:25.064 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:25.064 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.065 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.065 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:25.066 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.268606+0000", 2024-09-21T22:08:25.066 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2020000000000001e-06 2024-09-21T22:08:25.066 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.066 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.066 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:25.066 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.269149+0000", 2024-09-21T22:08:25.066 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00054327900000000003 2024-09-21T22:08:25.066 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.066 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.066 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:25.066 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.269173+0000", 2024-09-21T22:08:25.066 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4046999999999999e-05 2024-09-21T22:08:25.066 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.067 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.067 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:25.067 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.269187+0000", 2024-09-21T22:08:25.067 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4185999999999999e-05 2024-09-21T22:08:25.067 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.067 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.067 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:25.067 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.269303+0000", 2024-09-21T22:08:25.067 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000115831 2024-09-21T22:08:25.067 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.067 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.067 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:25.067 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.269320+0000", 2024-09-21T22:08:25.067 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6878999999999999e-05 2024-09-21T22:08:25.067 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.068 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.068 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:25.068 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.306991+0000", 2024-09-21T22:08:25.068 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.037671330000000003 2024-09-21T22:08:25.068 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:25.068 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:25.068 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:25.068 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.068 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.068 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.6 87/86 [PushOp(2:762dd302:ns::rbd_data.127f79bc213d.0000000000000007:head, version: 57'5, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:762dd302:ns::rbd_data.127f79bc213d.0000000000000007:head@57'5, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T22:08:25.068 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:08:07.310599+0000", 2024-09-21T22:08:25.068 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 17.682084862, 2024-09-21T22:08:25.068 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025565551999999998, 2024-09-21T22:08:25.068 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:25.069 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:25.069 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:08:25.069 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:25.069 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.069 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:25.069 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.310599+0000", 2024-09-21T22:08:25.069 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:25.069 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.069 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.069 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:25.069 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.310599+0000", 2024-09-21T22:08:25.069 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:25.069 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.069 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.069 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:25.070 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.310600+0000", 2024-09-21T22:08:25.070 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0559999999999999e-06 2024-09-21T22:08:25.070 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.070 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.070 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:25.070 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.311158+0000", 2024-09-21T22:08:25.070 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00055862200000000005 2024-09-21T22:08:25.070 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.070 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.070 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:25.070 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.311180+0000", 2024-09-21T22:08:25.070 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1577999999999999e-05 2024-09-21T22:08:25.070 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.070 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.071 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:25.071 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.311198+0000", 2024-09-21T22:08:25.071 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8359000000000002e-05 2024-09-21T22:08:25.071 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.071 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.071 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:25.071 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.311291+0000", 2024-09-21T22:08:25.071 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2887999999999996e-05 2024-09-21T22:08:25.071 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.071 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.071 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:25.071 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.311300+0000", 2024-09-21T22:08:25.071 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8640000000000002e-06 2024-09-21T22:08:25.071 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.071 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.072 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:25.072 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.336164+0000", 2024-09-21T22:08:25.072 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024864185 2024-09-21T22:08:25.072 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:25.072 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:25.072 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:25.072 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.072 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.072 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.6 87/86 [PushOp(2:762dd302:ns::rbd_data.127f79bc213d.0000000000000007:head, version: 57'5, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:762dd302:ns::rbd_data.127f79bc213d.0000000000000007:head@57'5, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:08:25.072 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:08:07.338463+0000", 2024-09-21T22:08:25.072 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 17.654220251000002, 2024-09-21T22:08:25.072 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026221639000000001, 2024-09-21T22:08:25.072 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:25.072 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:25.072 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:08:25.073 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:25.073 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.073 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:25.073 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.338463+0000", 2024-09-21T22:08:25.073 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:25.073 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.073 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.073 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:25.073 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.338463+0000", 2024-09-21T22:08:25.073 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:25.073 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.073 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.073 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:25.073 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.338464+0000", 2024-09-21T22:08:25.074 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.035e-06 2024-09-21T22:08:25.074 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.074 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.074 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:25.074 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.339044+0000", 2024-09-21T22:08:25.074 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00057999699999999996 2024-09-21T22:08:25.074 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.074 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.074 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:25.074 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.339070+0000", 2024-09-21T22:08:25.074 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.565e-05 2024-09-21T22:08:25.074 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.074 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.074 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:25.074 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.339088+0000", 2024-09-21T22:08:25.075 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8189000000000002e-05 2024-09-21T22:08:25.075 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.075 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.075 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:25.075 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.339184+0000", 2024-09-21T22:08:25.075 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5475000000000003e-05 2024-09-21T22:08:25.075 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.075 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.075 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:25.075 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.339193+0000", 2024-09-21T22:08:25.075 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7790000000000007e-06 2024-09-21T22:08:25.075 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.075 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.075 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:25.076 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.364685+0000", 2024-09-21T22:08:25.076 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025491514 2024-09-21T22:08:25.076 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:25.076 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:25.076 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:25.076 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.076 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.076 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5398.0:204 2.6 e89/86 2:633e2f34:::rbd_data.126483d75f2d.0000000000000001:head v 89'32, mlcod=88'29)", 2024-09-21T22:08:25.076 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:08:10.148990+0000", 2024-09-21T22:08:25.076 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 14.843693706, 2024-09-21T22:08:25.076 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.034897343999999997, 2024-09-21T22:08:25.076 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:25.076 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:25.076 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:25.076 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:25.077 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.077 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:25.077 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:10.148990+0000", 2024-09-21T22:08:25.077 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:25.077 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.077 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.077 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:25.077 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:10.148990+0000", 2024-09-21T22:08:25.077 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:25.077 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.077 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.077 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:25.077 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:10.148991+0000", 2024-09-21T22:08:25.077 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.61e-06 2024-09-21T22:08:25.077 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.078 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.078 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:25.078 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:10.149966+0000", 2024-09-21T22:08:25.078 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00097419800000000001 2024-09-21T22:08:25.078 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.078 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.078 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:25.078 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:10.149996+0000", 2024-09-21T22:08:25.078 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0762999999999997e-05 2024-09-21T22:08:25.078 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.078 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.078 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:25.078 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:10.150017+0000", 2024-09-21T22:08:25.078 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.109e-05 2024-09-21T22:08:25.079 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.079 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.079 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:25.079 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:10.150090+0000", 2024-09-21T22:08:25.079 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2835000000000003e-05 2024-09-21T22:08:25.079 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.079 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.079 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:25.079 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:10.150112+0000", 2024-09-21T22:08:25.079 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1746000000000002e-05 2024-09-21T22:08:25.079 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.079 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.079 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:25.079 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:10.183831+0000", 2024-09-21T22:08:25.079 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.033718571000000003 2024-09-21T22:08:25.080 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:25.080 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:25.080 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:25.080 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:10.183887+0000", 2024-09-21T22:08:25.080 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6530999999999998e-05 2024-09-21T22:08:25.080 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:25.080 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:25.080 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:25.080 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:25.080 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:25.080 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:08:25.082 INFO:tasks.workunit.client.0.smithi157.stderr:+ return 0 2024-09-21T22:08:25.082 INFO:tasks.workunit.client.0.smithi157.stderr:+ ps -C rbd-nbd 2024-09-21T22:08:25.082 INFO:tasks.workunit.client.0.smithi157.stderr:+ expect_false grep '^ *25100\b' 2024-09-21T22:08:25.082 INFO:tasks.workunit.client.0.smithi157.stderr:+ grep '^ *25100\b' 2024-09-21T22:08:25.082 INFO:tasks.workunit.client.0.smithi157.stderr:+ return 0 2024-09-21T22:08:25.082 INFO:tasks.workunit.client.0.smithi157.stderr:+ sleep 0.5 2024-09-21T22:08:25.083 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T22:08:25.083 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:08:25.077+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:08:25.182 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:08:25.179+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:08:25.259 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:08:25.259 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:08:25.260 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-09-21T22:08:25.260 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:08:25.274 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T22:08:25.311 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T22:08:25.318 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:08:25.315+0000 7f2075a06640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31720) UID: 0 2024-09-21T22:08:25.384 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:08:25.381+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:08:25.445 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:08:25.445 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:08:25.445 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-21T22:08:25.445 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-21T22:08:25.445 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:08:25.457 INFO:teuthology.orchestra.run.smithi039.stdout:osd.0: {} 2024-09-21T22:08:25.458 INFO:teuthology.orchestra.run.smithi039.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T22:08:25.460 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T22:08:25.474 INFO:teuthology.orchestra.run.smithi039.stdout:osd.1: {} 2024-09-21T22:08:25.474 INFO:teuthology.orchestra.run.smithi039.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T22:08:25.485 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:08:25.481+0000 7fe38af2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 34631) UID: 0 2024-09-21T22:08:25.490 INFO:teuthology.orchestra.run.smithi039.stdout:osd.2: {} 2024-09-21T22:08:25.490 INFO:teuthology.orchestra.run.smithi039.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T22:08:25.508 INFO:teuthology.orchestra.run.smithi039.stdout:osd.3: {} 2024-09-21T22:08:25.508 INFO:teuthology.orchestra.run.smithi039.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T22:08:25.524 INFO:tasks.workunit.client.0.smithi157.stdout:OK 2024-09-21T22:08:25.525 INFO:tasks.workunit.client.0.smithi157.stderr:+ DEV= 2024-09-21T22:08:25.525 INFO:tasks.workunit.client.0.smithi157.stderr:+ echo OK 2024-09-21T22:08:25.525 INFO:tasks.workunit.client.0.smithi157.stderr:+ cleanup 2024-09-21T22:08:25.525 INFO:tasks.workunit.client.0.smithi157.stderr:+ local ns s 2024-09-21T22:08:25.525 INFO:tasks.workunit.client.0.smithi157.stderr:+ set +e 2024-09-21T22:08:25.525 INFO:tasks.workunit.client.0.smithi157.stderr:+ mount 2024-09-21T22:08:25.526 INFO:tasks.workunit.client.0.smithi157.stderr:+ fgrep /tmp/tmp.GqKuv7rsZi/mnt 2024-09-21T22:08:25.526 INFO:teuthology.orchestra.run.smithi039.stdout:osd.4: {} 2024-09-21T22:08:25.526 INFO:teuthology.orchestra.run.smithi039.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T22:08:25.531 INFO:tasks.workunit.client.0.smithi157.stderr:+ rm -Rf /tmp/tmp.GqKuv7rsZi 2024-09-21T22:08:25.545 INFO:tasks.workunit.client.0.smithi157.stderr:+ '[' -n '' ']' 2024-09-21T22:08:25.545 INFO:tasks.workunit.client.0.smithi157.stderr:+ for ns in '' ${NS} 2024-09-21T22:08:25.545 INFO:tasks.workunit.client.0.smithi157.stderr:+ rbd -p rbd --namespace '' status testrbdnbd21553 2024-09-21T22:08:25.545 INFO:teuthology.orchestra.run.smithi039.stdout:osd.5: {} 2024-09-21T22:08:25.545 INFO:teuthology.orchestra.run.smithi039.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T22:08:25.586 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:08:25.583+0000 7f2075a06640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31720) UID: 0 2024-09-21T22:08:25.682 INFO:tasks.workunit.client.0.smithi157.stdout:Watchers: none 2024-09-21T22:08:25.682 INFO:tasks.workunit.client.0.smithi157.stderr:+ for s in 0.5 1 2 4 8 16 32 2024-09-21T22:08:25.682 INFO:tasks.workunit.client.0.smithi157.stderr:+ sleep 0.5 2024-09-21T22:08:25.683 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:25.683 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:25.683 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T22:08:25.726 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:08:25.723+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:08:25.788 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:08:25.785+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:08:25.815 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:25.815 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:25.830 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T22:08:25.920 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:08:25.917+0000 7fe38af2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 34631) UID: 0 2024-09-21T22:08:25.941 INFO:teuthology.orchestra.run.smithi039.stderr:noscrub is set 2024-09-21T22:08:26.003 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:26.003 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:26.003 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:08:25.997+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:08:26.017 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T22:08:26.104 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:08:26.101+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:08:26.109 INFO:tasks.workunit.client.0.smithi157.stderr:+ rbd -p rbd --namespace '' status testrbdnbd21553 2024-09-21T22:08:26.109 INFO:tasks.workunit.client.0.smithi157.stderr:+ grep 'Watchers: none' 2024-09-21T22:08:26.178 INFO:tasks.workunit.client.0.smithi157.stdout:Watchers: none 2024-09-21T22:08:26.178 INFO:tasks.workunit.client.0.smithi157.stderr:+ break 2024-09-21T22:08:26.178 INFO:tasks.workunit.client.0.smithi157.stderr:+ rbd -p rbd --namespace '' snap purge testrbdnbd21553 2024-09-21T22:08:26.190 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:26.190 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:26.191 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:08:26.185+0000 7fe38af2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 34631) UID: 0 2024-09-21T22:08:26.205 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T22:08:26.295 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:08:26.291+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:08:26.375 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:26.375 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:26.389 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T22:08:26.395 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:08:26.395+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:08:26.494 INFO:tasks.thrashosds.thrasher:in_osds: [3, 4, 5, 0] out_osds: [2, 1] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 2] 2024-09-21T22:08:26.494 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T22:08:26.495 INFO:tasks.thrashosds.thrasher:Adding osd 1 2024-09-21T22:08:26.495 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 1 2024-09-21T22:08:26.496 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:08:26.491+0000 7f2075a06640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31720) UID: 0 2024-09-21T22:08:26.516 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:26.516 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:26.531 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T22:08:26.616 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:08:26.613+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:08:26.695 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:26.696 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:26.698 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:08:26.699+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:08:26.710 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T22:08:26.799 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:08:26.793+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:08:26.808 INFO:tasks.ceph.mon.a.smithi039.stderr:2024-09-21T22:08:26.805+0000 7f9967196640 -1 mon.a@0(leader).osd e96 definitely_dead 0 2024-09-21T22:08:26.878 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:26.878 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:26.893 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T22:08:26.899 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:08:26.895+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:08:26.954 INFO:teuthology.orchestra.run.smithi039.stderr:marked in osd.1. 2024-09-21T22:08:26.986 INFO:tasks.thrashosds.thrasher:Added osd 1 2024-09-21T22:08:27.000 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:08:26.995+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:08:27.073 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:27.073 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:27.088 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T22:08:27.132 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:08:27.129+0000 7fe38af2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 34631) UID: 0 2024-09-21T22:08:27.202 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:08:27.199+0000 7f2075a06640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31720) UID: 0 2024-09-21T22:08:27.215 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:27.215 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:27.229 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T22:08:27.302 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:08:27.297+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:08:27.364 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:27.365 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:27.379 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T22:08:27.403 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:08:27.401+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:08:27.503 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:27.503 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:27.504 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:08:27.499+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:08:27.518 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T22:08:27.563 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T22:08:27.608 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:08:27.605+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:08:27.682 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:27.682 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:27.698 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T22:08:27.719 INFO:teuthology.orchestra.run.smithi039.stdout:osd.0: {} 2024-09-21T22:08:27.719 INFO:teuthology.orchestra.run.smithi039.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T22:08:27.736 INFO:teuthology.orchestra.run.smithi039.stdout:osd.1: {} 2024-09-21T22:08:27.736 INFO:teuthology.orchestra.run.smithi039.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T22:08:27.746 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:08:27.743+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:08:27.752 INFO:teuthology.orchestra.run.smithi039.stdout:osd.2: {} 2024-09-21T22:08:27.753 INFO:teuthology.orchestra.run.smithi039.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T22:08:27.772 INFO:teuthology.orchestra.run.smithi039.stdout:osd.3: {} 2024-09-21T22:08:27.773 INFO:teuthology.orchestra.run.smithi039.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T22:08:27.792 INFO:teuthology.orchestra.run.smithi039.stdout:osd.4: {} 2024-09-21T22:08:27.792 INFO:teuthology.orchestra.run.smithi039.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T22:08:27.807 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:08:27.803+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:08:27.814 INFO:teuthology.orchestra.run.smithi039.stdout:osd.5: {} 2024-09-21T22:08:27.814 INFO:teuthology.orchestra.run.smithi039.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T22:08:27.824 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:08:27.824 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:08:27.824 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-09-21T22:08:27.824 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:08:27.838 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T22:08:27.926 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:08:27.923+0000 7f2075a06640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31720) UID: 0 2024-09-21T22:08:27.964 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-21T22:08:28.008 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:08:28.003+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:08:28.015 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:08:28.015 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:08:28.015 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-21T22:08:28.015 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-21T22:08:28.015 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:08:28.029 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T22:08:28.108 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:08:28.105+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:08:28.210 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:08:28.210 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-21T22:08:28.210 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-21T22:08:28.210 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-21T22:08:28.210 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.211 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4840.0:676 2.7 e63/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 63'57, mlcod=63'55)", 2024-09-21T22:08:28.211 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.627276+0000", 2024-09-21T22:08:28.211 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 63.581346205999999, 2024-09-21T22:08:28.211 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.037500787000000001, 2024-09-21T22:08:28.211 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:28.211 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:28.211 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:28.211 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:28.211 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.211 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:28.211 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.627276+0000", 2024-09-21T22:08:28.211 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:28.211 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.212 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.212 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:28.212 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.627276+0000", 2024-09-21T22:08:28.212 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:28.212 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.212 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.212 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:28.212 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.627277+0000", 2024-09-21T22:08:28.212 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.068e-06 2024-09-21T22:08:28.212 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.212 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.212 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:28.212 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.628148+0000", 2024-09-21T22:08:28.212 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00087069999999999997 2024-09-21T22:08:28.213 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.213 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.213 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:28.213 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.628170+0000", 2024-09-21T22:08:28.213 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2147999999999999e-05 2024-09-21T22:08:28.213 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.213 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.213 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:28.213 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.628179+0000", 2024-09-21T22:08:28.213 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2450000000000004e-06 2024-09-21T22:08:28.213 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.213 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.213 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:28.213 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.628279+0000", 2024-09-21T22:08:28.214 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9298999999999999e-05 2024-09-21T22:08:28.214 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.214 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.214 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:28.214 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.628305+0000", 2024-09-21T22:08:28.214 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6316999999999999e-05 2024-09-21T22:08:28.214 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.214 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.214 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:28.214 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.664714+0000", 2024-09-21T22:08:28.214 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.036408923000000003 2024-09-21T22:08:28.214 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.214 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.214 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:28.215 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.664777+0000", 2024-09-21T22:08:28.215 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3087000000000001e-05 2024-09-21T22:08:28.215 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:28.215 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:28.215 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:28.215 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.215 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.215 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4840.0:678 2.7 e63/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 63'59, mlcod=63'57)", 2024-09-21T22:08:28.215 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.679689+0000", 2024-09-21T22:08:28.215 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 63.528933705999997, 2024-09-21T22:08:28.215 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.053308406000000003, 2024-09-21T22:08:28.216 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:28.216 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:28.216 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:28.216 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:28.216 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.216 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:28.216 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.679689+0000", 2024-09-21T22:08:28.216 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:28.216 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.216 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.216 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:28.216 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.679689+0000", 2024-09-21T22:08:28.217 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:28.217 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.217 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.217 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:28.217 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.679691+0000", 2024-09-21T22:08:28.217 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9400000000000001e-06 2024-09-21T22:08:28.217 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.217 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.217 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:28.217 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.680245+0000", 2024-09-21T22:08:28.217 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00055382999999999995 2024-09-21T22:08:28.217 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.218 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.218 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:28.218 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.680276+0000", 2024-09-21T22:08:28.218 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1123e-05 2024-09-21T22:08:28.218 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.218 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.218 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:28.218 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.680290+0000", 2024-09-21T22:08:28.218 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4056999999999999e-05 2024-09-21T22:08:28.218 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.218 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.218 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:28.218 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.680418+0000", 2024-09-21T22:08:28.219 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000127792 2024-09-21T22:08:28.219 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.219 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.219 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:28.219 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.680465+0000", 2024-09-21T22:08:28.219 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7360000000000001e-05 2024-09-21T22:08:28.219 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.219 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.219 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:28.219 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.732923+0000", 2024-09-21T22:08:28.219 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.052458312999999999 2024-09-21T22:08:28.219 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.219 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.219 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:28.220 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.732997+0000", 2024-09-21T22:08:28.220 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.3991000000000004e-05 2024-09-21T22:08:28.220 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:28.220 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:28.220 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:28.220 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.220 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.220 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4840.0:680 2.7 e63/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 63'61, mlcod=63'59)", 2024-09-21T22:08:28.220 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.745996+0000", 2024-09-21T22:08:28.220 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 63.462626192999998, 2024-09-21T22:08:28.220 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.061804132999999997, 2024-09-21T22:08:28.220 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:28.220 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:28.220 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:28.220 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:28.221 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.221 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:28.221 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.745996+0000", 2024-09-21T22:08:28.221 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:28.221 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.221 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.221 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:28.221 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.745996+0000", 2024-09-21T22:08:28.221 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:28.221 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.221 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.221 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:28.221 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.745998+0000", 2024-09-21T22:08:28.221 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.717e-06 2024-09-21T22:08:28.222 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.222 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.222 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:28.222 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.746523+0000", 2024-09-21T22:08:28.222 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00052525399999999996 2024-09-21T22:08:28.222 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.222 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.222 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:28.222 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.746554+0000", 2024-09-21T22:08:28.222 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0586000000000002e-05 2024-09-21T22:08:28.222 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.222 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.222 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:28.223 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.746577+0000", 2024-09-21T22:08:28.223 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.3079e-05 2024-09-21T22:08:28.223 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.223 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.223 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:28.223 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.746689+0000", 2024-09-21T22:08:28.223 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011177 2024-09-21T22:08:28.223 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.223 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.223 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:28.223 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.746722+0000", 2024-09-21T22:08:28.223 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2870000000000002e-05 2024-09-21T22:08:28.223 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.223 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.223 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:28.224 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.807740+0000", 2024-09-21T22:08:28.224 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.061018612 2024-09-21T22:08:28.224 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.224 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.224 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:28.224 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.807800+0000", 2024-09-21T22:08:28.224 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0244999999999997e-05 2024-09-21T22:08:28.224 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:28.224 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:28.224 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:28.224 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.224 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.224 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.4840.0:682 2.7 e63/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 63'63, mlcod=63'61)", 2024-09-21T22:08:28.224 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.824126+0000", 2024-09-21T22:08:28.225 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 63.384496624000001, 2024-09-21T22:08:28.225 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.038159327, 2024-09-21T22:08:28.225 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:28.225 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:28.225 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:28.225 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:28.225 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.225 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:28.225 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.824126+0000", 2024-09-21T22:08:28.225 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:28.225 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.225 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.225 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:28.226 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.824126+0000", 2024-09-21T22:08:28.226 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:28.226 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.226 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.226 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:28.226 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.824128+0000", 2024-09-21T22:08:28.226 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9659999999999999e-06 2024-09-21T22:08:28.226 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.226 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.226 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:28.226 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.824842+0000", 2024-09-21T22:08:28.226 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00071423999999999999 2024-09-21T22:08:28.226 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.226 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.227 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:28.227 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.824883+0000", 2024-09-21T22:08:28.227 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.1131999999999999e-05 2024-09-21T22:08:28.227 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.227 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.227 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:28.227 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.824931+0000", 2024-09-21T22:08:28.227 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7327000000000003e-05 2024-09-21T22:08:28.227 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.227 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.227 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:28.227 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.825058+0000", 2024-09-21T22:08:28.228 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012790499999999999 2024-09-21T22:08:28.228 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.228 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.228 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:28.228 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.825087+0000", 2024-09-21T22:08:28.228 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8192e-05 2024-09-21T22:08:28.228 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.228 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.228 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:28.228 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.862225+0000", 2024-09-21T22:08:28.228 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.037138309000000001 2024-09-21T22:08:28.228 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.228 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.228 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:28.228 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.862285+0000", 2024-09-21T22:08:28.229 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.0256e-05 2024-09-21T22:08:28.229 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:28.229 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:28.229 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:28.229 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.229 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.229 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5008.0:116 2.7 e71/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 71'148, mlcod=71'146)", 2024-09-21T22:08:28.229 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:37.867612+0000", 2024-09-21T22:08:28.229 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 50.341010128999997, 2024-09-21T22:08:28.229 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.045123155999999998, 2024-09-21T22:08:28.229 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:28.229 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:28.229 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:28.229 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:28.230 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.230 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:28.230 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.867612+0000", 2024-09-21T22:08:28.230 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:28.230 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.230 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.230 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:28.230 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.867612+0000", 2024-09-21T22:08:28.230 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:28.230 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.230 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.230 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:28.230 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.867613+0000", 2024-09-21T22:08:28.230 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1200000000000002e-07 2024-09-21T22:08:28.231 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.231 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.231 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:28.231 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.868135+0000", 2024-09-21T22:08:28.231 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00052132700000000003 2024-09-21T22:08:28.231 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.231 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.231 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:28.231 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.868150+0000", 2024-09-21T22:08:28.231 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5271e-05 2024-09-21T22:08:28.231 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.232 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.232 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:28.232 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.868156+0000", 2024-09-21T22:08:28.232 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.4969999999999998e-06 2024-09-21T22:08:28.232 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.232 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.232 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:28.232 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.868213+0000", 2024-09-21T22:08:28.232 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.7114999999999998e-05 2024-09-21T22:08:28.232 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.232 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.232 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:28.232 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.868238+0000", 2024-09-21T22:08:28.232 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4987999999999999e-05 2024-09-21T22:08:28.232 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.233 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.233 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:28.233 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.912677+0000", 2024-09-21T22:08:28.233 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.044439019000000003 2024-09-21T22:08:28.233 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.233 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.233 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:28.233 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.912736+0000", 2024-09-21T22:08:28.233 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8127000000000001e-05 2024-09-21T22:08:28.233 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:28.233 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:28.233 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:28.233 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.233 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.234 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5008.0:118 2.7 e71/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 71'150, mlcod=71'148)", 2024-09-21T22:08:28.234 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:37.924700+0000", 2024-09-21T22:08:28.234 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 50.283922938000003, 2024-09-21T22:08:28.234 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.034650561000000003, 2024-09-21T22:08:28.234 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:28.234 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:28.234 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:28.234 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:28.234 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.234 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:28.234 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.924700+0000", 2024-09-21T22:08:28.234 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:28.234 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.234 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.234 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:28.235 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.924700+0000", 2024-09-21T22:08:28.235 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:28.235 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.235 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.235 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:28.235 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.924702+0000", 2024-09-21T22:08:28.235 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9460000000000001e-06 2024-09-21T22:08:28.235 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.235 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.235 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:28.235 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.925132+0000", 2024-09-21T22:08:28.235 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00043021200000000002 2024-09-21T22:08:28.235 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.235 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.236 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:28.236 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.925156+0000", 2024-09-21T22:08:28.236 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4451999999999999e-05 2024-09-21T22:08:28.236 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.236 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.236 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:28.236 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.925168+0000", 2024-09-21T22:08:28.236 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2249000000000001e-05 2024-09-21T22:08:28.236 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.236 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.236 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:28.236 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.925241+0000", 2024-09-21T22:08:28.236 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2502000000000005e-05 2024-09-21T22:08:28.236 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.236 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.237 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:28.237 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.925267+0000", 2024-09-21T22:08:28.237 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6471000000000001e-05 2024-09-21T22:08:28.237 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.237 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.237 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:28.237 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.959217+0000", 2024-09-21T22:08:28.237 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.033949569999999998 2024-09-21T22:08:28.237 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.237 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.237 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:28.238 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.959350+0000", 2024-09-21T22:08:28.238 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013315899999999999 2024-09-21T22:08:28.238 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:28.238 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:28.238 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:28.238 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.238 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.238 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5008.0:120 2.7 e71/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 71'152, mlcod=71'150)", 2024-09-21T22:08:28.238 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:37.970837+0000", 2024-09-21T22:08:28.238 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 50.237785686999999, 2024-09-21T22:08:28.238 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.03044943, 2024-09-21T22:08:28.238 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:28.238 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:28.238 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:28.238 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:28.239 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.239 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:28.239 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.970837+0000", 2024-09-21T22:08:28.239 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:28.239 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.239 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.239 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:28.239 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.970837+0000", 2024-09-21T22:08:28.239 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:28.239 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.239 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.239 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:28.239 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.970838+0000", 2024-09-21T22:08:28.239 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4649999999999999e-06 2024-09-21T22:08:28.240 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.240 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.240 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:28.240 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.971265+0000", 2024-09-21T22:08:28.240 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00042665800000000001 2024-09-21T22:08:28.240 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.240 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.240 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:28.240 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.971285+0000", 2024-09-21T22:08:28.240 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9639999999999998e-05 2024-09-21T22:08:28.240 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.240 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.240 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:28.240 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.971294+0000", 2024-09-21T22:08:28.241 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7210000000000006e-06 2024-09-21T22:08:28.241 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.241 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.241 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:28.241 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.971354+0000", 2024-09-21T22:08:28.241 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.9613e-05 2024-09-21T22:08:28.241 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.241 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.241 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:28.241 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:37.971381+0000", 2024-09-21T22:08:28.241 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7538999999999998e-05 2024-09-21T22:08:28.241 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.241 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.241 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:28.241 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.001199+0000", 2024-09-21T22:08:28.242 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029817537000000002 2024-09-21T22:08:28.242 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.242 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.242 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:28.242 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.001286+0000", 2024-09-21T22:08:28.242 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7256999999999995e-05 2024-09-21T22:08:28.242 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:28.242 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:28.242 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:28.242 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.242 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.242 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5008.0:122 2.7 e71/16 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head v 71'154, mlcod=71'152)", 2024-09-21T22:08:28.242 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:38.014709+0000", 2024-09-21T22:08:28.242 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 50.193913811999998, 2024-09-21T22:08:28.242 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029567408999999999, 2024-09-21T22:08:28.243 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:28.243 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:28.243 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:28.243 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:28.243 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.243 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:28.243 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.014709+0000", 2024-09-21T22:08:28.243 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:28.243 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.243 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.243 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:28.243 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.014709+0000", 2024-09-21T22:08:28.243 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:28.243 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.244 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.244 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:28.244 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.014709+0000", 2024-09-21T22:08:28.244 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9599999999999999e-07 2024-09-21T22:08:28.244 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.244 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.244 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:28.244 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.015201+0000", 2024-09-21T22:08:28.244 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00049175599999999999 2024-09-21T22:08:28.244 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.244 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.244 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:28.244 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.015215+0000", 2024-09-21T22:08:28.244 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3692999999999999e-05 2024-09-21T22:08:28.244 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.245 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.245 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:28.245 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.015225+0000", 2024-09-21T22:08:28.245 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.815e-06 2024-09-21T22:08:28.245 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.245 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.245 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:28.245 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.015287+0000", 2024-09-21T22:08:28.245 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1858000000000003e-05 2024-09-21T22:08:28.245 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.245 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.245 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:28.245 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.015315+0000", 2024-09-21T22:08:28.245 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8490999999999999e-05 2024-09-21T22:08:28.245 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.246 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.246 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:28.246 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.044182+0000", 2024-09-21T22:08:28.246 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.028866936999999999 2024-09-21T22:08:28.246 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.246 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.246 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:28.246 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.044276+0000", 2024-09-21T22:08:28.246 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4163e-05 2024-09-21T22:08:28.246 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:28.246 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:28.246 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:28.246 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.246 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.247 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5207.0:70 2.7 e79/16 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head v 79'184, mlcod=79'181)", 2024-09-21T22:08:28.247 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:52.938385+0000", 2024-09-21T22:08:28.247 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 35.270237905999998, 2024-09-21T22:08:28.247 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023928324000000001, 2024-09-21T22:08:28.247 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:28.247 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:28.247 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:28.247 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:28.247 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.247 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:28.247 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938385+0000", 2024-09-21T22:08:28.248 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:28.248 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.248 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.248 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:28.248 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938385+0000", 2024-09-21T22:08:28.248 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:28.248 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.248 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.248 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:28.248 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938387+0000", 2024-09-21T22:08:28.248 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1639999999999999e-06 2024-09-21T22:08:28.248 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.248 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.248 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:28.249 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938511+0000", 2024-09-21T22:08:28.249 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000124485 2024-09-21T22:08:28.249 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.249 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.249 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:28.249 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938549+0000", 2024-09-21T22:08:28.249 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7347e-05 2024-09-21T22:08:28.249 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.249 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.249 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:28.249 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938570+0000", 2024-09-21T22:08:28.249 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1013e-05 2024-09-21T22:08:28.249 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.249 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.249 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:28.250 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938718+0000", 2024-09-21T22:08:28.250 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000148789 2024-09-21T22:08:28.250 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.250 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.250 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:28.250 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938769+0000", 2024-09-21T22:08:28.250 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1002999999999999e-05 2024-09-21T22:08:28.250 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.250 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.250 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:28.250 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.962199+0000", 2024-09-21T22:08:28.250 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.023429783999999999 2024-09-21T22:08:28.250 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.250 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.251 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:28.251 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.962313+0000", 2024-09-21T22:08:28.251 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000113739 2024-09-21T22:08:28.251 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:28.251 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:28.251 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:28.251 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.251 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.251 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5207.0:71 2.7 e79/16 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head v 79'185, mlcod=79'181)", 2024-09-21T22:08:28.251 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:52.951522+0000", 2024-09-21T22:08:28.251 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 35.257100014000002, 2024-09-21T22:08:28.251 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.021108028000000001, 2024-09-21T22:08:28.251 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:28.251 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:28.251 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:28.252 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:28.252 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.252 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:28.252 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.951522+0000", 2024-09-21T22:08:28.252 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:28.252 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.252 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.252 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:28.252 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.951522+0000", 2024-09-21T22:08:28.252 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:28.252 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.252 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.252 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:28.252 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.951524+0000", 2024-09-21T22:08:28.252 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9199999999999998e-06 2024-09-21T22:08:28.253 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.253 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.253 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:28.253 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.951686+0000", 2024-09-21T22:08:28.253 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016202499999999999 2024-09-21T22:08:28.253 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.253 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.253 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:28.253 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.951713+0000", 2024-09-21T22:08:28.253 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7007000000000001e-05 2024-09-21T22:08:28.253 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.253 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.253 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:28.253 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.951734+0000", 2024-09-21T22:08:28.254 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0228999999999999e-05 2024-09-21T22:08:28.254 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.254 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.254 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:28.254 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.952740+0000", 2024-09-21T22:08:28.254 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001006631 2024-09-21T22:08:28.254 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.254 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.254 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:28.254 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.952778+0000", 2024-09-21T22:08:28.254 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7316e-05 2024-09-21T22:08:28.254 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.254 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.254 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:28.255 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.972505+0000", 2024-09-21T22:08:28.255 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.019727776999999998 2024-09-21T22:08:28.255 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.255 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.255 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:28.255 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.972631+0000", 2024-09-21T22:08:28.255 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012512299999999999 2024-09-21T22:08:28.255 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:28.255 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:28.255 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:28.255 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.255 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.255 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5207.0:76 2.7 e79/16 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head v 79'190, mlcod=79'186)", 2024-09-21T22:08:28.256 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:53.001917+0000", 2024-09-21T22:08:28.256 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 35.206705565, 2024-09-21T22:08:28.256 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022783040000000001, 2024-09-21T22:08:28.256 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:28.256 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:28.256 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:28.256 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:28.256 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.256 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:28.256 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.001917+0000", 2024-09-21T22:08:28.256 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:28.256 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.256 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.256 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:28.257 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.001917+0000", 2024-09-21T22:08:28.257 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:28.257 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.257 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.257 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:28.257 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.001919+0000", 2024-09-21T22:08:28.257 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.703e-06 2024-09-21T22:08:28.257 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.257 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.257 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:28.257 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.002006+0000", 2024-09-21T22:08:28.257 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6978999999999995e-05 2024-09-21T22:08:28.258 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.258 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.258 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:28.258 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.002042+0000", 2024-09-21T22:08:28.258 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6847000000000001e-05 2024-09-21T22:08:28.258 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.258 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.258 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:28.258 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.002057+0000", 2024-09-21T22:08:28.258 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.419e-05 2024-09-21T22:08:28.258 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.258 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.259 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:28.259 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.004387+0000", 2024-09-21T22:08:28.259 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.002330365 2024-09-21T22:08:28.259 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.259 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.259 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:28.259 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.004420+0000", 2024-09-21T22:08:28.259 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.3170000000000003e-05 2024-09-21T22:08:28.259 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.259 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.259 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:28.259 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.024610+0000", 2024-09-21T22:08:28.259 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020190084000000001 2024-09-21T22:08:28.259 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.260 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.260 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:28.260 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.024700+0000", 2024-09-21T22:08:28.260 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9702000000000003e-05 2024-09-21T22:08:28.260 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:28.260 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:28.260 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:28.260 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.260 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.260 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5207.0:93 2.7 e79/16 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head v 79'193, mlcod=79'191)", 2024-09-21T22:08:28.260 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:53.149813+0000", 2024-09-21T22:08:28.260 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 35.058809506000003, 2024-09-21T22:08:28.260 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.060053738000000002, 2024-09-21T22:08:28.260 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:28.261 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:28.261 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:28.261 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:28.261 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.261 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:28.261 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.149813+0000", 2024-09-21T22:08:28.261 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:28.261 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.261 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.261 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:28.261 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.149813+0000", 2024-09-21T22:08:28.261 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:28.261 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.261 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.262 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:28.262 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.149815+0000", 2024-09-21T22:08:28.262 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9099999999999999e-06 2024-09-21T22:08:28.262 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.262 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.262 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:28.262 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.150828+0000", 2024-09-21T22:08:28.262 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0010129099999999999 2024-09-21T22:08:28.262 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.262 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.262 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:28.262 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.150860+0000", 2024-09-21T22:08:28.262 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2443000000000001e-05 2024-09-21T22:08:28.262 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.263 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.263 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:28.263 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.150874+0000", 2024-09-21T22:08:28.263 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3627e-05 2024-09-21T22:08:28.263 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.263 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.263 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:28.263 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.151010+0000", 2024-09-21T22:08:28.263 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013656599999999999 2024-09-21T22:08:28.263 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.263 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.263 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:28.263 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.151047+0000", 2024-09-21T22:08:28.263 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6495999999999998e-05 2024-09-21T22:08:28.263 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.264 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.264 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:28.264 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.209791+0000", 2024-09-21T22:08:28.264 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058744004000000002 2024-09-21T22:08:28.264 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.264 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.264 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:28.264 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.209867+0000", 2024-09-21T22:08:28.264 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5782000000000001e-05 2024-09-21T22:08:28.264 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:28.264 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:28.264 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:28.264 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.264 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.265 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5207.0:128 2.7 e83/16 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head v 83'198, mlcod=83'195)", 2024-09-21T22:08:28.265 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:08:01.058831+0000", 2024-09-21T22:08:28.265 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 27.149791871000001, 2024-09-21T22:08:28.265 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058096534999999998, 2024-09-21T22:08:28.265 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:28.265 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:28.265 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:28.265 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:28.265 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.265 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:28.265 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.058831+0000", 2024-09-21T22:08:28.265 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:28.265 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.265 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.266 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:28.266 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.058831+0000", 2024-09-21T22:08:28.266 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:28.266 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.266 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.266 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:28.266 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.058833+0000", 2024-09-21T22:08:28.266 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0660000000000002e-06 2024-09-21T22:08:28.266 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.266 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.266 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:28.266 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.059877+0000", 2024-09-21T22:08:28.266 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0010447 2024-09-21T22:08:28.266 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.267 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.267 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:28.267 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.059908+0000", 2024-09-21T22:08:28.267 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0800000000000003e-05 2024-09-21T22:08:28.267 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.267 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.267 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:28.267 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.059923+0000", 2024-09-21T22:08:28.267 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4497e-05 2024-09-21T22:08:28.267 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.267 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.267 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:28.267 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.060053+0000", 2024-09-21T22:08:28.268 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013002900000000001 2024-09-21T22:08:28.268 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.268 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.268 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:28.268 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.060091+0000", 2024-09-21T22:08:28.268 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8389999999999997e-05 2024-09-21T22:08:28.268 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.268 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.268 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:28.268 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.116840+0000", 2024-09-21T22:08:28.268 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.056749253999999999 2024-09-21T22:08:28.268 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.268 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.268 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:28.268 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.116927+0000", 2024-09-21T22:08:28.269 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6798999999999993e-05 2024-09-21T22:08:28.269 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:28.269 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:28.269 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:28.269 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.269 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.269 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.6 87/86 [PushOp(2:7e2189ff:ns::rbd_data.127f79bc213d.0000000000000001:head, version: 57'4, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:7e2189ff:ns::rbd_data.127f79bc213d.0000000000000001:head@57'4, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T22:08:28.269 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:08:07.095630+0000", 2024-09-21T22:08:28.269 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 21.112992915, 2024-09-21T22:08:28.269 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.063566191999999994, 2024-09-21T22:08:28.269 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:28.269 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:28.269 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:08:28.269 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:28.270 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.270 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:28.270 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.095630+0000", 2024-09-21T22:08:28.270 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:28.270 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.270 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.270 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:28.270 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.095630+0000", 2024-09-21T22:08:28.270 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:28.270 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.270 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.270 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:28.270 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.095631+0000", 2024-09-21T22:08:28.271 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3069999999999999e-06 2024-09-21T22:08:28.271 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.271 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.271 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:28.271 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.097075+0000", 2024-09-21T22:08:28.271 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001443831 2024-09-21T22:08:28.271 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.271 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.271 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:28.271 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.097153+0000", 2024-09-21T22:08:28.271 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.8485999999999994e-05 2024-09-21T22:08:28.272 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.272 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.272 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:28.272 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.097208+0000", 2024-09-21T22:08:28.272 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4716000000000003e-05 2024-09-21T22:08:28.272 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.272 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.272 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:28.272 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.097459+0000", 2024-09-21T22:08:28.272 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025113799999999998 2024-09-21T22:08:28.272 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.272 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.272 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:28.272 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.097481+0000", 2024-09-21T22:08:28.273 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2344000000000002e-05 2024-09-21T22:08:28.273 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.273 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.273 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:28.273 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.159196+0000", 2024-09-21T22:08:28.273 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.061714369999999998 2024-09-21T22:08:28.273 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:28.273 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:28.273 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:28.273 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.273 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.273 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.6 87/86 [PushOp(2:7e2189ff:ns::rbd_data.127f79bc213d.0000000000000001:head, version: 57'4, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:7e2189ff:ns::rbd_data.127f79bc213d.0000000000000001:head@57'4, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:08:28.273 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:08:07.163314+0000", 2024-09-21T22:08:28.273 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 21.045308860999999, 2024-09-21T22:08:28.274 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.044796923000000002, 2024-09-21T22:08:28.274 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:28.274 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:28.274 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:08:28.274 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:28.274 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.274 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:28.274 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.163314+0000", 2024-09-21T22:08:28.274 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:28.274 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.274 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.274 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:28.274 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.163314+0000", 2024-09-21T22:08:28.275 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:28.275 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.275 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.275 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:28.275 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.163315+0000", 2024-09-21T22:08:28.275 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7120000000000001e-06 2024-09-21T22:08:28.275 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.275 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.275 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:28.275 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.164270+0000", 2024-09-21T22:08:28.275 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000955034 2024-09-21T22:08:28.275 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.275 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.275 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:28.275 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.164326+0000", 2024-09-21T22:08:28.276 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.5627999999999997e-05 2024-09-21T22:08:28.276 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.276 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.276 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:28.276 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.164351+0000", 2024-09-21T22:08:28.276 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5219999999999999e-05 2024-09-21T22:08:28.276 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.276 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.276 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:28.276 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.164571+0000", 2024-09-21T22:08:28.276 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000219996 2024-09-21T22:08:28.276 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.276 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.276 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:28.277 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.164596+0000", 2024-09-21T22:08:28.277 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4564000000000001e-05 2024-09-21T22:08:28.277 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.277 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.277 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:28.277 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.208111+0000", 2024-09-21T22:08:28.277 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.043514769000000002 2024-09-21T22:08:28.277 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:28.277 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:28.277 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:28.277 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.277 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.277 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.6 87/86 [PushOp(2:7e2189ff:ns::rbd_data.127f79bc213d.0000000000000001:head, version: 57'4, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:7e2189ff:ns::rbd_data.127f79bc213d.0000000000000001:head@57'4, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:08:28.277 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:08:07.212243+0000", 2024-09-21T22:08:28.277 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 20.996379418, 2024-09-21T22:08:28.278 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.051665924000000002, 2024-09-21T22:08:28.278 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:28.278 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:28.278 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:08:28.278 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:28.278 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.278 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:28.278 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.212243+0000", 2024-09-21T22:08:28.278 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:28.278 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.278 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.278 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:28.278 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.212243+0000", 2024-09-21T22:08:28.278 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:28.279 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.279 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.279 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:28.279 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.212244+0000", 2024-09-21T22:08:28.279 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2839999999999999e-06 2024-09-21T22:08:28.279 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.279 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.279 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:28.279 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.212993+0000", 2024-09-21T22:08:28.279 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00074894100000000004 2024-09-21T22:08:28.279 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.279 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.279 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:28.279 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.213032+0000", 2024-09-21T22:08:28.279 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8948e-05 2024-09-21T22:08:28.280 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.280 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.280 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:28.280 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.213054+0000", 2024-09-21T22:08:28.280 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1937999999999999e-05 2024-09-21T22:08:28.280 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.280 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.280 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:28.280 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.213194+0000", 2024-09-21T22:08:28.280 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013942699999999999 2024-09-21T22:08:28.280 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.280 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.280 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:28.280 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.213210+0000", 2024-09-21T22:08:28.281 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5883999999999999e-05 2024-09-21T22:08:28.281 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.281 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.281 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:28.281 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.263909+0000", 2024-09-21T22:08:28.281 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.050699502 2024-09-21T22:08:28.281 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:28.281 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:28.281 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:28.281 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.281 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.281 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.6 87/86 [PushOp(2:7e2189ff:ns::rbd_data.127f79bc213d.0000000000000001:head, version: 57'4, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:7e2189ff:ns::rbd_data.127f79bc213d.0000000000000001:head@57'4, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 4194304, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:08:28.281 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:08:07.268604+0000", 2024-09-21T22:08:28.281 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 20.940018146, 2024-09-21T22:08:28.281 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.038386754000000002, 2024-09-21T22:08:28.282 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:28.282 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:28.282 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:08:28.282 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:28.282 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.282 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:28.282 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.268604+0000", 2024-09-21T22:08:28.282 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:28.282 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.282 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.282 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:28.282 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.268604+0000", 2024-09-21T22:08:28.282 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:28.282 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.283 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.283 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:28.283 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.268606+0000", 2024-09-21T22:08:28.283 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2020000000000001e-06 2024-09-21T22:08:28.283 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.283 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.283 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:28.283 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.269149+0000", 2024-09-21T22:08:28.283 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00054327900000000003 2024-09-21T22:08:28.283 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.283 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.283 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:28.283 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.269173+0000", 2024-09-21T22:08:28.283 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4046999999999999e-05 2024-09-21T22:08:28.284 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.284 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.284 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:28.284 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.269187+0000", 2024-09-21T22:08:28.284 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4185999999999999e-05 2024-09-21T22:08:28.284 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.284 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.284 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:28.284 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.269303+0000", 2024-09-21T22:08:28.284 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000115831 2024-09-21T22:08:28.284 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.284 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.284 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:28.284 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.269320+0000", 2024-09-21T22:08:28.284 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6878999999999999e-05 2024-09-21T22:08:28.285 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.285 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.285 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:28.285 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.306991+0000", 2024-09-21T22:08:28.285 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.037671330000000003 2024-09-21T22:08:28.285 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:28.285 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:28.285 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:28.285 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.285 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.285 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.6 87/86 [PushOp(2:762dd302:ns::rbd_data.127f79bc213d.0000000000000007:head, version: 57'5, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:762dd302:ns::rbd_data.127f79bc213d.0000000000000007:head@57'5, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T22:08:28.285 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:08:07.310599+0000", 2024-09-21T22:08:28.285 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 20.898023951999999, 2024-09-21T22:08:28.285 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025565551999999998, 2024-09-21T22:08:28.285 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:28.286 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:28.286 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:08:28.286 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:28.286 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.286 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:28.286 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.310599+0000", 2024-09-21T22:08:28.286 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:28.286 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.286 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.286 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:28.286 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.310599+0000", 2024-09-21T22:08:28.286 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:28.286 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.286 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.287 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:28.287 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.310600+0000", 2024-09-21T22:08:28.287 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0559999999999999e-06 2024-09-21T22:08:28.287 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.287 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.287 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:28.287 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.311158+0000", 2024-09-21T22:08:28.287 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00055862200000000005 2024-09-21T22:08:28.287 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.287 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.287 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:28.287 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.311180+0000", 2024-09-21T22:08:28.287 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1577999999999999e-05 2024-09-21T22:08:28.287 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.288 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.288 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:28.288 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.311198+0000", 2024-09-21T22:08:28.288 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8359000000000002e-05 2024-09-21T22:08:28.288 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.288 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.288 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:28.288 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.311291+0000", 2024-09-21T22:08:28.288 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2887999999999996e-05 2024-09-21T22:08:28.288 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.288 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.288 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:28.288 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.311300+0000", 2024-09-21T22:08:28.288 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8640000000000002e-06 2024-09-21T22:08:28.288 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.289 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.289 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:28.289 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.336164+0000", 2024-09-21T22:08:28.289 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.024864185 2024-09-21T22:08:28.289 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:28.289 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:28.289 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:28.289 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.289 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.289 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.6 87/86 [PushOp(2:762dd302:ns::rbd_data.127f79bc213d.0000000000000007:head, version: 57'5, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:762dd302:ns::rbd_data.127f79bc213d.0000000000000007:head@57'5, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:08:28.289 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:08:07.338463+0000", 2024-09-21T22:08:28.289 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 20.870159341000001, 2024-09-21T22:08:28.289 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026221639000000001, 2024-09-21T22:08:28.289 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:28.290 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:28.290 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:08:28.290 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:28.290 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.290 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:28.290 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.338463+0000", 2024-09-21T22:08:28.290 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:28.290 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.290 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.290 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:28.290 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.338463+0000", 2024-09-21T22:08:28.290 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:28.290 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.290 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.291 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:28.291 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.338464+0000", 2024-09-21T22:08:28.291 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.035e-06 2024-09-21T22:08:28.291 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.291 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.291 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:28.291 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.339044+0000", 2024-09-21T22:08:28.291 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00057999699999999996 2024-09-21T22:08:28.291 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.291 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.291 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:28.291 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.339070+0000", 2024-09-21T22:08:28.291 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.565e-05 2024-09-21T22:08:28.291 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.292 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.292 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:28.292 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.339088+0000", 2024-09-21T22:08:28.292 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8189000000000002e-05 2024-09-21T22:08:28.292 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.292 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.292 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:28.292 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.339184+0000", 2024-09-21T22:08:28.292 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5475000000000003e-05 2024-09-21T22:08:28.292 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.292 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.293 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:28.293 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.339193+0000", 2024-09-21T22:08:28.293 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7790000000000007e-06 2024-09-21T22:08:28.293 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.293 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.293 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:28.293 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.364685+0000", 2024-09-21T22:08:28.293 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.025491514 2024-09-21T22:08:28.293 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:28.293 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:28.293 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:28.293 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.293 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.293 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_repop(client.5398.0:204 2.6 e89/86 2:633e2f34:::rbd_data.126483d75f2d.0000000000000001:head v 89'32, mlcod=88'29)", 2024-09-21T22:08:28.294 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:08:10.148990+0000", 2024-09-21T22:08:28.294 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 18.059632795999999, 2024-09-21T22:08:28.294 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.034897343999999997, 2024-09-21T22:08:28.294 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:28.294 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:28.294 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:28.294 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:28.294 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.294 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:28.294 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:10.148990+0000", 2024-09-21T22:08:28.294 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:28.294 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.294 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.294 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:28.295 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:10.148990+0000", 2024-09-21T22:08:28.295 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:28.295 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.295 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.295 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:28.295 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:10.148991+0000", 2024-09-21T22:08:28.295 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.61e-06 2024-09-21T22:08:28.295 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.295 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.295 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:28.295 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:10.149966+0000", 2024-09-21T22:08:28.295 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00097419800000000001 2024-09-21T22:08:28.295 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.295 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.295 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:28.296 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:10.149996+0000", 2024-09-21T22:08:28.296 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0762999999999997e-05 2024-09-21T22:08:28.296 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.296 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.296 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:28.296 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:10.150017+0000", 2024-09-21T22:08:28.296 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.109e-05 2024-09-21T22:08:28.296 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.296 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.296 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:28.296 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:10.150090+0000", 2024-09-21T22:08:28.296 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.2835000000000003e-05 2024-09-21T22:08:28.296 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.296 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.297 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:28.297 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:10.150112+0000", 2024-09-21T22:08:28.297 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1746000000000002e-05 2024-09-21T22:08:28.297 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.297 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.297 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:28.297 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:10.183831+0000", 2024-09-21T22:08:28.297 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.033718571000000003 2024-09-21T22:08:28.297 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.297 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.297 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:28.297 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:10.183887+0000", 2024-09-21T22:08:28.297 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.6530999999999998e-05 2024-09-21T22:08:28.297 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:28.297 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:28.298 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:28.298 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:28.298 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:28.298 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:08:28.298 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:08:28.207+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:08:28.299 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T22:08:28.311 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:08:28.305+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:08:28.411 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:08:28.405+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:08:28.481 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:08:28.482 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-21T22:08:28.482 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.482 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.6183.0:28 2.2 2.b67abca (undecoded) ondisk+write+known_if_redirected+full_try+supports_pool_eio e98)", 2024-09-21T22:08:28.482 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:08:27.977642+0000", 2024-09-21T22:08:28.482 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 0.50533833900000003, 2024-09-21T22:08:28.482 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.505384422, 2024-09-21T22:08:28.482 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:28.482 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:28.482 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "waiting for peered", 2024-09-21T22:08:28.482 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:28.482 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.6183", 2024-09-21T22:08:28.483 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/2504500586", 2024-09-21T22:08:28.483 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 28 2024-09-21T22:08:28.483 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.483 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:28.483 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.483 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:28.483 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:27.977642+0000", 2024-09-21T22:08:28.483 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:28.483 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.483 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.483 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:28.483 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:27.977642+0000", 2024-09-21T22:08:28.483 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:28.483 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.484 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.484 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:28.484 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:27.977652+0000", 2024-09-21T22:08:28.484 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.7869999999999996e-06 2024-09-21T22:08:28.484 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.484 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.484 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:28.484 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:27.977661+0000", 2024-09-21T22:08:28.484 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9600000000000006e-06 2024-09-21T22:08:28.484 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.484 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.484 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:28.484 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:27.977688+0000", 2024-09-21T22:08:28.484 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7233000000000002e-05 2024-09-21T22:08:28.485 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.485 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.485 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:28.485 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:27.977703+0000", 2024-09-21T22:08:28.485 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5379999999999998e-05 2024-09-21T22:08:28.485 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.485 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.485 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:28.485 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:27.977787+0000", 2024-09-21T22:08:28.485 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4066000000000005e-05 2024-09-21T22:08:28.485 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.485 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.485 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-21T22:08:28.485 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:27.977798+0000", 2024-09-21T22:08:28.485 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1012999999999999e-05 2024-09-21T22:08:28.486 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:28.486 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:28.486 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:28.486 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:28.486 INFO:teuthology.orchestra.run.smithi064.stdout: ], 2024-09-21T22:08:28.486 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 1 2024-09-21T22:08:28.486 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:08:28.497 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T22:08:28.546 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:08:28.543+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:08:28.613 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:08:28.611+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:08:28.626 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:08:28.626 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:08:28.626 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-21T22:08:28.627 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-21T22:08:28.627 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:08:28.641 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T22:08:28.715 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:08:28.709+0000 7fe38af2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 34631) UID: 0 2024-09-21T22:08:28.815 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:08:28.813+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:08:28.816 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:08:28.816 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-21T22:08:28.817 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-21T22:08:28.817 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-21T22:08:28.817 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.817 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5031.0:3 2.2 2:53d5e6d0:::rbd_header.126483d75f2d:head [call rbd.get_size in=11b,call,call rbd.get_features in=15b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e65)", 2024-09-21T22:08:28.817 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:27.373436+0000", 2024-09-21T22:08:28.817 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 61.441411889999998, 2024-09-21T22:08:28.817 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.86524013700000002, 2024-09-21T22:08:28.817 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:28.817 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:28.817 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:08:28.817 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:28.817 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5031", 2024-09-21T22:08:28.817 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/3232705659", 2024-09-21T22:08:28.818 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3 2024-09-21T22:08:28.818 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.818 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:28.818 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.818 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:28.818 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:27.373436+0000", 2024-09-21T22:08:28.818 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:28.818 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.818 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.818 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:28.818 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:27.373436+0000", 2024-09-21T22:08:28.818 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:28.818 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.819 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.819 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:28.819 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:27.373439+0000", 2024-09-21T22:08:28.819 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.5969999999999999e-06 2024-09-21T22:08:28.819 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.819 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.819 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:28.819 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:27.373450+0000", 2024-09-21T22:08:28.819 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1326e-05 2024-09-21T22:08:28.819 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.819 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.819 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:28.819 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:27.373471+0000", 2024-09-21T22:08:28.819 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0169999999999998e-05 2024-09-21T22:08:28.820 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.820 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.820 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:28.820 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:27.373486+0000", 2024-09-21T22:08:28.820 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5483000000000001e-05 2024-09-21T22:08:28.820 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.820 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.820 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:28.820 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:27.373602+0000", 2024-09-21T22:08:28.820 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000115603 2024-09-21T22:08:28.820 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.820 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.820 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-21T22:08:28.820 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:27.373617+0000", 2024-09-21T22:08:28.821 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4956999999999999e-05 2024-09-21T22:08:28.821 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.821 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.821 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:28.821 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.234578+0000", 2024-09-21T22:08:28.821 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.86096097900000002 2024-09-21T22:08:28.821 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.821 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.821 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for missing object", 2024-09-21T22:08:28.821 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.234771+0000", 2024-09-21T22:08:28.821 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019394700000000001 2024-09-21T22:08:28.821 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.821 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.822 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:28.822 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.237470+0000", 2024-09-21T22:08:28.822 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0026981319999999998 2024-09-21T22:08:28.822 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.822 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.822 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:28.822 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.237791+0000", 2024-09-21T22:08:28.822 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00032138499999999999 2024-09-21T22:08:28.822 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.822 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.822 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:28.822 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.238677+0000", 2024-09-21T22:08:28.822 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00088555800000000005 2024-09-21T22:08:28.822 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:28.823 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:28.823 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:28.823 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.823 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.823 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.2 66/65 [PushOp(2:4481fad7:ns::rbd_data.127f79bc213d.0000000000000006:head, version: 57'9, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:4481fad7:ns::rbd_data.127f79bc213d.0000000000000006:head@57'9, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T22:08:28.823 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:28.247056+0000", 2024-09-21T22:08:28.823 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 60.567792226999998, 2024-09-21T22:08:28.823 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.045025012000000003, 2024-09-21T22:08:28.823 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:28.823 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:28.823 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:08:28.823 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:28.823 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.823 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:28.824 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.247056+0000", 2024-09-21T22:08:28.824 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:28.824 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.824 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.824 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:28.824 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.247056+0000", 2024-09-21T22:08:28.824 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:28.824 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.824 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.824 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:28.824 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.247057+0000", 2024-09-21T22:08:28.824 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2500000000000001e-06 2024-09-21T22:08:28.824 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.824 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.825 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:28.825 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.249384+0000", 2024-09-21T22:08:28.825 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0023265619999999999 2024-09-21T22:08:28.825 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.825 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.825 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:28.825 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.249429+0000", 2024-09-21T22:08:28.825 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5392999999999997e-05 2024-09-21T22:08:28.825 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.825 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.825 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:28.825 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.249476+0000", 2024-09-21T22:08:28.825 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6570000000000003e-05 2024-09-21T22:08:28.825 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.825 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.826 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:28.826 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.249611+0000", 2024-09-21T22:08:28.826 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000135143 2024-09-21T22:08:28.826 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.826 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.826 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:28.826 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.249626+0000", 2024-09-21T22:08:28.826 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.469e-05 2024-09-21T22:08:28.826 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.826 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.826 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:28.826 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.292081+0000", 2024-09-21T22:08:28.826 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.042455404000000002 2024-09-21T22:08:28.826 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:28.827 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:28.827 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:28.827 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.827 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.827 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5031.0:9 2.2 2:53d5e6d0:::rbd_header.126483d75f2d:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-21T22:08:28.827 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:28.250163+0000", 2024-09-21T22:08:28.827 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 60.564684833000001, 2024-09-21T22:08:28.827 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.042986430999999999, 2024-09-21T22:08:28.827 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:28.827 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:28.827 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:08:28.827 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:28.827 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5031", 2024-09-21T22:08:28.827 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/3232705659", 2024-09-21T22:08:28.828 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 9 2024-09-21T22:08:28.828 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.828 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:28.828 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.828 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:28.828 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.250163+0000", 2024-09-21T22:08:28.828 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:28.828 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.828 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.828 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:28.828 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.250163+0000", 2024-09-21T22:08:28.828 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:28.828 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.828 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.828 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:28.829 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.250166+0000", 2024-09-21T22:08:28.829 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7039999999999999e-06 2024-09-21T22:08:28.829 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.829 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.829 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:28.829 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.250175+0000", 2024-09-21T22:08:28.829 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7560000000000005e-06 2024-09-21T22:08:28.829 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.829 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.829 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:28.829 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.250199+0000", 2024-09-21T22:08:28.829 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4049999999999998e-05 2024-09-21T22:08:28.829 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.830 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.830 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:28.830 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.250218+0000", 2024-09-21T22:08:28.830 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9236e-05 2024-09-21T22:08:28.830 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.830 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.830 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:28.830 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.292193+0000", 2024-09-21T22:08:28.830 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.041974694 2024-09-21T22:08:28.830 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.830 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.830 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:28.830 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.292492+0000", 2024-09-21T22:08:28.830 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00029900100000000002 2024-09-21T22:08:28.831 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.831 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.831 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:28.831 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.293150+0000", 2024-09-21T22:08:28.831 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00065799000000000001 2024-09-21T22:08:28.831 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:28.831 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:28.831 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:28.831 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.831 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.831 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.2 66/65 [PushOp(2:4481fad7:ns::rbd_data.127f79bc213d.0000000000000006:head, version: 57'9, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:4481fad7:ns::rbd_data.127f79bc213d.0000000000000006:head@57'9, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:08:28.831 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:28.331166+0000", 2024-09-21T22:08:28.831 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 60.483682082999998, 2024-09-21T22:08:28.831 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.039995378999999998, 2024-09-21T22:08:28.832 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:28.832 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:28.832 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:08:28.832 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:28.832 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.832 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:28.832 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.331166+0000", 2024-09-21T22:08:28.832 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:28.832 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.832 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.832 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:28.832 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.331166+0000", 2024-09-21T22:08:28.832 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:28.832 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.832 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.833 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:28.833 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.331167+0000", 2024-09-21T22:08:28.833 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.9199999999999999e-07 2024-09-21T22:08:28.833 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.833 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.833 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:28.833 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.332127+0000", 2024-09-21T22:08:28.833 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00095969800000000004 2024-09-21T22:08:28.833 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.833 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.833 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:28.833 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.332149+0000", 2024-09-21T22:08:28.833 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1974999999999999e-05 2024-09-21T22:08:28.834 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.834 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.834 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:28.834 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.332161+0000", 2024-09-21T22:08:28.834 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2276000000000001e-05 2024-09-21T22:08:28.834 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.834 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.834 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:28.834 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.332290+0000", 2024-09-21T22:08:28.834 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000129214 2024-09-21T22:08:28.834 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.834 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.834 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:28.835 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.332312+0000", 2024-09-21T22:08:28.835 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1308999999999999e-05 2024-09-21T22:08:28.835 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.835 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.835 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:28.835 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.371162+0000", 2024-09-21T22:08:28.835 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.038850015000000002 2024-09-21T22:08:28.835 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:28.835 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:28.835 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:28.835 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.835 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.835 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4811.0:3 2.3 2:c4c92e5a:::rbd_trash:head [call rbd.trash_list in=13b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-21T22:08:28.835 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:28.475306+0000", 2024-09-21T22:08:28.835 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 60.339542539, 2024-09-21T22:08:28.836 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.77996670099999998, 2024-09-21T22:08:28.836 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:28.836 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:28.836 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:08:28.836 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:28.836 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4811", 2024-09-21T22:08:28.836 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/2028736390", 2024-09-21T22:08:28.836 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 3 2024-09-21T22:08:28.836 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.836 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:28.836 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.836 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:28.836 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.475306+0000", 2024-09-21T22:08:28.836 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:28.837 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.837 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.837 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:28.837 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.475306+0000", 2024-09-21T22:08:28.837 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:28.837 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.837 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.837 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:28.837 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.475308+0000", 2024-09-21T22:08:28.837 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8959999999999999e-06 2024-09-21T22:08:28.837 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.837 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.837 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:28.837 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.475314+0000", 2024-09-21T22:08:28.838 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.2280000000000003e-06 2024-09-21T22:08:28.838 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.838 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.838 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:28.838 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.475329+0000", 2024-09-21T22:08:28.838 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5054e-05 2024-09-21T22:08:28.838 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.838 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.838 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:28.838 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.475338+0000", 2024-09-21T22:08:28.838 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3770000000000002e-06 2024-09-21T22:08:28.838 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.838 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.838 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:28.839 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.475484+0000", 2024-09-21T22:08:28.839 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014612300000000001 2024-09-21T22:08:28.839 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.839 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.839 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-21T22:08:28.839 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:28.475506+0000", 2024-09-21T22:08:28.839 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1319999999999999e-05 2024-09-21T22:08:28.839 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.839 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.839 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:28.839 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.254433+0000", 2024-09-21T22:08:28.839 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.778927708 2024-09-21T22:08:28.839 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.839 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.840 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:28.840 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.254979+0000", 2024-09-21T22:08:28.840 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00054573100000000004 2024-09-21T22:08:28.840 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.840 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.840 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:28.840 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.255272+0000", 2024-09-21T22:08:28.840 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000293264 2024-09-21T22:08:28.840 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:28.840 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:28.840 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:28.840 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.840 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.840 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.3 67/66 [PushOp(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head, version: 63'11, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head@63'11, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T22:08:28.841 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:29.262595+0000", 2024-09-21T22:08:28.841 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 59.552253356000001, 2024-09-21T22:08:28.841 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.036027272999999999, 2024-09-21T22:08:28.841 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:28.841 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:28.841 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:08:28.841 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:28.841 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.841 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:28.841 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.262595+0000", 2024-09-21T22:08:28.841 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:28.841 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.841 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.841 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:28.841 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.262595+0000", 2024-09-21T22:08:28.842 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:28.842 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.842 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.842 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:28.842 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.262596+0000", 2024-09-21T22:08:28.842 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4300000000000001e-06 2024-09-21T22:08:28.842 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.842 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.842 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:28.842 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.264505+0000", 2024-09-21T22:08:28.842 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0019085650000000001 2024-09-21T22:08:28.842 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.842 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.843 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:28.843 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.264557+0000", 2024-09-21T22:08:28.843 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2040000000000003e-05 2024-09-21T22:08:28.843 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.843 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.843 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:28.843 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.264589+0000", 2024-09-21T22:08:28.843 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1636e-05 2024-09-21T22:08:28.843 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.843 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.843 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:28.843 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.264778+0000", 2024-09-21T22:08:28.843 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018935600000000001 2024-09-21T22:08:28.844 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.844 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.844 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:28.844 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.264803+0000", 2024-09-21T22:08:28.844 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4692999999999999e-05 2024-09-21T22:08:28.844 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.844 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.844 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:28.844 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.298622+0000", 2024-09-21T22:08:28.844 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.033819553000000002 2024-09-21T22:08:28.844 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:28.844 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:28.844 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:28.844 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.845 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.845 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.3 67/66 [PushOp(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head, version: 63'11, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head@63'11, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:08:28.845 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:29.301490+0000", 2024-09-21T22:08:28.845 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 59.513358570000001, 2024-09-21T22:08:28.845 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.041178293999999997, 2024-09-21T22:08:28.845 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:28.845 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:28.845 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:08:28.845 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:28.845 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.845 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:28.845 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.301490+0000", 2024-09-21T22:08:28.845 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:28.845 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.846 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.846 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:28.846 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.301490+0000", 2024-09-21T22:08:28.846 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:28.846 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.846 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.846 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:28.846 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.301491+0000", 2024-09-21T22:08:28.846 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.513e-06 2024-09-21T22:08:28.846 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.846 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.846 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:28.846 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.302815+0000", 2024-09-21T22:08:28.846 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001323722 2024-09-21T22:08:28.847 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.847 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.847 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:28.847 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.302863+0000", 2024-09-21T22:08:28.847 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8449999999999999e-05 2024-09-21T22:08:28.847 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.847 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.847 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:28.847 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.302895+0000", 2024-09-21T22:08:28.847 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1072999999999999e-05 2024-09-21T22:08:28.847 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.847 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.847 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:28.847 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.303014+0000", 2024-09-21T22:08:28.848 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011907400000000001 2024-09-21T22:08:28.848 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.848 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.848 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:28.848 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.303029+0000", 2024-09-21T22:08:28.848 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5322e-05 2024-09-21T22:08:28.848 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.848 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.848 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:28.848 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.342668+0000", 2024-09-21T22:08:28.848 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.039639140000000003 2024-09-21T22:08:28.848 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:28.848 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:28.848 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:28.849 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.849 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.849 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.3 67/66 [PushOp(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head, version: 63'11, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head@63'11, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:08:28.849 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:29.345626+0000", 2024-09-21T22:08:28.849 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 59.469221836000003, 2024-09-21T22:08:28.849 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.034820216000000001, 2024-09-21T22:08:28.849 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:28.849 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:28.849 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:08:28.849 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:28.849 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.849 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:28.849 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.345626+0000", 2024-09-21T22:08:28.849 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:28.850 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.850 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.850 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:28.850 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.345626+0000", 2024-09-21T22:08:28.850 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:28.850 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.850 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.850 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:28.850 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.345628+0000", 2024-09-21T22:08:28.850 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.3799999999999999e-06 2024-09-21T22:08:28.850 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.850 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.850 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:28.850 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.347215+0000", 2024-09-21T22:08:28.851 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0015868760000000001 2024-09-21T22:08:28.851 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.851 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.851 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:28.851 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.347283+0000", 2024-09-21T22:08:28.851 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8267000000000002e-05 2024-09-21T22:08:28.851 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.851 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.851 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:28.851 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.347336+0000", 2024-09-21T22:08:28.851 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.3455999999999998e-05 2024-09-21T22:08:28.851 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.851 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.851 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:28.852 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.347463+0000", 2024-09-21T22:08:28.852 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012614399999999999 2024-09-21T22:08:28.852 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.852 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.852 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:28.852 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.347473+0000", 2024-09-21T22:08:28.852 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0462000000000001e-05 2024-09-21T22:08:28.852 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.852 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.852 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:28.852 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.380447+0000", 2024-09-21T22:08:28.852 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.032973631000000003 2024-09-21T22:08:28.852 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:28.852 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:28.852 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:28.853 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.853 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.853 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.3 67/66 [PushOp(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head, version: 63'19, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head@63'19, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T22:08:28.853 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:29.420098+0000", 2024-09-21T22:08:28.853 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 59.394750000999998, 2024-09-21T22:08:28.853 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.067283846999999994, 2024-09-21T22:08:28.853 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:28.853 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:28.853 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:08:28.853 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:28.853 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.853 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:28.853 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.420098+0000", 2024-09-21T22:08:28.853 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:28.854 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.854 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.854 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:28.854 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.420098+0000", 2024-09-21T22:08:28.854 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:28.854 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.854 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.854 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:28.854 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.420100+0000", 2024-09-21T22:08:28.854 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.716e-06 2024-09-21T22:08:28.854 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.854 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.854 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:28.854 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.421949+0000", 2024-09-21T22:08:28.855 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0018485190000000001 2024-09-21T22:08:28.855 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.855 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.855 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:28.855 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.421996+0000", 2024-09-21T22:08:28.855 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7089000000000003e-05 2024-09-21T22:08:28.855 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.855 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.855 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:28.855 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.422042+0000", 2024-09-21T22:08:28.855 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.6010000000000002e-05 2024-09-21T22:08:28.855 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.855 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.855 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:28.856 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.422231+0000", 2024-09-21T22:08:28.856 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018960900000000001 2024-09-21T22:08:28.856 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.856 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.856 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:28.856 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.422254+0000", 2024-09-21T22:08:28.856 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2864e-05 2024-09-21T22:08:28.856 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.856 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.856 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:28.856 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.487382+0000", 2024-09-21T22:08:28.856 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.065128039999999998 2024-09-21T22:08:28.856 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:28.856 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:28.856 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:28.857 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.857 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.857 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.3 67/66 [PushOp(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head, version: 63'19, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head@63'19, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:08:28.857 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:29.491783+0000", 2024-09-21T22:08:28.857 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 59.323065655999997, 2024-09-21T22:08:28.857 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.04476596, 2024-09-21T22:08:28.857 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:28.857 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:28.857 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:08:28.857 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:28.857 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.857 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:28.857 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.491783+0000", 2024-09-21T22:08:28.857 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:28.858 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.858 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.858 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:28.858 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.491783+0000", 2024-09-21T22:08:28.858 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:28.858 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.858 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.858 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:28.858 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.491784+0000", 2024-09-21T22:08:28.858 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.17e-06 2024-09-21T22:08:28.858 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.858 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.858 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:28.858 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.493126+0000", 2024-09-21T22:08:28.858 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0013424489999999999 2024-09-21T22:08:28.859 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.859 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.859 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:28.859 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.493170+0000", 2024-09-21T22:08:28.859 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.4153000000000002e-05 2024-09-21T22:08:28.859 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.859 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.859 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:28.859 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.493218+0000", 2024-09-21T22:08:28.859 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7392999999999998e-05 2024-09-21T22:08:28.859 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.859 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.859 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:28.859 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.493392+0000", 2024-09-21T22:08:28.860 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017392699999999999 2024-09-21T22:08:28.860 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.860 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.860 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:28.860 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.493416+0000", 2024-09-21T22:08:28.860 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4117e-05 2024-09-21T22:08:28.860 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.860 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.860 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:28.860 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.536549+0000", 2024-09-21T22:08:28.860 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.043132750999999997 2024-09-21T22:08:28.860 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:28.860 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:28.860 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:28.861 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.861 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.861 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.3 67/66 [PushOp(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head, version: 63'19, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head@63'19, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:08:28.861 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:29.539831+0000", 2024-09-21T22:08:28.861 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 59.275016895999997, 2024-09-21T22:08:28.861 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.036233222000000002, 2024-09-21T22:08:28.861 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:28.861 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:28.861 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:08:28.861 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:28.861 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.861 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:28.861 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.539831+0000", 2024-09-21T22:08:28.861 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:28.862 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.862 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.862 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:28.862 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.539831+0000", 2024-09-21T22:08:28.862 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:28.862 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.862 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.862 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:28.862 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.539833+0000", 2024-09-21T22:08:28.862 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5200000000000001e-06 2024-09-21T22:08:28.862 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.862 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.862 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:28.862 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.541130+0000", 2024-09-21T22:08:28.862 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0012972719999999999 2024-09-21T22:08:28.863 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.863 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.863 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:28.863 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.541179+0000", 2024-09-21T22:08:28.863 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8507000000000002e-05 2024-09-21T22:08:28.863 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.863 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.863 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:28.863 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.541226+0000", 2024-09-21T22:08:28.863 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7663000000000001e-05 2024-09-21T22:08:28.863 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.863 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.863 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:28.863 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.541361+0000", 2024-09-21T22:08:28.864 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000135065 2024-09-21T22:08:28.864 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.864 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.864 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:28.864 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.541374+0000", 2024-09-21T22:08:28.864 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2295e-05 2024-09-21T22:08:28.864 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.864 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.864 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:28.864 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.576065+0000", 2024-09-21T22:08:28.864 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.034690899999999997 2024-09-21T22:08:28.864 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:28.864 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:28.864 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:28.864 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.865 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.865 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.3 67/66 [PushOp(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head, version: 63'19, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head@63'19, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 4194304, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:08:28.865 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:29.579455+0000", 2024-09-21T22:08:28.865 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 59.235393588000001, 2024-09-21T22:08:28.865 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.034295594999999998, 2024-09-21T22:08:28.865 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:28.865 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:28.865 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:08:28.865 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:28.865 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.865 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:28.865 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.579455+0000", 2024-09-21T22:08:28.865 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:28.866 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.866 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.866 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:28.866 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.579455+0000", 2024-09-21T22:08:28.866 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:28.866 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.866 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.866 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:28.866 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.579456+0000", 2024-09-21T22:08:28.866 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.404e-06 2024-09-21T22:08:28.866 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.866 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.866 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:28.866 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.580763+0000", 2024-09-21T22:08:28.867 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001307325 2024-09-21T22:08:28.867 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.867 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.867 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:28.867 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.580811+0000", 2024-09-21T22:08:28.867 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7843000000000003e-05 2024-09-21T22:08:28.867 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.867 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.867 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:28.867 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.580860+0000", 2024-09-21T22:08:28.867 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9036999999999998e-05 2024-09-21T22:08:28.867 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.867 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.867 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:28.868 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.580988+0000", 2024-09-21T22:08:28.868 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000127529 2024-09-21T22:08:28.868 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.868 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.868 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:28.868 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.580998+0000", 2024-09-21T22:08:28.868 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0617e-05 2024-09-21T22:08:28.868 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.868 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.868 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:28.868 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:29.613750+0000", 2024-09-21T22:08:28.868 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.032751839999999997 2024-09-21T22:08:28.868 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:28.868 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:28.869 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:28.869 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.869 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.869 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5008.0:125 2.3 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-21T22:08:28.869 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:38.066298+0000", 2024-09-21T22:08:28.869 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 50.748549924000002, 2024-09-21T22:08:28.869 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.062817136999999995, 2024-09-21T22:08:28.869 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:28.869 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:28.869 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:28.869 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:28.869 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5008", 2024-09-21T22:08:28.869 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/1400114666", 2024-09-21T22:08:28.869 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 125 2024-09-21T22:08:28.870 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.870 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:28.870 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.870 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:28.870 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.066298+0000", 2024-09-21T22:08:28.870 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:28.870 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.870 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.870 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:28.870 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.066298+0000", 2024-09-21T22:08:28.870 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:28.870 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.870 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.871 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:28.871 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.066300+0000", 2024-09-21T22:08:28.871 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.474e-06 2024-09-21T22:08:28.871 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.871 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.871 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:28.871 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.067443+0000", 2024-09-21T22:08:28.871 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011428009999999999 2024-09-21T22:08:28.871 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.871 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.871 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:28.871 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.067477+0000", 2024-09-21T22:08:28.871 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4059e-05 2024-09-21T22:08:28.871 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.871 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.872 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:28.872 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.067492+0000", 2024-09-21T22:08:28.872 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5054e-05 2024-09-21T22:08:28.872 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.872 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.872 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:28.872 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.067638+0000", 2024-09-21T22:08:28.872 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014583900000000001 2024-09-21T22:08:28.872 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.872 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.872 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:28.872 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.067882+0000", 2024-09-21T22:08:28.872 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024415500000000002 2024-09-21T22:08:28.872 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.873 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.873 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-09-21T22:08:28.873 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.068247+0000", 2024-09-21T22:08:28.873 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00036570800000000002 2024-09-21T22:08:28.873 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.873 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.873 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:28.873 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.119971+0000", 2024-09-21T22:08:28.873 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.051723866 2024-09-21T22:08:28.873 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.874 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.874 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:28.874 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.128849+0000", 2024-09-21T22:08:28.874 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0088779489999999996 2024-09-21T22:08:28.874 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.874 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.874 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:28.874 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.129000+0000", 2024-09-21T22:08:28.874 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015062499999999999 2024-09-21T22:08:28.874 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.874 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.874 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:28.874 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.129116+0000", 2024-09-21T22:08:28.875 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011560700000000001 2024-09-21T22:08:28.875 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:28.875 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:28.875 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:28.875 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.875 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.875 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5008.0:127 2.3 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-21T22:08:28.875 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:38.140308+0000", 2024-09-21T22:08:28.875 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 50.674539822, 2024-09-21T22:08:28.875 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.055290512999999999, 2024-09-21T22:08:28.875 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:28.875 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:28.875 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:28.875 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:28.876 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5008", 2024-09-21T22:08:28.876 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/1400114666", 2024-09-21T22:08:28.876 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 127 2024-09-21T22:08:28.876 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.876 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:28.876 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.876 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:28.876 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.140308+0000", 2024-09-21T22:08:28.876 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:28.876 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.876 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.876 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:28.876 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.140308+0000", 2024-09-21T22:08:28.876 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:28.876 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.877 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.877 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:28.877 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.140310+0000", 2024-09-21T22:08:28.877 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6509999999999999e-06 2024-09-21T22:08:28.877 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.877 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.877 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:28.877 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.141504+0000", 2024-09-21T22:08:28.877 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011935750000000001 2024-09-21T22:08:28.877 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.877 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.877 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:28.877 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.141522+0000", 2024-09-21T22:08:28.877 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8349e-05 2024-09-21T22:08:28.878 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.878 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.878 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:28.878 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.141531+0000", 2024-09-21T22:08:28.878 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3239999999999992e-06 2024-09-21T22:08:28.878 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.878 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.878 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:28.878 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.141661+0000", 2024-09-21T22:08:28.878 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013001000000000001 2024-09-21T22:08:28.878 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.878 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.878 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:28.879 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.141907+0000", 2024-09-21T22:08:28.879 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00024547499999999999 2024-09-21T22:08:28.879 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.879 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.879 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-09-21T22:08:28.879 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.142274+0000", 2024-09-21T22:08:28.879 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00036716400000000001 2024-09-21T22:08:28.879 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.879 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.879 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:28.879 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.189864+0000", 2024-09-21T22:08:28.879 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.047589617000000001 2024-09-21T22:08:28.879 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.880 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.880 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:28.880 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.195406+0000", 2024-09-21T22:08:28.880 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0055425860000000004 2024-09-21T22:08:28.880 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.880 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.880 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:28.880 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.195507+0000", 2024-09-21T22:08:28.880 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000100862 2024-09-21T22:08:28.880 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.880 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.880 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:28.880 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.195599+0000", 2024-09-21T22:08:28.880 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1899999999999998e-05 2024-09-21T22:08:28.881 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:28.881 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:28.881 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:28.881 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.881 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.881 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5008.0:129 2.3 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head [write 2228224~917504 in=917504b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-21T22:08:28.881 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:38.209297+0000", 2024-09-21T22:08:28.881 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 50.605550940999997, 2024-09-21T22:08:28.881 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.060441054000000001, 2024-09-21T22:08:28.881 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:28.881 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:28.881 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:28.881 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:28.881 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5008", 2024-09-21T22:08:28.881 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/1400114666", 2024-09-21T22:08:28.882 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 129 2024-09-21T22:08:28.882 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.882 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:28.882 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.882 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:28.882 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.209297+0000", 2024-09-21T22:08:28.882 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:28.882 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.882 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.882 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:28.882 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.209297+0000", 2024-09-21T22:08:28.882 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:28.882 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.882 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.883 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:28.883 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.209299+0000", 2024-09-21T22:08:28.883 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.274e-06 2024-09-21T22:08:28.883 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.883 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.883 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:28.883 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.210077+0000", 2024-09-21T22:08:28.883 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00077849799999999997 2024-09-21T22:08:28.883 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.883 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.883 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:28.883 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.210094+0000", 2024-09-21T22:08:28.883 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6772000000000001e-05 2024-09-21T22:08:28.883 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.883 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.884 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:28.884 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.210103+0000", 2024-09-21T22:08:28.884 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.6780000000000006e-06 2024-09-21T22:08:28.884 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.884 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.884 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:28.884 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.210254+0000", 2024-09-21T22:08:28.884 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015141699999999999 2024-09-21T22:08:28.884 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.884 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.884 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:28.884 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.210501+0000", 2024-09-21T22:08:28.884 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000246498 2024-09-21T22:08:28.884 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.885 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.885 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-09-21T22:08:28.885 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.210872+0000", 2024-09-21T22:08:28.885 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00037147500000000002 2024-09-21T22:08:28.885 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.885 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.885 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:28.885 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.269019+0000", 2024-09-21T22:08:28.885 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058147114 2024-09-21T22:08:28.885 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.885 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.885 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:28.885 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.269638+0000", 2024-09-21T22:08:28.885 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00061896499999999997 2024-09-21T22:08:28.886 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.886 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.886 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:28.886 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.269693+0000", 2024-09-21T22:08:28.886 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4786e-05 2024-09-21T22:08:28.886 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.886 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.886 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:28.886 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.269738+0000", 2024-09-21T22:08:28.886 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.5577000000000001e-05 2024-09-21T22:08:28.886 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:28.886 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:28.886 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:28.886 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.887 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.887 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5008.0:131 2.3 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-21T22:08:28.887 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:38.279879+0000", 2024-09-21T22:08:28.887 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 50.534969251, 2024-09-21T22:08:28.887 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.054184296999999999, 2024-09-21T22:08:28.887 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:28.887 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:28.887 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:28.887 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:28.887 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5008", 2024-09-21T22:08:28.887 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/1400114666", 2024-09-21T22:08:28.887 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 131 2024-09-21T22:08:28.887 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.887 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:28.888 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.888 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:28.888 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.279879+0000", 2024-09-21T22:08:28.888 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:28.888 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.888 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.888 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:28.888 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.279879+0000", 2024-09-21T22:08:28.888 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:28.888 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.888 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.888 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:28.888 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.279880+0000", 2024-09-21T22:08:28.888 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.136e-06 2024-09-21T22:08:28.888 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.889 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.889 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:28.889 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.280636+0000", 2024-09-21T22:08:28.889 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00075625900000000001 2024-09-21T22:08:28.889 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.889 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.889 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:28.889 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.280648+0000", 2024-09-21T22:08:28.889 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2026e-05 2024-09-21T22:08:28.889 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.889 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.889 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:28.889 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.280654+0000", 2024-09-21T22:08:28.889 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.8839999999999999e-06 2024-09-21T22:08:28.889 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.890 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.890 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:28.890 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.280739+0000", 2024-09-21T22:08:28.890 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.4196000000000005e-05 2024-09-21T22:08:28.890 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.890 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.890 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:28.890 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.280808+0000", 2024-09-21T22:08:28.890 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9685999999999997e-05 2024-09-21T22:08:28.890 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.890 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.890 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-09-21T22:08:28.890 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.280900+0000", 2024-09-21T22:08:28.890 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1407000000000001e-05 2024-09-21T22:08:28.891 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.891 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.891 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:28.891 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.307730+0000", 2024-09-21T22:08:28.891 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026830568999999999 2024-09-21T22:08:28.891 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.891 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.891 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:28.891 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.333901+0000", 2024-09-21T22:08:28.891 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026170787000000001 2024-09-21T22:08:28.891 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.891 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.892 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:28.892 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.333996+0000", 2024-09-21T22:08:28.892 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4723999999999996e-05 2024-09-21T22:08:28.892 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.892 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.892 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:28.892 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:38.334063+0000", 2024-09-21T22:08:28.892 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.7623e-05 2024-09-21T22:08:28.892 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:28.892 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:28.892 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:28.892 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.892 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.892 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.1 87/86 [PushOp(2:99eff0b8:ns::rbd_data.127f79bc213d.0000000000000002:head, version: 57'3, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:99eff0b8:ns::rbd_data.127f79bc213d.0000000000000002:head@57'3, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T22:08:28.893 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:08:07.091918+0000", 2024-09-21T22:08:28.893 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 21.722930808000001, 2024-09-21T22:08:28.893 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.071321166000000005, 2024-09-21T22:08:28.893 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:28.893 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:28.893 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:08:28.893 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:28.893 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.893 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:28.893 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.091918+0000", 2024-09-21T22:08:28.893 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:28.893 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.893 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.893 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:28.894 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.091918+0000", 2024-09-21T22:08:28.894 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:28.894 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.894 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.894 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:28.894 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.091919+0000", 2024-09-21T22:08:28.894 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.57e-06 2024-09-21T22:08:28.894 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.894 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.894 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:28.894 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.093627+0000", 2024-09-21T22:08:28.894 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001708162 2024-09-21T22:08:28.894 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.894 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.895 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:28.895 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.093675+0000", 2024-09-21T22:08:28.895 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.761e-05 2024-09-21T22:08:28.895 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.895 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.895 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:28.895 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.093707+0000", 2024-09-21T22:08:28.895 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1847000000000002e-05 2024-09-21T22:08:28.895 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.895 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.895 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:28.895 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.093899+0000", 2024-09-21T22:08:28.895 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000192449 2024-09-21T22:08:28.895 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.895 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.896 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:28.896 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.093922+0000", 2024-09-21T22:08:28.896 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2976000000000002e-05 2024-09-21T22:08:28.896 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.896 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.896 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:28.896 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.163239+0000", 2024-09-21T22:08:28.896 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.069316552000000003 2024-09-21T22:08:28.896 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:28.896 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:28.896 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:28.896 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.896 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.896 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.1 87/86 [PushOp(2:99eff0b8:ns::rbd_data.127f79bc213d.0000000000000002:head, version: 57'3, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:99eff0b8:ns::rbd_data.127f79bc213d.0000000000000002:head@57'3, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:08:28.897 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:08:07.167681+0000", 2024-09-21T22:08:28.897 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 21.647167222, 2024-09-21T22:08:28.897 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.043679153999999998, 2024-09-21T22:08:28.897 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:28.897 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:28.897 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:08:28.897 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:28.897 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.897 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:28.897 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.167681+0000", 2024-09-21T22:08:28.897 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:28.897 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.897 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.897 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:28.897 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.167681+0000", 2024-09-21T22:08:28.898 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:28.898 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.898 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.898 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:28.898 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.167683+0000", 2024-09-21T22:08:28.898 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.59e-06 2024-09-21T22:08:28.898 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.898 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.898 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:28.898 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.168793+0000", 2024-09-21T22:08:28.898 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011100229999999999 2024-09-21T22:08:28.898 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.898 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.898 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:28.899 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.168830+0000", 2024-09-21T22:08:28.899 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.7580999999999998e-05 2024-09-21T22:08:28.899 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.899 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.899 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:28.899 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.168852+0000", 2024-09-21T22:08:28.899 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1766000000000001e-05 2024-09-21T22:08:28.899 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.899 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.899 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:28.899 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.168987+0000", 2024-09-21T22:08:28.899 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000135203 2024-09-21T22:08:28.899 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.899 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.900 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:28.900 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.169003+0000", 2024-09-21T22:08:28.900 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.543e-05 2024-09-21T22:08:28.900 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.900 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.900 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:28.900 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.211360+0000", 2024-09-21T22:08:28.900 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.042357561000000002 2024-09-21T22:08:28.900 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:28.900 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:28.900 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:28.900 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.900 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.900 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.1 87/86 [PushOp(2:99eff0b8:ns::rbd_data.127f79bc213d.0000000000000002:head, version: 57'3, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:99eff0b8:ns::rbd_data.127f79bc213d.0000000000000002:head@57'3, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:08:28.900 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:08:07.215689+0000", 2024-09-21T22:08:28.901 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 21.599159067999999, 2024-09-21T22:08:28.901 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.052501277999999998, 2024-09-21T22:08:28.901 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:28.901 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:28.901 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:08:28.901 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:28.901 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.901 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:28.901 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.215689+0000", 2024-09-21T22:08:28.901 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:28.901 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.901 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.901 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:28.901 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.215689+0000", 2024-09-21T22:08:28.902 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:28.902 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.902 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.902 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:28.902 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.215690+0000", 2024-09-21T22:08:28.902 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1319999999999999e-06 2024-09-21T22:08:28.902 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.902 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.902 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:28.902 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.216711+0000", 2024-09-21T22:08:28.902 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0010210600000000001 2024-09-21T22:08:28.902 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.902 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.902 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:28.903 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.216748+0000", 2024-09-21T22:08:28.903 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.6776000000000002e-05 2024-09-21T22:08:28.903 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.903 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.903 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:28.903 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.216769+0000", 2024-09-21T22:08:28.903 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1189999999999999e-05 2024-09-21T22:08:28.903 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.903 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.903 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:28.903 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.216932+0000", 2024-09-21T22:08:28.903 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016283700000000001 2024-09-21T22:08:28.903 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.903 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.904 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:28.904 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.216957+0000", 2024-09-21T22:08:28.904 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4896999999999999e-05 2024-09-21T22:08:28.904 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.904 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.904 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:28.904 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.268191+0000", 2024-09-21T22:08:28.904 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.051233385999999999 2024-09-21T22:08:28.904 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:28.904 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:28.904 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:28.904 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.904 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.904 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.1 87/86 [PushOp(2:922055e2:ns::rbd_data.127f79bc213d.0000000000000009:head, version: 57'4, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:922055e2:ns::rbd_data.127f79bc213d.0000000000000009:head@57'4, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T22:08:28.905 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:08:07.302602+0000", 2024-09-21T22:08:28.905 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 21.512245932999999, 2024-09-21T22:08:28.905 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.038952326000000002, 2024-09-21T22:08:28.905 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:28.905 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:28.905 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:08:28.905 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:28.905 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.905 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:28.905 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.302602+0000", 2024-09-21T22:08:28.905 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:28.905 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.905 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.905 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:28.906 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.302602+0000", 2024-09-21T22:08:28.906 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:28.906 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.906 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.906 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:28.906 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.302604+0000", 2024-09-21T22:08:28.906 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9769999999999999e-06 2024-09-21T22:08:28.906 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.906 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.906 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:28.906 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.303971+0000", 2024-09-21T22:08:28.906 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001366719 2024-09-21T22:08:28.906 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.906 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.907 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:28.907 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.304022+0000", 2024-09-21T22:08:28.907 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.1307999999999997e-05 2024-09-21T22:08:28.907 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.907 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.907 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:28.907 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.304065+0000", 2024-09-21T22:08:28.907 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.214e-05 2024-09-21T22:08:28.907 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.907 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.907 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:28.907 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.304200+0000", 2024-09-21T22:08:28.907 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000135038 2024-09-21T22:08:28.907 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.908 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.908 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:28.908 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.304217+0000", 2024-09-21T22:08:28.908 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7856e-05 2024-09-21T22:08:28.908 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:28.908 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:28.908 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:28.908 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.341555+0000", 2024-09-21T22:08:28.908 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.037337288000000003 2024-09-21T22:08:28.908 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:28.908 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:28.908 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:28.908 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:28.908 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:28.908 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:08:28.910 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T22:08:28.956 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:08:28.953+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:08:28.972 INFO:teuthology.orchestra.run.smithi039.stderr:nodeep-scrub is set 2024-09-21T22:08:29.017 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:08:29.011+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:08:29.064 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:08:29.064 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:08:29.064 INFO:teuthology.orchestra.run.smithi039.stdout: "num_ops": 0 2024-09-21T22:08:29.064 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:08:29.079 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T22:08:29.124 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:08:29.121+0000 7fe38af2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 34631) UID: 0 2024-09-21T22:08:29.217 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:08:29.218 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:08:29.218 INFO:teuthology.orchestra.run.smithi039.stdout: "complaint_time": 30, 2024-09-21T22:08:29.218 INFO:teuthology.orchestra.run.smithi039.stdout: "num_blocked_ops": 0 2024-09-21T22:08:29.218 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:08:29.218 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:08:29.215+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:08:29.232 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T22:08:29.320 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:08:29.317+0000 7fe38af2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 34631) UID: 0 2024-09-21T22:08:29.403 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:08:29.403 INFO:teuthology.orchestra.run.smithi039.stdout: "size": 20, 2024-09-21T22:08:29.403 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 600, 2024-09-21T22:08:29.403 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [] 2024-09-21T22:08:29.403 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:08:29.418 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T22:08:29.419 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:08:29.415+0000 7f2075a06640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31720) UID: 0 2024-09-21T22:08:29.521 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:08:29.515+0000 7f2075a06640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31720) UID: 0 2024-09-21T22:08:29.582 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:08:29.582 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:08:29.583 INFO:teuthology.orchestra.run.smithi039.stdout: "num_ops": 0 2024-09-21T22:08:29.583 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:08:29.598 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T22:08:29.644 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:08:29.641+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:08:29.723 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:08:29.717+0000 7fe38af2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 34631) UID: 0 2024-09-21T22:08:29.734 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:08:29.735 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:08:29.735 INFO:teuthology.orchestra.run.smithi039.stdout: "complaint_time": 30, 2024-09-21T22:08:29.735 INFO:teuthology.orchestra.run.smithi039.stdout: "num_blocked_ops": 0 2024-09-21T22:08:29.735 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:08:29.751 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T22:08:29.834 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T22:08:29.848 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:08:29.845+0000 7fe38af2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 34631) UID: 0 2024-09-21T22:08:29.925 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:08:29.921+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:08:29.933 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:08:29.933 INFO:teuthology.orchestra.run.smithi039.stdout: "size": 20, 2024-09-21T22:08:29.933 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 600, 2024-09-21T22:08:29.933 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [ 2024-09-21T22:08:29.934 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.934 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:694 2.5 e63/37 2:b1d55633:::rbd_data.126483d75f2d.000000000000000b:head v 63'26, mlcod=63'24)", 2024-09-21T22:08:29.934 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.163199+0000", 2024-09-21T22:08:29.934 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 64.765944583999996, 2024-09-21T22:08:29.934 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.034676672999999998, 2024-09-21T22:08:29.935 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:29.935 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:29.935 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:29.935 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:29.935 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.935 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:29.936 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.163199+0000", 2024-09-21T22:08:29.936 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:29.936 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:29.936 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.937 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:29.937 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.163199+0000", 2024-09-21T22:08:29.937 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:29.937 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:29.937 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.937 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:29.938 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.163200+0000", 2024-09-21T22:08:29.938 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.703e-06 2024-09-21T22:08:29.938 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:29.938 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.938 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:29.938 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.164013+0000", 2024-09-21T22:08:29.939 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000812447 2024-09-21T22:08:29.939 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:29.939 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.939 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:29.939 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.164042+0000", 2024-09-21T22:08:29.939 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9006e-05 2024-09-21T22:08:29.940 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:29.940 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.940 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:29.940 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.164064+0000", 2024-09-21T22:08:29.941 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1719999999999999e-05 2024-09-21T22:08:29.941 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:29.941 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.941 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:29.941 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.164143+0000", 2024-09-21T22:08:29.941 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.9283999999999995e-05 2024-09-21T22:08:29.942 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:29.942 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.942 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:29.942 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.164165+0000", 2024-09-21T22:08:29.942 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.2379e-05 2024-09-21T22:08:29.942 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:29.943 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.943 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:29.943 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.197827+0000", 2024-09-21T22:08:29.943 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.033661782000000001 2024-09-21T22:08:29.943 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:29.943 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.944 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:29.944 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.197875+0000", 2024-09-21T22:08:29.944 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8352000000000001e-05 2024-09-21T22:08:29.944 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:29.944 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:29.944 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:29.944 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:29.945 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.945 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:696 2.5 e63/37 2:b1d55633:::rbd_data.126483d75f2d.000000000000000b:head v 63'28, mlcod=63'26)", 2024-09-21T22:08:29.945 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.218913+0000", 2024-09-21T22:08:29.945 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 64.710230138, 2024-09-21T22:08:29.945 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.056187599999999997, 2024-09-21T22:08:29.945 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:29.946 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:29.946 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:29.946 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:29.946 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.946 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:29.946 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.218913+0000", 2024-09-21T22:08:29.947 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:29.947 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:29.947 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.947 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:29.947 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.218913+0000", 2024-09-21T22:08:29.947 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:29.948 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:29.948 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.948 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:29.948 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.218914+0000", 2024-09-21T22:08:29.948 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.1930000000000001e-06 2024-09-21T22:08:29.948 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:29.949 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.949 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:29.949 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.219721+0000", 2024-09-21T22:08:29.949 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00080701500000000001 2024-09-21T22:08:29.949 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:29.949 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.950 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:29.950 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.219743+0000", 2024-09-21T22:08:29.950 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1364000000000001e-05 2024-09-21T22:08:29.950 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:29.950 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.950 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:29.951 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.219759+0000", 2024-09-21T22:08:29.951 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.611e-05 2024-09-21T22:08:29.951 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:29.951 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.951 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:29.951 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.219908+0000", 2024-09-21T22:08:29.952 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000149252 2024-09-21T22:08:29.952 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:29.952 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.952 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:29.952 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.219951+0000", 2024-09-21T22:08:29.952 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.2518999999999997e-05 2024-09-21T22:08:29.953 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:29.953 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.953 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:29.953 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.275031+0000", 2024-09-21T22:08:29.953 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.055080241000000002 2024-09-21T22:08:29.954 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:29.954 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.954 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:29.954 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.275101+0000", 2024-09-21T22:08:29.954 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.9906000000000005e-05 2024-09-21T22:08:29.954 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:29.955 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:29.955 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:29.955 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:29.955 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.955 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:698 2.5 e63/37 2:b1d55633:::rbd_data.126483d75f2d.000000000000000b:head v 63'30, mlcod=63'28)", 2024-09-21T22:08:29.955 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.289319+0000", 2024-09-21T22:08:29.956 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 64.639823929000002, 2024-09-21T22:08:29.956 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.053930944000000001, 2024-09-21T22:08:29.956 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:29.956 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:29.956 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:29.956 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:29.956 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.957 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:29.957 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.289319+0000", 2024-09-21T22:08:29.957 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:29.957 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:29.957 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.957 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:29.957 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.289319+0000", 2024-09-21T22:08:29.958 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:29.958 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:29.958 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.958 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:29.958 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.289321+0000", 2024-09-21T22:08:29.958 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.221e-06 2024-09-21T22:08:29.958 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:29.958 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.958 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:29.959 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.290091+0000", 2024-09-21T22:08:29.959 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00077069300000000003 2024-09-21T22:08:29.959 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:29.959 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.959 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:29.959 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.290109+0000", 2024-09-21T22:08:29.959 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7218999999999999e-05 2024-09-21T22:08:29.959 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:29.959 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.959 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:29.959 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.290122+0000", 2024-09-21T22:08:29.959 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3678e-05 2024-09-21T22:08:29.959 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:29.959 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.960 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:29.960 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.290274+0000", 2024-09-21T22:08:29.960 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00015205899999999999 2024-09-21T22:08:29.960 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:29.960 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.960 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:29.960 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.290313+0000", 2024-09-21T22:08:29.960 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.8636000000000001e-05 2024-09-21T22:08:29.960 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:29.960 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.960 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:29.960 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.343179+0000", 2024-09-21T22:08:29.960 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.052866199000000003 2024-09-21T22:08:29.960 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:29.961 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.961 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:29.961 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.343250+0000", 2024-09-21T22:08:29.961 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.1239e-05 2024-09-21T22:08:29.961 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:29.961 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:29.961 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:29.961 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:29.961 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.961 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:702 2.5 e63/37 2:a70ae99f:::rbd_data.126483d75f2d.000000000000000c:head v 63'34, mlcod=63'32)", 2024-09-21T22:08:29.961 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.392532+0000", 2024-09-21T22:08:29.961 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 64.536611340999997, 2024-09-21T22:08:29.961 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.035703990999999997, 2024-09-21T22:08:29.961 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:29.961 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:29.962 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:29.962 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:29.962 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.962 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:29.962 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.392532+0000", 2024-09-21T22:08:29.962 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:29.962 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:29.962 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.962 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:29.962 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.392532+0000", 2024-09-21T22:08:29.962 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:29.963 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:29.963 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.963 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:29.963 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.392534+0000", 2024-09-21T22:08:29.963 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.64e-06 2024-09-21T22:08:29.963 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:29.964 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.964 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:29.964 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.393321+0000", 2024-09-21T22:08:29.964 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00078705699999999997 2024-09-21T22:08:29.964 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:29.964 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.965 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:29.965 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.393347+0000", 2024-09-21T22:08:29.965 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.6735999999999999e-05 2024-09-21T22:08:29.965 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:29.965 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.965 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:29.965 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.393361+0000", 2024-09-21T22:08:29.965 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3346000000000001e-05 2024-09-21T22:08:29.965 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:29.965 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.965 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:29.965 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.393469+0000", 2024-09-21T22:08:29.966 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000108039 2024-09-21T22:08:29.966 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:29.966 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.966 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:29.966 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.393512+0000", 2024-09-21T22:08:29.966 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.2862999999999999e-05 2024-09-21T22:08:29.966 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:29.966 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.966 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:29.967 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.428170+0000", 2024-09-21T22:08:29.967 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.034658858000000001 2024-09-21T22:08:29.967 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:29.967 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.967 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:29.967 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.428236+0000", 2024-09-21T22:08:29.967 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.5451999999999996e-05 2024-09-21T22:08:29.968 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:29.968 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:29.968 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:29.968 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:29.968 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.968 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:708 2.1 e63/53 2:837ddd83:::rbd_data.126483d75f2d.000000000000000d:head v 63'6, mlcod=57'4)", 2024-09-21T22:08:29.968 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.517007+0000", 2024-09-21T22:08:29.968 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 64.412136415000006, 2024-09-21T22:08:29.968 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.032495584000000001, 2024-09-21T22:08:29.968 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:29.968 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:29.968 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:29.968 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:29.969 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.969 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:29.969 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.517007+0000", 2024-09-21T22:08:29.969 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:29.969 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:29.969 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.969 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:29.969 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.517007+0000", 2024-09-21T22:08:29.969 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:29.969 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:29.969 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.969 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:29.969 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.517008+0000", 2024-09-21T22:08:29.969 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.604e-06 2024-09-21T22:08:29.970 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:29.970 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.970 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:29.970 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.517603+0000", 2024-09-21T22:08:29.970 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00059461900000000001 2024-09-21T22:08:29.970 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:29.970 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.970 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:29.970 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.517632+0000", 2024-09-21T22:08:29.970 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8663e-05 2024-09-21T22:08:29.970 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:29.970 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.970 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:29.971 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.517647+0000", 2024-09-21T22:08:29.971 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4833e-05 2024-09-21T22:08:29.971 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:29.971 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.971 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:29.971 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.517714+0000", 2024-09-21T22:08:29.971 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.7669000000000006e-05 2024-09-21T22:08:29.971 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:29.971 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.972 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:29.972 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.517737+0000", 2024-09-21T22:08:29.972 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.2469000000000001e-05 2024-09-21T22:08:29.972 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:29.972 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.972 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:29.972 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.549370+0000", 2024-09-21T22:08:29.973 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.031633221000000003 2024-09-21T22:08:29.973 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:29.973 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.973 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:29.973 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.549502+0000", 2024-09-21T22:08:29.973 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013250600000000001 2024-09-21T22:08:29.973 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:29.973 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:29.973 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:29.973 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:29.973 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.973 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:712 2.1 e63/53 2:837ddd83:::rbd_data.126483d75f2d.000000000000000d:head v 63'10, mlcod=63'8)", 2024-09-21T22:08:29.973 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.604029+0000", 2024-09-21T22:08:29.973 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 64.325114475999996, 2024-09-21T22:08:29.974 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.037982371000000001, 2024-09-21T22:08:29.974 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:29.974 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:29.974 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:29.974 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:29.974 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.974 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:29.974 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.604029+0000", 2024-09-21T22:08:29.974 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:29.974 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:29.975 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.975 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:29.975 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.604029+0000", 2024-09-21T22:08:29.975 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:29.975 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:29.975 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.975 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:29.975 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.604031+0000", 2024-09-21T22:08:29.975 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8870000000000001e-06 2024-09-21T22:08:29.975 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:29.975 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.975 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:29.975 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.604395+0000", 2024-09-21T22:08:29.975 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00036422000000000001 2024-09-21T22:08:29.976 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:29.976 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.976 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:29.976 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.604424+0000", 2024-09-21T22:08:29.976 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8609e-05 2024-09-21T22:08:29.976 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:29.976 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.976 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:29.976 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.604437+0000", 2024-09-21T22:08:29.977 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3086e-05 2024-09-21T22:08:29.977 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:29.977 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.977 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:29.977 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.604509+0000", 2024-09-21T22:08:29.977 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.2448999999999997e-05 2024-09-21T22:08:29.977 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:29.977 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.977 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:29.978 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.604531+0000", 2024-09-21T22:08:29.978 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1602000000000001e-05 2024-09-21T22:08:29.978 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:29.978 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.978 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:29.978 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.641885+0000", 2024-09-21T22:08:29.978 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.037354545000000003 2024-09-21T22:08:29.978 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:29.978 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.978 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:29.978 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.642011+0000", 2024-09-21T22:08:29.979 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00012597300000000001 2024-09-21T22:08:29.979 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:29.979 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:29.979 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:29.979 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:29.979 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.979 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:720 2.1 e63/53 2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head v 63'18, mlcod=63'16)", 2024-09-21T22:08:29.979 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.805111+0000", 2024-09-21T22:08:29.979 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 64.124032416000006, 2024-09-21T22:08:29.979 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.046082864000000001, 2024-09-21T22:08:29.979 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:29.979 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:29.979 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:29.980 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:29.980 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.980 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:29.980 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.805111+0000", 2024-09-21T22:08:29.980 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:29.980 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:29.980 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.980 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:29.980 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.805111+0000", 2024-09-21T22:08:29.980 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:29.980 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:29.980 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.980 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:29.980 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.805112+0000", 2024-09-21T22:08:29.981 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.02e-06 2024-09-21T22:08:29.981 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:29.981 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.981 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:29.981 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.805429+0000", 2024-09-21T22:08:29.981 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00031740799999999998 2024-09-21T22:08:29.981 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:29.981 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.982 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:29.982 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.805459+0000", 2024-09-21T22:08:29.982 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.955e-05 2024-09-21T22:08:29.982 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:29.982 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.982 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:29.982 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.805472+0000", 2024-09-21T22:08:29.982 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3533000000000001e-05 2024-09-21T22:08:29.982 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:29.983 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.983 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:29.983 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.805581+0000", 2024-09-21T22:08:29.983 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000108194 2024-09-21T22:08:29.983 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:29.983 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.983 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:29.983 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.805615+0000", 2024-09-21T22:08:29.983 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.4310000000000002e-05 2024-09-21T22:08:29.983 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:29.983 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.984 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:29.984 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.851146+0000", 2024-09-21T22:08:29.984 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.045531092000000002 2024-09-21T22:08:29.984 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:29.984 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.984 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:29.984 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.851194+0000", 2024-09-21T22:08:29.984 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.7756999999999997e-05 2024-09-21T22:08:29.984 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:29.984 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:29.984 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:29.984 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:29.984 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.985 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:723 2.1 e63/53 2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head v 63'20, mlcod=63'18)", 2024-09-21T22:08:29.985 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.862341+0000", 2024-09-21T22:08:29.985 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 64.066802482, 2024-09-21T22:08:29.985 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.036454671000000001, 2024-09-21T22:08:29.985 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:29.985 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:29.985 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:29.985 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:29.985 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.985 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:29.985 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.862341+0000", 2024-09-21T22:08:29.985 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:29.985 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:29.985 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.986 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:29.986 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.862341+0000", 2024-09-21T22:08:29.986 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:29.986 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:29.986 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.986 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:29.987 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.862342+0000", 2024-09-21T22:08:29.987 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.0299999999999997e-07 2024-09-21T22:08:29.987 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:29.987 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.987 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:29.987 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.862721+0000", 2024-09-21T22:08:29.987 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00037889800000000002 2024-09-21T22:08:29.987 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:29.987 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.987 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:29.987 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.862738+0000", 2024-09-21T22:08:29.987 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6957999999999999e-05 2024-09-21T22:08:29.987 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:29.988 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.988 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:29.988 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.862747+0000", 2024-09-21T22:08:29.988 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.0729999999999994e-06 2024-09-21T22:08:29.988 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:29.988 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.988 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:29.988 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.862830+0000", 2024-09-21T22:08:29.988 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.3113000000000002e-05 2024-09-21T22:08:29.988 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:29.988 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.988 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:29.988 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.862860+0000", 2024-09-21T22:08:29.988 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9980000000000001e-05 2024-09-21T22:08:29.989 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:29.989 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.989 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:29.989 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.898705+0000", 2024-09-21T22:08:29.989 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.035845063000000003 2024-09-21T22:08:29.989 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:29.989 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.989 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:29.989 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.898795+0000", 2024-09-21T22:08:29.989 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.0682999999999999e-05 2024-09-21T22:08:29.989 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:29.989 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:29.989 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:29.989 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:29.990 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.990 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:136 2.5 e72/37 2:b1d55633:::rbd_data.126483d75f2d.000000000000000b:head v 72'44, mlcod=72'42)", 2024-09-21T22:08:29.990 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.386273+0000", 2024-09-21T22:08:29.990 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 51.542870366999999, 2024-09-21T22:08:29.990 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.033806744999999999, 2024-09-21T22:08:29.990 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:29.990 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:29.990 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:29.990 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:29.990 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.990 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:29.990 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.386273+0000", 2024-09-21T22:08:29.990 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:29.990 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:29.991 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.991 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:29.991 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.386273+0000", 2024-09-21T22:08:29.991 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:29.991 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:29.991 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.991 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:29.992 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.386274+0000", 2024-09-21T22:08:29.992 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.8299999999999996e-07 2024-09-21T22:08:29.992 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:29.992 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.992 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:29.992 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.387108+0000", 2024-09-21T22:08:29.992 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00083467800000000005 2024-09-21T22:08:29.992 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:29.992 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.992 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:29.992 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.387121+0000", 2024-09-21T22:08:29.992 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2823000000000001e-05 2024-09-21T22:08:29.992 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:29.993 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.993 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:29.993 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.387128+0000", 2024-09-21T22:08:29.993 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.404e-06 2024-09-21T22:08:29.993 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:29.993 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.993 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:29.993 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.387179+0000", 2024-09-21T22:08:29.993 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.1888000000000002e-05 2024-09-21T22:08:29.993 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:29.993 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.994 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:29.994 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.387201+0000", 2024-09-21T22:08:29.994 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.2019999999999999e-05 2024-09-21T22:08:29.994 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:29.994 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.994 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:29.994 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.419993+0000", 2024-09-21T22:08:29.994 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.032791182000000002 2024-09-21T22:08:29.994 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:29.994 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.994 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:29.994 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.420080+0000", 2024-09-21T22:08:29.994 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.7066999999999998e-05 2024-09-21T22:08:29.995 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:29.995 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:29.995 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:29.995 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:29.995 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.995 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:138 2.5 e72/37 2:b1d55633:::rbd_data.126483d75f2d.000000000000000b:head v 72'46, mlcod=72'44)", 2024-09-21T22:08:29.995 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.430104+0000", 2024-09-21T22:08:29.995 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 51.499039019000001, 2024-09-21T22:08:29.995 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.034087458000000001, 2024-09-21T22:08:29.995 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:29.995 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:29.995 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:29.996 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:29.996 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.996 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:29.996 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.430104+0000", 2024-09-21T22:08:29.996 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:29.996 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:29.996 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.997 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:29.997 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.430104+0000", 2024-09-21T22:08:29.997 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:29.997 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:29.997 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.997 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:29.997 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.430105+0000", 2024-09-21T22:08:29.997 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.1680000000000001e-06 2024-09-21T22:08:29.997 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:29.997 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.997 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:29.997 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.430864+0000", 2024-09-21T22:08:29.997 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00075819400000000001 2024-09-21T22:08:29.998 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:29.998 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.998 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:29.998 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.430881+0000", 2024-09-21T22:08:29.998 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7000999999999999e-05 2024-09-21T22:08:29.998 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:29.998 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.998 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:29.998 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.430889+0000", 2024-09-21T22:08:29.998 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.9540000000000004e-06 2024-09-21T22:08:29.998 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:29.998 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.998 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:29.998 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.430959+0000", 2024-09-21T22:08:29.999 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.0858999999999994e-05 2024-09-21T22:08:29.999 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:29.999 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.999 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:29.999 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.430996+0000", 2024-09-21T22:08:29.999 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.6443999999999999e-05 2024-09-21T22:08:29.999 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:29.999 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.999 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:29.999 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.464087+0000", 2024-09-21T22:08:29.999 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.033090799999999997 2024-09-21T22:08:29.999 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:29.999 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:29.999 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:30.000 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.464192+0000", 2024-09-21T22:08:30.000 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00010503800000000001 2024-09-21T22:08:30.000 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:30.000 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:30.000 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:30.000 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:30.000 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.000 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:140 2.5 e72/37 2:b1d55633:::rbd_data.126483d75f2d.000000000000000b:head v 72'48, mlcod=72'46)", 2024-09-21T22:08:30.000 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.475737+0000", 2024-09-21T22:08:30.000 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 51.453406071000003, 2024-09-21T22:08:30.000 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.037632025, 2024-09-21T22:08:30.000 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:30.001 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:30.001 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:30.001 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:30.001 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.001 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:30.001 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.475737+0000", 2024-09-21T22:08:30.001 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:30.001 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:30.002 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.002 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:30.002 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.475737+0000", 2024-09-21T22:08:30.002 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:30.002 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:30.002 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.002 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:30.002 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.475739+0000", 2024-09-21T22:08:30.002 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.666e-06 2024-09-21T22:08:30.002 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:30.002 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.002 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:30.002 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.476538+0000", 2024-09-21T22:08:30.003 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00079927399999999999 2024-09-21T22:08:30.003 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:30.003 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.003 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:30.003 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.476556+0000", 2024-09-21T22:08:30.003 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7666999999999999e-05 2024-09-21T22:08:30.003 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:30.003 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.003 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:30.003 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.476565+0000", 2024-09-21T22:08:30.003 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.0650000000000005e-06 2024-09-21T22:08:30.003 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:30.003 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.003 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:30.004 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.476651+0000", 2024-09-21T22:08:30.004 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.5916999999999997e-05 2024-09-21T22:08:30.004 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:30.004 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.004 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:30.004 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.476696+0000", 2024-09-21T22:08:30.004 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.5537000000000001e-05 2024-09-21T22:08:30.004 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:30.004 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.004 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:30.004 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.513300+0000", 2024-09-21T22:08:30.004 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.036603498999999998 2024-09-21T22:08:30.004 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:30.004 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.005 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:30.005 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.513369+0000", 2024-09-21T22:08:30.005 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.9400000000000006e-05 2024-09-21T22:08:30.005 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:30.005 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:30.005 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:30.005 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:30.005 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.005 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:145 2.5 e72/37 2:a70ae99f:::rbd_data.126483d75f2d.000000000000000c:head v 72'52, mlcod=72'50)", 2024-09-21T22:08:30.006 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.578171+0000", 2024-09-21T22:08:30.006 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 51.350972220999999, 2024-09-21T22:08:30.006 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.037181537000000001, 2024-09-21T22:08:30.006 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:30.006 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:30.006 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:30.006 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:30.007 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.007 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:30.007 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.578171+0000", 2024-09-21T22:08:30.007 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:30.007 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:30.007 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.007 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:30.007 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.578171+0000", 2024-09-21T22:08:30.007 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:30.007 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:30.007 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.008 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:30.008 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.578172+0000", 2024-09-21T22:08:30.008 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.017e-06 2024-09-21T22:08:30.008 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:30.008 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.008 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:30.008 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.579000+0000", 2024-09-21T22:08:30.008 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00082799499999999999 2024-09-21T22:08:30.008 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:30.008 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.008 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:30.008 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.579020+0000", 2024-09-21T22:08:30.008 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.0165000000000001e-05 2024-09-21T22:08:30.008 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:30.009 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.009 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:30.009 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.579036+0000", 2024-09-21T22:08:30.009 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5767e-05 2024-09-21T22:08:30.009 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:30.009 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.009 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:30.009 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.579106+0000", 2024-09-21T22:08:30.009 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.0091000000000004e-05 2024-09-21T22:08:30.009 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:30.009 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.009 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:30.010 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.579131+0000", 2024-09-21T22:08:30.010 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4978e-05 2024-09-21T22:08:30.010 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:30.010 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.010 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:30.010 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.615249+0000", 2024-09-21T22:08:30.011 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.036118355999999997 2024-09-21T22:08:30.011 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:30.011 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.011 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:30.011 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.615353+0000", 2024-09-21T22:08:30.011 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000103168 2024-09-21T22:08:30.011 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:30.011 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:30.012 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:30.012 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:30.012 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.012 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:147 2.5 e72/37 2:a70ae99f:::rbd_data.126483d75f2d.000000000000000c:head v 72'54, mlcod=72'52)", 2024-09-21T22:08:30.012 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.627203+0000", 2024-09-21T22:08:30.012 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 51.301940109, 2024-09-21T22:08:30.012 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.038203972000000003, 2024-09-21T22:08:30.012 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:30.012 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:30.012 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:30.012 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:30.012 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.012 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:30.012 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.627203+0000", 2024-09-21T22:08:30.013 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:30.013 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:30.013 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.013 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:30.013 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.627203+0000", 2024-09-21T22:08:30.013 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:30.013 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:30.013 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.013 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:30.013 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.627204+0000", 2024-09-21T22:08:30.013 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3069999999999999e-06 2024-09-21T22:08:30.013 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:30.013 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.013 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:30.014 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.628114+0000", 2024-09-21T22:08:30.014 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00090942099999999999 2024-09-21T22:08:30.014 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:30.014 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.014 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:30.014 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.628140+0000", 2024-09-21T22:08:30.014 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5882000000000001e-05 2024-09-21T22:08:30.014 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:30.014 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.015 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:30.015 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.628162+0000", 2024-09-21T22:08:30.015 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.2293000000000002e-05 2024-09-21T22:08:30.015 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:30.015 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.015 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:30.015 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.628250+0000", 2024-09-21T22:08:30.015 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.7973000000000006e-05 2024-09-21T22:08:30.016 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:30.016 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.016 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:30.016 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.628272+0000", 2024-09-21T22:08:30.016 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1682999999999999e-05 2024-09-21T22:08:30.016 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:30.016 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.016 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:30.016 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.665322+0000", 2024-09-21T22:08:30.016 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.037050650999999997 2024-09-21T22:08:30.016 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:30.016 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.016 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:30.016 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.665407+0000", 2024-09-21T22:08:30.017 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.4761999999999999e-05 2024-09-21T22:08:30.017 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:30.017 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:30.017 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:30.017 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:30.017 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.017 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:149 2.5 e72/37 2:a70ae99f:::rbd_data.126483d75f2d.000000000000000c:head v 72'56, mlcod=72'54)", 2024-09-21T22:08:30.017 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.690377+0000", 2024-09-21T22:08:30.017 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 51.238766366, 2024-09-21T22:08:30.017 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.049254799000000002, 2024-09-21T22:08:30.017 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:30.017 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:30.017 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:30.017 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:30.018 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.018 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:30.018 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.690377+0000", 2024-09-21T22:08:30.018 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:30.018 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:30.018 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.018 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:30.018 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.690377+0000", 2024-09-21T22:08:30.018 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:30.018 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:30.018 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.018 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:30.018 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.690378+0000", 2024-09-21T22:08:30.019 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.344e-06 2024-09-21T22:08:30.019 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:30.019 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.019 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:30.019 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.691302+0000", 2024-09-21T22:08:30.019 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00092322400000000005 2024-09-21T22:08:30.019 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:30.019 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.019 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:30.020 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.691321+0000", 2024-09-21T22:08:30.020 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9928999999999998e-05 2024-09-21T22:08:30.020 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:30.020 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.020 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:30.020 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.691345+0000", 2024-09-21T22:08:30.020 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.372e-05 2024-09-21T22:08:30.021 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:30.021 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.021 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:30.021 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.691442+0000", 2024-09-21T22:08:30.021 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.7270999999999998e-05 2024-09-21T22:08:30.021 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:30.022 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.022 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:30.022 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.691512+0000", 2024-09-21T22:08:30.022 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.9183000000000005e-05 2024-09-21T22:08:30.022 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:30.022 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.023 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:30.023 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.739545+0000", 2024-09-21T22:08:30.023 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.048033657 2024-09-21T22:08:30.023 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:30.023 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.023 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:30.024 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.739632+0000", 2024-09-21T22:08:30.024 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.6471000000000006e-05 2024-09-21T22:08:30.024 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:30.024 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:30.024 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:30.024 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:30.024 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.024 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:156 2.1 e72/53 2:837ddd83:::rbd_data.126483d75f2d.000000000000000d:head v 72'28, mlcod=72'26)", 2024-09-21T22:08:30.025 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.824856+0000", 2024-09-21T22:08:30.025 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 51.104286928000001, 2024-09-21T22:08:30.025 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.034320985999999998, 2024-09-21T22:08:30.025 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:30.025 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:30.025 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:30.025 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:30.026 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.026 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:30.026 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.824856+0000", 2024-09-21T22:08:30.026 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:30.026 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:30.026 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.026 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:30.027 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.824856+0000", 2024-09-21T22:08:30.027 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:30.027 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:30.027 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.027 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:30.027 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.824858+0000", 2024-09-21T22:08:30.027 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2979999999999999e-06 2024-09-21T22:08:30.027 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:30.027 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.027 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:30.028 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.825174+0000", 2024-09-21T22:08:30.028 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000316335 2024-09-21T22:08:30.028 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:30.028 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.028 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:30.028 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.825193+0000", 2024-09-21T22:08:30.028 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9436000000000002e-05 2024-09-21T22:08:30.028 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:30.028 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.028 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:30.028 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.825202+0000", 2024-09-21T22:08:30.028 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.8359999999999998e-06 2024-09-21T22:08:30.028 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:30.028 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.028 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:30.029 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.825286+0000", 2024-09-21T22:08:30.029 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.3683999999999994e-05 2024-09-21T22:08:30.029 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:30.029 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.029 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:30.029 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.825314+0000", 2024-09-21T22:08:30.029 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.7653000000000001e-05 2024-09-21T22:08:30.029 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:30.030 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.030 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:30.030 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.859122+0000", 2024-09-21T22:08:30.030 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.033808624000000002 2024-09-21T22:08:30.030 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:30.030 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.030 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:30.030 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.859177+0000", 2024-09-21T22:08:30.030 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.5120000000000001e-05 2024-09-21T22:08:30.031 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:30.031 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:30.031 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:30.031 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:30.031 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.031 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:158 2.1 e72/53 2:837ddd83:::rbd_data.126483d75f2d.000000000000000d:head v 72'30, mlcod=72'28)", 2024-09-21T22:08:30.031 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.869277+0000", 2024-09-21T22:08:30.031 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 51.059866731, 2024-09-21T22:08:30.031 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.040508390999999998, 2024-09-21T22:08:30.031 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:30.031 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:30.031 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:30.032 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:30.032 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.032 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:30.032 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.869277+0000", 2024-09-21T22:08:30.032 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:30.032 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:30.032 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.032 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:30.032 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.869277+0000", 2024-09-21T22:08:30.032 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:30.032 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:30.032 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.032 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:30.032 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.869278+0000", 2024-09-21T22:08:30.033 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5269999999999999e-06 2024-09-21T22:08:30.033 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:30.033 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.033 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:30.033 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.869902+0000", 2024-09-21T22:08:30.033 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00062399000000000005 2024-09-21T22:08:30.033 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:30.033 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.033 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:30.033 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.869924+0000", 2024-09-21T22:08:30.033 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1875999999999999e-05 2024-09-21T22:08:30.033 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:30.033 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.033 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:30.034 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.869937+0000", 2024-09-21T22:08:30.034 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2685e-05 2024-09-21T22:08:30.034 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:30.034 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.034 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:30.034 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.870058+0000", 2024-09-21T22:08:30.034 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000120862 2024-09-21T22:08:30.034 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:30.034 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.034 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:30.034 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.870093+0000", 2024-09-21T22:08:30.034 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.5896000000000003e-05 2024-09-21T22:08:30.035 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:30.035 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.035 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:30.035 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.909728+0000", 2024-09-21T22:08:30.035 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.039634461000000003 2024-09-21T22:08:30.035 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:30.035 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.035 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:30.035 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.909785+0000", 2024-09-21T22:08:30.036 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.7093999999999999e-05 2024-09-21T22:08:30.036 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:30.036 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:30.036 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:30.036 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:30.036 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.036 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:161 2.1 e72/53 2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head v 72'32, mlcod=72'30)", 2024-09-21T22:08:30.036 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.924455+0000", 2024-09-21T22:08:30.036 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 51.004688369, 2024-09-21T22:08:30.037 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.038366863000000001, 2024-09-21T22:08:30.037 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:30.037 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:30.037 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:30.037 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:30.037 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.037 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:30.037 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.924455+0000", 2024-09-21T22:08:30.037 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:30.037 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:30.037 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.037 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:30.037 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.924455+0000", 2024-09-21T22:08:30.038 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:30.038 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:30.038 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.038 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:30.038 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.924456+0000", 2024-09-21T22:08:30.038 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.003e-06 2024-09-21T22:08:30.038 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:30.038 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.038 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:30.038 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.924811+0000", 2024-09-21T22:08:30.038 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00035519499999999999 2024-09-21T22:08:30.038 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:30.038 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.039 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:30.039 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.924826+0000", 2024-09-21T22:08:30.039 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5296999999999999e-05 2024-09-21T22:08:30.039 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:30.039 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.039 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:30.039 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.924834+0000", 2024-09-21T22:08:30.039 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.5379999999999997e-06 2024-09-21T22:08:30.039 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:30.039 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.039 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:30.039 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.924910+0000", 2024-09-21T22:08:30.040 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.6488000000000004e-05 2024-09-21T22:08:30.040 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:30.040 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.040 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:30.040 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.924934+0000", 2024-09-21T22:08:30.040 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3598999999999999e-05 2024-09-21T22:08:30.040 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:30.040 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.040 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:30.040 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.962739+0000", 2024-09-21T22:08:30.040 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.037805370999999997 2024-09-21T22:08:30.040 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:30.041 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.041 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:30.041 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.962822+0000", 2024-09-21T22:08:30.041 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.2372000000000003e-05 2024-09-21T22:08:30.041 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:30.041 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:30.041 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:30.041 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:30.042 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.042 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:163 2.1 e72/53 2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head v 72'34, mlcod=72'32)", 2024-09-21T22:08:30.042 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.971683+0000", 2024-09-21T22:08:30.042 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 50.957459829000001, 2024-09-21T22:08:30.042 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.043171320999999999, 2024-09-21T22:08:30.042 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:30.042 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:30.043 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:30.043 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:30.043 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.043 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:30.043 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.971683+0000", 2024-09-21T22:08:30.043 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:30.044 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:30.044 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.044 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:30.044 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.971683+0000", 2024-09-21T22:08:30.044 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:30.044 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:30.045 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.045 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:30.045 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.971684+0000", 2024-09-21T22:08:30.045 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.6600000000000005e-07 2024-09-21T22:08:30.045 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:30.045 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.045 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:30.046 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.971945+0000", 2024-09-21T22:08:30.046 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00026101899999999998 2024-09-21T22:08:30.046 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:30.046 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.046 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:30.046 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.971959+0000", 2024-09-21T22:08:30.047 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4054e-05 2024-09-21T22:08:30.047 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:30.047 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.047 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:30.047 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.971969+0000", 2024-09-21T22:08:30.047 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.499e-06 2024-09-21T22:08:30.048 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:30.048 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.048 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:30.048 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.972037+0000", 2024-09-21T22:08:30.048 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.8448000000000006e-05 2024-09-21T22:08:30.048 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:30.048 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.049 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:30.049 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.972057+0000", 2024-09-21T22:08:30.049 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9817999999999999e-05 2024-09-21T22:08:30.049 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:30.049 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.049 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:30.050 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.014770+0000", 2024-09-21T22:08:30.050 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.042712464999999998 2024-09-21T22:08:30.050 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:30.050 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.050 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:30.050 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.014855+0000", 2024-09-21T22:08:30.051 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.5152000000000001e-05 2024-09-21T22:08:30.051 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:30.051 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:30.051 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:30.051 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:30.051 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.052 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:165 2.1 e72/53 2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head v 72'36, mlcod=72'34)", 2024-09-21T22:08:30.052 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:39.028631+0000", 2024-09-21T22:08:30.052 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 50.900512352, 2024-09-21T22:08:30.052 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.033430561999999997, 2024-09-21T22:08:30.052 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:30.052 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:30.053 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:30.053 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:30.053 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.053 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:30.053 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.028631+0000", 2024-09-21T22:08:30.054 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:30.054 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:30.054 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.054 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:30.054 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.028631+0000", 2024-09-21T22:08:30.054 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:30.055 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:30.055 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.055 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:30.055 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.028633+0000", 2024-09-21T22:08:30.055 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5799999999999999e-06 2024-09-21T22:08:30.055 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:30.055 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.056 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:30.056 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.029209+0000", 2024-09-21T22:08:30.056 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00057670799999999999 2024-09-21T22:08:30.056 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:30.056 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.056 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:30.056 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.029238+0000", 2024-09-21T22:08:30.057 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8943000000000001e-05 2024-09-21T22:08:30.057 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:30.057 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.057 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:30.057 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.029252+0000", 2024-09-21T22:08:30.057 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3672999999999999e-05 2024-09-21T22:08:30.057 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:30.058 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.058 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:30.058 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.029328+0000", 2024-09-21T22:08:30.058 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.6497999999999998e-05 2024-09-21T22:08:30.058 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:30.058 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.058 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:30.059 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.029353+0000", 2024-09-21T22:08:30.059 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5060999999999999e-05 2024-09-21T22:08:30.059 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:30.059 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.059 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:30.059 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.061993+0000", 2024-09-21T22:08:30.059 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.032639286000000003 2024-09-21T22:08:30.059 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:30.059 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.060 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:30.060 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.062062+0000", 2024-09-21T22:08:30.060 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.8813000000000007e-05 2024-09-21T22:08:30.060 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:30.060 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:30.060 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:30.060 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:30.060 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.060 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:167 2.1 e72/53 2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head v 72'38, mlcod=72'36)", 2024-09-21T22:08:30.060 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:39.077324+0000", 2024-09-21T22:08:30.060 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 50.851819538000001, 2024-09-21T22:08:30.060 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.036031961000000001, 2024-09-21T22:08:30.060 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:30.060 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:30.061 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:30.061 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:30.061 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.061 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:30.061 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.077324+0000", 2024-09-21T22:08:30.061 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:30.061 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:30.061 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.061 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:30.061 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.077324+0000", 2024-09-21T22:08:30.061 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:30.061 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:30.061 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.061 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:30.062 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.077326+0000", 2024-09-21T22:08:30.062 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.75e-06 2024-09-21T22:08:30.062 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:30.062 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.062 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:30.062 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.077719+0000", 2024-09-21T22:08:30.062 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00039322599999999999 2024-09-21T22:08:30.062 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:30.062 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.062 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:30.062 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.077749+0000", 2024-09-21T22:08:30.062 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9895000000000001e-05 2024-09-21T22:08:30.062 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:30.063 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.063 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:30.063 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.077763+0000", 2024-09-21T22:08:30.063 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4517e-05 2024-09-21T22:08:30.063 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:30.063 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.063 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:30.063 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.077843+0000", 2024-09-21T22:08:30.063 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.9994999999999996e-05 2024-09-21T22:08:30.063 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:30.064 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.064 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:30.064 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.077867+0000", 2024-09-21T22:08:30.064 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3640000000000001e-05 2024-09-21T22:08:30.064 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:30.064 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.064 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:30.064 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.113216+0000", 2024-09-21T22:08:30.064 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.035349168 2024-09-21T22:08:30.064 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:30.065 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:30.065 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:30.065 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.113356+0000", 2024-09-21T22:08:30.065 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013977 2024-09-21T22:08:30.065 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:30.065 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:30.065 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:30.065 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:30.065 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:30.065 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:08:30.067 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T22:08:30.067 INFO:teuthology.orchestra.run.smithi039.stdout:osd.0: {} 2024-09-21T22:08:30.068 INFO:teuthology.orchestra.run.smithi039.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T22:08:30.068 INFO:teuthology.orchestra.run.smithi039.stdout:osd.1: {} 2024-09-21T22:08:30.068 INFO:teuthology.orchestra.run.smithi039.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T22:08:30.068 INFO:teuthology.orchestra.run.smithi039.stdout:osd.2: {} 2024-09-21T22:08:30.069 INFO:teuthology.orchestra.run.smithi039.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T22:08:30.069 INFO:teuthology.orchestra.run.smithi039.stdout:osd.3: {} 2024-09-21T22:08:30.069 INFO:teuthology.orchestra.run.smithi039.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T22:08:30.069 INFO:teuthology.orchestra.run.smithi039.stdout:osd.4: {} 2024-09-21T22:08:30.070 INFO:teuthology.orchestra.run.smithi039.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T22:08:30.070 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:08:30.061+0000 7fe38af2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 34631) UID: 0 2024-09-21T22:08:30.087 INFO:teuthology.orchestra.run.smithi039.stdout:osd.5: {} 2024-09-21T22:08:30.087 INFO:teuthology.orchestra.run.smithi039.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T22:08:30.168 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:08:30.163+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:08:30.218 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:30.218 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:30.233 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T22:08:30.268 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:08:30.267+0000 7f2075a06640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31720) UID: 0 2024-09-21T22:08:30.357 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:30.357 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:30.368 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:08:30.365+0000 7fe38af2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 34631) UID: 0 2024-09-21T22:08:30.372 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T22:08:30.470 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:08:30.465+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:08:30.544 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:30.545 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:30.559 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T22:08:30.604 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:08:30.601+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:08:30.672 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:08:30.669+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:08:30.694 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:30.694 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:30.710 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T22:08:30.773 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:08:30.767+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:08:30.837 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:30.837 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:30.852 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T22:08:30.874 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:08:30.869+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:08:30.975 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:08:30.969+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:08:30.999 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-21T22:08:31.016 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:31.016 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:31.032 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T22:08:31.077 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:08:31.075+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:08:31.158 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:31.158 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:31.172 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T22:08:31.177 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:08:31.173+0000 7fe38af2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 34631) UID: 0 2024-09-21T22:08:31.278 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:08:31.273+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:08:31.340 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:31.340 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:31.356 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T22:08:31.378 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:08:31.373+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:08:31.479 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:08:31.475+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:08:31.481 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:31.482 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:31.496 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T22:08:31.582 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:08:31.579+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:08:31.666 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:31.666 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:31.681 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T22:08:31.681 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:08:31.677+0000 7fe38af2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 34631) UID: 0 2024-09-21T22:08:31.782 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:08:31.777+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:08:31.840 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:31.840 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:31.854 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T22:08:31.883 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:08:31.877+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:08:31.974 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:31.975 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:31.983 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:08:31.979+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:08:31.987 INFO:tasks.thrashosds.thrasher:in_osds: [3, 4, 5, 0, 1] out_osds: [2] dead_osds: [] live_osds: [0, 1, 3, 4, 5, 2] 2024-09-21T22:08:31.987 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T22:08:31.988 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-21T22:08:31.988 INFO:tasks.thrashosds.thrasher:Setting osd 3 primary_affinity to 0.340599 2024-09-21T22:08:31.988 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 0.34059935445152123 2024-09-21T22:08:31.989 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T22:08:32.027 INFO:teuthology.orchestra.run.smithi039.stderr:noscrub is unset 2024-09-21T22:08:32.084 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:08:32.079+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:08:32.106 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T22:08:32.130 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:32.130 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:32.149 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T22:08:32.192 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:08:32.189+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:08:32.260 INFO:teuthology.orchestra.run.smithi039.stdout:osd.0: {} 2024-09-21T22:08:32.261 INFO:teuthology.orchestra.run.smithi039.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T22:08:32.277 INFO:teuthology.orchestra.run.smithi039.stdout:osd.1: {} 2024-09-21T22:08:32.277 INFO:teuthology.orchestra.run.smithi039.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T22:08:32.285 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:08:32.281+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:08:32.296 INFO:teuthology.orchestra.run.smithi039.stdout:osd.2: {} 2024-09-21T22:08:32.296 INFO:teuthology.orchestra.run.smithi039.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T22:08:32.311 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:08:32.311 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:08:32.311 INFO:teuthology.orchestra.run.smithi039.stdout: "complaint_time": 30, 2024-09-21T22:08:32.311 INFO:teuthology.orchestra.run.smithi039.stdout: "num_blocked_ops": 0 2024-09-21T22:08:32.311 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:08:32.317 INFO:teuthology.orchestra.run.smithi039.stdout:osd.3: {} 2024-09-21T22:08:32.317 INFO:teuthology.orchestra.run.smithi039.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T22:08:32.326 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T22:08:32.338 INFO:teuthology.orchestra.run.smithi039.stdout:osd.4: {} 2024-09-21T22:08:32.338 INFO:teuthology.orchestra.run.smithi039.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T22:08:32.359 INFO:teuthology.orchestra.run.smithi039.stdout:osd.5: {} 2024-09-21T22:08:32.360 INFO:teuthology.orchestra.run.smithi039.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T22:08:32.387 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:08:32.383+0000 7f2075a06640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31720) UID: 0 2024-09-21T22:08:32.461 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:08:32.461 INFO:teuthology.orchestra.run.smithi039.stdout: "size": 20, 2024-09-21T22:08:32.461 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 600, 2024-09-21T22:08:32.461 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [] 2024-09-21T22:08:32.461 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:08:32.476 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T22:08:32.520 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:08:32.517+0000 7fe38af2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 34631) UID: 0 2024-09-21T22:08:32.588 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:08:32.587+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:08:32.603 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:08:32.603 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:08:32.603 INFO:teuthology.orchestra.run.smithi039.stdout: "num_ops": 0 2024-09-21T22:08:32.603 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:08:32.618 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T22:08:32.688 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:08:32.687+0000 7f2075a06640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31720) UID: 0 2024-09-21T22:08:32.786 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:08:32.787 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:08:32.787 INFO:teuthology.orchestra.run.smithi039.stdout: "complaint_time": 30, 2024-09-21T22:08:32.787 INFO:teuthology.orchestra.run.smithi039.stdout: "num_blocked_ops": 0 2024-09-21T22:08:32.787 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:08:32.789 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:08:32.785+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:08:32.801 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T22:08:32.890 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:08:32.887+0000 7f2075a06640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31720) UID: 0 2024-09-21T22:08:32.968 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:08:32.968 INFO:teuthology.orchestra.run.smithi039.stdout: "size": 20, 2024-09-21T22:08:32.968 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 600, 2024-09-21T22:08:32.968 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [ 2024-09-21T22:08:32.969 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:32.969 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:694 2.5 e63/37 2:b1d55633:::rbd_data.126483d75f2d.000000000000000b:head v 63'26, mlcod=63'24)", 2024-09-21T22:08:32.969 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.163199+0000", 2024-09-21T22:08:32.969 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 67.801145032999997, 2024-09-21T22:08:32.969 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.034676672999999998, 2024-09-21T22:08:32.969 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:32.969 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:32.969 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:32.969 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:32.969 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:32.969 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:32.969 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.163199+0000", 2024-09-21T22:08:32.970 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:32.970 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:32.970 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:32.970 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:32.970 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.163199+0000", 2024-09-21T22:08:32.970 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:32.970 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:32.970 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:32.970 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:32.970 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.163200+0000", 2024-09-21T22:08:32.970 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.703e-06 2024-09-21T22:08:32.970 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:32.971 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:32.971 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:32.971 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.164013+0000", 2024-09-21T22:08:32.971 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000812447 2024-09-21T22:08:32.971 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:32.971 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:32.971 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:32.971 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.164042+0000", 2024-09-21T22:08:32.971 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9006e-05 2024-09-21T22:08:32.971 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:32.971 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:32.971 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:32.971 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.164064+0000", 2024-09-21T22:08:32.971 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1719999999999999e-05 2024-09-21T22:08:32.971 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:32.972 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:32.972 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:32.972 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.164143+0000", 2024-09-21T22:08:32.972 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.9283999999999995e-05 2024-09-21T22:08:32.972 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:32.972 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:32.972 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:32.972 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.164165+0000", 2024-09-21T22:08:32.972 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.2379e-05 2024-09-21T22:08:32.972 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:32.972 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:32.972 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:32.972 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.197827+0000", 2024-09-21T22:08:32.972 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.033661782000000001 2024-09-21T22:08:32.973 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:32.973 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:32.973 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:32.973 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.197875+0000", 2024-09-21T22:08:32.973 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8352000000000001e-05 2024-09-21T22:08:32.973 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:32.973 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:32.973 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:32.973 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:32.973 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:32.973 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:696 2.5 e63/37 2:b1d55633:::rbd_data.126483d75f2d.000000000000000b:head v 63'28, mlcod=63'26)", 2024-09-21T22:08:32.973 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.218913+0000", 2024-09-21T22:08:32.973 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 67.745430587000001, 2024-09-21T22:08:32.973 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.056187599999999997, 2024-09-21T22:08:32.974 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:32.974 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:32.974 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:32.974 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:32.974 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:32.974 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:32.974 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.218913+0000", 2024-09-21T22:08:32.974 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:32.974 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:32.974 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:32.974 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:32.974 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.218913+0000", 2024-09-21T22:08:32.975 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:32.975 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:32.975 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:32.975 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:32.975 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.218914+0000", 2024-09-21T22:08:32.975 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.1930000000000001e-06 2024-09-21T22:08:32.975 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:32.975 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:32.975 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:32.975 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.219721+0000", 2024-09-21T22:08:32.975 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00080701500000000001 2024-09-21T22:08:32.975 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:32.975 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:32.975 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:32.976 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.219743+0000", 2024-09-21T22:08:32.976 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1364000000000001e-05 2024-09-21T22:08:32.976 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:32.976 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:32.976 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:32.976 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.219759+0000", 2024-09-21T22:08:32.976 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.611e-05 2024-09-21T22:08:32.976 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:32.976 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:32.976 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:32.976 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.219908+0000", 2024-09-21T22:08:32.976 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000149252 2024-09-21T22:08:32.976 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:32.976 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:32.976 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:32.977 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.219951+0000", 2024-09-21T22:08:32.977 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.2518999999999997e-05 2024-09-21T22:08:32.977 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:32.977 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:32.977 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:32.977 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.275031+0000", 2024-09-21T22:08:32.977 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.055080241000000002 2024-09-21T22:08:32.977 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:32.977 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:32.977 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:32.977 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.275101+0000", 2024-09-21T22:08:32.977 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.9906000000000005e-05 2024-09-21T22:08:32.977 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:32.977 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:32.978 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:32.978 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:32.978 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:32.978 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:698 2.5 e63/37 2:b1d55633:::rbd_data.126483d75f2d.000000000000000b:head v 63'30, mlcod=63'28)", 2024-09-21T22:08:32.978 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.289319+0000", 2024-09-21T22:08:32.978 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 67.675024378000003, 2024-09-21T22:08:32.978 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.053930944000000001, 2024-09-21T22:08:32.978 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:32.978 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:32.978 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:32.978 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:32.978 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:32.978 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:32.978 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.289319+0000", 2024-09-21T22:08:32.979 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:32.979 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:32.979 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:32.979 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:32.979 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.289319+0000", 2024-09-21T22:08:32.979 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:32.979 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:32.979 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:32.979 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:32.979 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.289321+0000", 2024-09-21T22:08:32.979 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.221e-06 2024-09-21T22:08:32.979 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:32.979 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:32.979 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:32.980 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.290091+0000", 2024-09-21T22:08:32.980 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00077069300000000003 2024-09-21T22:08:32.980 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:32.980 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:32.980 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:32.980 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.290109+0000", 2024-09-21T22:08:32.980 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7218999999999999e-05 2024-09-21T22:08:32.980 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:32.980 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:32.980 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:32.980 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.290122+0000", 2024-09-21T22:08:32.980 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3678e-05 2024-09-21T22:08:32.980 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:32.980 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:32.981 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:32.981 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.290274+0000", 2024-09-21T22:08:32.981 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00015205899999999999 2024-09-21T22:08:32.981 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:32.981 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:32.981 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:32.981 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.290313+0000", 2024-09-21T22:08:32.981 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.8636000000000001e-05 2024-09-21T22:08:32.981 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:32.981 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:32.981 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:32.981 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.343179+0000", 2024-09-21T22:08:32.981 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.052866199000000003 2024-09-21T22:08:32.981 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:32.981 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:32.982 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:32.982 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.343250+0000", 2024-09-21T22:08:32.982 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.1239e-05 2024-09-21T22:08:32.982 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:32.982 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:32.982 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:32.982 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:32.982 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:32.982 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:702 2.5 e63/37 2:a70ae99f:::rbd_data.126483d75f2d.000000000000000c:head v 63'34, mlcod=63'32)", 2024-09-21T22:08:32.982 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.392532+0000", 2024-09-21T22:08:32.982 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 67.571811789999998, 2024-09-21T22:08:32.982 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.035703990999999997, 2024-09-21T22:08:32.982 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:32.983 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:32.983 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:32.983 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:32.983 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:32.983 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:32.983 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.392532+0000", 2024-09-21T22:08:32.983 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:32.983 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:32.983 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:32.983 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:32.983 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.392532+0000", 2024-09-21T22:08:32.983 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:32.983 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:32.983 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:32.984 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:32.984 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.392534+0000", 2024-09-21T22:08:32.984 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.64e-06 2024-09-21T22:08:32.984 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:32.984 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:32.984 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:32.984 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.393321+0000", 2024-09-21T22:08:32.984 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00078705699999999997 2024-09-21T22:08:32.984 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:32.984 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:32.984 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:32.984 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.393347+0000", 2024-09-21T22:08:32.984 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.6735999999999999e-05 2024-09-21T22:08:32.984 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:32.985 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:32.985 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:32.985 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.393361+0000", 2024-09-21T22:08:32.985 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3346000000000001e-05 2024-09-21T22:08:32.985 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:32.985 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:32.985 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:32.985 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.393469+0000", 2024-09-21T22:08:32.985 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000108039 2024-09-21T22:08:32.985 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:32.985 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:32.985 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:32.985 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.393512+0000", 2024-09-21T22:08:32.985 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.2862999999999999e-05 2024-09-21T22:08:32.985 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:32.986 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:32.986 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:32.986 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.428170+0000", 2024-09-21T22:08:32.986 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.034658858000000001 2024-09-21T22:08:32.986 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:32.986 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:32.986 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:32.986 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.428236+0000", 2024-09-21T22:08:32.986 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.5451999999999996e-05 2024-09-21T22:08:32.986 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:32.986 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:32.986 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:32.987 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:32.987 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:32.987 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:708 2.1 e63/53 2:837ddd83:::rbd_data.126483d75f2d.000000000000000d:head v 63'6, mlcod=57'4)", 2024-09-21T22:08:32.987 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.517007+0000", 2024-09-21T22:08:32.987 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 67.447336863999993, 2024-09-21T22:08:32.987 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.032495584000000001, 2024-09-21T22:08:32.987 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:32.987 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:32.987 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:32.987 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:32.987 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:32.987 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:32.987 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.517007+0000", 2024-09-21T22:08:32.987 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:32.988 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:32.988 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:32.988 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:32.988 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.517007+0000", 2024-09-21T22:08:32.988 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:32.988 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:32.988 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:32.988 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:32.988 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.517008+0000", 2024-09-21T22:08:32.988 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.604e-06 2024-09-21T22:08:32.988 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:32.988 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:32.988 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:32.988 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.517603+0000", 2024-09-21T22:08:32.988 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00059461900000000001 2024-09-21T22:08:32.989 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:32.989 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:32.989 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:32.989 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.517632+0000", 2024-09-21T22:08:32.989 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8663e-05 2024-09-21T22:08:32.989 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:32.989 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:32.989 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:32.989 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.517647+0000", 2024-09-21T22:08:32.989 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4833e-05 2024-09-21T22:08:32.989 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:32.989 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:32.989 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:32.989 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.517714+0000", 2024-09-21T22:08:32.990 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.7669000000000006e-05 2024-09-21T22:08:32.990 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:32.990 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:32.990 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:32.990 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.517737+0000", 2024-09-21T22:08:32.990 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.2469000000000001e-05 2024-09-21T22:08:32.990 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:32.990 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:32.990 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:32.990 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.549370+0000", 2024-09-21T22:08:32.990 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.031633221000000003 2024-09-21T22:08:32.990 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:32.990 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:32.991 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:32.991 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.549502+0000", 2024-09-21T22:08:32.991 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013250600000000001 2024-09-21T22:08:32.991 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:32.991 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:32.991 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:32.991 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:32.991 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:32.991 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:712 2.1 e63/53 2:837ddd83:::rbd_data.126483d75f2d.000000000000000d:head v 63'10, mlcod=63'8)", 2024-09-21T22:08:32.991 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.604029+0000", 2024-09-21T22:08:32.991 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 67.360314924999997, 2024-09-21T22:08:32.991 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.037982371000000001, 2024-09-21T22:08:32.991 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:32.992 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:32.992 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:32.992 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:32.992 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:32.992 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:32.992 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.604029+0000", 2024-09-21T22:08:32.992 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:32.992 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:32.992 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:32.992 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:32.992 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.604029+0000", 2024-09-21T22:08:32.992 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:32.992 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:32.992 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:32.993 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:32.993 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.604031+0000", 2024-09-21T22:08:32.993 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8870000000000001e-06 2024-09-21T22:08:32.993 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:32.993 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:32.993 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:32.993 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.604395+0000", 2024-09-21T22:08:32.993 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00036422000000000001 2024-09-21T22:08:32.993 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:32.993 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:32.993 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:32.993 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.604424+0000", 2024-09-21T22:08:32.993 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8609e-05 2024-09-21T22:08:32.993 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:32.994 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:32.994 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:32.994 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.604437+0000", 2024-09-21T22:08:32.994 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3086e-05 2024-09-21T22:08:32.994 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:32.994 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:32.994 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:32.994 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.604509+0000", 2024-09-21T22:08:32.994 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.2448999999999997e-05 2024-09-21T22:08:32.994 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:32.994 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:32.994 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:32.995 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.604531+0000", 2024-09-21T22:08:32.995 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1602000000000001e-05 2024-09-21T22:08:32.995 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:32.995 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:32.995 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:32.995 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.641885+0000", 2024-09-21T22:08:32.995 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.037354545000000003 2024-09-21T22:08:32.995 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:32.995 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:32.995 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:32.995 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.642011+0000", 2024-09-21T22:08:32.995 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00012597300000000001 2024-09-21T22:08:32.995 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:32.995 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:32.996 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:32.996 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:32.996 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:32.996 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:720 2.1 e63/53 2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head v 63'18, mlcod=63'16)", 2024-09-21T22:08:32.996 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.805111+0000", 2024-09-21T22:08:32.996 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 67.159232865000007, 2024-09-21T22:08:32.996 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.046082864000000001, 2024-09-21T22:08:32.996 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:32.996 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:32.996 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:32.996 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:32.996 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:32.996 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:32.996 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.805111+0000", 2024-09-21T22:08:32.996 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:32.997 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:32.997 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:32.997 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:32.997 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.805111+0000", 2024-09-21T22:08:32.997 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:32.997 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:32.997 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:32.997 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:32.997 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.805112+0000", 2024-09-21T22:08:32.997 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.02e-06 2024-09-21T22:08:32.997 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:32.997 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:32.997 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:32.997 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.805429+0000", 2024-09-21T22:08:32.998 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00031740799999999998 2024-09-21T22:08:32.998 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:32.998 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:32.998 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:32.998 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.805459+0000", 2024-09-21T22:08:32.998 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.955e-05 2024-09-21T22:08:32.998 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:32.998 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:32.998 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:32.998 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.805472+0000", 2024-09-21T22:08:32.998 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3533000000000001e-05 2024-09-21T22:08:32.998 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:32.998 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:32.999 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:32.999 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.805581+0000", 2024-09-21T22:08:32.999 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000108194 2024-09-21T22:08:32.999 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:32.999 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:32.999 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:32.999 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.805615+0000", 2024-09-21T22:08:32.999 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.4310000000000002e-05 2024-09-21T22:08:32.999 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:32.999 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:32.999 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:32.999 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.851146+0000", 2024-09-21T22:08:32.999 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.045531092000000002 2024-09-21T22:08:32.999 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.000 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.000 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:33.000 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.851194+0000", 2024-09-21T22:08:33.000 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.7756999999999997e-05 2024-09-21T22:08:33.000 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:33.000 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:33.000 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:33.000 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.000 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.000 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:723 2.1 e63/53 2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head v 63'20, mlcod=63'18)", 2024-09-21T22:08:33.000 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.862341+0000", 2024-09-21T22:08:33.000 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 67.102002931000001, 2024-09-21T22:08:33.000 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.036454671000000001, 2024-09-21T22:08:33.001 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:33.001 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:33.001 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:33.001 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:33.001 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.001 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:33.001 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.862341+0000", 2024-09-21T22:08:33.001 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:33.001 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.001 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.001 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:33.001 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.862341+0000", 2024-09-21T22:08:33.001 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:33.001 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.002 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.002 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:33.002 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.862342+0000", 2024-09-21T22:08:33.002 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.0299999999999997e-07 2024-09-21T22:08:33.002 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.002 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.002 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:33.002 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.862721+0000", 2024-09-21T22:08:33.002 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00037889800000000002 2024-09-21T22:08:33.002 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.002 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.002 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:33.002 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.862738+0000", 2024-09-21T22:08:33.002 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6957999999999999e-05 2024-09-21T22:08:33.003 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.003 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.003 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:33.003 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.862747+0000", 2024-09-21T22:08:33.003 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.0729999999999994e-06 2024-09-21T22:08:33.003 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.003 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.003 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:33.003 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.862830+0000", 2024-09-21T22:08:33.003 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.3113000000000002e-05 2024-09-21T22:08:33.003 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.003 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.003 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:33.003 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.862860+0000", 2024-09-21T22:08:33.004 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9980000000000001e-05 2024-09-21T22:08:33.004 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.004 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.004 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:33.004 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.898705+0000", 2024-09-21T22:08:33.004 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.035845063000000003 2024-09-21T22:08:33.004 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.004 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.004 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:33.004 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.898795+0000", 2024-09-21T22:08:33.004 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.0682999999999999e-05 2024-09-21T22:08:33.004 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:33.004 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:33.004 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:33.004 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.005 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.005 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:136 2.5 e72/37 2:b1d55633:::rbd_data.126483d75f2d.000000000000000b:head v 72'44, mlcod=72'42)", 2024-09-21T22:08:33.005 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.386273+0000", 2024-09-21T22:08:33.005 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 54.578070816, 2024-09-21T22:08:33.005 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.033806744999999999, 2024-09-21T22:08:33.005 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:33.005 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:33.005 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:33.005 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:33.005 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.005 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:33.005 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.386273+0000", 2024-09-21T22:08:33.005 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:33.006 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.006 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.006 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:33.006 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.386273+0000", 2024-09-21T22:08:33.006 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:33.006 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.006 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.006 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:33.006 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.386274+0000", 2024-09-21T22:08:33.006 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.8299999999999996e-07 2024-09-21T22:08:33.006 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.006 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.006 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:33.007 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.387108+0000", 2024-09-21T22:08:33.007 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00083467800000000005 2024-09-21T22:08:33.007 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.007 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.007 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:33.007 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.387121+0000", 2024-09-21T22:08:33.007 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2823000000000001e-05 2024-09-21T22:08:33.007 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.007 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.007 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:33.007 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.387128+0000", 2024-09-21T22:08:33.007 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.404e-06 2024-09-21T22:08:33.007 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.007 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.008 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:33.008 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.387179+0000", 2024-09-21T22:08:33.008 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.1888000000000002e-05 2024-09-21T22:08:33.008 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.008 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.008 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:33.008 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.387201+0000", 2024-09-21T22:08:33.008 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.2019999999999999e-05 2024-09-21T22:08:33.008 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.008 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.008 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:33.008 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.419993+0000", 2024-09-21T22:08:33.008 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.032791182000000002 2024-09-21T22:08:33.009 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.009 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.009 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:33.009 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.420080+0000", 2024-09-21T22:08:33.009 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.7066999999999998e-05 2024-09-21T22:08:33.009 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:33.009 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:33.009 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:33.009 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.009 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.009 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:138 2.5 e72/37 2:b1d55633:::rbd_data.126483d75f2d.000000000000000b:head v 72'46, mlcod=72'44)", 2024-09-21T22:08:33.009 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.430104+0000", 2024-09-21T22:08:33.009 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 54.534239468000003, 2024-09-21T22:08:33.009 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.034087458000000001, 2024-09-21T22:08:33.009 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:33.010 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:33.010 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:33.010 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:33.010 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.010 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:33.010 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.430104+0000", 2024-09-21T22:08:33.010 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:33.010 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.010 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.010 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:33.010 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.430104+0000", 2024-09-21T22:08:33.010 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:33.010 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.010 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.011 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:33.011 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.430105+0000", 2024-09-21T22:08:33.011 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.1680000000000001e-06 2024-09-21T22:08:33.011 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.011 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.011 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:33.011 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.430864+0000", 2024-09-21T22:08:33.011 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00075819400000000001 2024-09-21T22:08:33.011 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.011 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.011 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:33.011 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.430881+0000", 2024-09-21T22:08:33.011 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7000999999999999e-05 2024-09-21T22:08:33.012 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.012 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.012 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:33.012 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.430889+0000", 2024-09-21T22:08:33.012 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.9540000000000004e-06 2024-09-21T22:08:33.012 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.012 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.012 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:33.012 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.430959+0000", 2024-09-21T22:08:33.012 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.0858999999999994e-05 2024-09-21T22:08:33.012 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.012 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.013 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:33.013 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.430996+0000", 2024-09-21T22:08:33.013 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.6443999999999999e-05 2024-09-21T22:08:33.013 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.013 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.013 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:33.013 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.464087+0000", 2024-09-21T22:08:33.013 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.033090799999999997 2024-09-21T22:08:33.013 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.013 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.013 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:33.013 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.464192+0000", 2024-09-21T22:08:33.013 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00010503800000000001 2024-09-21T22:08:33.013 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:33.014 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:33.014 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:33.014 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.014 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.014 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:140 2.5 e72/37 2:b1d55633:::rbd_data.126483d75f2d.000000000000000b:head v 72'48, mlcod=72'46)", 2024-09-21T22:08:33.014 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.475737+0000", 2024-09-21T22:08:33.014 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 54.488606519999998, 2024-09-21T22:08:33.014 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.037632025, 2024-09-21T22:08:33.014 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:33.014 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:33.014 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:33.014 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:33.014 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.015 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:33.015 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.475737+0000", 2024-09-21T22:08:33.015 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:33.015 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.015 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.015 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:33.015 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.475737+0000", 2024-09-21T22:08:33.015 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:33.015 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.015 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.015 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:33.015 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.475739+0000", 2024-09-21T22:08:33.015 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.666e-06 2024-09-21T22:08:33.015 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.016 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.016 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:33.016 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.476538+0000", 2024-09-21T22:08:33.016 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00079927399999999999 2024-09-21T22:08:33.016 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.016 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.016 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:33.016 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.476556+0000", 2024-09-21T22:08:33.016 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7666999999999999e-05 2024-09-21T22:08:33.016 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.016 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.016 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:33.016 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.476565+0000", 2024-09-21T22:08:33.016 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.0650000000000005e-06 2024-09-21T22:08:33.017 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.017 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.017 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:33.017 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.476651+0000", 2024-09-21T22:08:33.017 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.5916999999999997e-05 2024-09-21T22:08:33.017 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.017 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.017 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:33.017 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.476696+0000", 2024-09-21T22:08:33.017 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.5537000000000001e-05 2024-09-21T22:08:33.017 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.017 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.018 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:33.018 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.513300+0000", 2024-09-21T22:08:33.018 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.036603498999999998 2024-09-21T22:08:33.018 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.018 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.018 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:33.018 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.513369+0000", 2024-09-21T22:08:33.018 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.9400000000000006e-05 2024-09-21T22:08:33.018 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:33.018 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:33.018 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:33.018 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.019 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.019 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:145 2.5 e72/37 2:a70ae99f:::rbd_data.126483d75f2d.000000000000000c:head v 72'52, mlcod=72'50)", 2024-09-21T22:08:33.019 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.578171+0000", 2024-09-21T22:08:33.019 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 54.386172670000001, 2024-09-21T22:08:33.019 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.037181537000000001, 2024-09-21T22:08:33.019 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:33.019 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:33.019 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:33.019 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:33.019 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.019 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:33.019 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.578171+0000", 2024-09-21T22:08:33.020 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:33.020 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.020 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.020 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:33.020 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.578171+0000", 2024-09-21T22:08:33.020 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:33.020 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.020 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.020 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:33.020 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.578172+0000", 2024-09-21T22:08:33.020 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.017e-06 2024-09-21T22:08:33.021 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.021 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.021 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:33.021 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.579000+0000", 2024-09-21T22:08:33.021 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00082799499999999999 2024-09-21T22:08:33.021 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.021 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.021 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:33.021 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.579020+0000", 2024-09-21T22:08:33.021 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.0165000000000001e-05 2024-09-21T22:08:33.021 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.021 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.021 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:33.021 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.579036+0000", 2024-09-21T22:08:33.022 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5767e-05 2024-09-21T22:08:33.022 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.022 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.022 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:33.022 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.579106+0000", 2024-09-21T22:08:33.022 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.0091000000000004e-05 2024-09-21T22:08:33.022 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.022 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.022 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:33.022 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.579131+0000", 2024-09-21T22:08:33.022 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4978e-05 2024-09-21T22:08:33.022 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.022 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.022 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:33.023 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.615249+0000", 2024-09-21T22:08:33.023 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.036118355999999997 2024-09-21T22:08:33.023 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.023 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.023 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:33.023 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.615353+0000", 2024-09-21T22:08:33.023 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000103168 2024-09-21T22:08:33.023 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:33.023 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:33.023 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:33.023 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.023 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.023 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:147 2.5 e72/37 2:a70ae99f:::rbd_data.126483d75f2d.000000000000000c:head v 72'54, mlcod=72'52)", 2024-09-21T22:08:33.023 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.627203+0000", 2024-09-21T22:08:33.023 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 54.337140558000002, 2024-09-21T22:08:33.024 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.038203972000000003, 2024-09-21T22:08:33.024 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:33.024 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:33.024 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:33.024 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:33.024 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.024 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:33.024 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.627203+0000", 2024-09-21T22:08:33.024 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:33.024 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.024 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.024 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:33.024 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.627203+0000", 2024-09-21T22:08:33.024 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:33.024 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.025 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.025 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:33.025 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.627204+0000", 2024-09-21T22:08:33.025 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3069999999999999e-06 2024-09-21T22:08:33.025 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.025 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.025 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:33.025 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.628114+0000", 2024-09-21T22:08:33.025 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00090942099999999999 2024-09-21T22:08:33.025 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.025 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.025 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:33.025 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.628140+0000", 2024-09-21T22:08:33.025 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5882000000000001e-05 2024-09-21T22:08:33.026 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.026 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.026 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:33.026 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.628162+0000", 2024-09-21T22:08:33.026 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.2293000000000002e-05 2024-09-21T22:08:33.026 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.026 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.026 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:33.026 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.628250+0000", 2024-09-21T22:08:33.026 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.7973000000000006e-05 2024-09-21T22:08:33.026 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.026 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.026 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:33.026 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.628272+0000", 2024-09-21T22:08:33.027 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1682999999999999e-05 2024-09-21T22:08:33.027 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.027 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.027 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:33.027 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.665322+0000", 2024-09-21T22:08:33.027 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.037050650999999997 2024-09-21T22:08:33.027 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.027 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.027 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:33.027 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.665407+0000", 2024-09-21T22:08:33.027 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.4761999999999999e-05 2024-09-21T22:08:33.027 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:33.028 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:33.028 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:33.028 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.028 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.028 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:149 2.5 e72/37 2:a70ae99f:::rbd_data.126483d75f2d.000000000000000c:head v 72'56, mlcod=72'54)", 2024-09-21T22:08:33.028 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.690377+0000", 2024-09-21T22:08:33.028 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 54.273966815000001, 2024-09-21T22:08:33.028 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.049254799000000002, 2024-09-21T22:08:33.028 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:33.028 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:33.028 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:33.028 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:33.028 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.028 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:33.029 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.690377+0000", 2024-09-21T22:08:33.029 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:33.029 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.029 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.029 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:33.029 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.690377+0000", 2024-09-21T22:08:33.029 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:33.029 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.029 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.029 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:33.029 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.690378+0000", 2024-09-21T22:08:33.029 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.344e-06 2024-09-21T22:08:33.029 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.029 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.029 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:33.030 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.691302+0000", 2024-09-21T22:08:33.030 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00092322400000000005 2024-09-21T22:08:33.030 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.030 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.030 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:33.030 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.691321+0000", 2024-09-21T22:08:33.030 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9928999999999998e-05 2024-09-21T22:08:33.030 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.030 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.030 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:33.030 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.691345+0000", 2024-09-21T22:08:33.030 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.372e-05 2024-09-21T22:08:33.030 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.031 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.031 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:33.031 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.691442+0000", 2024-09-21T22:08:33.031 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.7270999999999998e-05 2024-09-21T22:08:33.031 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.031 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.031 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:33.031 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.691512+0000", 2024-09-21T22:08:33.031 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.9183000000000005e-05 2024-09-21T22:08:33.031 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.031 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.031 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:33.031 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.739545+0000", 2024-09-21T22:08:33.031 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.048033657 2024-09-21T22:08:33.032 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.032 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.032 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:33.032 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.739632+0000", 2024-09-21T22:08:33.032 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.6471000000000006e-05 2024-09-21T22:08:33.032 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:33.032 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:33.032 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:33.032 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.032 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.032 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:156 2.1 e72/53 2:837ddd83:::rbd_data.126483d75f2d.000000000000000d:head v 72'28, mlcod=72'26)", 2024-09-21T22:08:33.032 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.824856+0000", 2024-09-21T22:08:33.032 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 54.139487377000002, 2024-09-21T22:08:33.032 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.034320985999999998, 2024-09-21T22:08:33.033 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:33.033 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:33.033 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:33.033 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:33.033 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.033 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:33.033 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.824856+0000", 2024-09-21T22:08:33.033 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:33.033 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.033 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.033 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:33.033 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.824856+0000", 2024-09-21T22:08:33.033 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:33.033 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.034 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.034 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:33.034 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.824858+0000", 2024-09-21T22:08:33.034 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2979999999999999e-06 2024-09-21T22:08:33.034 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.034 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.034 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:33.034 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.825174+0000", 2024-09-21T22:08:33.034 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000316335 2024-09-21T22:08:33.034 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.034 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.034 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:33.034 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.825193+0000", 2024-09-21T22:08:33.034 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9436000000000002e-05 2024-09-21T22:08:33.035 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.035 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.035 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:33.035 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.825202+0000", 2024-09-21T22:08:33.035 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.8359999999999998e-06 2024-09-21T22:08:33.035 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.035 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.035 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:33.035 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.825286+0000", 2024-09-21T22:08:33.035 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.3683999999999994e-05 2024-09-21T22:08:33.035 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.035 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.035 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:33.035 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.825314+0000", 2024-09-21T22:08:33.036 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.7653000000000001e-05 2024-09-21T22:08:33.036 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.036 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.036 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:33.036 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.859122+0000", 2024-09-21T22:08:33.036 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.033808624000000002 2024-09-21T22:08:33.036 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.036 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.036 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:33.036 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.859177+0000", 2024-09-21T22:08:33.036 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.5120000000000001e-05 2024-09-21T22:08:33.036 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:33.036 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:33.036 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:33.037 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.037 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.037 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:158 2.1 e72/53 2:837ddd83:::rbd_data.126483d75f2d.000000000000000d:head v 72'30, mlcod=72'28)", 2024-09-21T22:08:33.037 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.869277+0000", 2024-09-21T22:08:33.037 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 54.095067180000001, 2024-09-21T22:08:33.037 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.040508390999999998, 2024-09-21T22:08:33.037 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:33.037 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:33.037 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:33.037 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:33.037 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.037 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:33.037 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.869277+0000", 2024-09-21T22:08:33.037 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:33.038 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.038 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.038 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:33.038 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.869277+0000", 2024-09-21T22:08:33.038 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:33.038 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.038 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.038 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:33.038 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.869278+0000", 2024-09-21T22:08:33.038 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5269999999999999e-06 2024-09-21T22:08:33.038 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.038 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.038 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:33.039 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.869902+0000", 2024-09-21T22:08:33.039 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00062399000000000005 2024-09-21T22:08:33.039 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.039 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.039 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:33.039 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.869924+0000", 2024-09-21T22:08:33.039 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1875999999999999e-05 2024-09-21T22:08:33.039 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.039 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.039 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:33.039 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.869937+0000", 2024-09-21T22:08:33.039 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2685e-05 2024-09-21T22:08:33.039 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.039 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.039 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:33.040 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.870058+0000", 2024-09-21T22:08:33.040 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000120862 2024-09-21T22:08:33.040 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.040 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.040 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:33.040 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.870093+0000", 2024-09-21T22:08:33.040 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.5896000000000003e-05 2024-09-21T22:08:33.040 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.040 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.040 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:33.040 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.909728+0000", 2024-09-21T22:08:33.040 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.039634461000000003 2024-09-21T22:08:33.040 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.040 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.041 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:33.041 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.909785+0000", 2024-09-21T22:08:33.041 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.7093999999999999e-05 2024-09-21T22:08:33.041 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:33.041 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:33.041 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:33.041 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.041 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.041 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:161 2.1 e72/53 2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head v 72'32, mlcod=72'30)", 2024-09-21T22:08:33.041 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.924455+0000", 2024-09-21T22:08:33.041 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 54.039888818000001, 2024-09-21T22:08:33.041 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.038366863000000001, 2024-09-21T22:08:33.041 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:33.041 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:33.042 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:33.042 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:33.042 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.042 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:33.042 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.924455+0000", 2024-09-21T22:08:33.042 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:33.042 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.042 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.042 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:33.042 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.924455+0000", 2024-09-21T22:08:33.042 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:33.042 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.042 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.043 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:33.043 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.924456+0000", 2024-09-21T22:08:33.043 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.003e-06 2024-09-21T22:08:33.043 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.043 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.043 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:33.043 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.924811+0000", 2024-09-21T22:08:33.043 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00035519499999999999 2024-09-21T22:08:33.043 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.043 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.043 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:33.043 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.924826+0000", 2024-09-21T22:08:33.043 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5296999999999999e-05 2024-09-21T22:08:33.043 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.044 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.044 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:33.044 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.924834+0000", 2024-09-21T22:08:33.044 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.5379999999999997e-06 2024-09-21T22:08:33.044 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.044 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.044 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:33.044 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.924910+0000", 2024-09-21T22:08:33.044 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.6488000000000004e-05 2024-09-21T22:08:33.044 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.044 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.044 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:33.044 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.924934+0000", 2024-09-21T22:08:33.044 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3598999999999999e-05 2024-09-21T22:08:33.044 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.045 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.045 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:33.045 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.962739+0000", 2024-09-21T22:08:33.045 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.037805370999999997 2024-09-21T22:08:33.045 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.045 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.045 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:33.045 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.962822+0000", 2024-09-21T22:08:33.045 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.2372000000000003e-05 2024-09-21T22:08:33.045 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:33.045 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:33.045 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:33.045 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.045 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.046 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:163 2.1 e72/53 2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head v 72'34, mlcod=72'32)", 2024-09-21T22:08:33.046 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.971683+0000", 2024-09-21T22:08:33.046 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 53.992660278000002, 2024-09-21T22:08:33.046 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.043171320999999999, 2024-09-21T22:08:33.046 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:33.046 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:33.046 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:33.046 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:33.046 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.046 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:33.046 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.971683+0000", 2024-09-21T22:08:33.046 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:33.047 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.047 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.047 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:33.047 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.971683+0000", 2024-09-21T22:08:33.047 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:33.047 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.047 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.047 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:33.047 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.971684+0000", 2024-09-21T22:08:33.047 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.6600000000000005e-07 2024-09-21T22:08:33.047 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.047 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.047 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:33.047 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.971945+0000", 2024-09-21T22:08:33.048 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00026101899999999998 2024-09-21T22:08:33.048 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.048 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.048 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:33.048 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.971959+0000", 2024-09-21T22:08:33.049 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4054e-05 2024-09-21T22:08:33.049 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.050 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.050 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:33.050 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.971969+0000", 2024-09-21T22:08:33.050 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.499e-06 2024-09-21T22:08:33.050 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.050 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.050 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:33.050 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.972037+0000", 2024-09-21T22:08:33.050 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.8448000000000006e-05 2024-09-21T22:08:33.050 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.050 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.050 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:33.050 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.972057+0000", 2024-09-21T22:08:33.050 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9817999999999999e-05 2024-09-21T22:08:33.051 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.051 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.051 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:33.051 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.014770+0000", 2024-09-21T22:08:33.051 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.042712464999999998 2024-09-21T22:08:33.051 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.051 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.051 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:33.051 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.014855+0000", 2024-09-21T22:08:33.051 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.5152000000000001e-05 2024-09-21T22:08:33.051 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:33.051 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:33.051 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:33.051 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.052 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.052 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:165 2.1 e72/53 2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head v 72'36, mlcod=72'34)", 2024-09-21T22:08:33.052 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:39.028631+0000", 2024-09-21T22:08:33.052 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 53.935712801000001, 2024-09-21T22:08:33.052 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.033430561999999997, 2024-09-21T22:08:33.052 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:33.052 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:33.052 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:33.052 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:33.052 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.052 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:33.052 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.028631+0000", 2024-09-21T22:08:33.053 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:33.053 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.053 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.053 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:33.053 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.028631+0000", 2024-09-21T22:08:33.053 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:33.053 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.053 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.053 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:33.053 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.028633+0000", 2024-09-21T22:08:33.053 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5799999999999999e-06 2024-09-21T22:08:33.053 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.053 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.053 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:33.054 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.029209+0000", 2024-09-21T22:08:33.054 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00057670799999999999 2024-09-21T22:08:33.054 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.054 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.054 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:33.054 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.029238+0000", 2024-09-21T22:08:33.054 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8943000000000001e-05 2024-09-21T22:08:33.054 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.054 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.054 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:33.054 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.029252+0000", 2024-09-21T22:08:33.054 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3672999999999999e-05 2024-09-21T22:08:33.054 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.054 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.055 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:33.055 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.029328+0000", 2024-09-21T22:08:33.055 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.6497999999999998e-05 2024-09-21T22:08:33.055 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.055 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.055 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:33.055 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.029353+0000", 2024-09-21T22:08:33.055 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5060999999999999e-05 2024-09-21T22:08:33.055 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.055 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.055 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:33.055 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.061993+0000", 2024-09-21T22:08:33.055 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.032639286000000003 2024-09-21T22:08:33.055 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.056 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.056 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:33.056 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.062062+0000", 2024-09-21T22:08:33.056 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 6.8813000000000007e-05 2024-09-21T22:08:33.056 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:33.056 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:33.056 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:33.056 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.056 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.056 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:167 2.1 e72/53 2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head v 72'38, mlcod=72'36)", 2024-09-21T22:08:33.056 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:39.077324+0000", 2024-09-21T22:08:33.056 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 53.887019987000002, 2024-09-21T22:08:33.056 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.036031961000000001, 2024-09-21T22:08:33.056 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:33.057 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:33.057 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:33.057 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:33.057 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.057 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:33.057 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.077324+0000", 2024-09-21T22:08:33.057 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:33.057 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.057 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.057 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:33.057 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.077324+0000", 2024-09-21T22:08:33.057 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:33.057 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.057 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.058 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:33.058 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.077326+0000", 2024-09-21T22:08:33.058 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.75e-06 2024-09-21T22:08:33.058 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.058 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.058 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:33.058 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.077719+0000", 2024-09-21T22:08:33.058 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00039322599999999999 2024-09-21T22:08:33.058 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.058 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.058 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:33.058 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.077749+0000", 2024-09-21T22:08:33.058 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9895000000000001e-05 2024-09-21T22:08:33.058 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.059 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.059 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:33.059 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.077763+0000", 2024-09-21T22:08:33.059 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4517e-05 2024-09-21T22:08:33.059 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.059 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.059 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:33.059 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.077843+0000", 2024-09-21T22:08:33.059 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.9994999999999996e-05 2024-09-21T22:08:33.059 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.059 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.059 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:33.059 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.077867+0000", 2024-09-21T22:08:33.059 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.3640000000000001e-05 2024-09-21T22:08:33.060 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.060 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.060 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:33.060 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.113216+0000", 2024-09-21T22:08:33.060 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.035349168 2024-09-21T22:08:33.060 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.060 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.060 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:33.060 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:39.113356+0000", 2024-09-21T22:08:33.060 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00013977 2024-09-21T22:08:33.060 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:33.060 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:33.060 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:33.060 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:33.060 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:33.061 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:08:33.062 INFO:tasks.workunit.client.0.smithi157.stderr: Removing all snapshots: 25% complete... Removing all snapshots: 50% complete... Removing all snapshots: 75% complete... Removing all snapshots: 100% complete... Removing all snapshots: 100% complete...done. 2024-09-21T22:08:33.063 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T22:08:33.063 INFO:teuthology.orchestra.run.smithi039.stderr:set osd.3 primary-affinity to 0.340599 (8223212) 2024-09-21T22:08:33.063 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:08:33.057+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:08:33.064 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-21T22:08:33.060+0000 7effbc87f640 0 -- 172.21.15.157:0/2504500586 >> [v2:172.21.15.64:6816/1547734524,v1:172.21.15.64:6817/1547734524] conn(0x7eff98008c60 msgr2=0x7eff9800b0f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-21T22:08:33.076 INFO:tasks.workunit.client.0.smithi157.stderr:+ rbd -p rbd --namespace '' remove testrbdnbd21553 2024-09-21T22:08:33.162 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:08:33.157+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:08:33.247 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:08:33.247 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:08:33.247 INFO:teuthology.orchestra.run.smithi039.stdout: "num_ops": 0 2024-09-21T22:08:33.247 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:08:33.262 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T22:08:33.264 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:08:33.259+0000 7f2075a06640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31720) UID: 0 2024-09-21T22:08:33.265 INFO:tasks.workunit.client.0.smithi157.stderr: Removing image: 6% complete... Removing image: 12% complete... Removing image: 18% complete... Removing image: 25% complete... Removing image: 31% complete... Removing image: 37% complete... Removing image: 43% complete... Removing image: 50% complete... Removing image: 56% complete... Removing image: 62% complete... Removing image: 68% complete... Removing image: 75% complete... Removing image: 81% complete... Removing image: 87% complete... Removing image: 93% complete...2024-09-21T22:08:33.260+0000 7fe232d40640 0 -- 172.21.15.157:0/2859749851 >> [v2:172.21.15.39:6816/2876495139,v1:172.21.15.39:6817/2876495139] conn(0x562ffb6b1cc0 msgr2=0x562ffb6b40b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-21T22:08:33.300 INFO:tasks.workunit.client.0.smithi157.stderr: Removing image: 100% complete...done. 2024-09-21T22:08:33.309 INFO:tasks.workunit.client.0.smithi157.stderr:+ for ns in '' ${NS} 2024-09-21T22:08:33.309 INFO:tasks.workunit.client.0.smithi157.stderr:+ rbd -p rbd --namespace ns status testrbdnbd21553 2024-09-21T22:08:33.364 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:08:33.361+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:08:33.368 INFO:tasks.workunit.client.0.smithi157.stdout:Watchers: none 2024-09-21T22:08:33.372 INFO:tasks.workunit.client.0.smithi157.stderr:+ for s in 0.5 1 2 4 8 16 32 2024-09-21T22:08:33.372 INFO:tasks.workunit.client.0.smithi157.stderr:+ sleep 0.5 2024-09-21T22:08:33.432 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:08:33.432 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [], 2024-09-21T22:08:33.432 INFO:teuthology.orchestra.run.smithi039.stdout: "complaint_time": 30, 2024-09-21T22:08:33.432 INFO:teuthology.orchestra.run.smithi039.stdout: "num_blocked_ops": 0 2024-09-21T22:08:33.432 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:08:33.447 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T22:08:33.466 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:08:33.463+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:08:33.566 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:08:33.563+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:08:33.618 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:08:33.618 INFO:teuthology.orchestra.run.smithi039.stdout: "size": 20, 2024-09-21T22:08:33.618 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 600, 2024-09-21T22:08:33.618 INFO:teuthology.orchestra.run.smithi039.stdout: "ops": [ 2024-09-21T22:08:33.618 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.618 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.3 55/54 [PushOp(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head, version: 52'2, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head@52'2, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T22:08:33.618 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.084379+0000", 2024-09-21T22:08:33.618 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 81.529220201000001, 2024-09-21T22:08:33.618 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.051020674000000002, 2024-09-21T22:08:33.618 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:33.619 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:33.619 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:08:33.619 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:33.619 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.619 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:33.619 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.084379+0000", 2024-09-21T22:08:33.619 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:33.619 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.619 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.619 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:33.619 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.084379+0000", 2024-09-21T22:08:33.619 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:33.619 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.619 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.620 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:33.620 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.084381+0000", 2024-09-21T22:08:33.620 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.73e-06 2024-09-21T22:08:33.620 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.620 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.620 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:33.620 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.085494+0000", 2024-09-21T22:08:33.620 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011131559999999999 2024-09-21T22:08:33.620 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.620 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.620 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:33.620 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.085542+0000", 2024-09-21T22:08:33.620 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8334999999999997e-05 2024-09-21T22:08:33.620 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.621 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.621 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:33.621 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.085591+0000", 2024-09-21T22:08:33.621 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.9165000000000001e-05 2024-09-21T22:08:33.621 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.621 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.621 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:33.621 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.085777+0000", 2024-09-21T22:08:33.621 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000186152 2024-09-21T22:08:33.621 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.621 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.621 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:33.621 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.085796+0000", 2024-09-21T22:08:33.621 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8612999999999999e-05 2024-09-21T22:08:33.621 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.622 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.622 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:33.622 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.135399+0000", 2024-09-21T22:08:33.622 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.049603522999999997 2024-09-21T22:08:33.622 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:33.622 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:33.622 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:33.622 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.622 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.622 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.3 55/54 [PushOp(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head, version: 52'2, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head@52'2, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:08:33.622 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.139554+0000", 2024-09-21T22:08:33.622 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 81.474045098000005, 2024-09-21T22:08:33.622 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.056215389999999997, 2024-09-21T22:08:33.622 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:33.623 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:33.623 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:08:33.623 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:33.623 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.623 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:33.623 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.139554+0000", 2024-09-21T22:08:33.623 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:33.623 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.623 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.623 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:33.623 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.139554+0000", 2024-09-21T22:08:33.623 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:33.623 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.623 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.624 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:33.624 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.139556+0000", 2024-09-21T22:08:33.624 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7999999999999999e-06 2024-09-21T22:08:33.624 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.624 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.624 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:33.624 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.140671+0000", 2024-09-21T22:08:33.624 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001115612 2024-09-21T22:08:33.624 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.624 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.624 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:33.624 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.140720+0000", 2024-09-21T22:08:33.624 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8631999999999998e-05 2024-09-21T22:08:33.624 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.625 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.625 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:33.625 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.140752+0000", 2024-09-21T22:08:33.625 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.2199000000000001e-05 2024-09-21T22:08:33.625 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.625 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.625 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:33.625 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.140900+0000", 2024-09-21T22:08:33.625 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00014786000000000001 2024-09-21T22:08:33.625 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.625 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.625 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:33.625 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.140917+0000", 2024-09-21T22:08:33.625 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6753999999999999e-05 2024-09-21T22:08:33.625 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.626 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.626 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:33.626 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.195769+0000", 2024-09-21T22:08:33.626 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.054852533000000002 2024-09-21T22:08:33.626 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:33.626 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:33.626 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:33.626 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.626 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.626 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.3 55/54 [PushOp(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head, version: 52'2, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head@52'2, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:08:33.626 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.199187+0000", 2024-09-21T22:08:33.626 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 81.414411553999997, 2024-09-21T22:08:33.626 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.051543975999999998, 2024-09-21T22:08:33.626 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:33.627 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:33.627 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:08:33.627 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:33.627 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.627 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:33.627 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.199187+0000", 2024-09-21T22:08:33.627 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:33.627 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.627 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.627 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:33.627 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.199187+0000", 2024-09-21T22:08:33.627 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:33.627 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.627 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.627 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:33.628 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.199189+0000", 2024-09-21T22:08:33.628 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7969999999999999e-06 2024-09-21T22:08:33.628 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.628 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.628 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:33.628 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.200355+0000", 2024-09-21T22:08:33.628 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011660139999999999 2024-09-21T22:08:33.628 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.628 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.628 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:33.628 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.200405+0000", 2024-09-21T22:08:33.628 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.0124000000000003e-05 2024-09-21T22:08:33.628 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.628 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.629 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:33.629 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.200457+0000", 2024-09-21T22:08:33.629 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.1177999999999997e-05 2024-09-21T22:08:33.629 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.629 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.629 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:33.629 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.200635+0000", 2024-09-21T22:08:33.629 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00017841299999999999 2024-09-21T22:08:33.629 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.629 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.629 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:33.629 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.200653+0000", 2024-09-21T22:08:33.629 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7944999999999998e-05 2024-09-21T22:08:33.629 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.630 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.630 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:33.630 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.250731+0000", 2024-09-21T22:08:33.630 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.050078505000000002 2024-09-21T22:08:33.630 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:33.630 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:33.630 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:33.630 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.630 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.630 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.3 55/54 [PushOp(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head, version: 52'3, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head@52'3, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:08:33.630 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.343754+0000", 2024-09-21T22:08:33.630 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 81.269844843000001, 2024-09-21T22:08:33.630 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.063560607000000005, 2024-09-21T22:08:33.630 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:33.631 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:33.631 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:08:33.631 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:33.631 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.631 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:33.631 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.343754+0000", 2024-09-21T22:08:33.631 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:33.631 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.631 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.631 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:33.631 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.343754+0000", 2024-09-21T22:08:33.631 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:33.631 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.631 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.632 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:33.632 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.343756+0000", 2024-09-21T22:08:33.632 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7069999999999999e-06 2024-09-21T22:08:33.632 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.632 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.632 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:33.632 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.344864+0000", 2024-09-21T22:08:33.632 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011085159999999999 2024-09-21T22:08:33.632 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.632 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.632 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:33.632 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.344914+0000", 2024-09-21T22:08:33.632 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.9462000000000002e-05 2024-09-21T22:08:33.632 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.633 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.633 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:33.633 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.344961+0000", 2024-09-21T22:08:33.633 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.7633999999999998e-05 2024-09-21T22:08:33.633 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.633 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.633 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:33.633 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.345171+0000", 2024-09-21T22:08:33.633 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000209652 2024-09-21T22:08:33.633 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.633 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.633 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:33.633 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.345193+0000", 2024-09-21T22:08:33.633 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.1985999999999999e-05 2024-09-21T22:08:33.633 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.634 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.634 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:33.634 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.407315+0000", 2024-09-21T22:08:33.634 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.06212165 2024-09-21T22:08:33.634 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:33.634 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:33.634 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:33.634 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.634 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.634 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.3 55/54 [PushOp(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head, version: 52'3, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head@52'3, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:08:33.634 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.410551+0000", 2024-09-21T22:08:33.634 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 81.203047712, 2024-09-21T22:08:33.634 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.054517522999999998, 2024-09-21T22:08:33.634 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:33.635 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:33.635 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:08:33.635 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:33.635 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.635 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:33.635 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.410551+0000", 2024-09-21T22:08:33.635 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:33.635 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.635 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.635 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:33.635 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.410551+0000", 2024-09-21T22:08:33.635 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:33.635 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.635 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.635 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:33.636 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.410553+0000", 2024-09-21T22:08:33.636 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5200000000000001e-06 2024-09-21T22:08:33.636 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.636 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.636 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:33.636 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.411747+0000", 2024-09-21T22:08:33.636 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0011946089999999999 2024-09-21T22:08:33.636 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.636 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.636 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:33.636 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.411796+0000", 2024-09-21T22:08:33.636 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8717999999999997e-05 2024-09-21T22:08:33.636 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.637 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.637 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:33.637 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.411844+0000", 2024-09-21T22:08:33.637 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8291999999999998e-05 2024-09-21T22:08:33.637 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.637 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.637 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:33.637 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.412066+0000", 2024-09-21T22:08:33.637 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00022115899999999999 2024-09-21T22:08:33.637 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.637 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.637 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:33.637 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.412081+0000", 2024-09-21T22:08:33.637 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.5654e-05 2024-09-21T22:08:33.638 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.638 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.638 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:33.638 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.465069+0000", 2024-09-21T22:08:33.638 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.052987570999999997 2024-09-21T22:08:33.638 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:33.638 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:33.638 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:33.638 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.638 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.638 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.3 55/54 [PushOp(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head, version: 52'3, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head@52'3, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 4194304, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:08:33.638 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:12.468339+0000", 2024-09-21T22:08:33.638 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 81.145260047999997, 2024-09-21T22:08:33.639 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.064010024999999998, 2024-09-21T22:08:33.639 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:33.639 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:33.639 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:08:33.639 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:33.639 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.639 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:33.639 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.468339+0000", 2024-09-21T22:08:33.639 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:33.639 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.639 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.639 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:33.639 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.468339+0000", 2024-09-21T22:08:33.639 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:33.639 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.640 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.640 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:33.640 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.468340+0000", 2024-09-21T22:08:33.640 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4330000000000001e-06 2024-09-21T22:08:33.640 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.640 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.640 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:33.640 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.469484+0000", 2024-09-21T22:08:33.640 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.001143535 2024-09-21T22:08:33.640 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.640 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.640 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:33.640 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.469533+0000", 2024-09-21T22:08:33.640 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8980999999999997e-05 2024-09-21T22:08:33.641 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.641 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.641 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:33.641 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.469581+0000", 2024-09-21T22:08:33.641 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.8551999999999999e-05 2024-09-21T22:08:33.641 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.641 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.641 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:33.641 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.469804+0000", 2024-09-21T22:08:33.641 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00022287699999999999 2024-09-21T22:08:33.641 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.641 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.641 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:33.641 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.469829+0000", 2024-09-21T22:08:33.642 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4766e-05 2024-09-21T22:08:33.642 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.642 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.642 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:33.642 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:12.532349+0000", 2024-09-21T22:08:33.642 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.062519880999999999 2024-09-21T22:08:33.642 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:33.642 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:33.642 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:33.642 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.642 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.642 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4840.0:670 2.3 2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:08:33.642 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:24.466980+0000", 2024-09-21T22:08:33.642 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 69.146618774000004, 2024-09-21T22:08:33.642 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.050913930000000003, 2024-09-21T22:08:33.643 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:33.643 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:33.643 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:33.643 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-21T22:08:33.643 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4840", 2024-09-21T22:08:33.643 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:08:33.643 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 670 2024-09-21T22:08:33.643 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.643 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:33.643 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.643 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:33.643 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.466980+0000", 2024-09-21T22:08:33.643 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:33.643 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.644 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.644 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:33.644 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.466980+0000", 2024-09-21T22:08:33.644 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:33.644 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.644 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.644 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:33.644 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.466981+0000", 2024-09-21T22:08:33.644 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0839999999999999e-06 2024-09-21T22:08:33.644 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.644 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.644 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:33.644 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.467815+0000", 2024-09-21T22:08:33.644 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00083330399999999999 2024-09-21T22:08:33.644 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.645 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.645 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:33.645 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.467832+0000", 2024-09-21T22:08:33.645 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7601e-05 2024-09-21T22:08:33.645 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.645 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.645 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:33.645 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.467840+0000", 2024-09-21T22:08:33.645 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.2220000000000002e-06 2024-09-21T22:08:33.645 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.645 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.645 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:33.645 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.467974+0000", 2024-09-21T22:08:33.645 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000133411 2024-09-21T22:08:33.646 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.646 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.646 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:33.646 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.468223+0000", 2024-09-21T22:08:33.646 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00024906400000000001 2024-09-21T22:08:33.646 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.646 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.646 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 2", 2024-09-21T22:08:33.646 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.468616+0000", 2024-09-21T22:08:33.646 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000393433 2024-09-21T22:08:33.646 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.646 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.646 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:33.646 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.517016+0000", 2024-09-21T22:08:33.646 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.048399708 2024-09-21T22:08:33.647 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.647 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.647 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-21T22:08:33.647 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.517693+0000", 2024-09-21T22:08:33.647 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00067681000000000002 2024-09-21T22:08:33.647 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.647 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.647 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:33.647 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.517789+0000", 2024-09-21T22:08:33.647 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.5778000000000004e-05 2024-09-21T22:08:33.647 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.647 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.647 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:33.647 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:24.517894+0000", 2024-09-21T22:08:33.648 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000105515 2024-09-21T22:08:33.648 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:33.648 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:33.648 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:33.648 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.648 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.648 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_op(client.4840.0:723 2.1 2:86869b00:::rbd_data.126483d75f2d.000000000000000e:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:08:33.648 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.860543+0000", 2024-09-21T22:08:33.648 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 67.753056326000006, 2024-09-21T22:08:33.648 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.062354957000000003, 2024-09-21T22:08:33.648 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:33.648 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:33.648 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:33.648 INFO:teuthology.orchestra.run.smithi039.stdout: "client_info": { 2024-09-21T22:08:33.649 INFO:teuthology.orchestra.run.smithi039.stdout: "client": "client.4840", 2024-09-21T22:08:33.649 INFO:teuthology.orchestra.run.smithi039.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:08:33.649 INFO:teuthology.orchestra.run.smithi039.stdout: "tid": 723 2024-09-21T22:08:33.649 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.649 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:33.649 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.649 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:33.649 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.860543+0000", 2024-09-21T22:08:33.649 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:33.649 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.649 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.649 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:33.649 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.860543+0000", 2024-09-21T22:08:33.649 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:33.650 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.650 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.650 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:33.650 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.860544+0000", 2024-09-21T22:08:33.650 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.1119999999999999e-06 2024-09-21T22:08:33.650 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.650 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.650 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:33.650 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.861400+0000", 2024-09-21T22:08:33.650 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000856393 2024-09-21T22:08:33.650 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.650 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.650 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:33.650 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.861419+0000", 2024-09-21T22:08:33.651 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8992e-05 2024-09-21T22:08:33.651 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.651 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.651 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:33.651 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.861428+0000", 2024-09-21T22:08:33.651 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.551e-06 2024-09-21T22:08:33.651 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.651 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.651 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:33.651 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.861555+0000", 2024-09-21T22:08:33.651 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000127286 2024-09-21T22:08:33.651 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.651 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.652 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:33.652 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.861816+0000", 2024-09-21T22:08:33.652 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00026100899999999998 2024-09-21T22:08:33.652 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.652 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.652 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "waiting for subops from 0", 2024-09-21T22:08:33.652 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.862189+0000", 2024-09-21T22:08:33.652 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00037261899999999998 2024-09-21T22:08:33.652 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.652 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.652 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:33.652 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.921514+0000", 2024-09-21T22:08:33.652 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.059325210000000003 2024-09-21T22:08:33.652 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.653 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.653 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "op_commit", 2024-09-21T22:08:33.653 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.922721+0000", 2024-09-21T22:08:33.653 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0012066819999999999 2024-09-21T22:08:33.653 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.653 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.653 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:33.653 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.922797+0000", 2024-09-21T22:08:33.653 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.6162999999999996e-05 2024-09-21T22:08:33.653 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.653 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.653 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:33.653 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.922898+0000", 2024-09-21T22:08:33.653 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00010094000000000001 2024-09-21T22:08:33.654 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:33.654 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:33.654 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:33.654 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.654 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.654 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:727 2.0 e63/16 2:08b4f96a:::rbd_data.126483d75f2d.000000000000000f:head v 63'35, mlcod=63'33)", 2024-09-21T22:08:33.654 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:25.998090+0000", 2024-09-21T22:08:33.654 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 67.615509407999994, 2024-09-21T22:08:33.654 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.052205704999999998, 2024-09-21T22:08:33.654 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:33.654 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:33.654 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:33.654 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:33.654 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.655 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:33.655 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.998090+0000", 2024-09-21T22:08:33.655 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:33.655 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.655 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.655 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:33.655 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.998090+0000", 2024-09-21T22:08:33.655 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:33.655 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.655 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.655 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:33.655 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.998091+0000", 2024-09-21T22:08:33.655 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6929999999999999e-06 2024-09-21T22:08:33.655 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.656 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.656 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:33.656 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.998940+0000", 2024-09-21T22:08:33.656 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00084875200000000004 2024-09-21T22:08:33.656 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.656 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.656 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:33.656 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.998970+0000", 2024-09-21T22:08:33.656 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9709999999999998e-05 2024-09-21T22:08:33.656 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.656 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.656 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:33.656 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.998984+0000", 2024-09-21T22:08:33.656 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3739e-05 2024-09-21T22:08:33.657 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.657 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.657 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:33.657 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.999097+0000", 2024-09-21T22:08:33.657 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000113484 2024-09-21T22:08:33.657 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.657 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.657 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:33.657 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:25.999135+0000", 2024-09-21T22:08:33.657 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.8405999999999998e-05 2024-09-21T22:08:33.657 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.657 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.657 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:33.657 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.050245+0000", 2024-09-21T22:08:33.657 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.051109935000000002 2024-09-21T22:08:33.658 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.658 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.658 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:33.658 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.050295+0000", 2024-09-21T22:08:33.658 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.9985999999999999e-05 2024-09-21T22:08:33.658 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:33.658 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:33.658 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:33.658 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.658 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.658 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.4840.0:731 2.0 e63/16 2:08b4f96a:::rbd_data.126483d75f2d.000000000000000f:head v 63'39, mlcod=63'37)", 2024-09-21T22:08:33.658 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:26.135946+0000", 2024-09-21T22:08:33.659 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 67.477653278999995, 2024-09-21T22:08:33.659 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.054290359000000003, 2024-09-21T22:08:33.659 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:33.659 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:33.659 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:33.659 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:33.659 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.659 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:33.659 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.135946+0000", 2024-09-21T22:08:33.659 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:33.659 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.659 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.659 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:33.659 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.135946+0000", 2024-09-21T22:08:33.660 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:33.660 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.660 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.660 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:33.660 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.135948+0000", 2024-09-21T22:08:33.660 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7999999999999999e-06 2024-09-21T22:08:33.660 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.660 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.660 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:33.660 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.136730+0000", 2024-09-21T22:08:33.660 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00078271899999999997 2024-09-21T22:08:33.660 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.660 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.660 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:33.660 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.136760+0000", 2024-09-21T22:08:33.661 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9876e-05 2024-09-21T22:08:33.661 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.661 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.661 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:33.661 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.136774+0000", 2024-09-21T22:08:33.661 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3478999999999999e-05 2024-09-21T22:08:33.661 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.661 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.661 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:33.661 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.136944+0000", 2024-09-21T22:08:33.661 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000170626 2024-09-21T22:08:33.661 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.661 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.662 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:33.662 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.136985+0000", 2024-09-21T22:08:33.662 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.0377999999999999e-05 2024-09-21T22:08:33.662 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.662 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.662 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:33.662 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.190137+0000", 2024-09-21T22:08:33.662 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.053152438000000003 2024-09-21T22:08:33.662 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.662 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.662 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:33.662 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:26.190236+0000", 2024-09-21T22:08:33.662 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 9.9042999999999993e-05 2024-09-21T22:08:33.662 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:33.663 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:33.663 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:33.663 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.663 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.663 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:125 2.3 e71/66 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head v 71'31, mlcod=71'29)", 2024-09-21T22:08:33.663 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.066855+0000", 2024-09-21T22:08:33.663 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 55.546743843999998, 2024-09-21T22:08:33.663 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.059856059000000003, 2024-09-21T22:08:33.663 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:33.663 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:33.663 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:33.663 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:33.663 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.664 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:33.664 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.066855+0000", 2024-09-21T22:08:33.664 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:33.664 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.664 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.664 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:33.664 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.066855+0000", 2024-09-21T22:08:33.664 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:33.664 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.664 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.664 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:33.664 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.066857+0000", 2024-09-21T22:08:33.664 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.807e-06 2024-09-21T22:08:33.664 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.664 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.665 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:33.665 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.068365+0000", 2024-09-21T22:08:33.665 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0015077910000000001 2024-09-21T22:08:33.665 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.665 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.665 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:33.665 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.068395+0000", 2024-09-21T22:08:33.665 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.0326000000000001e-05 2024-09-21T22:08:33.665 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.665 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.665 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:33.665 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.068415+0000", 2024-09-21T22:08:33.665 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9871999999999999e-05 2024-09-21T22:08:33.665 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.666 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.666 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:33.666 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.068580+0000", 2024-09-21T22:08:33.666 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.000165317 2024-09-21T22:08:33.666 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.666 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.666 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:33.666 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.068618+0000", 2024-09-21T22:08:33.666 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.7753000000000003e-05 2024-09-21T22:08:33.666 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.666 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.666 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:33.666 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.126623+0000", 2024-09-21T22:08:33.667 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.058004642000000002 2024-09-21T22:08:33.669 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.669 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.670 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:33.670 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.126711+0000", 2024-09-21T22:08:33.670 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.8551e-05 2024-09-21T22:08:33.670 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:33.670 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:33.670 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:33.670 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.670 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.670 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:127 2.3 e71/66 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head v 71'33, mlcod=71'31)", 2024-09-21T22:08:33.670 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.140864+0000", 2024-09-21T22:08:33.670 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 55.472734815999999, 2024-09-21T22:08:33.670 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.052512380999999997, 2024-09-21T22:08:33.670 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:33.671 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:33.671 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:33.671 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:33.671 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.671 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:33.671 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.140864+0000", 2024-09-21T22:08:33.671 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:33.671 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.671 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.671 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:33.671 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.140864+0000", 2024-09-21T22:08:33.671 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:33.671 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.672 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.672 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:33.672 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.140866+0000", 2024-09-21T22:08:33.672 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.694e-06 2024-09-21T22:08:33.672 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.672 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.672 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:33.672 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.141712+0000", 2024-09-21T22:08:33.672 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00084581499999999998 2024-09-21T22:08:33.672 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.672 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.672 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:33.672 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.141742+0000", 2024-09-21T22:08:33.673 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9865999999999998e-05 2024-09-21T22:08:33.673 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.673 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.673 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:33.673 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.141761+0000", 2024-09-21T22:08:33.673 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9473000000000001e-05 2024-09-21T22:08:33.673 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.673 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.673 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:33.673 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.141919+0000", 2024-09-21T22:08:33.673 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00015758300000000001 2024-09-21T22:08:33.673 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.673 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.673 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:33.673 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.141955+0000", 2024-09-21T22:08:33.674 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.6613000000000003e-05 2024-09-21T22:08:33.674 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.674 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.674 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:33.674 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.193333+0000", 2024-09-21T22:08:33.674 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.051377260000000001 2024-09-21T22:08:33.674 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.674 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.674 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:33.674 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.193377+0000", 2024-09-21T22:08:33.674 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.4076999999999998e-05 2024-09-21T22:08:33.674 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:33.674 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:33.675 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:33.675 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.675 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.675 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:129 2.3 e71/66 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head v 71'35, mlcod=71'33)", 2024-09-21T22:08:33.675 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.209458+0000", 2024-09-21T22:08:33.675 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 55.404141066999998, 2024-09-21T22:08:33.675 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.056595578000000001, 2024-09-21T22:08:33.675 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:33.675 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:33.675 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:33.675 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:33.676 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.676 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:33.676 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.209458+0000", 2024-09-21T22:08:33.676 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:33.676 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.676 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.676 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:33.676 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.209458+0000", 2024-09-21T22:08:33.676 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:33.676 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.676 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.676 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:33.676 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.209460+0000", 2024-09-21T22:08:33.676 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7600000000000001e-06 2024-09-21T22:08:33.677 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.677 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.677 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:33.677 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.210317+0000", 2024-09-21T22:08:33.677 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00085771700000000003 2024-09-21T22:08:33.677 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.677 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.677 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:33.677 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.210346+0000", 2024-09-21T22:08:33.677 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8334e-05 2024-09-21T22:08:33.677 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.677 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.677 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:33.677 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.210365+0000", 2024-09-21T22:08:33.677 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9439000000000001e-05 2024-09-21T22:08:33.678 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.678 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.678 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:33.678 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.210544+0000", 2024-09-21T22:08:33.678 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00017845599999999999 2024-09-21T22:08:33.678 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.678 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.678 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:33.678 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.210584+0000", 2024-09-21T22:08:33.678 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.0423000000000002e-05 2024-09-21T22:08:33.678 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.678 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.678 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:33.679 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.265970+0000", 2024-09-21T22:08:33.679 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.055385765000000003 2024-09-21T22:08:33.679 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.679 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.679 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:33.679 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.266054+0000", 2024-09-21T22:08:33.679 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 8.3683999999999994e-05 2024-09-21T22:08:33.679 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:33.679 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:33.679 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:33.679 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.679 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.679 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "osd_repop(client.5008.0:131 2.3 e71/66 2:deb19dad:::rbd_data.126483d75f2d.000000000000000a:head v 71'37, mlcod=71'35)", 2024-09-21T22:08:33.680 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:07:38.279550+0000", 2024-09-21T22:08:33.680 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 55.334049229000001, 2024-09-21T22:08:33.680 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.052290419999999997, 2024-09-21T22:08:33.680 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:33.680 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:33.680 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:33.680 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:33.680 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.680 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:33.680 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.279550+0000", 2024-09-21T22:08:33.680 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:33.680 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.680 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.680 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:33.680 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.279550+0000", 2024-09-21T22:08:33.681 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:33.681 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.681 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.681 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:33.681 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.279552+0000", 2024-09-21T22:08:33.681 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9439999999999999e-06 2024-09-21T22:08:33.681 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.681 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.681 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:33.681 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.280296+0000", 2024-09-21T22:08:33.681 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00074417000000000005 2024-09-21T22:08:33.681 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.681 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.682 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:33.682 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.280324+0000", 2024-09-21T22:08:33.682 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.8045999999999999e-05 2024-09-21T22:08:33.682 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.682 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.682 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:33.682 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.280345+0000", 2024-09-21T22:08:33.682 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.0849999999999999e-05 2024-09-21T22:08:33.682 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.682 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.682 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:33.682 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.280497+0000", 2024-09-21T22:08:33.682 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00015252500000000001 2024-09-21T22:08:33.682 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.683 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.683 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:33.683 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.280531+0000", 2024-09-21T22:08:33.683 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.4034e-05 2024-09-21T22:08:33.683 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.683 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.683 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "commit_sent", 2024-09-21T22:08:33.683 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.331769+0000", 2024-09-21T22:08:33.683 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.051237999999999999 2024-09-21T22:08:33.683 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.683 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.683 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:33.683 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:07:38.331840+0000", 2024-09-21T22:08:33.683 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 7.0851000000000003e-05 2024-09-21T22:08:33.683 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:33.684 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:33.684 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:33.684 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.684 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.684 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.6 98/97 [PushOp(2:7e2189ff:ns::rbd_data.127f79bc213d.0000000000000001:head, version: 57'4, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:7e2189ff:ns::rbd_data.127f79bc213d.0000000000000001:head@57'4, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T22:08:33.684 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:08:28.006861+0000", 2024-09-21T22:08:33.684 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 5.6067377480000005, 2024-09-21T22:08:33.684 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.062159881, 2024-09-21T22:08:33.684 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:33.684 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:33.684 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:08:33.684 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:33.684 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.684 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:33.685 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:08:28.006861+0000", 2024-09-21T22:08:33.685 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:33.685 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.685 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.685 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:33.685 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:08:28.006861+0000", 2024-09-21T22:08:33.685 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:33.685 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.685 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.685 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:33.685 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:08:28.006863+0000", 2024-09-21T22:08:33.685 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.35e-06 2024-09-21T22:08:33.685 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.686 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.686 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:33.686 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:08:28.008478+0000", 2024-09-21T22:08:33.686 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0016154299999999999 2024-09-21T22:08:33.686 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.686 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.686 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:33.686 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:08:28.008524+0000", 2024-09-21T22:08:33.686 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.5707000000000001e-05 2024-09-21T22:08:33.686 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.686 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.686 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:33.686 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:08:28.008556+0000", 2024-09-21T22:08:33.686 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.2162000000000001e-05 2024-09-21T22:08:33.687 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.687 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.687 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:33.687 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:08:28.008788+0000", 2024-09-21T22:08:33.687 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00023188700000000001 2024-09-21T22:08:33.687 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.687 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.687 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:33.687 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:08:28.008832+0000", 2024-09-21T22:08:33.687 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.4487000000000003e-05 2024-09-21T22:08:33.687 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.687 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.687 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:33.687 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:08:28.069021+0000", 2024-09-21T22:08:33.688 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.060188857999999998 2024-09-21T22:08:33.688 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:33.688 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:33.688 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:33.688 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.688 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.688 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.0 98/97 [PushOp(2:06303f52:::rbd_data.126483d75f2d.0000000000000005:6, version: 88'78, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 1, recovery_info: ObjectRecoveryInfo(2:06303f52:::rbd_data.126483d75f2d.0000000000000005:6@88'78, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 6=[]:{6: [6, 5, 3]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T22:08:33.688 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:08:28.696433+0000", 2024-09-21T22:08:33.688 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 4.9171655770000005, 2024-09-21T22:08:33.688 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.050951600999999999, 2024-09-21T22:08:33.688 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:33.688 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:33.688 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:08:33.688 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:33.688 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.689 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:33.689 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:08:28.696433+0000", 2024-09-21T22:08:33.689 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:33.689 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.689 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.689 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:33.689 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:08:28.696433+0000", 2024-09-21T22:08:33.689 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:33.689 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.689 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.689 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:33.689 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:08:28.696435+0000", 2024-09-21T22:08:33.689 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.4929999999999999e-06 2024-09-21T22:08:33.689 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.690 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.690 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:33.690 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:08:28.697414+0000", 2024-09-21T22:08:33.690 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00097895399999999998 2024-09-21T22:08:33.690 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.690 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.690 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:33.690 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:08:28.697461+0000", 2024-09-21T22:08:33.690 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.7250000000000003e-05 2024-09-21T22:08:33.690 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.690 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.690 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:33.690 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:08:28.697491+0000", 2024-09-21T22:08:33.690 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.9969000000000001e-05 2024-09-21T22:08:33.691 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.691 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.691 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:33.691 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:08:28.697669+0000", 2024-09-21T22:08:33.691 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00017747100000000001 2024-09-21T22:08:33.691 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.691 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.691 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:33.691 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:08:28.697694+0000", 2024-09-21T22:08:33.691 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5106e-05 2024-09-21T22:08:33.691 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.691 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.691 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:33.691 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:08:28.747385+0000", 2024-09-21T22:08:33.692 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.049691357999999998 2024-09-21T22:08:33.692 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:33.692 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:33.692 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:33.692 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.692 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.692 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.1 99/97 [PushOp(2:837ddd83:::rbd_data.126483d75f2d.000000000000000d:6, version: 88'39, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 1, recovery_info: ObjectRecoveryInfo(2:837ddd83:::rbd_data.126483d75f2d.000000000000000d:6@88'39, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 6=[]:{6: [6, 5, 3]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T22:08:33.692 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:08:29.044328+0000", 2024-09-21T22:08:33.692 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 4.5692712259999997, 2024-09-21T22:08:33.692 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.062851167999999999, 2024-09-21T22:08:33.692 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:33.692 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:33.692 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:08:33.692 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:33.692 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.693 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:33.693 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:08:29.044328+0000", 2024-09-21T22:08:33.693 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:33.693 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.693 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.693 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:33.693 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:08:29.044328+0000", 2024-09-21T22:08:33.693 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:33.693 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.693 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.693 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:33.693 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:08:29.044330+0000", 2024-09-21T22:08:33.693 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.8700000000000001e-06 2024-09-21T22:08:33.693 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.694 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.694 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:33.694 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:08:29.045570+0000", 2024-09-21T22:08:33.694 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.0012407589999999999 2024-09-21T22:08:33.694 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.694 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.694 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:33.694 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:08:29.045616+0000", 2024-09-21T22:08:33.694 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 4.5856999999999998e-05 2024-09-21T22:08:33.694 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.694 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.694 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:33.694 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:08:29.045651+0000", 2024-09-21T22:08:33.695 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.4233000000000002e-05 2024-09-21T22:08:33.695 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.695 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.695 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:33.695 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:08:29.077051+0000", 2024-09-21T22:08:33.695 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.031400477000000003 2024-09-21T22:08:33.695 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.695 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.695 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:33.695 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:08:29.077071+0000", 2024-09-21T22:08:33.695 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.0251999999999999e-05 2024-09-21T22:08:33.695 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.695 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.695 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:33.696 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:08:29.107179+0000", 2024-09-21T22:08:33.696 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.030107720000000001 2024-09-21T22:08:33.696 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:33.696 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:33.696 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:33.696 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.696 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.696 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.1 99/97 [PushOp(2:922055e2:ns::rbd_data.127f79bc213d.0000000000000009:head, version: 57'4, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:922055e2:ns::rbd_data.127f79bc213d.0000000000000009:head@57'4, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 4194304, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:08:33.696 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:08:29.080177+0000", 2024-09-21T22:08:33.696 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 4.5334215789999996, 2024-09-21T22:08:33.696 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.053168184, 2024-09-21T22:08:33.696 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:33.696 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:33.696 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:08:33.697 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:33.697 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.697 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:33.697 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:08:29.080177+0000", 2024-09-21T22:08:33.697 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:33.697 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.697 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.697 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:33.697 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:08:29.080177+0000", 2024-09-21T22:08:33.697 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:33.697 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.697 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.697 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:33.697 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:08:29.080179+0000", 2024-09-21T22:08:33.697 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.6169999999999999e-06 2024-09-21T22:08:33.698 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.698 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.698 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:33.698 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:08:29.080517+0000", 2024-09-21T22:08:33.698 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00033829999999999998 2024-09-21T22:08:33.698 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.698 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.698 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:33.698 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:08:29.080550+0000", 2024-09-21T22:08:33.698 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 3.2486e-05 2024-09-21T22:08:33.698 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.698 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.698 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:33.698 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:08:29.080576+0000", 2024-09-21T22:08:33.699 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.5871999999999999e-05 2024-09-21T22:08:33.699 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.699 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.699 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:33.699 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:08:29.107264+0000", 2024-09-21T22:08:33.699 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.026688686 2024-09-21T22:08:33.699 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.699 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.699 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:33.699 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:08:29.107276+0000", 2024-09-21T22:08:33.699 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2011000000000001e-05 2024-09-21T22:08:33.699 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.699 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.699 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:33.700 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:08:29.133346+0000", 2024-09-21T22:08:33.700 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.026069212000000001 2024-09-21T22:08:33.700 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:33.700 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:33.700 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:33.700 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.700 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.700 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.1 99/97 [PushOp(2:86869b00:::rbd_data.126483d75f2d.000000000000000e:6, version: 88'41, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 1, recovery_info: ObjectRecoveryInfo(2:86869b00:::rbd_data.126483d75f2d.000000000000000e:6@88'41, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 6=[]:{6: [6, 5, 3]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T22:08:33.700 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:08:29.172206+0000", 2024-09-21T22:08:33.700 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 4.4413928489999996, 2024-09-21T22:08:33.700 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.070834772000000004, 2024-09-21T22:08:33.700 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:33.700 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:33.700 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:08:33.700 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:33.701 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.701 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:33.701 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:08:29.172206+0000", 2024-09-21T22:08:33.701 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:33.701 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.701 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.701 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:33.701 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:08:29.172206+0000", 2024-09-21T22:08:33.701 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:33.701 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.701 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.701 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:33.701 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:08:29.172207+0000", 2024-09-21T22:08:33.701 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.2389999999999999e-06 2024-09-21T22:08:33.702 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.702 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.702 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:33.702 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:08:29.173151+0000", 2024-09-21T22:08:33.702 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00094336599999999995 2024-09-21T22:08:33.702 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.702 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.702 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:33.702 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:08:29.173175+0000", 2024-09-21T22:08:33.702 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.4439000000000001e-05 2024-09-21T22:08:33.702 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.702 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.702 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:33.702 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:08:29.173198+0000", 2024-09-21T22:08:33.702 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 2.2535999999999999e-05 2024-09-21T22:08:33.703 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.703 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.703 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:33.703 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:08:29.214584+0000", 2024-09-21T22:08:33.703 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.041385845999999997 2024-09-21T22:08:33.703 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.703 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.703 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:33.703 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:08:29.214603+0000", 2024-09-21T22:08:33.703 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.9511999999999999e-05 2024-09-21T22:08:33.703 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.703 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.703 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:33.703 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:08:29.243041+0000", 2024-09-21T22:08:33.704 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.028437833999999999 2024-09-21T22:08:33.704 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:33.704 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:33.704 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:33.704 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.704 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.704 INFO:teuthology.orchestra.run.smithi039.stdout: "description": "MOSDPGPush(2.1 99/97 [PushOp(2:86869b00:::rbd_data.126483d75f2d.000000000000000e:6, version: 88'41, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:86869b00:::rbd_data.126483d75f2d.000000000000000e:6@88'41, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 6=[]:{6: [6, 5, 3]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T22:08:33.704 INFO:teuthology.orchestra.run.smithi039.stdout: "initiated_at": "2024-09-21T22:08:29.246299+0000", 2024-09-21T22:08:33.704 INFO:teuthology.orchestra.run.smithi039.stdout: "age": 4.3672995830000003, 2024-09-21T22:08:33.704 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.063768796000000003, 2024-09-21T22:08:33.704 INFO:teuthology.orchestra.run.smithi039.stdout: "continuous": false, 2024-09-21T22:08:33.704 INFO:teuthology.orchestra.run.smithi039.stdout: "type_data": { 2024-09-21T22:08:33.704 INFO:teuthology.orchestra.run.smithi039.stdout: "flag_point": "started", 2024-09-21T22:08:33.704 INFO:teuthology.orchestra.run.smithi039.stdout: "events": [ 2024-09-21T22:08:33.704 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.705 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "initiated", 2024-09-21T22:08:33.705 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:08:29.246299+0000", 2024-09-21T22:08:33.705 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:33.705 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.705 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.705 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "header_read", 2024-09-21T22:08:33.705 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:08:29.246299+0000", 2024-09-21T22:08:33.705 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0 2024-09-21T22:08:33.705 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.705 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.705 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "throttled", 2024-09-21T22:08:33.705 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:08:29.246300+0000", 2024-09-21T22:08:33.705 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 5.6499999999999999e-07 2024-09-21T22:08:33.705 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.705 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.706 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "all_read", 2024-09-21T22:08:33.706 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:08:29.247228+0000", 2024-09-21T22:08:33.706 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.00092752999999999998 2024-09-21T22:08:33.706 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.706 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.706 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "dispatched", 2024-09-21T22:08:33.706 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:08:29.247245+0000", 2024-09-21T22:08:33.706 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.7155000000000001e-05 2024-09-21T22:08:33.706 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.706 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.706 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "queued_for_pg", 2024-09-21T22:08:33.706 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:08:29.247255+0000", 2024-09-21T22:08:33.706 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.0285000000000001e-05 2024-09-21T22:08:33.706 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.707 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.707 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "reached_pg", 2024-09-21T22:08:33.707 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:08:29.270647+0000", 2024-09-21T22:08:33.707 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.023391553999999998 2024-09-21T22:08:33.707 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.707 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.707 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "started", 2024-09-21T22:08:33.707 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:08:29.270660+0000", 2024-09-21T22:08:33.707 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 1.3952e-05 2024-09-21T22:08:33.707 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-09-21T22:08:33.707 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-09-21T22:08:33.707 INFO:teuthology.orchestra.run.smithi039.stdout: "event": "done", 2024-09-21T22:08:33.707 INFO:teuthology.orchestra.run.smithi039.stdout: "time": "2024-09-21T22:08:29.310068+0000", 2024-09-21T22:08:33.707 INFO:teuthology.orchestra.run.smithi039.stdout: "duration": 0.039407755000000003 2024-09-21T22:08:33.707 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:33.708 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:33.708 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:33.708 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-09-21T22:08:33.708 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-09-21T22:08:33.708 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:08:33.709 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T22:08:33.710 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:08:33.705+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:08:33.810 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:08:33.807+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:08:33.845 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:08:33.845 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:08:33.845 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-09-21T22:08:33.845 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:08:33.859 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T22:08:33.874 INFO:tasks.workunit.client.0.smithi157.stderr:+ grep 'Watchers: none' 2024-09-21T22:08:33.874 INFO:tasks.workunit.client.0.smithi157.stderr:+ rbd -p rbd --namespace ns status testrbdnbd21553 2024-09-21T22:08:33.941 INFO:tasks.workunit.client.0.smithi157.stdout:Watchers: none 2024-09-21T22:08:33.942 INFO:tasks.workunit.client.0.smithi157.stderr:+ break 2024-09-21T22:08:33.942 INFO:tasks.workunit.client.0.smithi157.stderr:+ rbd -p rbd --namespace ns snap purge testrbdnbd21553 2024-09-21T22:08:33.946 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:08:33.943+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:08:34.012 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:08:34.009+0000 7fe38af2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 34631) UID: 0 2024-09-21T22:08:34.025 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:08:34.025 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:08:34.025 INFO:teuthology.orchestra.run.smithi064.stdout: "complaint_time": 30, 2024-09-21T22:08:34.025 INFO:teuthology.orchestra.run.smithi064.stdout: "num_blocked_ops": 0 2024-09-21T22:08:34.025 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:08:34.040 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T22:08:34.060 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-21T22:08:34.063 INFO:tasks.workunit.client.0.smithi157.stderr: Removing all snapshots: 100% complete... Removing all snapshots: 100% complete...done. 2024-09-21T22:08:34.079 INFO:tasks.workunit.client.0.smithi157.stderr:+ rbd -p rbd --namespace ns remove testrbdnbd21553 2024-09-21T22:08:34.130 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:08:34.127+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:08:34.213 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:08:34.209+0000 7fe38af2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 34631) UID: 0 2024-09-21T22:08:34.217 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:08:34.217 INFO:teuthology.orchestra.run.smithi064.stdout: "size": 20, 2024-09-21T22:08:34.218 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 600, 2024-09-21T22:08:34.218 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [ 2024-09-21T22:08:34.218 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.218 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4195.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e14)", 2024-09-21T22:08:34.218 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:06:02.036355+0000", 2024-09-21T22:08:34.218 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 152.178328725, 2024-09-21T22:08:34.218 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.062441934999999997, 2024-09-21T22:08:34.218 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:34.218 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:34.218 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:34.218 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:34.218 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4195", 2024-09-21T22:08:34.219 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.64:0/728381110", 2024-09-21T22:08:34.219 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 1 2024-09-21T22:08:34.219 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.219 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:34.219 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.219 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:34.219 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036355+0000", 2024-09-21T22:08:34.219 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:34.219 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.219 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.219 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:34.219 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036355+0000", 2024-09-21T22:08:34.219 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:34.219 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.220 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.220 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:34.220 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036357+0000", 2024-09-21T22:08:34.220 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.446e-06 2024-09-21T22:08:34.220 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.220 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.220 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:34.220 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036373+0000", 2024-09-21T22:08:34.220 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5447e-05 2024-09-21T22:08:34.220 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.221 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.221 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:34.221 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036399+0000", 2024-09-21T22:08:34.221 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.6423000000000001e-05 2024-09-21T22:08:34.221 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.221 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.221 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:34.221 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036420+0000", 2024-09-21T22:08:34.221 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1129999999999999e-05 2024-09-21T22:08:34.221 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.221 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.221 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:34.221 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036557+0000", 2024-09-21T22:08:34.221 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013686899999999999 2024-09-21T22:08:34.222 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.222 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.222 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for peered", 2024-09-21T22:08:34.222 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.036585+0000", 2024-09-21T22:08:34.222 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8362999999999999e-05 2024-09-21T22:08:34.222 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.222 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.222 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:34.222 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.095068+0000", 2024-09-21T22:08:34.222 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058482972000000001 2024-09-21T22:08:34.222 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.222 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.222 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:34.223 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.095516+0000", 2024-09-21T22:08:34.223 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00044802999999999998 2024-09-21T22:08:34.223 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.223 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.223 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:08:34.223 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.095869+0000", 2024-09-21T22:08:34.223 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000352104 2024-09-21T22:08:34.223 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.223 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.223 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:34.223 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.097055+0000", 2024-09-21T22:08:34.223 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0011865599999999999 2024-09-21T22:08:34.223 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.224 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.224 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:34.224 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.098637+0000", 2024-09-21T22:08:34.224 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001581458 2024-09-21T22:08:34.224 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.224 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.224 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:34.224 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.098728+0000", 2024-09-21T22:08:34.224 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1043000000000002e-05 2024-09-21T22:08:34.224 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.224 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.224 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:34.224 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:06:02.098797+0000", 2024-09-21T22:08:34.225 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.9090000000000004e-05 2024-09-21T22:08:34.225 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:34.225 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:34.225 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:34.225 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.225 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.225 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4840.0:680 2.7 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:08:34.225 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.744506+0000", 2024-09-21T22:08:34.225 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 69.470177871000004, 2024-09-21T22:08:34.225 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.063865055000000004, 2024-09-21T22:08:34.225 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:34.225 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:34.225 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:34.225 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:34.226 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4840", 2024-09-21T22:08:34.226 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:08:34.226 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 680 2024-09-21T22:08:34.226 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.226 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:34.226 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.226 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:34.226 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.744506+0000", 2024-09-21T22:08:34.226 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:34.226 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.226 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.226 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:34.227 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.744506+0000", 2024-09-21T22:08:34.227 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:34.227 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.227 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.227 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:34.227 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.744508+0000", 2024-09-21T22:08:34.227 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.103e-06 2024-09-21T22:08:34.227 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.227 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.227 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:34.227 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.745371+0000", 2024-09-21T22:08:34.227 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00086322199999999997 2024-09-21T22:08:34.227 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.228 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.228 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:34.228 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.745401+0000", 2024-09-21T22:08:34.228 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9955999999999999e-05 2024-09-21T22:08:34.228 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.228 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.228 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:34.228 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.745416+0000", 2024-09-21T22:08:34.228 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4873e-05 2024-09-21T22:08:34.228 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.228 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.228 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:34.228 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.745509+0000", 2024-09-21T22:08:34.229 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3171000000000006e-05 2024-09-21T22:08:34.229 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.229 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.229 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:34.229 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.745632+0000", 2024-09-21T22:08:34.229 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012272699999999999 2024-09-21T22:08:34.229 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.229 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.229 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-09-21T22:08:34.229 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.745789+0000", 2024-09-21T22:08:34.229 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015781399999999999 2024-09-21T22:08:34.229 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.229 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.229 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:34.230 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.785494+0000", 2024-09-21T22:08:34.230 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.039704401 2024-09-21T22:08:34.230 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.230 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.230 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:34.230 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.808090+0000", 2024-09-21T22:08:34.230 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.022596207 2024-09-21T22:08:34.230 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.230 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.230 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:34.230 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.808246+0000", 2024-09-21T22:08:34.230 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015567200000000001 2024-09-21T22:08:34.230 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.230 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.231 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:34.231 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.808371+0000", 2024-09-21T22:08:34.231 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012490900000000001 2024-09-21T22:08:34.231 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:34.231 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:34.231 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:34.231 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.231 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.231 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4840.0:682 2.7 2:eceef27f:::rbd_data.126483d75f2d.0000000000000009:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:08:34.231 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:24.822415+0000", 2024-09-21T22:08:34.231 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 69.392268307999998, 2024-09-21T22:08:34.231 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.065044633000000004, 2024-09-21T22:08:34.231 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:34.231 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:34.232 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:34.232 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:34.232 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4840", 2024-09-21T22:08:34.232 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:08:34.232 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 682 2024-09-21T22:08:34.232 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.232 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:34.232 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.232 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:34.232 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.822415+0000", 2024-09-21T22:08:34.232 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:34.232 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.232 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.232 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:34.233 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.822415+0000", 2024-09-21T22:08:34.233 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:34.233 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.233 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.233 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:34.233 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.822417+0000", 2024-09-21T22:08:34.233 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.8029999999999999e-06 2024-09-21T22:08:34.233 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.233 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.233 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:34.233 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.823308+0000", 2024-09-21T22:08:34.233 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00089146500000000003 2024-09-21T22:08:34.233 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.233 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.234 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:34.234 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.823339+0000", 2024-09-21T22:08:34.234 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.0493000000000001e-05 2024-09-21T22:08:34.234 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.234 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.234 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:34.234 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.823354+0000", 2024-09-21T22:08:34.234 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5472999999999999e-05 2024-09-21T22:08:34.234 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.234 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.234 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:34.234 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.823476+0000", 2024-09-21T22:08:34.234 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000121209 2024-09-21T22:08:34.235 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.235 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.235 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:34.235 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.823650+0000", 2024-09-21T22:08:34.235 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017394599999999999 2024-09-21T22:08:34.235 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.235 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.235 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-09-21T22:08:34.235 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.823873+0000", 2024-09-21T22:08:34.235 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022314899999999999 2024-09-21T22:08:34.235 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.235 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.235 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:34.235 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.884396+0000", 2024-09-21T22:08:34.236 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.060523674999999999 2024-09-21T22:08:34.236 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.236 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.236 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:34.236 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.887238+0000", 2024-09-21T22:08:34.236 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0028420450000000001 2024-09-21T22:08:34.236 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.236 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.236 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:34.236 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.887369+0000", 2024-09-21T22:08:34.236 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013010699999999999 2024-09-21T22:08:34.236 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.236 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.236 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:34.237 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:24.887460+0000", 2024-09-21T22:08:34.237 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1267999999999994e-05 2024-09-21T22:08:34.237 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:34.237 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:34.237 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:34.237 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.237 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.237 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4840.0:692 2.5 2:b1d55633:::rbd_data.126483d75f2d.000000000000000b:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:08:34.237 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:25.091048+0000", 2024-09-21T22:08:34.237 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 69.123635773999993, 2024-09-21T22:08:34.237 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.060917092999999999, 2024-09-21T22:08:34.237 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:34.237 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:34.237 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:34.237 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:34.238 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4840", 2024-09-21T22:08:34.238 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:08:34.238 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 692 2024-09-21T22:08:34.238 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.238 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:34.238 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.238 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:34.238 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.091048+0000", 2024-09-21T22:08:34.238 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:34.238 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.238 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.238 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:34.238 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.091048+0000", 2024-09-21T22:08:34.239 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:34.239 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.239 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.239 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:34.239 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.091050+0000", 2024-09-21T22:08:34.239 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.03e-06 2024-09-21T22:08:34.239 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.239 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.239 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:34.239 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.091989+0000", 2024-09-21T22:08:34.239 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00093932100000000001 2024-09-21T22:08:34.239 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.239 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.239 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:34.240 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.092023+0000", 2024-09-21T22:08:34.240 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.4310000000000002e-05 2024-09-21T22:08:34.240 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.240 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.240 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:34.240 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.092040+0000", 2024-09-21T22:08:34.240 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6426000000000002e-05 2024-09-21T22:08:34.240 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.240 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.240 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:34.240 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.092175+0000", 2024-09-21T22:08:34.240 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013500299999999999 2024-09-21T22:08:34.240 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.240 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.240 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:34.241 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.092307+0000", 2024-09-21T22:08:34.241 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000132562 2024-09-21T22:08:34.241 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.241 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.241 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:08:34.241 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.092499+0000", 2024-09-21T22:08:34.241 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000191972 2024-09-21T22:08:34.241 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.241 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.241 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:34.241 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.151065+0000", 2024-09-21T22:08:34.241 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058565514999999999 2024-09-21T22:08:34.241 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.241 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.242 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:34.242 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.151844+0000", 2024-09-21T22:08:34.242 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00077930800000000004 2024-09-21T22:08:34.242 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.242 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.242 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:34.242 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.151916+0000", 2024-09-21T22:08:34.242 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1921999999999993e-05 2024-09-21T22:08:34.242 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.242 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.242 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:34.242 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.151965+0000", 2024-09-21T22:08:34.242 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8724000000000003e-05 2024-09-21T22:08:34.242 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:34.243 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:34.243 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:34.243 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.243 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.243 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4840.0:696 2.5 2:b1d55633:::rbd_data.126483d75f2d.000000000000000b:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:08:34.243 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:25.218641+0000", 2024-09-21T22:08:34.243 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 68.996042505999995, 2024-09-21T22:08:34.243 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058363669999999999, 2024-09-21T22:08:34.243 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:34.243 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:34.243 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:34.243 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:34.243 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4840", 2024-09-21T22:08:34.243 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:08:34.243 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 696 2024-09-21T22:08:34.244 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.244 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:34.244 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.244 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:34.244 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.218641+0000", 2024-09-21T22:08:34.244 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:34.244 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.244 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.244 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:34.244 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.218641+0000", 2024-09-21T22:08:34.244 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:34.244 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.244 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.244 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:34.244 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.218642+0000", 2024-09-21T22:08:34.245 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.266e-06 2024-09-21T22:08:34.245 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.245 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.245 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:34.245 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.219432+0000", 2024-09-21T22:08:34.245 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00078986300000000002 2024-09-21T22:08:34.245 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.245 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.245 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:34.245 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.219450+0000", 2024-09-21T22:08:34.245 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.7526000000000001e-05 2024-09-21T22:08:34.245 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.245 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.246 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:34.246 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.219459+0000", 2024-09-21T22:08:34.246 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3000000000000007e-06 2024-09-21T22:08:34.246 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.246 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.246 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:34.246 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.219655+0000", 2024-09-21T22:08:34.246 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019599099999999999 2024-09-21T22:08:34.246 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.246 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.246 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:34.246 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.219917+0000", 2024-09-21T22:08:34.246 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026198200000000001 2024-09-21T22:08:34.246 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.247 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.247 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 0", 2024-09-21T22:08:34.247 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.220198+0000", 2024-09-21T22:08:34.247 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00028154500000000002 2024-09-21T22:08:34.247 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.247 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.247 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:34.247 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.268856+0000", 2024-09-21T22:08:34.247 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.048657687999999998 2024-09-21T22:08:34.247 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.247 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.247 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:34.248 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.276903+0000", 2024-09-21T22:08:34.248 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0080472249999999999 2024-09-21T22:08:34.248 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.248 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.248 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:34.248 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.276958+0000", 2024-09-21T22:08:34.248 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4184999999999997e-05 2024-09-21T22:08:34.248 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.248 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.248 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:34.248 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:25.277005+0000", 2024-09-21T22:08:34.248 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.7098999999999997e-05 2024-09-21T22:08:34.248 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:34.248 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:34.249 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:34.249 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.249 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.249 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4840.0:729 2.0 2:08b4f96a:::rbd_data.126483d75f2d.000000000000000f:head [write 2228224~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:08:34.249 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:26.062012+0000", 2024-09-21T22:08:34.249 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 68.152671272999996, 2024-09-21T22:08:34.249 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.060656823999999998, 2024-09-21T22:08:34.249 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:34.249 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:34.249 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:34.249 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:34.249 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4840", 2024-09-21T22:08:34.250 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:08:34.250 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 729 2024-09-21T22:08:34.250 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.250 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:34.250 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.250 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:34.250 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.062012+0000", 2024-09-21T22:08:34.250 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:34.250 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.250 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.250 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:34.250 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.062012+0000", 2024-09-21T22:08:34.250 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:34.250 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.251 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.251 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:34.251 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.062013+0000", 2024-09-21T22:08:34.251 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.243e-06 2024-09-21T22:08:34.251 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.251 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.251 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:34.251 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.062827+0000", 2024-09-21T22:08:34.251 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00081374799999999999 2024-09-21T22:08:34.251 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.251 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.251 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:34.252 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.062845+0000", 2024-09-21T22:08:34.252 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.808e-05 2024-09-21T22:08:34.252 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.252 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.252 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:34.252 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.062854+0000", 2024-09-21T22:08:34.252 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1039999999999996e-06 2024-09-21T22:08:34.252 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.252 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.252 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:34.252 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.062970+0000", 2024-09-21T22:08:34.252 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011601 2024-09-21T22:08:34.252 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.252 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.253 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:34.253 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.063230+0000", 2024-09-21T22:08:34.253 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025918799999999999 2024-09-21T22:08:34.253 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.253 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.253 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-09-21T22:08:34.253 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.063641+0000", 2024-09-21T22:08:34.253 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00041152799999999998 2024-09-21T22:08:34.253 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.253 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.253 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:34.253 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.122015+0000", 2024-09-21T22:08:34.253 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.058374246999999997 2024-09-21T22:08:34.254 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.254 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.254 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:34.254 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.122451+0000", 2024-09-21T22:08:34.254 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000435771 2024-09-21T22:08:34.254 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.254 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.254 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:34.254 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.122576+0000", 2024-09-21T22:08:34.254 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012469099999999999 2024-09-21T22:08:34.254 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.254 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.254 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:34.255 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.122669+0000", 2024-09-21T22:08:34.255 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3214000000000006e-05 2024-09-21T22:08:34.255 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:34.255 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:34.255 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:34.255 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.255 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.255 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.4840.0:731 2.0 2:08b4f96a:::rbd_data.126483d75f2d.000000000000000f:head [write 3276800~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e63)", 2024-09-21T22:08:34.255 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:26.135603+0000", 2024-09-21T22:08:34.255 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 68.079080773000001, 2024-09-21T22:08:34.255 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.056838474999999999, 2024-09-21T22:08:34.255 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:34.255 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:34.256 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:34.256 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:34.256 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.4840", 2024-09-21T22:08:34.256 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/947995813", 2024-09-21T22:08:34.256 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 731 2024-09-21T22:08:34.256 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.256 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:34.256 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.256 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:34.256 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.135603+0000", 2024-09-21T22:08:34.256 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:34.256 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.256 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.256 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:34.257 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.135603+0000", 2024-09-21T22:08:34.257 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:34.257 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.257 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.257 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:34.257 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.135605+0000", 2024-09-21T22:08:34.257 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9700000000000002e-06 2024-09-21T22:08:34.257 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.257 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.257 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:34.257 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.136554+0000", 2024-09-21T22:08:34.257 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00094886700000000005 2024-09-21T22:08:34.257 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.257 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.258 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:34.258 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.136582+0000", 2024-09-21T22:08:34.258 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.8393000000000001e-05 2024-09-21T22:08:34.258 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.258 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.258 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:34.258 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.136596+0000", 2024-09-21T22:08:34.258 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.43e-05 2024-09-21T22:08:34.258 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.258 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.258 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:34.258 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.136691+0000", 2024-09-21T22:08:34.258 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.4843e-05 2024-09-21T22:08:34.258 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.259 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.259 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:34.259 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.136873+0000", 2024-09-21T22:08:34.259 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00018237099999999999 2024-09-21T22:08:34.259 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.259 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.259 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 1", 2024-09-21T22:08:34.259 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.137140+0000", 2024-09-21T22:08:34.259 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00026642700000000002 2024-09-21T22:08:34.259 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.259 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.259 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:34.259 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.180580+0000", 2024-09-21T22:08:34.259 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.043440078 2024-09-21T22:08:34.260 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.260 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.260 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:34.260 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.192175+0000", 2024-09-21T22:08:34.260 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011594959 2024-09-21T22:08:34.260 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.260 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.260 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:34.260 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.192353+0000", 2024-09-21T22:08:34.260 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017850200000000001 2024-09-21T22:08:34.260 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.260 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.260 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:34.261 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:26.192441+0000", 2024-09-21T22:08:34.261 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.7764999999999997e-05 2024-09-21T22:08:34.261 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:34.261 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:34.261 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:34.261 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.261 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.261 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5207.0:34 2.7 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head [write 262144~131072] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e78)", 2024-09-21T22:08:34.261 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:52.587708+0000", 2024-09-21T22:08:34.261 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 41.626975752, 2024-09-21T22:08:34.261 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.056690213000000003, 2024-09-21T22:08:34.261 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:34.261 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:34.261 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:34.262 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:34.262 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5207", 2024-09-21T22:08:34.262 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/357731919", 2024-09-21T22:08:34.262 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 34 2024-09-21T22:08:34.262 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.262 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:34.262 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.262 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:34.262 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.587708+0000", 2024-09-21T22:08:34.262 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:34.262 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.262 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.263 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:34.263 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.587708+0000", 2024-09-21T22:08:34.263 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:34.263 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.263 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.263 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:34.263 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.587709+0000", 2024-09-21T22:08:34.263 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.72e-06 2024-09-21T22:08:34.263 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.263 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.263 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:34.263 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588017+0000", 2024-09-21T22:08:34.263 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00030767499999999998 2024-09-21T22:08:34.264 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.264 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.264 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:34.264 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588037+0000", 2024-09-21T22:08:34.264 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.9865999999999999e-05 2024-09-21T22:08:34.264 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.264 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.264 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:34.264 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588048+0000", 2024-09-21T22:08:34.264 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.1384e-05 2024-09-21T22:08:34.264 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.264 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.264 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:34.265 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.614263+0000", 2024-09-21T22:08:34.265 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.026214232000000001 2024-09-21T22:08:34.265 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.265 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.265 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:34.265 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.614397+0000", 2024-09-21T22:08:34.265 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00013396299999999999 2024-09-21T22:08:34.265 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.265 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.265 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-09-21T22:08:34.265 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.614670+0000", 2024-09-21T22:08:34.265 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027295500000000002 2024-09-21T22:08:34.265 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.265 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.265 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:34.266 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.644046+0000", 2024-09-21T22:08:34.266 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.029376973000000001 2024-09-21T22:08:34.266 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.266 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.266 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:34.266 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.644208+0000", 2024-09-21T22:08:34.266 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016169900000000001 2024-09-21T22:08:34.266 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.266 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.266 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:34.266 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.644335+0000", 2024-09-21T22:08:34.266 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012656000000000001 2024-09-21T22:08:34.266 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.267 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.267 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:34.267 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.644398+0000", 2024-09-21T22:08:34.267 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.3186000000000001e-05 2024-09-21T22:08:34.267 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:34.267 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:34.267 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:34.267 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.267 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.267 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5207.0:35 2.7 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head [write 393216~131072] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e78)", 2024-09-21T22:08:34.267 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:52.588070+0000", 2024-09-21T22:08:34.267 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 41.626613853999999, 2024-09-21T22:08:34.267 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.076411232999999995, 2024-09-21T22:08:34.267 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:34.268 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:34.268 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:34.268 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:34.268 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5207", 2024-09-21T22:08:34.268 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/357731919", 2024-09-21T22:08:34.268 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 35 2024-09-21T22:08:34.268 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.268 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:34.268 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.268 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:34.268 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588070+0000", 2024-09-21T22:08:34.268 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:34.268 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.268 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.269 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:34.269 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588070+0000", 2024-09-21T22:08:34.269 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:34.269 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.269 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.269 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:34.269 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588071+0000", 2024-09-21T22:08:34.269 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1299999999999998e-07 2024-09-21T22:08:34.269 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.269 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.269 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:34.269 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588327+0000", 2024-09-21T22:08:34.269 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025630199999999997 2024-09-21T22:08:34.269 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.270 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.270 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:34.270 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588343+0000", 2024-09-21T22:08:34.270 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6140000000000001e-05 2024-09-21T22:08:34.270 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.270 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.270 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:34.270 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588353+0000", 2024-09-21T22:08:34.270 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.0043e-05 2024-09-21T22:08:34.270 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.270 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.270 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:34.270 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.623978+0000", 2024-09-21T22:08:34.271 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.035625281000000002 2024-09-21T22:08:34.271 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.271 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.271 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:34.271 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.624104+0000", 2024-09-21T22:08:34.271 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012591300000000001 2024-09-21T22:08:34.271 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.271 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.271 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-09-21T22:08:34.271 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.624296+0000", 2024-09-21T22:08:34.271 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00019226200000000001 2024-09-21T22:08:34.271 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.271 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.271 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:34.272 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.654488+0000", 2024-09-21T22:08:34.272 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.030191551 2024-09-21T22:08:34.272 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.272 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.272 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:34.272 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.664329+0000", 2024-09-21T22:08:34.272 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0098405119999999992 2024-09-21T22:08:34.272 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.272 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.272 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:34.272 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.664414+0000", 2024-09-21T22:08:34.272 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5385999999999999e-05 2024-09-21T22:08:34.272 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.272 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.273 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:34.273 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.664481+0000", 2024-09-21T22:08:34.273 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.6929999999999998e-05 2024-09-21T22:08:34.273 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:34.273 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:34.273 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:34.273 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.273 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.273 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5207.0:36 2.7 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head [write 524288~131072] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e78)", 2024-09-21T22:08:34.273 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:52.588370+0000", 2024-09-21T22:08:34.273 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 41.626313076000002, 2024-09-21T22:08:34.273 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.076552001999999994, 2024-09-21T22:08:34.274 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:34.274 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:34.274 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:34.274 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:34.274 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5207", 2024-09-21T22:08:34.274 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/357731919", 2024-09-21T22:08:34.274 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 36 2024-09-21T22:08:34.274 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.274 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:34.274 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.274 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:34.274 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588370+0000", 2024-09-21T22:08:34.274 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:34.275 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.275 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.275 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:34.275 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588370+0000", 2024-09-21T22:08:34.275 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:34.275 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.275 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.275 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:34.275 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588372+0000", 2024-09-21T22:08:34.275 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.153e-06 2024-09-21T22:08:34.275 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.275 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.275 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:34.275 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588716+0000", 2024-09-21T22:08:34.276 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00034479200000000002 2024-09-21T22:08:34.276 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.276 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.276 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:34.276 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588752+0000", 2024-09-21T22:08:34.276 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.5865999999999998e-05 2024-09-21T22:08:34.276 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.276 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.276 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:34.276 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588777+0000", 2024-09-21T22:08:34.276 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4583000000000002e-05 2024-09-21T22:08:34.276 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.276 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.277 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:34.277 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.633886+0000", 2024-09-21T22:08:34.277 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.045108941999999999 2024-09-21T22:08:34.277 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.277 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.277 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:34.277 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.634059+0000", 2024-09-21T22:08:34.277 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000173046 2024-09-21T22:08:34.277 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.277 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.277 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-09-21T22:08:34.277 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.634314+0000", 2024-09-21T22:08:34.277 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025482400000000001 2024-09-21T22:08:34.278 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.278 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.278 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:34.278 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.664541+0000", 2024-09-21T22:08:34.278 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.030227222000000002 2024-09-21T22:08:34.278 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.278 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.278 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:34.278 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.664653+0000", 2024-09-21T22:08:34.278 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00011192900000000001 2024-09-21T22:08:34.278 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.278 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.279 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:34.279 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.664798+0000", 2024-09-21T22:08:34.279 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014499900000000001 2024-09-21T22:08:34.279 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.279 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.279 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:34.279 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.664922+0000", 2024-09-21T22:08:34.279 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012464600000000001 2024-09-21T22:08:34.279 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:34.279 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:34.279 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:34.279 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.279 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.280 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5207.0:37 2.7 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head [write 655360~131072] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e78)", 2024-09-21T22:08:34.280 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:52.588813+0000", 2024-09-21T22:08:34.280 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 41.625870061999997, 2024-09-21T22:08:34.280 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.087074796999999995, 2024-09-21T22:08:34.280 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:34.280 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:34.280 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:34.280 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:34.280 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5207", 2024-09-21T22:08:34.280 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/357731919", 2024-09-21T22:08:34.280 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 37 2024-09-21T22:08:34.280 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.280 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:34.281 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.281 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:34.281 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588813+0000", 2024-09-21T22:08:34.281 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:34.281 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.281 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.281 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:34.281 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588813+0000", 2024-09-21T22:08:34.281 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:34.281 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.281 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.281 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:34.281 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.588815+0000", 2024-09-21T22:08:34.282 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.643e-06 2024-09-21T22:08:34.282 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.282 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.282 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:34.282 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.589071+0000", 2024-09-21T22:08:34.282 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025580200000000002 2024-09-21T22:08:34.282 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.282 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.282 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:34.282 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.589098+0000", 2024-09-21T22:08:34.282 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.7336000000000001e-05 2024-09-21T22:08:34.282 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.282 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.283 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:34.283 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.589121+0000", 2024-09-21T22:08:34.283 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.2373000000000001e-05 2024-09-21T22:08:34.283 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.283 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.283 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:34.283 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.644491+0000", 2024-09-21T22:08:34.283 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.055369994999999998 2024-09-21T22:08:34.283 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.283 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.283 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:34.283 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.644707+0000", 2024-09-21T22:08:34.283 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000216488 2024-09-21T22:08:34.284 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.284 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.284 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-09-21T22:08:34.284 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.644908+0000", 2024-09-21T22:08:34.284 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000201196 2024-09-21T22:08:34.284 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.284 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.284 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:34.284 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.675419+0000", 2024-09-21T22:08:34.284 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.03051071 2024-09-21T22:08:34.284 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.284 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.284 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:34.284 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.675591+0000", 2024-09-21T22:08:34.285 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000172545 2024-09-21T22:08:34.285 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.285 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.285 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:34.285 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.675761+0000", 2024-09-21T22:08:34.285 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00016942300000000001 2024-09-21T22:08:34.285 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.285 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.285 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:34.285 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.675888+0000", 2024-09-21T22:08:34.285 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000127286 2024-09-21T22:08:34.285 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:34.285 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:34.285 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:34.286 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.286 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.286 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5207.0:38 2.7 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head [write 786432~131072] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e78)", 2024-09-21T22:08:34.286 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:52.589156+0000", 2024-09-21T22:08:34.286 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 41.625527157999997, 2024-09-21T22:08:34.286 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.087126880000000004, 2024-09-21T22:08:34.286 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:34.286 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:34.286 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:34.286 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:34.286 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5207", 2024-09-21T22:08:34.286 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/357731919", 2024-09-21T22:08:34.286 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 38 2024-09-21T22:08:34.286 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.287 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:34.287 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.287 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:34.287 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.589156+0000", 2024-09-21T22:08:34.287 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:34.287 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.287 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.287 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:34.287 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.589156+0000", 2024-09-21T22:08:34.287 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:34.287 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.287 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.287 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:34.288 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.589158+0000", 2024-09-21T22:08:34.288 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.717e-06 2024-09-21T22:08:34.288 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.288 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.288 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:34.288 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.589417+0000", 2024-09-21T22:08:34.288 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025860500000000002 2024-09-21T22:08:34.288 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.288 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.288 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:34.288 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.589447+0000", 2024-09-21T22:08:34.288 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.9916e-05 2024-09-21T22:08:34.288 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.288 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.289 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:34.289 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.589467+0000", 2024-09-21T22:08:34.289 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0894e-05 2024-09-21T22:08:34.289 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.289 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.289 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:34.289 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.654576+0000", 2024-09-21T22:08:34.289 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.065108579999999999 2024-09-21T22:08:34.289 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.289 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.289 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:34.289 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.654694+0000", 2024-09-21T22:08:34.289 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000118083 2024-09-21T22:08:34.289 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.290 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.290 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-09-21T22:08:34.290 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.654880+0000", 2024-09-21T22:08:34.290 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000186069 2024-09-21T22:08:34.290 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.290 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.290 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:34.290 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.675453+0000", 2024-09-21T22:08:34.290 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.020572498000000002 2024-09-21T22:08:34.290 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.290 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.290 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:34.290 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.676055+0000", 2024-09-21T22:08:34.291 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00060247000000000002 2024-09-21T22:08:34.291 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.291 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.291 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:34.291 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.676178+0000", 2024-09-21T22:08:34.291 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000122395 2024-09-21T22:08:34.291 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.291 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.291 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:34.291 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.676283+0000", 2024-09-21T22:08:34.291 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000105653 2024-09-21T22:08:34.291 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:34.291 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:34.291 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:34.292 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.292 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.292 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5207.0:39 2.7 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head [write 917504~131072] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e78)", 2024-09-21T22:08:34.292 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:52.589499+0000", 2024-09-21T22:08:34.292 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 41.62518455, 2024-09-21T22:08:34.292 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.087365371999999997, 2024-09-21T22:08:34.292 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:34.292 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:34.292 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:34.292 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:34.292 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5207", 2024-09-21T22:08:34.292 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/357731919", 2024-09-21T22:08:34.292 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 39 2024-09-21T22:08:34.292 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.293 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:34.293 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.293 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:34.293 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.589499+0000", 2024-09-21T22:08:34.293 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:34.293 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.293 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.293 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:34.293 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.589499+0000", 2024-09-21T22:08:34.293 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:34.293 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.293 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.293 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:34.294 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.589500+0000", 2024-09-21T22:08:34.294 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.547e-06 2024-09-21T22:08:34.294 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.294 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.294 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:34.294 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.589752+0000", 2024-09-21T22:08:34.294 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025112799999999998 2024-09-21T22:08:34.294 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.294 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.294 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:34.294 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.589783+0000", 2024-09-21T22:08:34.294 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1557000000000003e-05 2024-09-21T22:08:34.294 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.295 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.295 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:34.295 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.589832+0000", 2024-09-21T22:08:34.295 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.9175999999999998e-05 2024-09-21T22:08:34.295 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.295 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.295 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:34.295 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.665028+0000", 2024-09-21T22:08:34.295 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.075195293999999996 2024-09-21T22:08:34.295 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.295 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.295 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:34.295 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.665169+0000", 2024-09-21T22:08:34.296 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000141299 2024-09-21T22:08:34.296 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.296 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.296 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-09-21T22:08:34.296 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.665464+0000", 2024-09-21T22:08:34.296 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00029459899999999999 2024-09-21T22:08:34.296 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.296 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.296 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:34.296 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.676551+0000", 2024-09-21T22:08:34.296 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.011087107000000001 2024-09-21T22:08:34.296 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.296 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.296 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:34.297 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.676700+0000", 2024-09-21T22:08:34.297 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000149853 2024-09-21T22:08:34.297 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.297 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.297 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:34.297 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.676802+0000", 2024-09-21T22:08:34.297 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00010194199999999999 2024-09-21T22:08:34.297 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.297 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.297 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:34.297 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.676864+0000", 2024-09-21T22:08:34.297 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1870000000000002e-05 2024-09-21T22:08:34.297 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:34.297 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:34.297 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:34.298 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.298 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.298 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5207.0:74 2.7 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head [write 524288~131072] snapc 5=[5,3] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-21T22:08:34.298 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:52.938124+0000", 2024-09-21T22:08:34.298 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 41.276559337999998, 2024-09-21T22:08:34.298 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.071435100000000001, 2024-09-21T22:08:34.298 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:34.298 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:34.298 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:34.298 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:34.298 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5207", 2024-09-21T22:08:34.298 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/357731919", 2024-09-21T22:08:34.299 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 74 2024-09-21T22:08:34.299 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.299 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:34.299 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.299 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:34.299 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938124+0000", 2024-09-21T22:08:34.299 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:34.299 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.299 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.299 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:34.299 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938124+0000", 2024-09-21T22:08:34.299 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:34.299 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.299 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.300 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:34.300 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938125+0000", 2024-09-21T22:08:34.300 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.5600000000000005e-07 2024-09-21T22:08:34.300 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.300 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.300 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:34.300 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938218+0000", 2024-09-21T22:08:34.300 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.3239999999999995e-05 2024-09-21T22:08:34.300 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.300 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.300 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:34.300 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938231+0000", 2024-09-21T22:08:34.300 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.272e-05 2024-09-21T22:08:34.301 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.301 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.301 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:34.301 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938239+0000", 2024-09-21T22:08:34.301 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.5059999999999995e-06 2024-09-21T22:08:34.301 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.301 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.301 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:34.301 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.983312+0000", 2024-09-21T22:08:34.301 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.045072252 2024-09-21T22:08:34.301 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.301 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.301 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:34.301 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.983458+0000", 2024-09-21T22:08:34.302 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014628899999999999 2024-09-21T22:08:34.302 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.302 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.302 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-09-21T22:08:34.302 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.983682+0000", 2024-09-21T22:08:34.302 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00022402899999999999 2024-09-21T22:08:34.302 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.302 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.302 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:34.302 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.993679+0000", 2024-09-21T22:08:34.302 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0099972740000000004 2024-09-21T22:08:34.302 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.302 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.302 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:34.303 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.009436+0000", 2024-09-21T22:08:34.303 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015756867000000001 2024-09-21T22:08:34.303 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.303 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.303 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:34.303 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.009506+0000", 2024-09-21T22:08:34.303 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0277999999999994e-05 2024-09-21T22:08:34.303 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.303 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.303 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:34.303 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.009559+0000", 2024-09-21T22:08:34.303 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.2889000000000001e-05 2024-09-21T22:08:34.303 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:34.304 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:34.304 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:34.304 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.304 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.304 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5207.0:75 2.7 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head [write 655360~131072] snapc 5=[5,3] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-21T22:08:34.304 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:52.938253+0000", 2024-09-21T22:08:34.304 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 41.276430282, 2024-09-21T22:08:34.304 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.082064041000000004, 2024-09-21T22:08:34.304 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:34.304 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:34.304 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:34.304 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:34.304 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5207", 2024-09-21T22:08:34.304 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/357731919", 2024-09-21T22:08:34.305 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 75 2024-09-21T22:08:34.305 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.305 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:34.305 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.305 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:34.305 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938253+0000", 2024-09-21T22:08:34.305 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:34.305 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.305 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.305 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:34.305 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938253+0000", 2024-09-21T22:08:34.305 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:34.305 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.305 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.306 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:34.306 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938254+0000", 2024-09-21T22:08:34.306 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8599999999999998e-07 2024-09-21T22:08:34.306 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.306 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.306 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:34.306 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938354+0000", 2024-09-21T22:08:34.306 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.9696000000000002e-05 2024-09-21T22:08:34.306 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.306 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.306 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:34.306 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938381+0000", 2024-09-21T22:08:34.306 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.705e-05 2024-09-21T22:08:34.307 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.307 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.307 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:34.307 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938397+0000", 2024-09-21T22:08:34.307 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5996999999999999e-05 2024-09-21T22:08:34.307 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.307 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.307 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:34.307 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.993739+0000", 2024-09-21T22:08:34.307 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.055342823999999999 2024-09-21T22:08:34.307 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.307 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.307 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:34.307 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.993835+0000", 2024-09-21T22:08:34.308 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.5919000000000001e-05 2024-09-21T22:08:34.308 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.308 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.308 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-09-21T22:08:34.308 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.993993+0000", 2024-09-21T22:08:34.308 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015729900000000001 2024-09-21T22:08:34.308 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.308 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.308 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:34.308 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.009579+0000", 2024-09-21T22:08:34.308 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.015586723 2024-09-21T22:08:34.308 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.308 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.308 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:34.309 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.020134+0000", 2024-09-21T22:08:34.309 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010554908 2024-09-21T22:08:34.309 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.309 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.309 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:34.309 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.020241+0000", 2024-09-21T22:08:34.309 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000106246 2024-09-21T22:08:34.309 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.309 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.309 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:34.309 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.020317+0000", 2024-09-21T22:08:34.309 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.6693000000000006e-05 2024-09-21T22:08:34.309 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:34.309 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:34.309 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:34.310 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.310 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.310 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5207.0:76 2.7 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head [write 786432~131072] snapc 5=[5,3] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-21T22:08:34.310 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:52.938410+0000", 2024-09-21T22:08:34.310 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 41.276273113000002, 2024-09-21T22:08:34.310 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.086926370000000003, 2024-09-21T22:08:34.310 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:34.310 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:34.310 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:34.310 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:34.310 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5207", 2024-09-21T22:08:34.310 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/357731919", 2024-09-21T22:08:34.310 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 76 2024-09-21T22:08:34.310 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.311 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:34.311 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.311 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:34.311 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938410+0000", 2024-09-21T22:08:34.311 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:34.311 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.311 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.311 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:34.311 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938410+0000", 2024-09-21T22:08:34.311 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:34.311 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.311 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.311 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:34.311 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938411+0000", 2024-09-21T22:08:34.312 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.0299999999999998e-07 2024-09-21T22:08:34.312 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.312 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.312 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:34.312 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938502+0000", 2024-09-21T22:08:34.312 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.0562000000000005e-05 2024-09-21T22:08:34.312 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.312 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.312 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:34.312 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938514+0000", 2024-09-21T22:08:34.312 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.2327000000000001e-05 2024-09-21T22:08:34.312 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.312 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.312 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:34.312 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938522+0000", 2024-09-21T22:08:34.313 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.1170000000000004e-06 2024-09-21T22:08:34.313 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.313 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.313 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:34.313 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.001427+0000", 2024-09-21T22:08:34.313 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.062904871000000001 2024-09-21T22:08:34.313 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.313 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.313 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:34.313 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.001519+0000", 2024-09-21T22:08:34.313 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.1798000000000004e-05 2024-09-21T22:08:34.313 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.313 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.313 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-09-21T22:08:34.314 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.001671+0000", 2024-09-21T22:08:34.314 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015195900000000001 2024-09-21T22:08:34.314 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.314 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.314 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:34.314 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.020388+0000", 2024-09-21T22:08:34.314 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.018717428000000001 2024-09-21T22:08:34.314 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.314 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.314 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:34.314 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.025041+0000", 2024-09-21T22:08:34.314 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0046524039999999997 2024-09-21T22:08:34.314 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.315 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.315 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:34.315 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.025213+0000", 2024-09-21T22:08:34.315 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017286499999999999 2024-09-21T22:08:34.315 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.315 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.315 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:34.315 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.025337+0000", 2024-09-21T22:08:34.315 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00012333600000000001 2024-09-21T22:08:34.315 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:34.315 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:34.315 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:34.315 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.316 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.316 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5207.0:77 2.7 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head [write 917504~131072] snapc 5=[5,3] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-21T22:08:34.316 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:52.938534+0000", 2024-09-21T22:08:34.316 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 41.276149298, 2024-09-21T22:08:34.316 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.087709436000000002, 2024-09-21T22:08:34.316 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:34.316 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:34.316 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:34.316 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:34.316 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5207", 2024-09-21T22:08:34.316 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/357731919", 2024-09-21T22:08:34.316 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 77 2024-09-21T22:08:34.316 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.317 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:34.317 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.317 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:34.317 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938534+0000", 2024-09-21T22:08:34.317 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:34.317 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.317 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.317 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:34.317 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938534+0000", 2024-09-21T22:08:34.317 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:34.317 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.317 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.317 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:34.317 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938535+0000", 2024-09-21T22:08:34.318 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.1399999999999996e-07 2024-09-21T22:08:34.318 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.318 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.318 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:34.318 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938623+0000", 2024-09-21T22:08:34.318 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 8.8226000000000005e-05 2024-09-21T22:08:34.318 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.318 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.318 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:34.318 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938645+0000", 2024-09-21T22:08:34.318 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.1948999999999999e-05 2024-09-21T22:08:34.318 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.318 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.318 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:34.319 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:52.938660+0000", 2024-09-21T22:08:34.319 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.4776999999999999e-05 2024-09-21T22:08:34.319 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.319 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.319 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:34.319 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.009669+0000", 2024-09-21T22:08:34.319 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.071009116999999997 2024-09-21T22:08:34.319 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.319 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.319 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:34.319 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.009949+0000", 2024-09-21T22:08:34.319 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00027962899999999999 2024-09-21T22:08:34.319 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.319 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.320 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-09-21T22:08:34.320 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.010355+0000", 2024-09-21T22:08:34.320 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000406334 2024-09-21T22:08:34.320 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.320 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.320 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:34.320 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.020875+0000", 2024-09-21T22:08:34.320 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.010520144 2024-09-21T22:08:34.320 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.320 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.320 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:34.320 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.026086+0000", 2024-09-21T22:08:34.320 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0052107569999999999 2024-09-21T22:08:34.320 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.321 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.321 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:34.321 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.026178+0000", 2024-09-21T22:08:34.321 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 9.2113000000000004e-05 2024-09-21T22:08:34.321 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.321 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.321 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:34.321 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.026244+0000", 2024-09-21T22:08:34.321 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.5675999999999999e-05 2024-09-21T22:08:34.321 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:34.321 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:34.321 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:34.321 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.322 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.322 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5207.0:93 2.7 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head [write 131072~917504] snapc 5=[5,3] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-21T22:08:34.322 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:07:53.147757+0000", 2024-09-21T22:08:34.322 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 41.066926119999998, 2024-09-21T22:08:34.322 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.062493043999999998, 2024-09-21T22:08:34.322 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:34.322 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:34.322 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:34.322 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:34.322 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5207", 2024-09-21T22:08:34.322 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/357731919", 2024-09-21T22:08:34.322 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 93 2024-09-21T22:08:34.322 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.322 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:34.323 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.323 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:34.323 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.147757+0000", 2024-09-21T22:08:34.323 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:34.323 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.323 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.323 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:34.323 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.147757+0000", 2024-09-21T22:08:34.323 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:34.323 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.323 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.323 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:34.323 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.147759+0000", 2024-09-21T22:08:34.323 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.0159999999999998e-06 2024-09-21T22:08:34.324 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.324 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.324 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:34.324 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.149009+0000", 2024-09-21T22:08:34.324 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001250082 2024-09-21T22:08:34.324 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.324 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.324 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:34.324 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.149042+0000", 2024-09-21T22:08:34.324 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.2617000000000003e-05 2024-09-21T22:08:34.324 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.324 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.324 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:34.325 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.149058+0000", 2024-09-21T22:08:34.325 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6113e-05 2024-09-21T22:08:34.325 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.325 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.325 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:34.325 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.149237+0000", 2024-09-21T22:08:34.325 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00017834900000000001 2024-09-21T22:08:34.325 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.325 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.325 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:34.325 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.149378+0000", 2024-09-21T22:08:34.325 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00014099199999999999 2024-09-21T22:08:34.325 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.325 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.326 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-09-21T22:08:34.326 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.149582+0000", 2024-09-21T22:08:34.326 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00020414900000000001 2024-09-21T22:08:34.326 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.326 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.326 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:34.326 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.208648+0000", 2024-09-21T22:08:34.326 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.059066397999999999 2024-09-21T22:08:34.326 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.326 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.326 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:34.326 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.210125+0000", 2024-09-21T22:08:34.326 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001477119 2024-09-21T22:08:34.326 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.327 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.327 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:34.327 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.210200+0000", 2024-09-21T22:08:34.327 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 7.4973000000000002e-05 2024-09-21T22:08:34.327 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.327 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.327 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:34.327 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:07:53.210250+0000", 2024-09-21T22:08:34.327 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.0235999999999998e-05 2024-09-21T22:08:34.327 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:34.327 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:34.327 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:34.327 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.327 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.328 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "osd_op(client.5207.0:128 2.7 2:f590a3a8:::rbd_data.126483d75f2d.0000000000000000:head [write 131072~917504] snapc 6=[6,5,3] ondisk+write+known_if_redirected+supports_pool_eio e83)", 2024-09-21T22:08:34.328 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:08:01.056777+0000", 2024-09-21T22:08:34.328 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 33.157906650000001, 2024-09-21T22:08:34.328 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.062294929999999998, 2024-09-21T22:08:34.328 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:34.328 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:34.328 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T22:08:34.328 INFO:teuthology.orchestra.run.smithi064.stdout: "client_info": { 2024-09-21T22:08:34.328 INFO:teuthology.orchestra.run.smithi064.stdout: "client": "client.5207", 2024-09-21T22:08:34.328 INFO:teuthology.orchestra.run.smithi064.stdout: "client_addr": "172.21.15.157:0/357731919", 2024-09-21T22:08:34.328 INFO:teuthology.orchestra.run.smithi064.stdout: "tid": 128 2024-09-21T22:08:34.328 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.328 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:34.329 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.329 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:34.329 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.056777+0000", 2024-09-21T22:08:34.329 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:34.329 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.329 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.329 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:34.329 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.056777+0000", 2024-09-21T22:08:34.329 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:34.329 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.329 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.329 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:34.329 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.056779+0000", 2024-09-21T22:08:34.330 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 2.4669999999999998e-06 2024-09-21T22:08:34.330 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.330 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.330 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:34.330 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.058094+0000", 2024-09-21T22:08:34.330 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001314602 2024-09-21T22:08:34.330 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.330 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.330 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:34.330 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.058126+0000", 2024-09-21T22:08:34.330 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.1625999999999998e-05 2024-09-21T22:08:34.330 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.330 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.331 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:34.331 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.058141+0000", 2024-09-21T22:08:34.331 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.5274e-05 2024-09-21T22:08:34.331 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.331 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.331 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:34.331 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.058267+0000", 2024-09-21T22:08:34.331 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000126682 2024-09-21T22:08:34.331 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.331 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.331 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:34.331 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.058420+0000", 2024-09-21T22:08:34.331 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00015212599999999999 2024-09-21T22:08:34.331 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.332 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.332 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "waiting for subops from 4", 2024-09-21T22:08:34.332 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.058644+0000", 2024-09-21T22:08:34.332 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.000224749 2024-09-21T22:08:34.332 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.332 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.332 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "sub_op_commit_rec", 2024-09-21T22:08:34.332 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.117883+0000", 2024-09-21T22:08:34.332 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.059238617 2024-09-21T22:08:34.332 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.332 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.332 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "op_commit", 2024-09-21T22:08:34.332 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.118942+0000", 2024-09-21T22:08:34.332 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.0010588780000000001 2024-09-21T22:08:34.333 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.333 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.333 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "commit_sent", 2024-09-21T22:08:34.333 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.119011+0000", 2024-09-21T22:08:34.333 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.8802999999999998e-05 2024-09-21T22:08:34.333 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.333 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.333 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:34.333 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:01.119072+0000", 2024-09-21T22:08:34.333 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 6.1105999999999994e-05 2024-09-21T22:08:34.333 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:34.333 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:34.333 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:34.333 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.333 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.334 INFO:teuthology.orchestra.run.smithi064.stdout: "description": "MOSDPGPush(2.3 87/86 [PushOp(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head, version: 71'29, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:d0224986:::rbd_data.126483d75f2d.0000000000000008:head@71'29, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T22:08:34.334 INFO:teuthology.orchestra.run.smithi064.stdout: "initiated_at": "2024-09-21T22:08:07.107088+0000", 2024-09-21T22:08:34.334 INFO:teuthology.orchestra.run.smithi064.stdout: "age": 27.107595908, 2024-09-21T22:08:34.334 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.070273375999999999, 2024-09-21T22:08:34.334 INFO:teuthology.orchestra.run.smithi064.stdout: "continuous": false, 2024-09-21T22:08:34.334 INFO:teuthology.orchestra.run.smithi064.stdout: "type_data": { 2024-09-21T22:08:34.334 INFO:teuthology.orchestra.run.smithi064.stdout: "flag_point": "started", 2024-09-21T22:08:34.334 INFO:teuthology.orchestra.run.smithi064.stdout: "events": [ 2024-09-21T22:08:34.334 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.334 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "initiated", 2024-09-21T22:08:34.334 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.107088+0000", 2024-09-21T22:08:34.334 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:34.334 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.335 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.335 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "header_read", 2024-09-21T22:08:34.335 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.107088+0000", 2024-09-21T22:08:34.335 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0 2024-09-21T22:08:34.335 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.335 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.335 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "throttled", 2024-09-21T22:08:34.335 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.107089+0000", 2024-09-21T22:08:34.335 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 1.6559999999999999e-06 2024-09-21T22:08:34.335 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.335 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.335 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "all_read", 2024-09-21T22:08:34.335 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.108876+0000", 2024-09-21T22:08:34.335 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.001786489 2024-09-21T22:08:34.335 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.336 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.336 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "dispatched", 2024-09-21T22:08:34.336 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.108925+0000", 2024-09-21T22:08:34.336 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 4.8843e-05 2024-09-21T22:08:34.336 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.336 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.336 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "queued_for_pg", 2024-09-21T22:08:34.336 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.108979+0000", 2024-09-21T22:08:34.336 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 5.4945999999999999e-05 2024-09-21T22:08:34.336 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.336 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.336 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "reached_pg", 2024-09-21T22:08:34.336 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.109230+0000", 2024-09-21T22:08:34.336 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.00025086599999999998 2024-09-21T22:08:34.336 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.337 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.337 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "started", 2024-09-21T22:08:34.337 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.109269+0000", 2024-09-21T22:08:34.337 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 3.8216000000000001e-05 2024-09-21T22:08:34.337 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-09-21T22:08:34.337 INFO:teuthology.orchestra.run.smithi064.stdout: { 2024-09-21T22:08:34.337 INFO:teuthology.orchestra.run.smithi064.stdout: "event": "done", 2024-09-21T22:08:34.337 INFO:teuthology.orchestra.run.smithi064.stdout: "time": "2024-09-21T22:08:07.177361+0000", 2024-09-21T22:08:34.337 INFO:teuthology.orchestra.run.smithi064.stdout: "duration": 0.068092360000000005 2024-09-21T22:08:34.337 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:34.337 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:34.337 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:34.337 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-09-21T22:08:34.338 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-09-21T22:08:34.338 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:08:34.340 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T22:08:34.341 INFO:tasks.workunit.client.0.smithi157.stderr:2024-09-21T22:08:34.228+0000 7f6d23978640 0 -- 172.21.15.157:0/451418257 >> [v2:172.21.15.39:6816/2876495139,v1:172.21.15.39:6817/2876495139] conn(0x7f6d0403b7b0 msgr2=0x7f6d0403dbe0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-21T22:08:34.341 INFO:tasks.workunit.client.0.smithi157.stderr: Removing image: 6% complete... Removing image: 12% complete... Removing image: 18% complete... Removing image: 25% complete... Removing image: 31% complete... Removing image: 37% complete... Removing image: 43% complete... Removing image: 50% complete... Removing image: 56% complete... Removing image: 62% complete... Removing image: 68% complete... Removing image: 75% complete... Removing image: 81% complete... Removing image: 87% complete... Removing image: 93% complete...2024-09-21T22:08:34.240+0000 7f6d23978640 0 -- 172.21.15.157:0/451418257 >> [v2:172.21.15.39:6816/2876495139,v1:172.21.15.39:6817/2876495139] conn(0x555896a09e10 msgr2=0x7f6d040409b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-21T22:08:34.341 INFO:tasks.workunit.client.0.smithi157.stderr: Removing image: 100% complete...done. 2024-09-21T22:08:34.341 INFO:tasks.workunit.client.0.smithi157.stderr:+ rbd namespace remove rbd/ns 2024-09-21T22:08:34.341 INFO:tasks.workunit.client.0.smithi157.stderr:+ rbd config global rm global rbd_default_pool 2024-09-21T22:08:34.342 INFO:tasks.ceph.osd.0.smithi039.stderr:2024-09-21T22:08:34.333+0000 7fa1f0bbe640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25154) UID: 0 2024-09-21T22:08:34.378 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T22:08:34.382 INFO:tasks.workunit.client.0.smithi157.stderr:+ ceph osd pool delete new_default_pool21553 new_default_pool21553 --yes-i-really-really-mean-it 2024-09-21T22:08:34.440 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:08:34.437+0000 7fe38af2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 34631) UID: 0 2024-09-21T22:08:34.515 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-09-21T22:08:34.515 INFO:teuthology.orchestra.run.smithi064.stdout: "ops": [], 2024-09-21T22:08:34.515 INFO:teuthology.orchestra.run.smithi064.stdout: "num_ops": 0 2024-09-21T22:08:34.515 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-09-21T22:08:34.529 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T22:08:34.530 INFO:teuthology.orchestra.run.smithi039.stdout:osd.0: {} 2024-09-21T22:08:34.531 INFO:teuthology.orchestra.run.smithi039.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T22:08:34.547 INFO:teuthology.orchestra.run.smithi039.stdout:osd.1: {} 2024-09-21T22:08:34.548 INFO:teuthology.orchestra.run.smithi039.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T22:08:34.564 INFO:teuthology.orchestra.run.smithi039.stdout:osd.2: {} 2024-09-21T22:08:34.564 INFO:teuthology.orchestra.run.smithi039.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T22:08:34.573 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:08:34.571+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:08:34.582 INFO:teuthology.orchestra.run.smithi039.stdout:osd.3: {} 2024-09-21T22:08:34.582 INFO:teuthology.orchestra.run.smithi039.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T22:08:34.599 INFO:teuthology.orchestra.run.smithi039.stdout:osd.4: {} 2024-09-21T22:08:34.599 INFO:teuthology.orchestra.run.smithi039.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T22:08:34.619 INFO:teuthology.orchestra.run.smithi039.stdout:osd.5: {} 2024-09-21T22:08:34.620 INFO:teuthology.orchestra.run.smithi039.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T22:08:34.642 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:08:34.639+0000 7f2075a06640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31720) UID: 0 2024-09-21T22:08:34.655 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:34.655 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:34.669 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T22:08:34.743 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:08:34.737+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:08:34.833 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:34.833 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:34.844 INFO:tasks.ceph.osd.5.smithi064.stderr:2024-09-21T22:08:34.839+0000 7f2075a06640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31720) UID: 0 2024-09-21T22:08:34.848 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T22:08:34.944 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:08:34.941+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:08:35.009 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:35.009 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:35.023 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T22:08:35.045 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:08:35.041+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:08:35.048 INFO:teuthology.orchestra.run.smithi039.stderr:nodeep-scrub is unset 2024-09-21T22:08:35.145 INFO:tasks.workunit.client.0.smithi157.stderr:pool 'new_default_pool21553' does not exist 2024-09-21T22:08:35.145 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:08:35.143+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:08:35.152 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:35.152 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:35.162 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-21T22:08:35.162 DEBUG:teuthology.orchestra.run.smithi157:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-21T22:08:35.167 DEBUG:teuthology.orchestra.run.smithi064:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T22:08:35.218 INFO:tasks.workunit:Stopping ['rbd/rbd-nbd.sh'] on client.0... 2024-09-21T22:08:35.218 DEBUG:teuthology.orchestra.run.smithi157:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-21T22:08:35.247 INFO:tasks.ceph.osd.2.smithi039.stderr:2024-09-21T22:08:35.241+0000 7fe38af2f640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 34631) UID: 0 2024-09-21T22:08:35.335 INFO:teuthology.orchestra.run.smithi064.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:35.335 INFO:teuthology.orchestra.run.smithi064.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:35.347 INFO:tasks.ceph.osd.1.smithi039.stderr:2024-09-21T22:08:35.345+0000 7fb10a0be640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25156) UID: 0 2024-09-21T22:08:35.349 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T22:08:35.448 INFO:tasks.ceph.osd.3.smithi064.stderr:2024-09-21T22:08:35.443+0000 7fcc5daf5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25291) UID: 0 2024-09-21T22:08:35.485 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:35.485 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:35.499 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T22:08:35.547 DEBUG:teuthology.parallel:result is None 2024-09-21T22:08:35.547 DEBUG:teuthology.orchestra.run.smithi157:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-09-21T22:08:35.549 INFO:tasks.ceph.osd.4.smithi064.stderr:2024-09-21T22:08:35.543+0000 7f6e7c4aa640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25290) UID: 0 2024-09-21T22:08:35.559 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-09-21T22:08:35.559 DEBUG:teuthology.orchestra.run.smithi157:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-09-21T22:08:35.620 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-09-21T22:08:35.620 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-09-21T22:08:35.629 INFO:tasks.thrashosds:joining thrashosds 2024-09-21T22:08:35.630 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:35.630 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:35.636 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T22:08:35.766 INFO:teuthology.orchestra.run.smithi039.stdout:ERROR: (22) Invalid argument 2024-09-21T22:08:35.766 INFO:teuthology.orchestra.run.smithi039.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T22:08:37.076 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-21T22:08:38.071 INFO:teuthology.orchestra.run.smithi039.stderr:noscrub is unset 2024-09-21T22:08:38.072 INFO:tasks.thrashosds.thrasher:inning osd 2024-09-21T22:08:38.072 INFO:tasks.thrashosds.thrasher:Adding osd 2 2024-09-21T22:08:38.072 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 2 2024-09-21T22:08:38.099 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-21T22:08:38.385 INFO:tasks.ceph.mon.a.smithi039.stderr:2024-09-21T22:08:38.381+0000 7f9967196640 -1 mon.a@0(leader).osd e106 definitely_dead 0 2024-09-21T22:08:39.114 INFO:teuthology.orchestra.run.smithi039.stderr:marked in osd.2. 2024-09-21T22:08:39.114 INFO:teuthology.orchestra.run.smithi039.stderr:nodeep-scrub is unset 2024-09-21T22:08:39.141 INFO:tasks.thrashosds.thrasher:Added osd 2 2024-09-21T22:08:39.141 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-09-21T22:08:39.278 INFO:teuthology.orchestra.run.smithi039.stdout:mon.a: {} 2024-09-21T22:08:39.279 INFO:teuthology.orchestra.run.smithi039.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-21T22:08:39.292 INFO:teuthology.orchestra.run.smithi039.stdout:mon.b: {} 2024-09-21T22:08:39.292 INFO:teuthology.orchestra.run.smithi039.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-21T22:08:39.301 INFO:teuthology.orchestra.run.smithi039.stdout:mon.c: {} 2024-09-21T22:08:39.301 INFO:teuthology.orchestra.run.smithi039.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-21T22:08:39.317 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-09-21T22:08:40.038 INFO:teuthology.orchestra.run.smithi039.stderr:reweighted osd.0 to 1 (10000) 2024-09-21T22:08:40.069 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1 2024-09-21T22:08:41.043 INFO:teuthology.orchestra.run.smithi039.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-09-21T22:08:41.068 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-09-21T22:08:42.143 INFO:teuthology.orchestra.run.smithi039.stderr:reweighted osd.1 to 1 (10000) 2024-09-21T22:08:42.168 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2024-09-21T22:08:43.152 INFO:teuthology.orchestra.run.smithi039.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-09-21T22:08:43.177 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-09-21T22:08:44.170 INFO:teuthology.orchestra.run.smithi039.stderr:reweighted osd.3 to 1 (10000) 2024-09-21T22:08:44.197 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 1 2024-09-21T22:08:45.193 INFO:teuthology.orchestra.run.smithi039.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-09-21T22:08:45.220 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-09-21T22:08:46.196 INFO:teuthology.orchestra.run.smithi039.stderr:reweighted osd.4 to 1 (10000) 2024-09-21T22:08:46.224 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 4 1 2024-09-21T22:08:47.216 INFO:teuthology.orchestra.run.smithi039.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-09-21T22:08:47.244 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-09-21T22:08:48.236 INFO:teuthology.orchestra.run.smithi039.stderr:reweighted osd.5 to 1 (10000) 2024-09-21T22:08:48.263 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 1 2024-09-21T22:08:49.244 INFO:teuthology.orchestra.run.smithi039.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-09-21T22:08:49.273 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-09-21T22:08:50.264 INFO:teuthology.orchestra.run.smithi039.stderr:reweighted osd.2 to 1 (10000) 2024-09-21T22:08:50.294 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 1 2024-09-21T22:08:51.293 INFO:teuthology.orchestra.run.smithi039.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-09-21T22:08:51.319 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-09-21T22:08:51.319 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-09-21T22:08:51.319 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-09-21T22:08:51.319 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-09-21T22:08:51.320 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-21T22:08:51.320 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-21T22:08:51.628 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T22:08:51.628 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":119,"fsid":"00bb09a3-e410-4e18-a2b2-08eaf61978e6","created":"2024-09-21T22:05:45.303341+0000","modified":"2024-09-21T22:08:51.282331+0000","last_up_change":"2024-09-21T22:08:18.467986+0000","last_in_change":"2024-09-21T22:08:39.090072+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T22:06:00.868591+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-21T22:06:03.535857+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"104","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":7,"snap_epoch":104,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8336c397-2018-4c75-ab31-2e5f6ad564b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":3054061468},{"type":"v1","addr":"172.21.15.39:6801","nonce":3054061468}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":3054061468},{"type":"v1","addr":"172.21.15.39:6803","nonce":3054061468}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":3054061468},{"type":"v1","addr":"172.21.15.39:6810","nonce":3054061468}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":3054061468},{"type":"v1","addr":"172.21.15.39:6806","nonce":3054061468}]},"public_addr":"172.21.15.39:6801/3054061468","cluster_addr":"172.21.15.39:6803/3054061468","heartbeat_back_addr":"172.21.15.39:6810/3054061468","heartbeat_front_addr":"172.21.15.39:6806/3054061468","state":["exists","up"]},{"osd":1,"uuid":"bca64af6-7427-4c5d-97a4-a183b1f3a7d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":2876495139},{"type":"v1","addr":"172.21.15.39:6817","nonce":2876495139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":2876495139},{"type":"v1","addr":"172.21.15.39:6819","nonce":2876495139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":2876495139},{"type":"v1","addr":"172.21.15.39:6823","nonce":2876495139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":2876495139},{"type":"v1","addr":"172.21.15.39:6821","nonce":2876495139}]},"public_addr":"172.21.15.39:6817/2876495139","cluster_addr":"172.21.15.39:6819/2876495139","heartbeat_back_addr":"172.21.15.39:6823/2876495139","heartbeat_front_addr":"172.21.15.39:6821/2876495139","state":["exists","up"]},{"osd":2,"uuid":"5f2678ed-6a23-4a67-96a4-c0ca91254bd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":11,"last_clean_end":80,"up_from":93,"up_thru":110,"down_at":81,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6805","nonce":1877094708},{"type":"v1","addr":"172.21.15.39:6807","nonce":1877094708}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6809","nonce":1877094708},{"type":"v1","addr":"172.21.15.39:6811","nonce":1877094708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":1877094708},{"type":"v1","addr":"172.21.15.39:6815","nonce":1877094708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":1877094708},{"type":"v1","addr":"172.21.15.39:6813","nonce":1877094708}]},"public_addr":"172.21.15.39:6807/1877094708","cluster_addr":"172.21.15.39:6811/1877094708","heartbeat_back_addr":"172.21.15.39:6815/1877094708","heartbeat_front_addr":"172.21.15.39:6813/1877094708","state":["exists","up"]},{"osd":3,"uuid":"aba30555-ef9e-47c9-a38b-ee1a4b8092f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":1132198485},{"type":"v1","addr":"172.21.15.64:6801","nonce":1132198485}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":1132198485},{"type":"v1","addr":"172.21.15.64:6803","nonce":1132198485}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":1132198485},{"type":"v1","addr":"172.21.15.64:6807","nonce":1132198485}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":1132198485},{"type":"v1","addr":"172.21.15.64:6805","nonce":1132198485}]},"public_addr":"172.21.15.64:6801/1132198485","cluster_addr":"172.21.15.64:6803/1132198485","heartbeat_back_addr":"172.21.15.64:6807/1132198485","heartbeat_front_addr":"172.21.15.64:6805/1132198485","state":["exists","up"]},{"osd":4,"uuid":"2a1288d7-618d-4f44-8346-930cf56d4e80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":3500786459},{"type":"v1","addr":"172.21.15.64:6809","nonce":3500786459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":3500786459},{"type":"v1","addr":"172.21.15.64:6811","nonce":3500786459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":3500786459},{"type":"v1","addr":"172.21.15.64:6815","nonce":3500786459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":3500786459},{"type":"v1","addr":"172.21.15.64:6813","nonce":3500786459}]},"public_addr":"172.21.15.64:6809/3500786459","cluster_addr":"172.21.15.64:6811/3500786459","heartbeat_back_addr":"172.21.15.64:6815/3500786459","heartbeat_front_addr":"172.21.15.64:6813/3500786459","state":["exists","up"]},{"osd":5,"uuid":"3ce719a8-34e9-4890-bd40-d22201e7bad8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":11,"last_clean_end":41,"up_from":65,"up_thru":107,"down_at":42,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":1547734524},{"type":"v1","addr":"172.21.15.64:6817","nonce":1547734524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":1547734524},{"type":"v1","addr":"172.21.15.64:6819","nonce":1547734524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":1547734524},{"type":"v1","addr":"172.21.15.64:6823","nonce":1547734524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":1547734524},{"type":"v1","addr":"172.21.15.64:6821","nonce":1547734524}]},"public_addr":"172.21.15.64:6817/1547734524","cluster_addr":"172.21.15.64:6819/1547734524","heartbeat_back_addr":"172.21.15.64:6823/1547734524","heartbeat_front_addr":"172.21.15.64:6821/1547734524","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T22:05:54.224881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T22:05:54.668978+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T22:05:54.315876+0000","dead_epoch":81},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T22:05:54.314795+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T22:05:54.322403+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T22:05:54.750633+0000","dead_epoch":42}],"pg_upmap":[{"pgid":"1.0","osds":[3,2]}],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T22:08:51.647 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-21T22:08:51.647 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-21T22:08:51.948 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T22:08:51.948 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":119,"fsid":"00bb09a3-e410-4e18-a2b2-08eaf61978e6","created":"2024-09-21T22:05:45.303341+0000","modified":"2024-09-21T22:08:51.282331+0000","last_up_change":"2024-09-21T22:08:18.467986+0000","last_in_change":"2024-09-21T22:08:39.090072+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T22:06:00.868591+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-21T22:06:03.535857+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"104","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":7,"snap_epoch":104,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8336c397-2018-4c75-ab31-2e5f6ad564b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":3054061468},{"type":"v1","addr":"172.21.15.39:6801","nonce":3054061468}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":3054061468},{"type":"v1","addr":"172.21.15.39:6803","nonce":3054061468}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":3054061468},{"type":"v1","addr":"172.21.15.39:6810","nonce":3054061468}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":3054061468},{"type":"v1","addr":"172.21.15.39:6806","nonce":3054061468}]},"public_addr":"172.21.15.39:6801/3054061468","cluster_addr":"172.21.15.39:6803/3054061468","heartbeat_back_addr":"172.21.15.39:6810/3054061468","heartbeat_front_addr":"172.21.15.39:6806/3054061468","state":["exists","up"]},{"osd":1,"uuid":"bca64af6-7427-4c5d-97a4-a183b1f3a7d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":2876495139},{"type":"v1","addr":"172.21.15.39:6817","nonce":2876495139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":2876495139},{"type":"v1","addr":"172.21.15.39:6819","nonce":2876495139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":2876495139},{"type":"v1","addr":"172.21.15.39:6823","nonce":2876495139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":2876495139},{"type":"v1","addr":"172.21.15.39:6821","nonce":2876495139}]},"public_addr":"172.21.15.39:6817/2876495139","cluster_addr":"172.21.15.39:6819/2876495139","heartbeat_back_addr":"172.21.15.39:6823/2876495139","heartbeat_front_addr":"172.21.15.39:6821/2876495139","state":["exists","up"]},{"osd":2,"uuid":"5f2678ed-6a23-4a67-96a4-c0ca91254bd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":11,"last_clean_end":80,"up_from":93,"up_thru":110,"down_at":81,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6805","nonce":1877094708},{"type":"v1","addr":"172.21.15.39:6807","nonce":1877094708}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6809","nonce":1877094708},{"type":"v1","addr":"172.21.15.39:6811","nonce":1877094708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":1877094708},{"type":"v1","addr":"172.21.15.39:6815","nonce":1877094708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":1877094708},{"type":"v1","addr":"172.21.15.39:6813","nonce":1877094708}]},"public_addr":"172.21.15.39:6807/1877094708","cluster_addr":"172.21.15.39:6811/1877094708","heartbeat_back_addr":"172.21.15.39:6815/1877094708","heartbeat_front_addr":"172.21.15.39:6813/1877094708","state":["exists","up"]},{"osd":3,"uuid":"aba30555-ef9e-47c9-a38b-ee1a4b8092f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":1132198485},{"type":"v1","addr":"172.21.15.64:6801","nonce":1132198485}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":1132198485},{"type":"v1","addr":"172.21.15.64:6803","nonce":1132198485}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":1132198485},{"type":"v1","addr":"172.21.15.64:6807","nonce":1132198485}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":1132198485},{"type":"v1","addr":"172.21.15.64:6805","nonce":1132198485}]},"public_addr":"172.21.15.64:6801/1132198485","cluster_addr":"172.21.15.64:6803/1132198485","heartbeat_back_addr":"172.21.15.64:6807/1132198485","heartbeat_front_addr":"172.21.15.64:6805/1132198485","state":["exists","up"]},{"osd":4,"uuid":"2a1288d7-618d-4f44-8346-930cf56d4e80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":3500786459},{"type":"v1","addr":"172.21.15.64:6809","nonce":3500786459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":3500786459},{"type":"v1","addr":"172.21.15.64:6811","nonce":3500786459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":3500786459},{"type":"v1","addr":"172.21.15.64:6815","nonce":3500786459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":3500786459},{"type":"v1","addr":"172.21.15.64:6813","nonce":3500786459}]},"public_addr":"172.21.15.64:6809/3500786459","cluster_addr":"172.21.15.64:6811/3500786459","heartbeat_back_addr":"172.21.15.64:6815/3500786459","heartbeat_front_addr":"172.21.15.64:6813/3500786459","state":["exists","up"]},{"osd":5,"uuid":"3ce719a8-34e9-4890-bd40-d22201e7bad8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":11,"last_clean_end":41,"up_from":65,"up_thru":107,"down_at":42,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":1547734524},{"type":"v1","addr":"172.21.15.64:6817","nonce":1547734524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":1547734524},{"type":"v1","addr":"172.21.15.64:6819","nonce":1547734524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":1547734524},{"type":"v1","addr":"172.21.15.64:6823","nonce":1547734524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":1547734524},{"type":"v1","addr":"172.21.15.64:6821","nonce":1547734524}]},"public_addr":"172.21.15.64:6817/1547734524","cluster_addr":"172.21.15.64:6819/1547734524","heartbeat_back_addr":"172.21.15.64:6823/1547734524","heartbeat_front_addr":"172.21.15.64:6821/1547734524","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T22:05:54.224881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T22:05:54.668978+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T22:05:54.315876+0000","dead_epoch":81},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T22:05:54.314795+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T22:05:54.322403+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T22:05:54.750633+0000","dead_epoch":42}],"pg_upmap":[{"pgid":"1.0","osds":[3,2]}],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T22:08:51.968 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-21T22:08:51.968 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-21T22:08:51.968 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-21T22:08:51.969 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-21T22:08:51.969 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-21T22:08:51.969 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-21T22:08:52.126 INFO:teuthology.orchestra.run.smithi039.stdout:47244640298 2024-09-21T22:08:52.127 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-21T22:08:52.153 INFO:teuthology.orchestra.run.smithi039.stdout:47244640298 2024-09-21T22:08:52.153 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-21T22:08:52.185 INFO:teuthology.orchestra.run.smithi039.stdout:399431958537 2024-09-21T22:08:52.186 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-21T22:08:52.200 INFO:teuthology.orchestra.run.smithi039.stdout:47244640297 2024-09-21T22:08:52.200 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-21T22:08:52.214 INFO:teuthology.orchestra.run.smithi039.stdout:47244640295 2024-09-21T22:08:52.214 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-21T22:08:52.220 INFO:teuthology.orchestra.run.smithi039.stdout:279172874259 2024-09-21T22:08:52.220 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-21T22:08:52.510 INFO:teuthology.orchestra.run.smithi039.stdout:399431958536 2024-09-21T22:08:52.525 INFO:teuthology.orchestra.run.smithi039.stdout:47244640297 2024-09-21T22:08:52.529 INFO:tasks.ceph.ceph_manager.ceph:need seq 399431958537 got 399431958536 for osd.2 2024-09-21T22:08:52.543 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640298 got 47244640297 for osd.3 2024-09-21T22:08:52.580 INFO:teuthology.orchestra.run.smithi039.stdout:47244640297 2024-09-21T22:08:52.602 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640298 got 47244640297 for osd.0 2024-09-21T22:08:52.648 INFO:teuthology.orchestra.run.smithi039.stdout:47244640296 2024-09-21T22:08:52.665 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640297 got 47244640296 for osd.1 2024-09-21T22:08:52.673 INFO:teuthology.orchestra.run.smithi039.stdout:47244640294 2024-09-21T22:08:52.684 INFO:teuthology.orchestra.run.smithi039.stdout:279172874257 2024-09-21T22:08:52.691 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640295 got 47244640294 for osd.4 2024-09-21T22:08:52.701 INFO:tasks.ceph.ceph_manager.ceph:need seq 279172874259 got 279172874257 for osd.5 2024-09-21T22:08:53.531 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-21T22:08:53.544 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-21T22:08:53.604 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-21T22:08:53.666 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-21T22:08:53.691 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-21T22:08:53.702 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-21T22:08:53.946 INFO:teuthology.orchestra.run.smithi039.stdout:399431958537 2024-09-21T22:08:53.948 INFO:teuthology.orchestra.run.smithi039.stdout:47244640298 2024-09-21T22:08:53.976 INFO:tasks.ceph.ceph_manager.ceph:need seq 399431958537 got 399431958537 for osd.2 2024-09-21T22:08:53.976 DEBUG:teuthology.parallel:result is None 2024-09-21T22:08:53.976 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640298 got 47244640298 for osd.0 2024-09-21T22:08:53.977 DEBUG:teuthology.parallel:result is None 2024-09-21T22:08:54.001 INFO:teuthology.orchestra.run.smithi039.stdout:47244640297 2024-09-21T22:08:54.009 INFO:teuthology.orchestra.run.smithi039.stdout:47244640298 2024-09-21T22:08:54.019 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640297 got 47244640297 for osd.1 2024-09-21T22:08:54.019 DEBUG:teuthology.parallel:result is None 2024-09-21T22:08:54.032 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640298 got 47244640298 for osd.3 2024-09-21T22:08:54.033 DEBUG:teuthology.parallel:result is None 2024-09-21T22:08:54.105 INFO:teuthology.orchestra.run.smithi039.stdout:47244640295 2024-09-21T22:08:54.122 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640295 got 47244640295 for osd.4 2024-09-21T22:08:54.122 DEBUG:teuthology.parallel:result is None 2024-09-21T22:08:54.127 INFO:teuthology.orchestra.run.smithi039.stdout:279172874259 2024-09-21T22:08:54.145 INFO:tasks.ceph.ceph_manager.ceph:need seq 279172874259 got 279172874259 for osd.5 2024-09-21T22:08:54.145 DEBUG:teuthology.parallel:result is None 2024-09-21T22:08:54.145 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-09-21T22:08:54.145 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T22:08:54.485 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T22:08:54.485 INFO:teuthology.orchestra.run.smithi039.stderr:dumped all 2024-09-21T22:08:54.501 INFO:teuthology.orchestra.run.smithi039.stdout:{"pg_ready":true,"pg_map":{"version":201,"stamp":"2024-09-21T22:08:52.832633+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":7548,"num_read_kb":163179,"num_write":1146,"num_write_kb":275375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":231131616,"num_keys_recovered":61,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1240,"ondisk_log_size":1240,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":197436,"kb_used_data":8360,"kb_used_omap":16,"kb_used_meta":188911,"kb_avail":566033604,"statfs":{"total":579820584960,"available":579618410496,"internally_reserved":0,"allocated":8560640,"data_stored":4453044,"data_compressed":13990,"data_compressed_allocated":73728,"data_compressed_original":1163264,"omap_allocated":16441,"internal_metadata":193445831},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.606855"},"pg_stats":[{"pgid":"1.0","version":"15'117","reported_seq":406,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-21T22:08:51.294841+0000","last_change":"2024-09-21T22:08:46.212886+0000","last_active":"2024-09-21T22:08:51.294841+0000","last_peered":"2024-09-21T22:08:51.294841+0000","last_clean":"2024-09-21T22:08:51.294841+0000","last_became_active":"2024-09-21T22:08:46.212212+0000","last_became_peered":"2024-09-21T22:08:46.212212+0000","last_unstale":"2024-09-21T22:08:51.294841+0000","last_undegraded":"2024-09-21T22:08:51.294841+0000","last_fullsized":"2024-09-21T22:08:51.294841+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:00.990331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:00.990331+0000","last_clean_scrub_stamp":"2024-09-21T22:06:00.990331+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T05:09:41.935962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"103'62","reported_seq":286,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-21T22:08:51.291117+0000","last_change":"2024-09-21T22:08:40.076840+0000","last_active":"2024-09-21T22:08:51.291117+0000","last_peered":"2024-09-21T22:08:51.291117+0000","last_clean":"2024-09-21T22:08:51.291117+0000","last_became_active":"2024-09-21T22:08:40.053267+0000","last_became_peered":"2024-09-21T22:08:40.053267+0000","last_unstale":"2024-09-21T22:08:51.291117+0000","last_undegraded":"2024-09-21T22:08:51.291117+0000","last_fullsized":"2024-09-21T22:08:51.291117+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:05:04.598630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036861000000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":94,"num_read_kb":8213,"num_write":46,"num_write_kb":24581,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":33554508,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"104'117","reported_seq":1123,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-21T22:08:51.294765+0000","last_change":"2024-09-21T22:08:34.057726+0000","last_active":"2024-09-21T22:08:51.294765+0000","last_peered":"2024-09-21T22:08:51.294765+0000","last_clean":"2024-09-21T22:08:51.294765+0000","last_became_active":"2024-09-21T22:08:27.993129+0000","last_became_peered":"2024-09-21T22:08:27.993129+0000","last_unstale":"2024-09-21T22:08:51.294765+0000","last_undegraded":"2024-09-21T22:08:51.294765+0000","last_fullsized":"2024-09-21T22:08:51.294765+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T06:21:07.580957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00099297199999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":721,"num_read_kb":41715,"num_write":87,"num_write_kb":57411,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":50397184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"104'60","reported_seq":47,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-21T22:08:51.507719+0000","last_change":"2024-09-21T22:08:40.066003+0000","last_active":"2024-09-21T22:08:51.507719+0000","last_peered":"2024-09-21T22:08:51.507719+0000","last_clean":"2024-09-21T22:08:51.507719+0000","last_became_active":"2024-09-21T22:08:40.052140+0000","last_became_peered":"2024-09-21T22:08:40.052140+0000","last_unstale":"2024-09-21T22:08:51.507719+0000","last_undegraded":"2024-09-21T22:08:51.507719+0000","last_fullsized":"2024-09-21T22:08:51.507719+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T04:30:58.743113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011545399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":81,"num_read_kb":16395,"num_write":44,"num_write_kb":32769,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":41943040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"104'269","reported_seq":1794,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-21T22:08:51.291107+0000","last_change":"2024-09-21T22:08:34.054445+0000","last_active":"2024-09-21T22:08:51.291107+0000","last_peered":"2024-09-21T22:08:51.291107+0000","last_clean":"2024-09-21T22:08:51.291107+0000","last_became_active":"2024-09-21T22:08:31.007410+0000","last_became_peered":"2024-09-21T22:08:31.007410+0000","last_unstale":"2024-09-21T22:08:51.291107+0000","last_undegraded":"2024-09-21T22:08:51.291107+0000","last_fullsized":"2024-09-21T22:08:51.291107+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":269,"log_dups_size":0,"ondisk_log_size":269,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T09:57:16.015269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056494499999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4024,"num_read_kb":3421,"num_write":160,"num_write_kb":8272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":25166100,"num_keys_recovered":31,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"104'107","reported_seq":219,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-21T22:08:51.827176+0000","last_change":"2024-09-21T22:08:34.055487+0000","last_active":"2024-09-21T22:08:51.827176+0000","last_peered":"2024-09-21T22:08:51.827176+0000","last_clean":"2024-09-21T22:08:51.827176+0000","last_became_active":"2024-09-21T22:08:27.987130+0000","last_became_peered":"2024-09-21T22:08:27.987130+0000","last_unstale":"2024-09-21T22:08:51.827176+0000","last_undegraded":"2024-09-21T22:08:51.827176+0000","last_fullsized":"2024-09-21T22:08:51.827176+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:42:38.530832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002184208,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":551,"num_read_kb":24994,"num_write":73,"num_write_kb":32826,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":33603584,"num_keys_recovered":24,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"104'93","reported_seq":443,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-21T22:08:51.294904+0000","last_change":"2024-09-21T22:08:34.058368+0000","last_active":"2024-09-21T22:08:51.294904+0000","last_peered":"2024-09-21T22:08:51.294904+0000","last_clean":"2024-09-21T22:08:51.294904+0000","last_became_active":"2024-09-21T22:06:44.874697+0000","last_became_peered":"2024-09-21T22:06:44.874697+0000","last_unstale":"2024-09-21T22:08:51.294904+0000","last_undegraded":"2024-09-21T22:08:51.294904+0000","last_fullsized":"2024-09-21T22:08:51.294904+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T08:45:09.814190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015356479999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":231,"num_read_kb":16507,"num_write":94,"num_write_kb":28712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"104'64","reported_seq":119,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-21T22:08:51.827199+0000","last_change":"2024-09-21T22:08:34.055376+0000","last_active":"2024-09-21T22:08:51.827199+0000","last_peered":"2024-09-21T22:08:51.827199+0000","last_clean":"2024-09-21T22:08:51.827199+0000","last_became_active":"2024-09-21T22:08:27.983818+0000","last_became_peered":"2024-09-21T22:08:27.983818+0000","last_unstale":"2024-09-21T22:08:51.827199+0000","last_undegraded":"2024-09-21T22:08:51.827199+0000","last_fullsized":"2024-09-21T22:08:51.827199+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T06:39:39.444028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015450220000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":88,"num_read_kb":13397,"num_write":53,"num_write_kb":30748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":44105728,"num_keys_recovered":6,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"104'351","reported_seq":1415,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-21T22:08:51.294902+0000","last_change":"2024-09-21T22:08:34.057700+0000","last_active":"2024-09-21T22:08:51.294902+0000","last_peered":"2024-09-21T22:08:51.294902+0000","last_clean":"2024-09-21T22:08:51.294902+0000","last_became_active":"2024-09-21T22:06:05.220996+0000","last_became_peered":"2024-09-21T22:06:05.220996+0000","last_unstale":"2024-09-21T22:08:51.294902+0000","last_undegraded":"2024-09-21T22:08:51.294902+0000","last_fullsized":"2024-09-21T22:08:51.294902+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":351,"log_dups_size":0,"ondisk_log_size":351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T08:56:07.384975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095003999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1642,"num_read_kb":38437,"num_write":430,"num_write_kb":57640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7432,"num_read_kb":163079,"num_write":987,"num_write_kb":272959,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":228770144,"num_keys_recovered":61,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1174,"internal_metadata":0},"log_size":1123,"ondisk_log_size":1123,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":2361472,"data_compressed":28848,"data_compressed_allocated":147456,"data_compressed_original":2342912,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":65,"seq":279172874259,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29696,"kb_used_data":1384,"kb_used_omap":5,"kb_used_meta":28282,"kb_avail":94342144,"statfs":{"total":96636764160,"available":96606355456,"internally_reserved":0,"allocated":1417216,"data_stored":546933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5607,"internal_metadata":28961305},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640295,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32572,"kb_used_data":1380,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339268,"statfs":{"total":96636764160,"available":96603410432,"internally_reserved":0,"allocated":1413120,"data_stored":546914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640298,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32632,"kb_used_data":1436,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339208,"statfs":{"total":96636764160,"available":96603348992,"internally_reserved":0,"allocated":1470464,"data_stored":1137282,"data_compressed":6561,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":93,"seq":399431958537,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29092,"kb_used_data":1420,"kb_used_omap":4,"kb_used_meta":27643,"kb_avail":94342748,"statfs":{"total":96636764160,"available":96606973952,"internally_reserved":0,"allocated":1454080,"data_stored":1137282,"data_compressed":7429,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":4456,"internal_metadata":28307096},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640297,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36720,"kb_used_data":1372,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335120,"statfs":{"total":96636764160,"available":96599162880,"internally_reserved":0,"allocated":1404928,"data_stored":542326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640298,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36724,"kb_used_data":1368,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335116,"statfs":{"total":96636764160,"available":96599158784,"internally_reserved":0,"allocated":1400832,"data_stored":542307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":7429,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":7429,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":7429,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6561,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1174,"internal_metadata":0}]}} 2024-09-21T22:08:54.502 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T22:08:54.784 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T22:08:54.784 INFO:teuthology.orchestra.run.smithi039.stderr:dumped all 2024-09-21T22:08:54.801 INFO:teuthology.orchestra.run.smithi039.stdout:{"pg_ready":true,"pg_map":{"version":201,"stamp":"2024-09-21T22:08:52.832633+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":7548,"num_read_kb":163179,"num_write":1146,"num_write_kb":275375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":231131616,"num_keys_recovered":61,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1240,"ondisk_log_size":1240,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":197436,"kb_used_data":8360,"kb_used_omap":16,"kb_used_meta":188911,"kb_avail":566033604,"statfs":{"total":579820584960,"available":579618410496,"internally_reserved":0,"allocated":8560640,"data_stored":4453044,"data_compressed":13990,"data_compressed_allocated":73728,"data_compressed_original":1163264,"omap_allocated":16441,"internal_metadata":193445831},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.606855"},"pg_stats":[{"pgid":"1.0","version":"15'117","reported_seq":406,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-21T22:08:51.294841+0000","last_change":"2024-09-21T22:08:46.212886+0000","last_active":"2024-09-21T22:08:51.294841+0000","last_peered":"2024-09-21T22:08:51.294841+0000","last_clean":"2024-09-21T22:08:51.294841+0000","last_became_active":"2024-09-21T22:08:46.212212+0000","last_became_peered":"2024-09-21T22:08:46.212212+0000","last_unstale":"2024-09-21T22:08:51.294841+0000","last_undegraded":"2024-09-21T22:08:51.294841+0000","last_fullsized":"2024-09-21T22:08:51.294841+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:00.990331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:00.990331+0000","last_clean_scrub_stamp":"2024-09-21T22:06:00.990331+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T05:09:41.935962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"103'62","reported_seq":286,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-21T22:08:51.291117+0000","last_change":"2024-09-21T22:08:40.076840+0000","last_active":"2024-09-21T22:08:51.291117+0000","last_peered":"2024-09-21T22:08:51.291117+0000","last_clean":"2024-09-21T22:08:51.291117+0000","last_became_active":"2024-09-21T22:08:40.053267+0000","last_became_peered":"2024-09-21T22:08:40.053267+0000","last_unstale":"2024-09-21T22:08:51.291117+0000","last_undegraded":"2024-09-21T22:08:51.291117+0000","last_fullsized":"2024-09-21T22:08:51.291117+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:05:04.598630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036861000000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":94,"num_read_kb":8213,"num_write":46,"num_write_kb":24581,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":33554508,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"104'117","reported_seq":1123,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-21T22:08:51.294765+0000","last_change":"2024-09-21T22:08:34.057726+0000","last_active":"2024-09-21T22:08:51.294765+0000","last_peered":"2024-09-21T22:08:51.294765+0000","last_clean":"2024-09-21T22:08:51.294765+0000","last_became_active":"2024-09-21T22:08:27.993129+0000","last_became_peered":"2024-09-21T22:08:27.993129+0000","last_unstale":"2024-09-21T22:08:51.294765+0000","last_undegraded":"2024-09-21T22:08:51.294765+0000","last_fullsized":"2024-09-21T22:08:51.294765+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T06:21:07.580957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00099297199999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":721,"num_read_kb":41715,"num_write":87,"num_write_kb":57411,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":50397184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"104'60","reported_seq":47,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-21T22:08:51.507719+0000","last_change":"2024-09-21T22:08:40.066003+0000","last_active":"2024-09-21T22:08:51.507719+0000","last_peered":"2024-09-21T22:08:51.507719+0000","last_clean":"2024-09-21T22:08:51.507719+0000","last_became_active":"2024-09-21T22:08:40.052140+0000","last_became_peered":"2024-09-21T22:08:40.052140+0000","last_unstale":"2024-09-21T22:08:51.507719+0000","last_undegraded":"2024-09-21T22:08:51.507719+0000","last_fullsized":"2024-09-21T22:08:51.507719+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T04:30:58.743113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011545399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":81,"num_read_kb":16395,"num_write":44,"num_write_kb":32769,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":41943040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"104'269","reported_seq":1794,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-21T22:08:51.291107+0000","last_change":"2024-09-21T22:08:34.054445+0000","last_active":"2024-09-21T22:08:51.291107+0000","last_peered":"2024-09-21T22:08:51.291107+0000","last_clean":"2024-09-21T22:08:51.291107+0000","last_became_active":"2024-09-21T22:08:31.007410+0000","last_became_peered":"2024-09-21T22:08:31.007410+0000","last_unstale":"2024-09-21T22:08:51.291107+0000","last_undegraded":"2024-09-21T22:08:51.291107+0000","last_fullsized":"2024-09-21T22:08:51.291107+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":269,"log_dups_size":0,"ondisk_log_size":269,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T09:57:16.015269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056494499999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4024,"num_read_kb":3421,"num_write":160,"num_write_kb":8272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":25166100,"num_keys_recovered":31,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"104'107","reported_seq":219,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-21T22:08:51.827176+0000","last_change":"2024-09-21T22:08:34.055487+0000","last_active":"2024-09-21T22:08:51.827176+0000","last_peered":"2024-09-21T22:08:51.827176+0000","last_clean":"2024-09-21T22:08:51.827176+0000","last_became_active":"2024-09-21T22:08:27.987130+0000","last_became_peered":"2024-09-21T22:08:27.987130+0000","last_unstale":"2024-09-21T22:08:51.827176+0000","last_undegraded":"2024-09-21T22:08:51.827176+0000","last_fullsized":"2024-09-21T22:08:51.827176+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:42:38.530832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002184208,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":551,"num_read_kb":24994,"num_write":73,"num_write_kb":32826,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":33603584,"num_keys_recovered":24,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"104'93","reported_seq":443,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-21T22:08:51.294904+0000","last_change":"2024-09-21T22:08:34.058368+0000","last_active":"2024-09-21T22:08:51.294904+0000","last_peered":"2024-09-21T22:08:51.294904+0000","last_clean":"2024-09-21T22:08:51.294904+0000","last_became_active":"2024-09-21T22:06:44.874697+0000","last_became_peered":"2024-09-21T22:06:44.874697+0000","last_unstale":"2024-09-21T22:08:51.294904+0000","last_undegraded":"2024-09-21T22:08:51.294904+0000","last_fullsized":"2024-09-21T22:08:51.294904+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T08:45:09.814190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015356479999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":231,"num_read_kb":16507,"num_write":94,"num_write_kb":28712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"104'64","reported_seq":119,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-21T22:08:51.827199+0000","last_change":"2024-09-21T22:08:34.055376+0000","last_active":"2024-09-21T22:08:51.827199+0000","last_peered":"2024-09-21T22:08:51.827199+0000","last_clean":"2024-09-21T22:08:51.827199+0000","last_became_active":"2024-09-21T22:08:27.983818+0000","last_became_peered":"2024-09-21T22:08:27.983818+0000","last_unstale":"2024-09-21T22:08:51.827199+0000","last_undegraded":"2024-09-21T22:08:51.827199+0000","last_fullsized":"2024-09-21T22:08:51.827199+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T06:39:39.444028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015450220000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":88,"num_read_kb":13397,"num_write":53,"num_write_kb":30748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":44105728,"num_keys_recovered":6,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"104'351","reported_seq":1415,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-21T22:08:51.294902+0000","last_change":"2024-09-21T22:08:34.057700+0000","last_active":"2024-09-21T22:08:51.294902+0000","last_peered":"2024-09-21T22:08:51.294902+0000","last_clean":"2024-09-21T22:08:51.294902+0000","last_became_active":"2024-09-21T22:06:05.220996+0000","last_became_peered":"2024-09-21T22:06:05.220996+0000","last_unstale":"2024-09-21T22:08:51.294902+0000","last_undegraded":"2024-09-21T22:08:51.294902+0000","last_fullsized":"2024-09-21T22:08:51.294902+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":351,"log_dups_size":0,"ondisk_log_size":351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T08:56:07.384975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095003999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1642,"num_read_kb":38437,"num_write":430,"num_write_kb":57640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7432,"num_read_kb":163079,"num_write":987,"num_write_kb":272959,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":228770144,"num_keys_recovered":61,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1174,"internal_metadata":0},"log_size":1123,"ondisk_log_size":1123,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":2361472,"data_compressed":28848,"data_compressed_allocated":147456,"data_compressed_original":2342912,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":65,"seq":279172874259,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29696,"kb_used_data":1384,"kb_used_omap":5,"kb_used_meta":28282,"kb_avail":94342144,"statfs":{"total":96636764160,"available":96606355456,"internally_reserved":0,"allocated":1417216,"data_stored":546933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5607,"internal_metadata":28961305},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640295,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32572,"kb_used_data":1380,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339268,"statfs":{"total":96636764160,"available":96603410432,"internally_reserved":0,"allocated":1413120,"data_stored":546914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640298,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32632,"kb_used_data":1436,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339208,"statfs":{"total":96636764160,"available":96603348992,"internally_reserved":0,"allocated":1470464,"data_stored":1137282,"data_compressed":6561,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":93,"seq":399431958537,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29092,"kb_used_data":1420,"kb_used_omap":4,"kb_used_meta":27643,"kb_avail":94342748,"statfs":{"total":96636764160,"available":96606973952,"internally_reserved":0,"allocated":1454080,"data_stored":1137282,"data_compressed":7429,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":4456,"internal_metadata":28307096},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640297,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36720,"kb_used_data":1372,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335120,"statfs":{"total":96636764160,"available":96599162880,"internally_reserved":0,"allocated":1404928,"data_stored":542326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640298,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36724,"kb_used_data":1368,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335116,"statfs":{"total":96636764160,"available":96599158784,"internally_reserved":0,"allocated":1400832,"data_stored":542307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":7429,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":7429,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":7429,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6561,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1174,"internal_metadata":0}]}} 2024-09-21T22:08:54.802 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-09-21T22:08:54.802 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-09-21T22:08:55.108 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-21T22:08:55.118 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-21T22:08:55.118 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T22:08:55.401 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T22:08:55.401 INFO:teuthology.orchestra.run.smithi039.stderr:dumped all 2024-09-21T22:08:55.418 INFO:teuthology.orchestra.run.smithi039.stdout:{"pg_ready":true,"pg_map":{"version":202,"stamp":"2024-09-21T22:08:54.833203+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":7548,"num_read_kb":163179,"num_write":1146,"num_write_kb":275375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":231131616,"num_keys_recovered":61,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1240,"ondisk_log_size":1240,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":197436,"kb_used_data":8360,"kb_used_omap":16,"kb_used_meta":188911,"kb_avail":566033604,"statfs":{"total":579820584960,"available":579618410496,"internally_reserved":0,"allocated":8560640,"data_stored":4453044,"data_compressed":13990,"data_compressed_allocated":73728,"data_compressed_original":1163264,"omap_allocated":16441,"internal_metadata":193445831},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001560"},"pg_stats":[{"pgid":"1.0","version":"15'117","reported_seq":406,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-21T22:08:51.294841+0000","last_change":"2024-09-21T22:08:46.212886+0000","last_active":"2024-09-21T22:08:51.294841+0000","last_peered":"2024-09-21T22:08:51.294841+0000","last_clean":"2024-09-21T22:08:51.294841+0000","last_became_active":"2024-09-21T22:08:46.212212+0000","last_became_peered":"2024-09-21T22:08:46.212212+0000","last_unstale":"2024-09-21T22:08:51.294841+0000","last_undegraded":"2024-09-21T22:08:51.294841+0000","last_fullsized":"2024-09-21T22:08:51.294841+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:00.990331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:00.990331+0000","last_clean_scrub_stamp":"2024-09-21T22:06:00.990331+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T05:09:41.935962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"103'62","reported_seq":286,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-21T22:08:51.291117+0000","last_change":"2024-09-21T22:08:40.076840+0000","last_active":"2024-09-21T22:08:51.291117+0000","last_peered":"2024-09-21T22:08:51.291117+0000","last_clean":"2024-09-21T22:08:51.291117+0000","last_became_active":"2024-09-21T22:08:40.053267+0000","last_became_peered":"2024-09-21T22:08:40.053267+0000","last_unstale":"2024-09-21T22:08:51.291117+0000","last_undegraded":"2024-09-21T22:08:51.291117+0000","last_fullsized":"2024-09-21T22:08:51.291117+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:05:04.598630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036861000000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":94,"num_read_kb":8213,"num_write":46,"num_write_kb":24581,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":33554508,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"104'117","reported_seq":1123,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-21T22:08:51.294765+0000","last_change":"2024-09-21T22:08:34.057726+0000","last_active":"2024-09-21T22:08:51.294765+0000","last_peered":"2024-09-21T22:08:51.294765+0000","last_clean":"2024-09-21T22:08:51.294765+0000","last_became_active":"2024-09-21T22:08:27.993129+0000","last_became_peered":"2024-09-21T22:08:27.993129+0000","last_unstale":"2024-09-21T22:08:51.294765+0000","last_undegraded":"2024-09-21T22:08:51.294765+0000","last_fullsized":"2024-09-21T22:08:51.294765+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T06:21:07.580957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00099297199999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":721,"num_read_kb":41715,"num_write":87,"num_write_kb":57411,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":50397184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"104'60","reported_seq":47,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-21T22:08:51.507719+0000","last_change":"2024-09-21T22:08:40.066003+0000","last_active":"2024-09-21T22:08:51.507719+0000","last_peered":"2024-09-21T22:08:51.507719+0000","last_clean":"2024-09-21T22:08:51.507719+0000","last_became_active":"2024-09-21T22:08:40.052140+0000","last_became_peered":"2024-09-21T22:08:40.052140+0000","last_unstale":"2024-09-21T22:08:51.507719+0000","last_undegraded":"2024-09-21T22:08:51.507719+0000","last_fullsized":"2024-09-21T22:08:51.507719+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T04:30:58.743113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011545399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":81,"num_read_kb":16395,"num_write":44,"num_write_kb":32769,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":41943040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"104'269","reported_seq":1794,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-21T22:08:51.291107+0000","last_change":"2024-09-21T22:08:34.054445+0000","last_active":"2024-09-21T22:08:51.291107+0000","last_peered":"2024-09-21T22:08:51.291107+0000","last_clean":"2024-09-21T22:08:51.291107+0000","last_became_active":"2024-09-21T22:08:31.007410+0000","last_became_peered":"2024-09-21T22:08:31.007410+0000","last_unstale":"2024-09-21T22:08:51.291107+0000","last_undegraded":"2024-09-21T22:08:51.291107+0000","last_fullsized":"2024-09-21T22:08:51.291107+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":269,"log_dups_size":0,"ondisk_log_size":269,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T09:57:16.015269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056494499999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4024,"num_read_kb":3421,"num_write":160,"num_write_kb":8272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":25166100,"num_keys_recovered":31,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"104'107","reported_seq":219,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-21T22:08:51.827176+0000","last_change":"2024-09-21T22:08:34.055487+0000","last_active":"2024-09-21T22:08:51.827176+0000","last_peered":"2024-09-21T22:08:51.827176+0000","last_clean":"2024-09-21T22:08:51.827176+0000","last_became_active":"2024-09-21T22:08:27.987130+0000","last_became_peered":"2024-09-21T22:08:27.987130+0000","last_unstale":"2024-09-21T22:08:51.827176+0000","last_undegraded":"2024-09-21T22:08:51.827176+0000","last_fullsized":"2024-09-21T22:08:51.827176+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:42:38.530832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002184208,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":551,"num_read_kb":24994,"num_write":73,"num_write_kb":32826,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":33603584,"num_keys_recovered":24,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"104'93","reported_seq":443,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-21T22:08:51.294904+0000","last_change":"2024-09-21T22:08:34.058368+0000","last_active":"2024-09-21T22:08:51.294904+0000","last_peered":"2024-09-21T22:08:51.294904+0000","last_clean":"2024-09-21T22:08:51.294904+0000","last_became_active":"2024-09-21T22:06:44.874697+0000","last_became_peered":"2024-09-21T22:06:44.874697+0000","last_unstale":"2024-09-21T22:08:51.294904+0000","last_undegraded":"2024-09-21T22:08:51.294904+0000","last_fullsized":"2024-09-21T22:08:51.294904+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T08:45:09.814190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015356479999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":231,"num_read_kb":16507,"num_write":94,"num_write_kb":28712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"104'64","reported_seq":119,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-21T22:08:51.827199+0000","last_change":"2024-09-21T22:08:34.055376+0000","last_active":"2024-09-21T22:08:51.827199+0000","last_peered":"2024-09-21T22:08:51.827199+0000","last_clean":"2024-09-21T22:08:51.827199+0000","last_became_active":"2024-09-21T22:08:27.983818+0000","last_became_peered":"2024-09-21T22:08:27.983818+0000","last_unstale":"2024-09-21T22:08:51.827199+0000","last_undegraded":"2024-09-21T22:08:51.827199+0000","last_fullsized":"2024-09-21T22:08:51.827199+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T06:39:39.444028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015450220000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":88,"num_read_kb":13397,"num_write":53,"num_write_kb":30748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":44105728,"num_keys_recovered":6,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"104'351","reported_seq":1415,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-21T22:08:51.294902+0000","last_change":"2024-09-21T22:08:34.057700+0000","last_active":"2024-09-21T22:08:51.294902+0000","last_peered":"2024-09-21T22:08:51.294902+0000","last_clean":"2024-09-21T22:08:51.294902+0000","last_became_active":"2024-09-21T22:06:05.220996+0000","last_became_peered":"2024-09-21T22:06:05.220996+0000","last_unstale":"2024-09-21T22:08:51.294902+0000","last_undegraded":"2024-09-21T22:08:51.294902+0000","last_fullsized":"2024-09-21T22:08:51.294902+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":351,"log_dups_size":0,"ondisk_log_size":351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T08:56:07.384975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095003999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1642,"num_read_kb":38437,"num_write":430,"num_write_kb":57640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7432,"num_read_kb":163079,"num_write":987,"num_write_kb":272959,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":228770144,"num_keys_recovered":61,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1174,"internal_metadata":0},"log_size":1123,"ondisk_log_size":1123,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":2361472,"data_compressed":28848,"data_compressed_allocated":147456,"data_compressed_original":2342912,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":65,"seq":279172874259,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29696,"kb_used_data":1384,"kb_used_omap":5,"kb_used_meta":28282,"kb_avail":94342144,"statfs":{"total":96636764160,"available":96606355456,"internally_reserved":0,"allocated":1417216,"data_stored":546933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5607,"internal_metadata":28961305},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640295,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32572,"kb_used_data":1380,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339268,"statfs":{"total":96636764160,"available":96603410432,"internally_reserved":0,"allocated":1413120,"data_stored":546914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640298,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32632,"kb_used_data":1436,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339208,"statfs":{"total":96636764160,"available":96603348992,"internally_reserved":0,"allocated":1470464,"data_stored":1137282,"data_compressed":6561,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":93,"seq":399431958538,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29092,"kb_used_data":1420,"kb_used_omap":4,"kb_used_meta":27643,"kb_avail":94342748,"statfs":{"total":96636764160,"available":96606973952,"internally_reserved":0,"allocated":1454080,"data_stored":1137282,"data_compressed":7429,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":4456,"internal_metadata":28307096},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640297,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36720,"kb_used_data":1372,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335120,"statfs":{"total":96636764160,"available":96599162880,"internally_reserved":0,"allocated":1404928,"data_stored":542326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640298,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36724,"kb_used_data":1368,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335116,"statfs":{"total":96636764160,"available":96599158784,"internally_reserved":0,"allocated":1400832,"data_stored":542307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":7429,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":7429,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":7429,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6561,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1174,"internal_metadata":0}]}} 2024-09-21T22:08:55.419 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T22:08:55.719 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T22:08:55.720 INFO:teuthology.orchestra.run.smithi039.stderr:dumped all 2024-09-21T22:08:55.739 INFO:teuthology.orchestra.run.smithi039.stdout:{"pg_ready":true,"pg_map":{"version":202,"stamp":"2024-09-21T22:08:54.833203+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":7548,"num_read_kb":163179,"num_write":1146,"num_write_kb":275375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":231131616,"num_keys_recovered":61,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1240,"ondisk_log_size":1240,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":197436,"kb_used_data":8360,"kb_used_omap":16,"kb_used_meta":188911,"kb_avail":566033604,"statfs":{"total":579820584960,"available":579618410496,"internally_reserved":0,"allocated":8560640,"data_stored":4453044,"data_compressed":13990,"data_compressed_allocated":73728,"data_compressed_original":1163264,"omap_allocated":16441,"internal_metadata":193445831},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001560"},"pg_stats":[{"pgid":"1.0","version":"15'117","reported_seq":406,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-21T22:08:51.294841+0000","last_change":"2024-09-21T22:08:46.212886+0000","last_active":"2024-09-21T22:08:51.294841+0000","last_peered":"2024-09-21T22:08:51.294841+0000","last_clean":"2024-09-21T22:08:51.294841+0000","last_became_active":"2024-09-21T22:08:46.212212+0000","last_became_peered":"2024-09-21T22:08:46.212212+0000","last_unstale":"2024-09-21T22:08:51.294841+0000","last_undegraded":"2024-09-21T22:08:51.294841+0000","last_fullsized":"2024-09-21T22:08:51.294841+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:00.990331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:00.990331+0000","last_clean_scrub_stamp":"2024-09-21T22:06:00.990331+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T05:09:41.935962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"103'62","reported_seq":286,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-21T22:08:51.291117+0000","last_change":"2024-09-21T22:08:40.076840+0000","last_active":"2024-09-21T22:08:51.291117+0000","last_peered":"2024-09-21T22:08:51.291117+0000","last_clean":"2024-09-21T22:08:51.291117+0000","last_became_active":"2024-09-21T22:08:40.053267+0000","last_became_peered":"2024-09-21T22:08:40.053267+0000","last_unstale":"2024-09-21T22:08:51.291117+0000","last_undegraded":"2024-09-21T22:08:51.291117+0000","last_fullsized":"2024-09-21T22:08:51.291117+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:05:04.598630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036861000000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":94,"num_read_kb":8213,"num_write":46,"num_write_kb":24581,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":33554508,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"104'117","reported_seq":1123,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-21T22:08:51.294765+0000","last_change":"2024-09-21T22:08:34.057726+0000","last_active":"2024-09-21T22:08:51.294765+0000","last_peered":"2024-09-21T22:08:51.294765+0000","last_clean":"2024-09-21T22:08:51.294765+0000","last_became_active":"2024-09-21T22:08:27.993129+0000","last_became_peered":"2024-09-21T22:08:27.993129+0000","last_unstale":"2024-09-21T22:08:51.294765+0000","last_undegraded":"2024-09-21T22:08:51.294765+0000","last_fullsized":"2024-09-21T22:08:51.294765+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T06:21:07.580957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00099297199999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":721,"num_read_kb":41715,"num_write":87,"num_write_kb":57411,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":50397184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"104'60","reported_seq":47,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-21T22:08:51.507719+0000","last_change":"2024-09-21T22:08:40.066003+0000","last_active":"2024-09-21T22:08:51.507719+0000","last_peered":"2024-09-21T22:08:51.507719+0000","last_clean":"2024-09-21T22:08:51.507719+0000","last_became_active":"2024-09-21T22:08:40.052140+0000","last_became_peered":"2024-09-21T22:08:40.052140+0000","last_unstale":"2024-09-21T22:08:51.507719+0000","last_undegraded":"2024-09-21T22:08:51.507719+0000","last_fullsized":"2024-09-21T22:08:51.507719+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T04:30:58.743113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011545399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":81,"num_read_kb":16395,"num_write":44,"num_write_kb":32769,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":41943040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"104'269","reported_seq":1794,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-21T22:08:51.291107+0000","last_change":"2024-09-21T22:08:34.054445+0000","last_active":"2024-09-21T22:08:51.291107+0000","last_peered":"2024-09-21T22:08:51.291107+0000","last_clean":"2024-09-21T22:08:51.291107+0000","last_became_active":"2024-09-21T22:08:31.007410+0000","last_became_peered":"2024-09-21T22:08:31.007410+0000","last_unstale":"2024-09-21T22:08:51.291107+0000","last_undegraded":"2024-09-21T22:08:51.291107+0000","last_fullsized":"2024-09-21T22:08:51.291107+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":269,"log_dups_size":0,"ondisk_log_size":269,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T09:57:16.015269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056494499999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4024,"num_read_kb":3421,"num_write":160,"num_write_kb":8272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":25166100,"num_keys_recovered":31,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"104'107","reported_seq":219,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-21T22:08:51.827176+0000","last_change":"2024-09-21T22:08:34.055487+0000","last_active":"2024-09-21T22:08:51.827176+0000","last_peered":"2024-09-21T22:08:51.827176+0000","last_clean":"2024-09-21T22:08:51.827176+0000","last_became_active":"2024-09-21T22:08:27.987130+0000","last_became_peered":"2024-09-21T22:08:27.987130+0000","last_unstale":"2024-09-21T22:08:51.827176+0000","last_undegraded":"2024-09-21T22:08:51.827176+0000","last_fullsized":"2024-09-21T22:08:51.827176+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:42:38.530832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002184208,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":551,"num_read_kb":24994,"num_write":73,"num_write_kb":32826,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":33603584,"num_keys_recovered":24,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"104'93","reported_seq":443,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-21T22:08:51.294904+0000","last_change":"2024-09-21T22:08:34.058368+0000","last_active":"2024-09-21T22:08:51.294904+0000","last_peered":"2024-09-21T22:08:51.294904+0000","last_clean":"2024-09-21T22:08:51.294904+0000","last_became_active":"2024-09-21T22:06:44.874697+0000","last_became_peered":"2024-09-21T22:06:44.874697+0000","last_unstale":"2024-09-21T22:08:51.294904+0000","last_undegraded":"2024-09-21T22:08:51.294904+0000","last_fullsized":"2024-09-21T22:08:51.294904+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T08:45:09.814190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015356479999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":231,"num_read_kb":16507,"num_write":94,"num_write_kb":28712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"104'64","reported_seq":119,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-21T22:08:51.827199+0000","last_change":"2024-09-21T22:08:34.055376+0000","last_active":"2024-09-21T22:08:51.827199+0000","last_peered":"2024-09-21T22:08:51.827199+0000","last_clean":"2024-09-21T22:08:51.827199+0000","last_became_active":"2024-09-21T22:08:27.983818+0000","last_became_peered":"2024-09-21T22:08:27.983818+0000","last_unstale":"2024-09-21T22:08:51.827199+0000","last_undegraded":"2024-09-21T22:08:51.827199+0000","last_fullsized":"2024-09-21T22:08:51.827199+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T06:39:39.444028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015450220000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":88,"num_read_kb":13397,"num_write":53,"num_write_kb":30748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":44105728,"num_keys_recovered":6,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"104'351","reported_seq":1415,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-21T22:08:51.294902+0000","last_change":"2024-09-21T22:08:34.057700+0000","last_active":"2024-09-21T22:08:51.294902+0000","last_peered":"2024-09-21T22:08:51.294902+0000","last_clean":"2024-09-21T22:08:51.294902+0000","last_became_active":"2024-09-21T22:06:05.220996+0000","last_became_peered":"2024-09-21T22:06:05.220996+0000","last_unstale":"2024-09-21T22:08:51.294902+0000","last_undegraded":"2024-09-21T22:08:51.294902+0000","last_fullsized":"2024-09-21T22:08:51.294902+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":351,"log_dups_size":0,"ondisk_log_size":351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T08:56:07.384975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095003999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1642,"num_read_kb":38437,"num_write":430,"num_write_kb":57640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7432,"num_read_kb":163079,"num_write":987,"num_write_kb":272959,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":228770144,"num_keys_recovered":61,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1174,"internal_metadata":0},"log_size":1123,"ondisk_log_size":1123,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":2361472,"data_compressed":28848,"data_compressed_allocated":147456,"data_compressed_original":2342912,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":65,"seq":279172874259,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29696,"kb_used_data":1384,"kb_used_omap":5,"kb_used_meta":28282,"kb_avail":94342144,"statfs":{"total":96636764160,"available":96606355456,"internally_reserved":0,"allocated":1417216,"data_stored":546933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5607,"internal_metadata":28961305},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640295,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32572,"kb_used_data":1380,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339268,"statfs":{"total":96636764160,"available":96603410432,"internally_reserved":0,"allocated":1413120,"data_stored":546914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640298,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32632,"kb_used_data":1436,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339208,"statfs":{"total":96636764160,"available":96603348992,"internally_reserved":0,"allocated":1470464,"data_stored":1137282,"data_compressed":6561,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":93,"seq":399431958538,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29092,"kb_used_data":1420,"kb_used_omap":4,"kb_used_meta":27643,"kb_avail":94342748,"statfs":{"total":96636764160,"available":96606973952,"internally_reserved":0,"allocated":1454080,"data_stored":1137282,"data_compressed":7429,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":4456,"internal_metadata":28307096},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640297,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36720,"kb_used_data":1372,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335120,"statfs":{"total":96636764160,"available":96599162880,"internally_reserved":0,"allocated":1404928,"data_stored":542326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640298,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36724,"kb_used_data":1368,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335116,"statfs":{"total":96636764160,"available":96599158784,"internally_reserved":0,"allocated":1400832,"data_stored":542307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":7429,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":7429,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":7429,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6561,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1174,"internal_metadata":0}]}} 2024-09-21T22:08:55.740 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-21T22:08:55.740 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T22:08:56.045 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T22:08:56.046 INFO:teuthology.orchestra.run.smithi039.stderr:dumped all 2024-09-21T22:08:56.064 INFO:teuthology.orchestra.run.smithi039.stdout:{"pg_ready":true,"pg_map":{"version":202,"stamp":"2024-09-21T22:08:54.833203+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":7548,"num_read_kb":163179,"num_write":1146,"num_write_kb":275375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":231131616,"num_keys_recovered":61,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1240,"ondisk_log_size":1240,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":197436,"kb_used_data":8360,"kb_used_omap":16,"kb_used_meta":188911,"kb_avail":566033604,"statfs":{"total":579820584960,"available":579618410496,"internally_reserved":0,"allocated":8560640,"data_stored":4453044,"data_compressed":13990,"data_compressed_allocated":73728,"data_compressed_original":1163264,"omap_allocated":16441,"internal_metadata":193445831},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001560"},"pg_stats":[{"pgid":"1.0","version":"15'117","reported_seq":406,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-21T22:08:51.294841+0000","last_change":"2024-09-21T22:08:46.212886+0000","last_active":"2024-09-21T22:08:51.294841+0000","last_peered":"2024-09-21T22:08:51.294841+0000","last_clean":"2024-09-21T22:08:51.294841+0000","last_became_active":"2024-09-21T22:08:46.212212+0000","last_became_peered":"2024-09-21T22:08:46.212212+0000","last_unstale":"2024-09-21T22:08:51.294841+0000","last_undegraded":"2024-09-21T22:08:51.294841+0000","last_fullsized":"2024-09-21T22:08:51.294841+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:00.990331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:00.990331+0000","last_clean_scrub_stamp":"2024-09-21T22:06:00.990331+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T05:09:41.935962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"103'62","reported_seq":286,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-21T22:08:51.291117+0000","last_change":"2024-09-21T22:08:40.076840+0000","last_active":"2024-09-21T22:08:51.291117+0000","last_peered":"2024-09-21T22:08:51.291117+0000","last_clean":"2024-09-21T22:08:51.291117+0000","last_became_active":"2024-09-21T22:08:40.053267+0000","last_became_peered":"2024-09-21T22:08:40.053267+0000","last_unstale":"2024-09-21T22:08:51.291117+0000","last_undegraded":"2024-09-21T22:08:51.291117+0000","last_fullsized":"2024-09-21T22:08:51.291117+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:05:04.598630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036861000000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":94,"num_read_kb":8213,"num_write":46,"num_write_kb":24581,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":33554508,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"104'117","reported_seq":1123,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-21T22:08:51.294765+0000","last_change":"2024-09-21T22:08:34.057726+0000","last_active":"2024-09-21T22:08:51.294765+0000","last_peered":"2024-09-21T22:08:51.294765+0000","last_clean":"2024-09-21T22:08:51.294765+0000","last_became_active":"2024-09-21T22:08:27.993129+0000","last_became_peered":"2024-09-21T22:08:27.993129+0000","last_unstale":"2024-09-21T22:08:51.294765+0000","last_undegraded":"2024-09-21T22:08:51.294765+0000","last_fullsized":"2024-09-21T22:08:51.294765+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T06:21:07.580957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00099297199999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":721,"num_read_kb":41715,"num_write":87,"num_write_kb":57411,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":50397184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"104'60","reported_seq":47,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-21T22:08:51.507719+0000","last_change":"2024-09-21T22:08:40.066003+0000","last_active":"2024-09-21T22:08:51.507719+0000","last_peered":"2024-09-21T22:08:51.507719+0000","last_clean":"2024-09-21T22:08:51.507719+0000","last_became_active":"2024-09-21T22:08:40.052140+0000","last_became_peered":"2024-09-21T22:08:40.052140+0000","last_unstale":"2024-09-21T22:08:51.507719+0000","last_undegraded":"2024-09-21T22:08:51.507719+0000","last_fullsized":"2024-09-21T22:08:51.507719+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T04:30:58.743113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011545399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":81,"num_read_kb":16395,"num_write":44,"num_write_kb":32769,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":41943040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"104'269","reported_seq":1794,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-21T22:08:51.291107+0000","last_change":"2024-09-21T22:08:34.054445+0000","last_active":"2024-09-21T22:08:51.291107+0000","last_peered":"2024-09-21T22:08:51.291107+0000","last_clean":"2024-09-21T22:08:51.291107+0000","last_became_active":"2024-09-21T22:08:31.007410+0000","last_became_peered":"2024-09-21T22:08:31.007410+0000","last_unstale":"2024-09-21T22:08:51.291107+0000","last_undegraded":"2024-09-21T22:08:51.291107+0000","last_fullsized":"2024-09-21T22:08:51.291107+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":269,"log_dups_size":0,"ondisk_log_size":269,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T09:57:16.015269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056494499999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4024,"num_read_kb":3421,"num_write":160,"num_write_kb":8272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":25166100,"num_keys_recovered":31,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"104'107","reported_seq":219,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-21T22:08:51.827176+0000","last_change":"2024-09-21T22:08:34.055487+0000","last_active":"2024-09-21T22:08:51.827176+0000","last_peered":"2024-09-21T22:08:51.827176+0000","last_clean":"2024-09-21T22:08:51.827176+0000","last_became_active":"2024-09-21T22:08:27.987130+0000","last_became_peered":"2024-09-21T22:08:27.987130+0000","last_unstale":"2024-09-21T22:08:51.827176+0000","last_undegraded":"2024-09-21T22:08:51.827176+0000","last_fullsized":"2024-09-21T22:08:51.827176+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:42:38.530832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002184208,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":551,"num_read_kb":24994,"num_write":73,"num_write_kb":32826,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":33603584,"num_keys_recovered":24,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"104'93","reported_seq":443,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-21T22:08:51.294904+0000","last_change":"2024-09-21T22:08:34.058368+0000","last_active":"2024-09-21T22:08:51.294904+0000","last_peered":"2024-09-21T22:08:51.294904+0000","last_clean":"2024-09-21T22:08:51.294904+0000","last_became_active":"2024-09-21T22:06:44.874697+0000","last_became_peered":"2024-09-21T22:06:44.874697+0000","last_unstale":"2024-09-21T22:08:51.294904+0000","last_undegraded":"2024-09-21T22:08:51.294904+0000","last_fullsized":"2024-09-21T22:08:51.294904+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T08:45:09.814190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015356479999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":231,"num_read_kb":16507,"num_write":94,"num_write_kb":28712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"104'64","reported_seq":119,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-21T22:08:51.827199+0000","last_change":"2024-09-21T22:08:34.055376+0000","last_active":"2024-09-21T22:08:51.827199+0000","last_peered":"2024-09-21T22:08:51.827199+0000","last_clean":"2024-09-21T22:08:51.827199+0000","last_became_active":"2024-09-21T22:08:27.983818+0000","last_became_peered":"2024-09-21T22:08:27.983818+0000","last_unstale":"2024-09-21T22:08:51.827199+0000","last_undegraded":"2024-09-21T22:08:51.827199+0000","last_fullsized":"2024-09-21T22:08:51.827199+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T06:39:39.444028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015450220000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":88,"num_read_kb":13397,"num_write":53,"num_write_kb":30748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":44105728,"num_keys_recovered":6,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"104'351","reported_seq":1415,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-21T22:08:51.294902+0000","last_change":"2024-09-21T22:08:34.057700+0000","last_active":"2024-09-21T22:08:51.294902+0000","last_peered":"2024-09-21T22:08:51.294902+0000","last_clean":"2024-09-21T22:08:51.294902+0000","last_became_active":"2024-09-21T22:06:05.220996+0000","last_became_peered":"2024-09-21T22:06:05.220996+0000","last_unstale":"2024-09-21T22:08:51.294902+0000","last_undegraded":"2024-09-21T22:08:51.294902+0000","last_fullsized":"2024-09-21T22:08:51.294902+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":351,"log_dups_size":0,"ondisk_log_size":351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T08:56:07.384975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095003999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1642,"num_read_kb":38437,"num_write":430,"num_write_kb":57640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7432,"num_read_kb":163079,"num_write":987,"num_write_kb":272959,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":228770144,"num_keys_recovered":61,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1174,"internal_metadata":0},"log_size":1123,"ondisk_log_size":1123,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":2361472,"data_compressed":28848,"data_compressed_allocated":147456,"data_compressed_original":2342912,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":65,"seq":279172874259,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29696,"kb_used_data":1384,"kb_used_omap":5,"kb_used_meta":28282,"kb_avail":94342144,"statfs":{"total":96636764160,"available":96606355456,"internally_reserved":0,"allocated":1417216,"data_stored":546933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5607,"internal_metadata":28961305},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640295,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32572,"kb_used_data":1380,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339268,"statfs":{"total":96636764160,"available":96603410432,"internally_reserved":0,"allocated":1413120,"data_stored":546914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640298,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32632,"kb_used_data":1436,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339208,"statfs":{"total":96636764160,"available":96603348992,"internally_reserved":0,"allocated":1470464,"data_stored":1137282,"data_compressed":6561,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":93,"seq":399431958538,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29092,"kb_used_data":1420,"kb_used_omap":4,"kb_used_meta":27643,"kb_avail":94342748,"statfs":{"total":96636764160,"available":96606973952,"internally_reserved":0,"allocated":1454080,"data_stored":1137282,"data_compressed":7429,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":4456,"internal_metadata":28307096},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640297,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36720,"kb_used_data":1372,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335120,"statfs":{"total":96636764160,"available":96599162880,"internally_reserved":0,"allocated":1404928,"data_stored":542326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640298,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36724,"kb_used_data":1368,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335116,"statfs":{"total":96636764160,"available":96599158784,"internally_reserved":0,"allocated":1400832,"data_stored":542307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":7429,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":7429,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":7429,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6561,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1174,"internal_metadata":0}]}} 2024-09-21T22:08:56.065 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-21T22:08:56.365 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T22:08:56.365 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":119,"fsid":"00bb09a3-e410-4e18-a2b2-08eaf61978e6","created":"2024-09-21T22:05:45.303341+0000","modified":"2024-09-21T22:08:51.282331+0000","last_up_change":"2024-09-21T22:08:18.467986+0000","last_in_change":"2024-09-21T22:08:39.090072+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T22:06:00.868591+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-21T22:06:03.535857+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"104","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":7,"snap_epoch":104,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"8336c397-2018-4c75-ab31-2e5f6ad564b5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":3054061468},{"type":"v1","addr":"172.21.15.39:6801","nonce":3054061468}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":3054061468},{"type":"v1","addr":"172.21.15.39:6803","nonce":3054061468}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":3054061468},{"type":"v1","addr":"172.21.15.39:6810","nonce":3054061468}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":3054061468},{"type":"v1","addr":"172.21.15.39:6806","nonce":3054061468}]},"public_addr":"172.21.15.39:6801/3054061468","cluster_addr":"172.21.15.39:6803/3054061468","heartbeat_back_addr":"172.21.15.39:6810/3054061468","heartbeat_front_addr":"172.21.15.39:6806/3054061468","state":["exists","up"]},{"osd":1,"uuid":"bca64af6-7427-4c5d-97a4-a183b1f3a7d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":2876495139},{"type":"v1","addr":"172.21.15.39:6817","nonce":2876495139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":2876495139},{"type":"v1","addr":"172.21.15.39:6819","nonce":2876495139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":2876495139},{"type":"v1","addr":"172.21.15.39:6823","nonce":2876495139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":2876495139},{"type":"v1","addr":"172.21.15.39:6821","nonce":2876495139}]},"public_addr":"172.21.15.39:6817/2876495139","cluster_addr":"172.21.15.39:6819/2876495139","heartbeat_back_addr":"172.21.15.39:6823/2876495139","heartbeat_front_addr":"172.21.15.39:6821/2876495139","state":["exists","up"]},{"osd":2,"uuid":"5f2678ed-6a23-4a67-96a4-c0ca91254bd7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":11,"last_clean_end":80,"up_from":93,"up_thru":110,"down_at":81,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6805","nonce":1877094708},{"type":"v1","addr":"172.21.15.39:6807","nonce":1877094708}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6809","nonce":1877094708},{"type":"v1","addr":"172.21.15.39:6811","nonce":1877094708}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":1877094708},{"type":"v1","addr":"172.21.15.39:6815","nonce":1877094708}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":1877094708},{"type":"v1","addr":"172.21.15.39:6813","nonce":1877094708}]},"public_addr":"172.21.15.39:6807/1877094708","cluster_addr":"172.21.15.39:6811/1877094708","heartbeat_back_addr":"172.21.15.39:6815/1877094708","heartbeat_front_addr":"172.21.15.39:6813/1877094708","state":["exists","up"]},{"osd":3,"uuid":"aba30555-ef9e-47c9-a38b-ee1a4b8092f5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":113,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":1132198485},{"type":"v1","addr":"172.21.15.64:6801","nonce":1132198485}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":1132198485},{"type":"v1","addr":"172.21.15.64:6803","nonce":1132198485}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":1132198485},{"type":"v1","addr":"172.21.15.64:6807","nonce":1132198485}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":1132198485},{"type":"v1","addr":"172.21.15.64:6805","nonce":1132198485}]},"public_addr":"172.21.15.64:6801/1132198485","cluster_addr":"172.21.15.64:6803/1132198485","heartbeat_back_addr":"172.21.15.64:6807/1132198485","heartbeat_front_addr":"172.21.15.64:6805/1132198485","state":["exists","up"]},{"osd":4,"uuid":"2a1288d7-618d-4f44-8346-930cf56d4e80","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":3500786459},{"type":"v1","addr":"172.21.15.64:6809","nonce":3500786459}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":3500786459},{"type":"v1","addr":"172.21.15.64:6811","nonce":3500786459}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":3500786459},{"type":"v1","addr":"172.21.15.64:6815","nonce":3500786459}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":3500786459},{"type":"v1","addr":"172.21.15.64:6813","nonce":3500786459}]},"public_addr":"172.21.15.64:6809/3500786459","cluster_addr":"172.21.15.64:6811/3500786459","heartbeat_back_addr":"172.21.15.64:6815/3500786459","heartbeat_front_addr":"172.21.15.64:6813/3500786459","state":["exists","up"]},{"osd":5,"uuid":"3ce719a8-34e9-4890-bd40-d22201e7bad8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":11,"last_clean_end":41,"up_from":65,"up_thru":107,"down_at":42,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":1547734524},{"type":"v1","addr":"172.21.15.64:6817","nonce":1547734524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":1547734524},{"type":"v1","addr":"172.21.15.64:6819","nonce":1547734524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":1547734524},{"type":"v1","addr":"172.21.15.64:6823","nonce":1547734524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":1547734524},{"type":"v1","addr":"172.21.15.64:6821","nonce":1547734524}]},"public_addr":"172.21.15.64:6817/1547734524","cluster_addr":"172.21.15.64:6819/1547734524","heartbeat_back_addr":"172.21.15.64:6823/1547734524","heartbeat_front_addr":"172.21.15.64:6821/1547734524","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T22:05:54.224881+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T22:05:54.668978+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T22:05:54.315876+0000","dead_epoch":81},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T22:05:54.314795+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T22:05:54.322403+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T22:05:54.750633+0000","dead_epoch":42}],"pg_upmap":[{"pgid":"1.0","osds":[3,2]}],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T22:08:57.384 INFO:tasks.ceph:Scrubbing osd.0 2024-09-21T22:08:57.384 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-21T22:08:57.548 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:08:57.548 INFO:teuthology.orchestra.run.smithi039.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-21T22:08:57.549 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:08:57.565 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-21T22:08:57.846 INFO:teuthology.orchestra.run.smithi039.stderr:instructed osd(s) 0 to deep-scrub 2024-09-21T22:08:57.863 INFO:tasks.ceph:Scrubbing osd.1 2024-09-21T22:08:57.863 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-21T22:08:58.002 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:08:58.002 INFO:teuthology.orchestra.run.smithi039.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-21T22:08:58.002 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:08:58.017 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-21T22:08:58.303 INFO:teuthology.orchestra.run.smithi039.stderr:instructed osd(s) 1 to deep-scrub 2024-09-21T22:08:58.320 INFO:tasks.ceph:Scrubbing osd.2 2024-09-21T22:08:58.321 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-21T22:08:58.461 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:08:58.462 INFO:teuthology.orchestra.run.smithi039.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-21T22:08:58.462 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:08:58.477 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-21T22:08:58.756 INFO:teuthology.orchestra.run.smithi039.stderr:instructed osd(s) 2 to deep-scrub 2024-09-21T22:08:58.773 INFO:tasks.ceph:Scrubbing osd.3 2024-09-21T22:08:58.774 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-09-21T22:08:58.916 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:08:58.916 INFO:teuthology.orchestra.run.smithi039.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-21T22:08:58.917 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:08:58.931 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-09-21T22:08:59.214 INFO:teuthology.orchestra.run.smithi039.stderr:instructed osd(s) 3 to deep-scrub 2024-09-21T22:08:59.233 INFO:tasks.ceph:Scrubbing osd.4 2024-09-21T22:08:59.233 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-09-21T22:08:59.382 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:08:59.382 INFO:teuthology.orchestra.run.smithi039.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-21T22:08:59.382 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:08:59.397 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-09-21T22:08:59.683 INFO:teuthology.orchestra.run.smithi039.stderr:instructed osd(s) 4 to deep-scrub 2024-09-21T22:08:59.701 INFO:tasks.ceph:Scrubbing osd.5 2024-09-21T22:08:59.701 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-09-21T22:08:59.853 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-09-21T22:08:59.853 INFO:teuthology.orchestra.run.smithi039.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-21T22:08:59.853 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-09-21T22:08:59.869 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-09-21T22:09:00.145 INFO:teuthology.orchestra.run.smithi039.stderr:instructed osd(s) 5 to deep-scrub 2024-09-21T22:09:00.163 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T22:09:00.465 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T22:09:00.465 INFO:teuthology.orchestra.run.smithi039.stderr:dumped all 2024-09-21T22:09:00.483 INFO:teuthology.orchestra.run.smithi039.stdout:{"pg_ready":true,"pg_map":{"version":204,"stamp":"2024-09-21T22:08:58.834271+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":7548,"num_read_kb":163179,"num_write":1146,"num_write_kb":275375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":231131616,"num_keys_recovered":61,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1240,"ondisk_log_size":1240,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":197456,"kb_used_data":8384,"kb_used_omap":16,"kb_used_meta":188911,"kb_avail":566033584,"statfs":{"total":579820584960,"available":579618390016,"internally_reserved":0,"allocated":8585216,"data_stored":4462258,"data_compressed":13990,"data_compressed_allocated":73728,"data_compressed_original":1163264,"omap_allocated":16441,"internal_metadata":193445831},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.566613"},"pg_stats":[{"pgid":"1.0","version":"15'117","reported_seq":406,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-21T22:08:51.294841+0000","last_change":"2024-09-21T22:08:46.212886+0000","last_active":"2024-09-21T22:08:51.294841+0000","last_peered":"2024-09-21T22:08:51.294841+0000","last_clean":"2024-09-21T22:08:51.294841+0000","last_became_active":"2024-09-21T22:08:46.212212+0000","last_became_peered":"2024-09-21T22:08:46.212212+0000","last_unstale":"2024-09-21T22:08:51.294841+0000","last_undegraded":"2024-09-21T22:08:51.294841+0000","last_fullsized":"2024-09-21T22:08:51.294841+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:00.990331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:00.990331+0000","last_clean_scrub_stamp":"2024-09-21T22:06:00.990331+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T05:09:41.935962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"103'62","reported_seq":286,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-21T22:08:51.291117+0000","last_change":"2024-09-21T22:08:40.076840+0000","last_active":"2024-09-21T22:08:51.291117+0000","last_peered":"2024-09-21T22:08:51.291117+0000","last_clean":"2024-09-21T22:08:51.291117+0000","last_became_active":"2024-09-21T22:08:40.053267+0000","last_became_peered":"2024-09-21T22:08:40.053267+0000","last_unstale":"2024-09-21T22:08:51.291117+0000","last_undegraded":"2024-09-21T22:08:51.291117+0000","last_fullsized":"2024-09-21T22:08:51.291117+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:05:04.598630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036861000000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":94,"num_read_kb":8213,"num_write":46,"num_write_kb":24581,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":33554508,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"104'117","reported_seq":1123,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-21T22:08:51.294765+0000","last_change":"2024-09-21T22:08:34.057726+0000","last_active":"2024-09-21T22:08:51.294765+0000","last_peered":"2024-09-21T22:08:51.294765+0000","last_clean":"2024-09-21T22:08:51.294765+0000","last_became_active":"2024-09-21T22:08:27.993129+0000","last_became_peered":"2024-09-21T22:08:27.993129+0000","last_unstale":"2024-09-21T22:08:51.294765+0000","last_undegraded":"2024-09-21T22:08:51.294765+0000","last_fullsized":"2024-09-21T22:08:51.294765+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T06:21:07.580957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00099297199999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":721,"num_read_kb":41715,"num_write":87,"num_write_kb":57411,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":50397184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"104'60","reported_seq":47,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-21T22:08:51.507719+0000","last_change":"2024-09-21T22:08:40.066003+0000","last_active":"2024-09-21T22:08:51.507719+0000","last_peered":"2024-09-21T22:08:51.507719+0000","last_clean":"2024-09-21T22:08:51.507719+0000","last_became_active":"2024-09-21T22:08:40.052140+0000","last_became_peered":"2024-09-21T22:08:40.052140+0000","last_unstale":"2024-09-21T22:08:51.507719+0000","last_undegraded":"2024-09-21T22:08:51.507719+0000","last_fullsized":"2024-09-21T22:08:51.507719+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T04:30:58.743113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011545399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":81,"num_read_kb":16395,"num_write":44,"num_write_kb":32769,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":41943040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"104'269","reported_seq":1794,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-21T22:08:51.291107+0000","last_change":"2024-09-21T22:08:34.054445+0000","last_active":"2024-09-21T22:08:51.291107+0000","last_peered":"2024-09-21T22:08:51.291107+0000","last_clean":"2024-09-21T22:08:51.291107+0000","last_became_active":"2024-09-21T22:08:31.007410+0000","last_became_peered":"2024-09-21T22:08:31.007410+0000","last_unstale":"2024-09-21T22:08:51.291107+0000","last_undegraded":"2024-09-21T22:08:51.291107+0000","last_fullsized":"2024-09-21T22:08:51.291107+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":269,"log_dups_size":0,"ondisk_log_size":269,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T09:57:16.015269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056494499999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4024,"num_read_kb":3421,"num_write":160,"num_write_kb":8272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":25166100,"num_keys_recovered":31,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"104'107","reported_seq":219,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-21T22:08:51.827176+0000","last_change":"2024-09-21T22:08:34.055487+0000","last_active":"2024-09-21T22:08:51.827176+0000","last_peered":"2024-09-21T22:08:51.827176+0000","last_clean":"2024-09-21T22:08:51.827176+0000","last_became_active":"2024-09-21T22:08:27.987130+0000","last_became_peered":"2024-09-21T22:08:27.987130+0000","last_unstale":"2024-09-21T22:08:51.827176+0000","last_undegraded":"2024-09-21T22:08:51.827176+0000","last_fullsized":"2024-09-21T22:08:51.827176+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:42:38.530832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002184208,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":551,"num_read_kb":24994,"num_write":73,"num_write_kb":32826,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":33603584,"num_keys_recovered":24,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"104'93","reported_seq":443,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-21T22:08:51.294904+0000","last_change":"2024-09-21T22:08:34.058368+0000","last_active":"2024-09-21T22:08:51.294904+0000","last_peered":"2024-09-21T22:08:51.294904+0000","last_clean":"2024-09-21T22:08:51.294904+0000","last_became_active":"2024-09-21T22:06:44.874697+0000","last_became_peered":"2024-09-21T22:06:44.874697+0000","last_unstale":"2024-09-21T22:08:51.294904+0000","last_undegraded":"2024-09-21T22:08:51.294904+0000","last_fullsized":"2024-09-21T22:08:51.294904+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T08:45:09.814190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015356479999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":231,"num_read_kb":16507,"num_write":94,"num_write_kb":28712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"104'64","reported_seq":119,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-21T22:08:51.827199+0000","last_change":"2024-09-21T22:08:34.055376+0000","last_active":"2024-09-21T22:08:51.827199+0000","last_peered":"2024-09-21T22:08:51.827199+0000","last_clean":"2024-09-21T22:08:51.827199+0000","last_became_active":"2024-09-21T22:08:27.983818+0000","last_became_peered":"2024-09-21T22:08:27.983818+0000","last_unstale":"2024-09-21T22:08:51.827199+0000","last_undegraded":"2024-09-21T22:08:51.827199+0000","last_fullsized":"2024-09-21T22:08:51.827199+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T06:39:39.444028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015450220000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":88,"num_read_kb":13397,"num_write":53,"num_write_kb":30748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":44105728,"num_keys_recovered":6,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"104'351","reported_seq":1415,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-21T22:08:51.294902+0000","last_change":"2024-09-21T22:08:34.057700+0000","last_active":"2024-09-21T22:08:51.294902+0000","last_peered":"2024-09-21T22:08:51.294902+0000","last_clean":"2024-09-21T22:08:51.294902+0000","last_became_active":"2024-09-21T22:06:05.220996+0000","last_became_peered":"2024-09-21T22:06:05.220996+0000","last_unstale":"2024-09-21T22:08:51.294902+0000","last_undegraded":"2024-09-21T22:08:51.294902+0000","last_fullsized":"2024-09-21T22:08:51.294902+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T22:06:04.058935+0000","last_clean_scrub_stamp":"2024-09-21T22:06:04.058935+0000","objects_scrubbed":0,"log_size":351,"log_dups_size":0,"ondisk_log_size":351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T08:56:07.384975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095003999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1642,"num_read_kb":38437,"num_write":430,"num_write_kb":57640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7432,"num_read_kb":163079,"num_write":987,"num_write_kb":272959,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":228770144,"num_keys_recovered":61,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1174,"internal_metadata":0},"log_size":1123,"ondisk_log_size":1123,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":2361472,"data_compressed":28848,"data_compressed_allocated":147456,"data_compressed_original":2342912,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":65,"seq":279172874260,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29696,"kb_used_data":1384,"kb_used_omap":5,"kb_used_meta":28282,"kb_avail":94342144,"statfs":{"total":96636764160,"available":96606355456,"internally_reserved":0,"allocated":1417216,"data_stored":546933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5607,"internal_metadata":28961305},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640296,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32572,"kb_used_data":1380,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339268,"statfs":{"total":96636764160,"available":96603410432,"internally_reserved":0,"allocated":1413120,"data_stored":546914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640299,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32632,"kb_used_data":1436,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339208,"statfs":{"total":96636764160,"available":96603348992,"internally_reserved":0,"allocated":1470464,"data_stored":1137282,"data_compressed":6561,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":93,"seq":399431958539,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29092,"kb_used_data":1420,"kb_used_omap":4,"kb_used_meta":27643,"kb_avail":94342748,"statfs":{"total":96636764160,"available":96606973952,"internally_reserved":0,"allocated":1454080,"data_stored":1137282,"data_compressed":7429,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":4456,"internal_metadata":28307096},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640298,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36736,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335104,"statfs":{"total":96636764160,"available":96599146496,"internally_reserved":0,"allocated":1417216,"data_stored":546933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640299,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36728,"kb_used_data":1380,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335112,"statfs":{"total":96636764160,"available":96599154688,"internally_reserved":0,"allocated":1413120,"data_stored":546914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":7429,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":7429,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":7429,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6561,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1174,"internal_metadata":0}]}} 2024-09-21T22:09:00.486 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-21T22:06:00.990331+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=22, tm_min=6, tm_sec=0, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=22, tm_min=8, tm_sec=56, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T22:09:00.486 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-21T22:06:04.058935+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=22, tm_min=6, tm_sec=4, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=22, tm_min=8, tm_sec=56, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T22:09:00.486 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-21T22:06:04.058935+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=22, tm_min=6, tm_sec=4, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=22, tm_min=8, tm_sec=56, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T22:09:00.486 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-21T22:06:04.058935+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=22, tm_min=6, tm_sec=4, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=22, tm_min=8, tm_sec=56, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T22:09:00.486 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-21T22:06:04.058935+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=22, tm_min=6, tm_sec=4, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=22, tm_min=8, tm_sec=56, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T22:09:00.487 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-21T22:06:04.058935+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=22, tm_min=6, tm_sec=4, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=22, tm_min=8, tm_sec=56, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T22:09:00.487 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-21T22:06:04.058935+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=22, tm_min=6, tm_sec=4, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=22, tm_min=8, tm_sec=56, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T22:09:00.487 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-21T22:06:04.058935+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=22, tm_min=6, tm_sec=4, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=22, tm_min=8, tm_sec=56, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T22:09:00.487 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-21T22:06:04.058935+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=22, tm_min=6, tm_sec=4, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=22, tm_min=8, tm_sec=56, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T22:09:00.487 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-21T22:09:20.488 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T22:09:20.803 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-09-21T22:09:20.804 INFO:teuthology.orchestra.run.smithi039.stderr:dumped all 2024-09-21T22:09:20.822 INFO:teuthology.orchestra.run.smithi039.stdout:{"pg_ready":true,"pg_map":{"version":214,"stamp":"2024-09-21T22:09:18.839540+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":7548,"num_read_kb":163179,"num_write":1146,"num_write_kb":275375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":122,"num_bytes_recovered":231131616,"num_keys_recovered":61,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1243,"ondisk_log_size":1243,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":197416,"kb_used_data":8384,"kb_used_omap":16,"kb_used_meta":188911,"kb_avail":566033624,"statfs":{"total":579820584960,"available":579618430976,"internally_reserved":0,"allocated":8585216,"data_stored":4462258,"data_compressed":13990,"data_compressed_allocated":73728,"data_compressed_original":1163264,"omap_allocated":16441,"internal_metadata":193445831},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003207"},"pg_stats":[{"pgid":"1.0","version":"119'119","reported_seq":417,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-21T22:09:03.018812+0000","last_change":"2024-09-21T22:09:03.018889+0000","last_active":"2024-09-21T22:09:03.018812+0000","last_peered":"2024-09-21T22:09:03.018812+0000","last_clean":"2024-09-21T22:09:03.018812+0000","last_became_active":"2024-09-21T22:08:46.212212+0000","last_became_peered":"2024-09-21T22:08:46.212212+0000","last_unstale":"2024-09-21T22:09:03.018812+0000","last_undegraded":"2024-09-21T22:09:03.018812+0000","last_fullsized":"2024-09-21T22:09:03.018812+0000","mapping_epoch":113,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":114,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'119","last_scrub_stamp":"2024-09-21T22:09:03.018617+0000","last_deep_scrub":"119'119","last_deep_scrub_stamp":"2024-09-21T22:09:03.018617+0000","last_clean_scrub_stamp":"2024-09-21T22:09:03.018617+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T07:10:33.227497+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"103'62","reported_seq":295,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-21T22:09:01.728883+0000","last_change":"2024-09-21T22:09:01.728945+0000","last_active":"2024-09-21T22:09:01.728883+0000","last_peered":"2024-09-21T22:09:01.728883+0000","last_clean":"2024-09-21T22:09:01.728883+0000","last_became_active":"2024-09-21T22:08:40.053267+0000","last_became_peered":"2024-09-21T22:08:40.053267+0000","last_unstale":"2024-09-21T22:09:01.728883+0000","last_undegraded":"2024-09-21T22:09:01.728883+0000","last_fullsized":"2024-09-21T22:09:01.728883+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'62","last_scrub_stamp":"2024-09-21T22:09:01.728658+0000","last_deep_scrub":"103'62","last_deep_scrub_stamp":"2024-09-21T22:09:01.728658+0000","last_clean_scrub_stamp":"2024-09-21T22:09:01.728658+0000","objects_scrubbed":1,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T23:05:10.600530+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00036861000000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":94,"num_read_kb":8213,"num_write":46,"num_write_kb":24581,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":33554508,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"104'117","reported_seq":1132,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-21T22:09:01.988030+0000","last_change":"2024-09-21T22:09:01.988217+0000","last_active":"2024-09-21T22:09:01.988030+0000","last_peered":"2024-09-21T22:09:01.988030+0000","last_clean":"2024-09-21T22:09:01.988030+0000","last_became_active":"2024-09-21T22:08:27.993129+0000","last_became_peered":"2024-09-21T22:08:27.993129+0000","last_unstale":"2024-09-21T22:09:01.988030+0000","last_undegraded":"2024-09-21T22:09:01.988030+0000","last_fullsized":"2024-09-21T22:09:01.988030+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'117","last_scrub_stamp":"2024-09-21T22:09:01.987791+0000","last_deep_scrub":"104'117","last_deep_scrub_stamp":"2024-09-21T22:09:01.987791+0000","last_clean_scrub_stamp":"2024-09-21T22:09:01.987791+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T04:06:54.997924+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00099297199999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":721,"num_read_kb":41715,"num_write":87,"num_write_kb":57411,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":50397184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"104'60","reported_seq":56,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-21T22:08:59.731242+0000","last_change":"2024-09-21T22:08:59.731274+0000","last_active":"2024-09-21T22:08:59.731242+0000","last_peered":"2024-09-21T22:08:59.731242+0000","last_clean":"2024-09-21T22:08:59.731242+0000","last_became_active":"2024-09-21T22:08:40.052140+0000","last_became_peered":"2024-09-21T22:08:40.052140+0000","last_unstale":"2024-09-21T22:08:59.731242+0000","last_undegraded":"2024-09-21T22:08:59.731242+0000","last_fullsized":"2024-09-21T22:08:59.731242+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'60","last_scrub_stamp":"2024-09-21T22:08:59.731140+0000","last_deep_scrub":"104'60","last_deep_scrub_stamp":"2024-09-21T22:08:59.731140+0000","last_clean_scrub_stamp":"2024-09-21T22:08:59.731140+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T07:56:12.177491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011545399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":81,"num_read_kb":16395,"num_write":44,"num_write_kb":32769,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":41943040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"119'270","reported_seq":1804,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-21T22:09:00.765213+0000","last_change":"2024-09-21T22:09:00.765401+0000","last_active":"2024-09-21T22:09:00.765213+0000","last_peered":"2024-09-21T22:09:00.765213+0000","last_clean":"2024-09-21T22:09:00.765213+0000","last_became_active":"2024-09-21T22:08:31.007410+0000","last_became_peered":"2024-09-21T22:08:31.007410+0000","last_unstale":"2024-09-21T22:09:00.765213+0000","last_undegraded":"2024-09-21T22:09:00.765213+0000","last_fullsized":"2024-09-21T22:09:00.765213+0000","mapping_epoch":100,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":101,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'270","last_scrub_stamp":"2024-09-21T22:09:00.764970+0000","last_deep_scrub":"119'270","last_deep_scrub_stamp":"2024-09-21T22:09:00.764970+0000","last_clean_scrub_stamp":"2024-09-21T22:09:00.764970+0000","objects_scrubbed":1,"log_size":270,"log_dups_size":0,"ondisk_log_size":270,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T09:04:37.430763+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.00056494499999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4024,"num_read_kb":3421,"num_write":160,"num_write_kb":8272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":25166100,"num_keys_recovered":31,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"104'107","reported_seq":228,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-21T22:08:59.526238+0000","last_change":"2024-09-21T22:08:59.526264+0000","last_active":"2024-09-21T22:08:59.526238+0000","last_peered":"2024-09-21T22:08:59.526238+0000","last_clean":"2024-09-21T22:08:59.526238+0000","last_became_active":"2024-09-21T22:08:27.987130+0000","last_became_peered":"2024-09-21T22:08:27.987130+0000","last_unstale":"2024-09-21T22:08:59.526238+0000","last_undegraded":"2024-09-21T22:08:59.526238+0000","last_fullsized":"2024-09-21T22:08:59.526238+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":99,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'107","last_scrub_stamp":"2024-09-21T22:08:59.526122+0000","last_deep_scrub":"104'107","last_deep_scrub_stamp":"2024-09-21T22:08:59.526122+0000","last_clean_scrub_stamp":"2024-09-21T22:08:59.526122+0000","objects_scrubbed":2,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:30:33.910678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002184208,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":551,"num_read_kb":24994,"num_write":73,"num_write_kb":32826,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":33603584,"num_keys_recovered":24,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"104'93","reported_seq":452,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-21T22:09:00.974125+0000","last_change":"2024-09-21T22:09:00.974269+0000","last_active":"2024-09-21T22:09:00.974125+0000","last_peered":"2024-09-21T22:09:00.974125+0000","last_clean":"2024-09-21T22:09:00.974125+0000","last_became_active":"2024-09-21T22:06:44.874697+0000","last_became_peered":"2024-09-21T22:06:44.874697+0000","last_unstale":"2024-09-21T22:09:00.974125+0000","last_undegraded":"2024-09-21T22:09:00.974125+0000","last_fullsized":"2024-09-21T22:09:00.974125+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'93","last_scrub_stamp":"2024-09-21T22:09:00.973888+0000","last_deep_scrub":"104'93","last_deep_scrub_stamp":"2024-09-21T22:09:00.973888+0000","last_clean_scrub_stamp":"2024-09-21T22:09:00.973888+0000","objects_scrubbed":0,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T06:07:58.066591+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0015356479999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":231,"num_read_kb":16507,"num_write":94,"num_write_kb":28712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"104'64","reported_seq":128,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-21T22:08:58.507992+0000","last_change":"2024-09-21T22:08:58.508041+0000","last_active":"2024-09-21T22:08:58.507992+0000","last_peered":"2024-09-21T22:08:58.507992+0000","last_clean":"2024-09-21T22:08:58.507992+0000","last_became_active":"2024-09-21T22:08:27.983818+0000","last_became_peered":"2024-09-21T22:08:27.983818+0000","last_unstale":"2024-09-21T22:08:58.507992+0000","last_undegraded":"2024-09-21T22:08:58.507992+0000","last_fullsized":"2024-09-21T22:08:58.507992+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'64","last_scrub_stamp":"2024-09-21T22:08:58.507859+0000","last_deep_scrub":"104'64","last_deep_scrub_stamp":"2024-09-21T22:08:58.507859+0000","last_clean_scrub_stamp":"2024-09-21T22:08:58.507859+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:41:35.301154+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0015450220000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":88,"num_read_kb":13397,"num_write":53,"num_write_kb":30748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":44105728,"num_keys_recovered":6,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"104'351","reported_seq":1424,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-21T22:09:00.005611+0000","last_change":"2024-09-21T22:09:00.005673+0000","last_active":"2024-09-21T22:09:00.005611+0000","last_peered":"2024-09-21T22:09:00.005611+0000","last_clean":"2024-09-21T22:09:00.005611+0000","last_became_active":"2024-09-21T22:06:05.220996+0000","last_became_peered":"2024-09-21T22:06:05.220996+0000","last_unstale":"2024-09-21T22:09:00.005611+0000","last_undegraded":"2024-09-21T22:09:00.005611+0000","last_fullsized":"2024-09-21T22:09:00.005611+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'351","last_scrub_stamp":"2024-09-21T22:09:00.005373+0000","last_deep_scrub":"104'351","last_deep_scrub_stamp":"2024-09-21T22:09:00.005373+0000","last_clean_scrub_stamp":"2024-09-21T22:09:00.005373+0000","objects_scrubbed":0,"log_size":351,"log_dups_size":0,"ondisk_log_size":351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:34:31.584010+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00095003999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1642,"num_read_kb":38437,"num_write":430,"num_write_kb":57640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7432,"num_read_kb":163079,"num_write":987,"num_write_kb":272959,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":114,"num_bytes_recovered":228770144,"num_keys_recovered":61,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1174,"internal_metadata":0},"log_size":1124,"ondisk_log_size":1124,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":2361472,"data_compressed":28848,"data_compressed_allocated":147456,"data_compressed_original":2342912,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":65,"seq":279172874264,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29688,"kb_used_data":1384,"kb_used_omap":5,"kb_used_meta":28282,"kb_avail":94342152,"statfs":{"total":96636764160,"available":96606363648,"internally_reserved":0,"allocated":1417216,"data_stored":546933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5607,"internal_metadata":28961305},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640300,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32572,"kb_used_data":1380,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339268,"statfs":{"total":96636764160,"available":96603410432,"internally_reserved":0,"allocated":1413120,"data_stored":546914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640303,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32620,"kb_used_data":1436,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339220,"statfs":{"total":96636764160,"available":96603361280,"internally_reserved":0,"allocated":1470464,"data_stored":1137282,"data_compressed":6561,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":93,"seq":399431958543,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29084,"kb_used_data":1420,"kb_used_omap":4,"kb_used_meta":27643,"kb_avail":94342756,"statfs":{"total":96636764160,"available":96606982144,"internally_reserved":0,"allocated":1454080,"data_stored":1137282,"data_compressed":7429,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":4456,"internal_metadata":28307096},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640302,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36728,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335112,"statfs":{"total":96636764160,"available":96599154688,"internally_reserved":0,"allocated":1417216,"data_stored":546933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":11,"seq":47244640303,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36724,"kb_used_data":1380,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335116,"statfs":{"total":96636764160,"available":96599158784,"internally_reserved":0,"allocated":1413120,"data_stored":546914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":7429,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":7429,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":590368,"data_compressed":7429,"data_compressed_allocated":36864,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":590368,"data_compressed":6561,"data_compressed_allocated":36864,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1174,"internal_metadata":0}]}} 2024-09-21T22:09:20.823 DEBUG:teuthology.orchestra.run.smithi039:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-21T22:09:21.158 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-21T22:09:21.159 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-21T22:09:21.159 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-21T22:09:21.159 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T22:09:21.259 INFO:tasks.ceph.osd.0:Stopped 2024-09-21T22:09:21.259 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-21T22:09:21.260 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T22:09:21.339 INFO:tasks.ceph.osd.1:Stopped 2024-09-21T22:09:21.339 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-21T22:09:21.340 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T22:09:21.413 INFO:tasks.ceph.osd.2:Stopped 2024-09-21T22:09:21.413 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-21T22:09:21.413 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T22:09:21.508 INFO:tasks.ceph.osd.3:Stopped 2024-09-21T22:09:21.508 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-09-21T22:09:21.508 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T22:09:26.068 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~0s 2024-09-21T22:09:27.610 INFO:tasks.ceph.osd.4:Stopped 2024-09-21T22:09:27.611 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-09-21T22:09:27.611 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T22:09:27.680 INFO:tasks.ceph.osd.5:Stopped 2024-09-21T22:09:27.680 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-21T22:09:27.680 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-21T22:09:27.680 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T22:09:27.711 INFO:tasks.ceph.mgr.x:Stopped 2024-09-21T22:09:27.711 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-21T22:09:27.711 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-21T22:09:27.712 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T22:09:27.735 INFO:tasks.ceph.mon.a:Stopped 2024-09-21T22:09:27.736 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-21T22:09:27.736 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T22:09:27.803 INFO:tasks.ceph.mon.c:Stopped 2024-09-21T22:09:27.872 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-21T22:09:27.872 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T22:09:27.892 INFO:tasks.ceph.mon.b:Stopped 2024-09-21T22:09:27.892 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-21T22:09:27.893 DEBUG:teuthology.orchestra.run.smithi039:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v 'out of quorum' | egrep -v noscrub | egrep -v nodeep-scrub | head -n 1 2024-09-21T22:09:27.912 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi039.front.sepia.ceph.com 2024-09-21T22:09:27.912 DEBUG:teuthology.orchestra.run.smithi039:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-21T22:09:28.133 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi039.front.sepia.ceph.com 2024-09-21T22:09:28.133 DEBUG:teuthology.orchestra.run.smithi039:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-21T22:09:28.280 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi039.front.sepia.ceph.com 2024-09-21T22:09:28.280 DEBUG:teuthology.orchestra.run.smithi039:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-21T22:09:28.394 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi064.front.sepia.ceph.com 2024-09-21T22:09:28.394 DEBUG:teuthology.orchestra.run.smithi064:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-09-21T22:09:28.641 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi064.front.sepia.ceph.com 2024-09-21T22:09:28.641 DEBUG:teuthology.orchestra.run.smithi064:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-09-21T22:09:28.855 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi064.front.sepia.ceph.com 2024-09-21T22:09:28.856 DEBUG:teuthology.orchestra.run.smithi064:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-09-21T22:09:29.132 INFO:tasks.ceph:Archiving mon data... 2024-09-21T22:09:29.133 DEBUG:teuthology.misc:Transferring archived files from smithi039:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/teuthology-2024-09-21_21:16:02-rbd-squid-distro-default-smithi/7914449/data/mon.a.tgz 2024-09-21T22:09:29.133 DEBUG:teuthology.orchestra.run.smithi039:> mktemp 2024-09-21T22:09:29.139 INFO:teuthology.orchestra.run.smithi039.stdout:/tmp/tmp.plXelgPxLA 2024-09-21T22:09:29.139 DEBUG:teuthology.orchestra.run.smithi039:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.plXelgPxLA 2024-09-21T22:09:29.266 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod 0666 /tmp/tmp.plXelgPxLA 2024-09-21T22:09:29.332 DEBUG:teuthology.orchestra.remote:smithi039:/tmp/tmp.plXelgPxLA is 288KB 2024-09-21T22:09:29.393 DEBUG:teuthology.orchestra.run.smithi039:> rm -fr /tmp/tmp.plXelgPxLA 2024-09-21T22:09:29.398 DEBUG:teuthology.misc:Transferring archived files from smithi039:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/teuthology-2024-09-21_21:16:02-rbd-squid-distro-default-smithi/7914449/data/mon.c.tgz 2024-09-21T22:09:29.399 DEBUG:teuthology.orchestra.run.smithi039:> mktemp 2024-09-21T22:09:29.449 INFO:teuthology.orchestra.run.smithi039.stdout:/tmp/tmp.vNCvzcFVQN 2024-09-21T22:09:29.449 DEBUG:teuthology.orchestra.run.smithi039:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.vNCvzcFVQN 2024-09-21T22:09:29.600 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod 0666 /tmp/tmp.vNCvzcFVQN 2024-09-21T22:09:29.663 DEBUG:teuthology.orchestra.remote:smithi039:/tmp/tmp.vNCvzcFVQN is 400KB 2024-09-21T22:09:29.725 DEBUG:teuthology.orchestra.run.smithi039:> rm -fr /tmp/tmp.vNCvzcFVQN 2024-09-21T22:09:29.731 DEBUG:teuthology.misc:Transferring archived files from smithi064:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/teuthology-2024-09-21_21:16:02-rbd-squid-distro-default-smithi/7914449/data/mon.b.tgz 2024-09-21T22:09:29.731 DEBUG:teuthology.orchestra.run.smithi064:> mktemp 2024-09-21T22:09:29.737 INFO:teuthology.orchestra.run.smithi064.stdout:/tmp/tmp.jercVlaezE 2024-09-21T22:09:29.737 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.jercVlaezE 2024-09-21T22:09:29.882 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 0666 /tmp/tmp.jercVlaezE 2024-09-21T22:09:29.945 DEBUG:teuthology.orchestra.remote:smithi064:/tmp/tmp.jercVlaezE is 400KB 2024-09-21T22:09:30.003 DEBUG:teuthology.orchestra.run.smithi064:> rm -fr /tmp/tmp.jercVlaezE 2024-09-21T22:09:30.007 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-21T22:09:30.008 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-21T22:09:30.010 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-21T22:09:30.050 DEBUG:teuthology.orchestra.run.smithi157:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-21T22:09:30.071 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-21T22:09:30.105 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-21T22:09:30.114 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-21T22:09:30.114 INFO:tasks.ceph:Archiving crash dumps... 2024-09-21T22:09:30.116 DEBUG:teuthology.misc:Transferring archived files from smithi039:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-21_21:16:02-rbd-squid-distro-default-smithi/7914449/remote/smithi039/crash 2024-09-21T22:09:30.117 DEBUG:teuthology.orchestra.run.smithi039:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-21T22:09:30.131 DEBUG:teuthology.misc:Transferring archived files from smithi064:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-21_21:16:02-rbd-squid-distro-default-smithi/7914449/remote/smithi064/crash 2024-09-21T22:09:30.132 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-21T22:09:30.161 DEBUG:teuthology.misc:Transferring archived files from smithi157:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-21_21:16:02-rbd-squid-distro-default-smithi/7914449/remote/smithi157/crash 2024-09-21T22:09:30.162 DEBUG:teuthology.orchestra.run.smithi157:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-21T22:09:30.175 INFO:tasks.ceph:Compressing logs... 2024-09-21T22:09:30.176 DEBUG:teuthology.orchestra.run.smithi039:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-21T22:09:30.180 DEBUG:teuthology.orchestra.run.smithi064:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-21T22:09:30.206 DEBUG:teuthology.orchestra.run.smithi157:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-21T22:09:30.212 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32240.log 2024-09-21T22:09:30.213 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32961.log 2024-09-21T22:09:30.214 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29919.log 2024-09-21T22:09:30.214 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.32240.log: /var/log/ceph/ceph-client.admin.32961.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37743.log 2024-09-21T22:09:30.214 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32240.log.gz 2024-09-21T22:09:30.214 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32961.log.gz 2024-09-21T22:09:30.215 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.29919.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29919.log.gz 2024-09-21T22:09:30.215 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32440.log 2024-09-21T22:09:30.215 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37379.log 2024-09-21T22:09:30.217 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.37743.log: gzip -5 --verbose --/var/log/ceph/ceph-client.admin.32440.log: /var/log/ceph/ceph-client.admin.37056.log 2024-09-21T22:09:30.217 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32440.log.gz 2024-09-21T22:09:30.217 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37743.log.gz 2024-09-21T22:09:30.217 INFO:teuthology.orchestra.run.smithi039.stderr:gzip/var/log/ceph/ceph-client.admin.37379.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.37639.log 2024-09-21T22:09:30.217 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37379.log.gz 2024-09-21T22:09:30.217 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.37056.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37056.log.gz 2024-09-21T22:09:30.217 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37175.log 2024-09-21T22:09:30.217 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.37639.log: gzip -5 --verbose 0.0% -- /var/log/ceph/ceph-client.admin.32869.log -- replaced with /var/log/ceph/ceph-client.admin.37639.log.gz 2024-09-21T22:09:30.218 INFO:teuthology.orchestra.run.smithi039.stderr: 2024-09-21T22:09:30.218 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.37175.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36221.log 2024-09-21T22:09:30.218 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37175.log.gz 2024-09-21T22:09:30.218 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.32869.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36573.log 2024-09-21T22:09:30.218 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32869.log.gz 2024-09-21T22:09:30.221 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.36221.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36221.log.gz 2024-09-21T22:09:30.221 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32141.log 2024-09-21T22:09:30.221 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30255.log 2024-09-21T22:09:30.221 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.32141.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33441.log 2024-09-21T22:09:30.221 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.36573.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32141.log.gz 2024-09-21T22:09:30.221 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36573.log.gz 2024-09-21T22:09:30.221 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.30255.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30255.log.gz 2024-09-21T22:09:30.222 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32191.log 2024-09-21T22:09:30.222 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29567.log 2024-09-21T22:09:30.222 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.33441.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33441.log.gz 2024-09-21T22:09:30.222 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.32191.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.32191.log.gz 2024-09-21T22:09:30.222 INFO:teuthology.orchestra.run.smithi039.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.25240.log 2024-09-21T22:09:30.222 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.29567.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-09-21T22:09:30.222 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29567.log.gz 2024-09-21T22:09:30.222 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.25240.log: gzip 0.0% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.25240.log.gz -- 2024-09-21T22:09:30.222 INFO:teuthology.orchestra.run.smithi039.stderr: /var/log/ceph/ceph-client.admin.37405.log 2024-09-21T22:09:30.222 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28396.log 2024-09-21T22:09:30.222 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.37405.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37405.log.gz 2024-09-21T22:09:30.222 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36879.log 2024-09-21T22:09:30.228 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.28396.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37230.log 2024-09-21T22:09:30.228 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28396.log.gz 2024-09-21T22:09:30.228 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30713.log 2024-09-21T22:09:30.228 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.36879.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36879.log.gz 2024-09-21T22:09:30.228 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.37230.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36547.log 2024-09-21T22:09:30.228 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37230.log.gz 2024-09-21T22:09:30.228 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.30713.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30713.log.gz 2024-09-21T22:09:30.228 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32315.log 2024-09-21T22:09:30.229 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.36547.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36547.log.gz 2024-09-21T22:09:30.229 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33141.log 2024-09-21T22:09:30.229 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27720.log 2024-09-21T22:09:30.229 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.32315.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32315.log.gz 2024-09-21T22:09:30.229 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.33141.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36521.log 2024-09-21T22:09:30.229 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33141.log.gz 2024-09-21T22:09:30.229 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.27720.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27720.log.gz 2024-09-21T22:09:30.229 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29344.log 2024-09-21T22:09:30.229 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31312.log 2024-09-21T22:09:30.229 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.36521.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36521.log.gz 2024-09-21T22:09:30.229 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.29344.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33018.log 2024-09-21T22:09:30.229 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29344.log.gz 2024-09-21T22:09:30.229 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.31312.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31312.log.gz 2024-09-21T22:09:30.229 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32349.log 2024-09-21T22:09:30.229 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31608.log 2024-09-21T22:09:30.230 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.33018.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33018.log.gz 2024-09-21T22:09:30.230 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.32349.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27998.log 2024-09-21T22:09:30.230 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32349.log.gz 2024-09-21T22:09:30.230 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.31608.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28039.log 2024-09-21T22:09:30.230 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31608.log.gz 2024-09-21T22:09:30.230 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.27998.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27998.log.gz 2024-09-21T22:09:30.230 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28129.log 2024-09-21T22:09:30.230 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30179.log 2024-09-21T22:09:30.230 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.28039.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28039.log.gz 2024-09-21T22:09:30.230 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.28129.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28544.log 2024-09-21T22:09:30.230 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28129.log.gz 2024-09-21T22:09:30.230 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.30179.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29956.log 2024-09-21T22:09:30.230 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30179.log.gz 2024-09-21T22:09:30.240 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25144.log 2024-09-21T22:09:30.240 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25240.log 2024-09-21T22:09:30.240 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/objectstore_tool.28451.log 2024-09-21T22:09:30.240 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.25144.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25144.log.gz 2024-09-21T22:09:30.240 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.25240.log: gzip -5 --verbose -- /var/log/ceph/objectstore_tool.29969.log 2024-09-21T22:09:30.240 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25240.log.gz 2024-09-21T22:09:30.240 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/objectstore_tool.28451.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-09-21T22:09:30.240 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/objectstore_tool.29969.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25264.log 2024-09-21T22:09:30.240 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-09-21T22:09:30.241 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.25264.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-09-21T22:09:30.241 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25264.log.gz 2024-09-21T22:09:30.241 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-21T22:09:30.241 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25192.log 2024-09-21T22:09:30.241 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/objectstore_tool.29212.log 2024-09-21T22:09:30.241 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-client.admin.25192.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-21T22:09:30.241 INFO:teuthology.orchestra.run.smithi064.stderr: -- replaced with /var/log/ceph/ceph-client.admin.25192.log.gz 2024-09-21T22:09:30.241 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/objectstore_tool.29212.log: 85.7% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-21T22:09:30.241 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25168.log 2024-09-21T22:09:30.241 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-09-21T22:09:30.241 INFO:teuthology.orchestra.run.smithi064.stderr: 95.2% -- replaced with /var/log/ceph/objectstore_tool.28451.log.gz 2024-09-21T22:09:30.241 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.25168.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25168.log.gz 2024-09-21T22:09:30.241 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-09-21T22:09:30.241 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25216.log 2024-09-21T22:09:30.241 INFO:teuthology.orchestra.run.smithi064.stderr: 89.0% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-21T22:09:30.242 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/objectstore_tool.30737.log 2024-09-21T22:09:30.242 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/ceph-client.admin.25216.log: 95.4% -- replaced with /var/log/ceph/objectstore_tool.29969.log.gz 2024-09-21T22:09:30.242 INFO:teuthology.orchestra.run.smithi064.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25216.log.gz 2024-09-21T22:09:30.243 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/objectstore_tool.30737.log: 95.2% -- replaced with /var/log/ceph/objectstore_tool.29212.log.gz 2024-09-21T22:09:30.244 INFO:teuthology.orchestra.run.smithi064.stderr: 95.1% -- replaced with /var/log/ceph/objectstore_tool.30737.log.gz 2024-09-21T22:09:30.256 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36950.log 2024-09-21T22:09:30.256 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.28544.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28544.log.gz 2024-09-21T22:09:30.256 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.29956.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29956.log.gz 2024-09-21T22:09:30.256 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28721.log 2024-09-21T22:09:30.256 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36872.log 2024-09-21T22:09:30.256 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.36950.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36950.log.gz 2024-09-21T22:09:30.257 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.28721.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28721.log.gz 2024-09-21T22:09:30.257 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28878.log 2024-09-21T22:09:30.257 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.36872.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32605.log 2024-09-21T22:09:30.257 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36872.log.gz 2024-09-21T22:09:30.257 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34491.log 2024-09-21T22:09:30.257 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.28878.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28878.log.gz 2024-09-21T22:09:30.257 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.32605.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32605.log.gz 2024-09-21T22:09:30.257 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33054.log 2024-09-21T22:09:30.257 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.34491.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36412.log 2024-09-21T22:09:30.257 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34491.log.gz 2024-09-21T22:09:30.257 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.33054.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33054.log.gz 2024-09-21T22:09:30.257 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-09-21T22:09:30.257 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.36412.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36412.log.gz 2024-09-21T22:09:30.257 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34658.log 2024-09-21T22:09:30.257 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32005.log 2024-09-21T22:09:30.258 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-osd.2.log: /var/log/ceph/ceph-client.admin.34658.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28336.log 2024-09-21T22:09:30.258 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34658.log.gz 2024-09-21T22:09:30.258 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.32005.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37457.log 2024-09-21T22:09:30.258 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32005.log.gz 2024-09-21T22:09:30.258 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.28336.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32851.log 2024-09-21T22:09:30.258 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28336.log.gz 2024-09-21T22:09:30.258 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.37457.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36651.log 2024-09-21T22:09:30.258 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37457.log.gz 2024-09-21T22:09:30.258 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.32851.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32851.log.gz 2024-09-21T22:09:30.258 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29203.log 2024-09-21T22:09:30.258 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.36651.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33348.log 2024-09-21T22:09:30.258 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36651.log.gz 2024-09-21T22:09:30.258 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.29203.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29203.log.gz 2024-09-21T22:09:30.258 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35384.log 2024-09-21T22:09:30.259 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.33348.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32522.log 2024-09-21T22:09:30.259 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33348.log.gz 2024-09-21T22:09:30.259 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.35384.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29719.log 2024-09-21T22:09:30.259 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35384.log.gz 2024-09-21T22:09:30.259 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.32522.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32029.log 2024-09-21T22:09:30.259 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32522.log.gz 2024-09-21T22:09:30.259 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.29719.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29719.log.gz 2024-09-21T22:09:30.259 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36599.log 2024-09-21T22:09:30.259 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.32029.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29402.log 2024-09-21T22:09:30.259 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32029.log.gz 2024-09-21T22:09:30.259 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.36599.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36599.log.gz 2024-09-21T22:09:30.259 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37795.log 2024-09-21T22:09:30.259 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32883.log 2024-09-21T22:09:30.259 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.29402.log: /var/log/ceph/ceph-client.admin.37795.log: gzip 0.0% -5 --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.29402.log.gz /var/log/ceph/ceph-mon.a.log 2024-09-21T22:09:30.260 INFO:teuthology.orchestra.run.smithi039.stderr: 2024-09-21T22:09:30.260 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37795.log.gz 2024-09-21T22:09:30.260 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.32883.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25046.log 2024-09-21T22:09:30.260 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32883.log.gz 2024-09-21T22:09:30.260 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35735.log 2024-09-21T22:09:30.260 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.25046.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32104.log 2024-09-21T22:09:30.260 INFO:teuthology.orchestra.run.smithi039.stderr: 27.8% -- replaced with /var/log/ceph/ceph-client.admin.25046.log.gz 2024-09-21T22:09:30.260 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.35735.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36311.log 2024-09-21T22:09:30.260 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35735.log.gz 2024-09-21T22:09:30.260 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.32104.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32104.log.gz 2024-09-21T22:09:30.260 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34726.log 2024-09-21T22:09:30.260 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31566.log 2024-09-21T22:09:30.260 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.36311.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36311.log.gz 2024-09-21T22:09:30.260 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.34726.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37013.log 2024-09-21T22:09:30.260 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34726.log.gz 2024-09-21T22:09:30.261 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.31566.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31566.log.gz 2024-09-21T22:09:30.261 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35785.log 2024-09-21T22:09:30.261 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37509.log 2024-09-21T22:09:30.261 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.37013.log: /var/log/ceph/ceph-client.admin.35785.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37013.log.gz 2024-09-21T22:09:30.261 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35785.log.gz 2024-09-21T22:09:30.261 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36464.log 2024-09-21T22:09:30.261 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35698.log 2024-09-21T22:09:30.261 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.37509.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37509.log.gz 2024-09-21T22:09:30.261 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.36464.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36464.log.gz 2024-09-21T22:09:30.261 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31695.log 2024-09-21T22:09:30.261 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28939.log 2024-09-21T22:09:30.261 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.35698.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35698.log.gz/var/log/ceph/ceph-client.admin.31695.log: 2024-09-21T22:09:30.261 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31695.log.gzgzip 2024-09-21T22:09:30.261 INFO:teuthology.orchestra.run.smithi039.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.28980.log 2024-09-21T22:09:30.262 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31090.log 2024-09-21T22:09:30.262 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.28939.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28939.log.gz 2024-09-21T22:09:30.262 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.28980.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28980.log.gz 2024-09-21T22:09:30.262 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28180.log 2024-09-21T22:09:30.262 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31026.log 2024-09-21T22:09:30.262 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.31090.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31090.log.gz 2024-09-21T22:09:30.262 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.28180.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29489.log 2024-09-21T22:09:30.262 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28180.log.gz 2024-09-21T22:09:30.262 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.31026.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31026.log.gz 2024-09-21T22:09:30.262 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36833.log 2024-09-21T22:09:30.262 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25163.log 2024-09-21T22:09:30.262 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.29489.log: /var/log/ceph/ceph-client.admin.36833.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29489.log.gz 2024-09-21T22:09:30.262 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36833.log.gz 2024-09-21T22:09:30.262 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37483.log 2024-09-21T22:09:30.262 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37769.log 2024-09-21T22:09:30.263 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.25163.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25163.log.gz 2024-09-21T22:09:30.263 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.37483.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37483.log.gz 2024-09-21T22:09:30.263 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37431.log 2024-09-21T22:09:30.263 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30663.log 2024-09-21T22:09:30.263 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.37769.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37769.log.gz 2024-09-21T22:09:30.263 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.37431.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37431.log.gz 2024-09-21T22:09:30.263 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28340.log 2024-09-21T22:09:30.263 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28492.log 2024-09-21T22:09:30.263 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.30663.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30663.log.gz 2024-09-21T22:09:30.263 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.28340.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31455.log 2024-09-21T22:09:30.263 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28340.log.gz 2024-09-21T22:09:30.263 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.28492.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37054.log 2024-09-21T22:09:30.263 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28492.log.gz 2024-09-21T22:09:30.263 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.31455.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30259.log 2024-09-21T22:09:30.264 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31455.log.gz 2024-09-21T22:09:30.264 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.37054.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30205.log 2024-09-21T22:09:30.264 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37054.log.gz 2024-09-21T22:09:30.264 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.30259.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30259.log.gz 2024-09-21T22:09:30.264 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36087.log 2024-09-21T22:09:30.264 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34283.log 2024-09-21T22:09:30.264 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.30205.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30205.log.gz 2024-09-21T22:09:30.264 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.36087.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36087.log.gz 2024-09-21T22:09:30.264 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37691.log 2024-09-21T22:09:30.264 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29715.log 2024-09-21T22:09:30.264 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.34283.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34283.log.gz 2024-09-21T22:09:30.264 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.37691.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37691.log.gz 2024-09-21T22:09:30.264 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28300.log 2024-09-21T22:09:30.264 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29299.log 2024-09-21T22:09:30.265 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.29715.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29715.log.gz 2024-09-21T22:09:30.265 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.28300.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28300.log.gz 2024-09-21T22:09:30.265 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36186.log 2024-09-21T22:09:30.265 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37871.log 2024-09-21T22:09:30.265 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.29299.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29299.log.gz 2024-09-21T22:09:30.265 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.36186.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36186.log.gz -5 2024-09-21T22:09:30.265 INFO:teuthology.orchestra.run.smithi039.stderr: --verbose -- /var/log/ceph/ceph-client.admin.32936.log 2024-09-21T22:09:30.265 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31140.log 2024-09-21T22:09:30.265 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.37871.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37871.log.gz 2024-09-21T22:09:30.265 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.32936.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32936.log.gz 2024-09-21T22:09:30.265 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28518.log 2024-09-21T22:09:30.265 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30768.log 2024-09-21T22:09:30.265 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.31140.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31140.log.gz 2024-09-21T22:09:30.265 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.28518.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-09-21T22:09:30.265 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28518.log.gz 2024-09-21T22:09:30.266 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.30768.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30768.log.gz 2024-09-21T22:09:30.266 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37023.log 2024-09-21T22:09:30.266 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.21446.log 2024-09-21T22:09:30.266 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-osd.1.log: /var/log/ceph/ceph-client.admin.37023.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28622.log 2024-09-21T22:09:30.266 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37023.log.gz 2024-09-21T22:09:30.266 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph.tmp-client.admin.21446.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph.tmp-client.admin.21446.log.gz --verbose 2024-09-21T22:09:30.266 INFO:teuthology.orchestra.run.smithi039.stderr: -- /var/log/ceph/ceph-client.admin.28719.log 2024-09-21T22:09:30.287 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37171.log 2024-09-21T22:09:30.287 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.28622.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28622.log.gz 2024-09-21T22:09:30.287 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.28719.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28719.log.gz 2024-09-21T22:09:30.287 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34375.log 2024-09-21T22:09:30.287 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.37171.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32266.log 2024-09-21T22:09:30.287 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37171.log.gz 2024-09-21T22:09:30.287 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.34375.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31285.log 2024-09-21T22:09:30.287 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34375.log.gz 2024-09-21T22:09:30.288 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.32266.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34490.log 2024-09-21T22:09:30.288 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32266.log.gz 2024-09-21T22:09:30.288 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.31285.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37231.log 2024-09-21T22:09:30.288 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31285.log.gz 2024-09-21T22:09:30.288 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.34490.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.34490.log.gz -5 2024-09-21T22:09:30.288 INFO:teuthology.orchestra.run.smithi039.stderr: --verbose -- /var/log/ceph/ceph-client.admin.29038.log 2024-09-21T22:09:30.288 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37821.log 2024-09-21T22:09:30.288 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.37231.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37231.log.gz 2024-09-21T22:09:30.288 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.29038.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29851.log 2024-09-21T22:09:30.288 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29038.log.gz 2024-09-21T22:09:30.288 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.37821.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37821.log.gz 2024-09-21T22:09:30.288 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31408.log 2024-09-21T22:09:30.288 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29643.log 2024-09-21T22:09:30.288 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.29851.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29851.log.gz 2024-09-21T22:09:30.289 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.31408.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34236.log 2024-09-21T22:09:30.289 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31408.log.gz 2024-09-21T22:09:30.289 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.29643.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31853.log 2024-09-21T22:09:30.289 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29643.log.gz 2024-09-21T22:09:30.289 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34607.log 2024-09-21T22:09:30.289 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.34236.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34236.log.gz 2024-09-21T22:09:30.289 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.31853.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30810.log 2024-09-21T22:09:30.289 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31853.log.gz 2024-09-21T22:09:30.289 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.34607.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34607.log.gz 2024-09-21T22:09:30.289 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33303.log 2024-09-21T22:09:30.289 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-21T22:09:30.289 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.30810.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30810.log.gz 2024-09-21T22:09:30.289 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.33303.log: gzip 0.0% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.33303.log.gz -- 2024-09-21T22:09:30.289 INFO:teuthology.orchestra.run.smithi039.stderr: /var/log/ceph/ceph-client.admin.30529.log 2024-09-21T22:09:30.289 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29619.log 2024-09-21T22:09:30.290 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.30529.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30032.log 2024-09-21T22:09:30.290 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30529.log.gz 2024-09-21T22:09:30.290 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.29619.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29619.log.gz 2024-09-21T22:09:30.290 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37535.log 2024-09-21T22:09:30.290 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28211.log 2024-09-21T22:09:30.290 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.30032.log: /var/log/ceph/ceph-client.admin.37535.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30032.log.gz 2024-09-21T22:09:30.290 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33081.log 2024-09-21T22:09:30.290 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37535.log.gz 2024-09-21T22:09:30.290 INFO:teuthology.orchestra.run.smithi039.stderr: 92.0% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-21T22:09:30.290 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36335.log 2024-09-21T22:09:30.290 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.28211.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28211.log.gz 2024-09-21T22:09:30.290 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.33081.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30133.log 2024-09-21T22:09:30.290 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33081.log.gz 2024-09-21T22:09:30.290 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.36335.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36416.log 2024-09-21T22:09:30.291 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36335.log.gz 2024-09-21T22:09:30.291 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.30133.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36625.log 2024-09-21T22:09:30.291 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30133.log.gz 2024-09-21T22:09:30.291 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.36416.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30954.log 2024-09-21T22:09:30.291 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36416.log.gz 2024-09-21T22:09:30.291 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.36625.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37179.log 2024-09-21T22:09:30.291 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36625.log.gz 2024-09-21T22:09:30.291 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.30954.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33182.log 2024-09-21T22:09:30.291 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30954.log.gz 2024-09-21T22:09:30.291 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.37179.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28370.log 2024-09-21T22:09:30.291 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37179.log.gz 2024-09-21T22:09:30.291 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27926.log 2024-09-21T22:09:30.291 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.33182.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33182.log.gz 2024-09-21T22:09:30.291 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.28370.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29855.log 2024-09-21T22:09:30.291 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28370.log.gz 2024-09-21T22:09:30.292 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27946.log 2024-09-21T22:09:30.292 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.27926.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27926.log.gz 2024-09-21T22:09:30.292 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.29855.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.29855.log.gz --verbose 2024-09-21T22:09:30.292 INFO:teuthology.orchestra.run.smithi039.stderr: -- /var/log/ceph/ceph-client.admin.29143.log 2024-09-21T22:09:30.292 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35872.log 2024-09-21T22:09:30.292 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.27946.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27946.log.gz 2024-09-21T22:09:30.292 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.29143.log: gzip -5 --verbose 0.0% -- -- replaced with /var/log/ceph/ceph-client.admin.29143.log.gz /var/log/ceph/ceph-client.admin.34317.log 2024-09-21T22:09:30.292 INFO:teuthology.orchestra.run.smithi039.stderr: 2024-09-21T22:09:30.292 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33218.log 2024-09-21T22:09:30.292 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.35872.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35872.log.gz 2024-09-21T22:09:30.292 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.34317.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.34317.log.gz 2024-09-21T22:09:30.292 INFO:teuthology.orchestra.run.smithi039.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.27972.log 2024-09-21T22:09:30.292 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-09-21T22:09:30.292 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.33218.log: /var/log/ceph/ceph-client.admin.27972.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33218.log.gz 2024-09-21T22:09:30.293 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27972.log.gz 2024-09-21T22:09:30.293 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32415.log 2024-09-21T22:09:30.293 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37665.log 2024-09-21T22:09:30.293 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-mon.c.log: /var/log/ceph/ceph-client.admin.32415.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30231.log 2024-09-21T22:09:30.293 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32415.log.gz 2024-09-21T22:09:30.293 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.37665.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37665.log.gz 2024-09-21T22:09:30.293 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36864.log 2024-09-21T22:09:30.293 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34434.log 2024-09-21T22:09:30.293 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.30231.log: /var/log/ceph/ceph-client.admin.36864.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30086.log 2024-09-21T22:09:30.293 INFO:teuthology.orchestra.run.smithi039.stderr: -- replaced with /var/log/ceph/ceph-client.admin.30231.log.gz 0.0% 2024-09-21T22:09:30.293 INFO:teuthology.orchestra.run.smithi039.stderr: -- replaced with /var/log/ceph/ceph-client.admin.36864.log.gz 2024-09-21T22:09:30.293 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.34434.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34434.log.gz 2024-09-21T22:09:30.293 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30393.log 2024-09-21T22:09:30.293 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.30086.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37203.log 2024-09-21T22:09:30.293 INFO:teuthology.orchestra.run.smithi039.stderr: -- replaced with /var/log/ceph/ceph-client.admin.30086.log.gz 2024-09-21T22:09:30.294 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.30393.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37847.log 2024-09-21T22:09:30.294 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30393.log.gz 2024-09-21T22:09:30.294 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.37203.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32699.log 2024-09-21T22:09:30.294 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37203.log.gz 2024-09-21T22:09:30.294 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.37847.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37847.log.gz 2024-09-21T22:09:30.294 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28214.log 2024-09-21T22:09:30.294 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.32699.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28040.log 2024-09-21T22:09:30.294 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32699.log.gz 2024-09-21T22:09:30.294 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.28214.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35834.log 2024-09-21T22:09:30.294 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28214.log.gz 2024-09-21T22:09:30.294 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.28040.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36677.log 2024-09-21T22:09:30.294 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28040.log.gz 2024-09-21T22:09:30.294 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.35834.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35834.log.gz 2024-09-21T22:09:30.294 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36755.log 2024-09-21T22:09:30.295 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.36677.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.36677.log.gz /var/log/ceph/ceph-client.admin.36386.log 2024-09-21T22:09:30.295 INFO:teuthology.orchestra.run.smithi039.stderr: 2024-09-21T22:09:30.295 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.36755.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30987.log 2024-09-21T22:09:30.295 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36755.log.gz 2024-09-21T22:09:30.295 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.36386.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28186.log 2024-09-21T22:09:30.295 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36386.log.gz 2024-09-21T22:09:30.295 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34401.log 2024-09-21T22:09:30.295 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.30987.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30987.log.gz 2024-09-21T22:09:30.295 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.28186.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31908.log 2024-09-21T22:09:30.295 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28186.log.gz 2024-09-21T22:09:30.295 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29208.log 2024-09-21T22:09:30.295 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.34401.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34401.log.gz 2024-09-21T22:09:30.295 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.31908.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28570.log 2024-09-21T22:09:30.295 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31908.log.gz 2024-09-21T22:09:30.295 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.29208.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29208.log.gz 2024-09-21T22:09:30.296 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36495.log 2024-09-21T22:09:30.296 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.28570.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35925.log 2024-09-21T22:09:30.296 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28570.log.gz 2024-09-21T22:09:30.296 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.36495.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35950.log 2024-09-21T22:09:30.296 INFO:teuthology.orchestra.run.smithi039.stderr: -- replaced with /var/log/ceph/ceph-client.admin.36495.log.gz 2024-09-21T22:09:30.296 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.35925.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30356.log 2024-09-21T22:09:30.296 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35925.log.gz 2024-09-21T22:09:30.296 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.35950.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-21T22:09:30.296 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35950.log.gz 2024-09-21T22:09:30.296 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.30356.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30356.log.gz 2024-09-21T22:09:30.296 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37613.log 2024-09-21T22:09:30.296 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28844.log 2024-09-21T22:09:30.296 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.37613.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37327.log 2024-09-21T22:09:30.296 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37613.log.gz 2024-09-21T22:09:30.296 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.28844.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28844.log.gz 2024-09-21T22:09:30.297 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36807.log 2024-09-21T22:09:30.297 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.37327.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36871.log 2024-09-21T22:09:30.297 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37327.log.gz 2024-09-21T22:09:30.297 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.36807.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36807.log.gz 2024-09-21T22:09:30.297 INFO:teuthology.orchestra.run.smithi039.stderr: 93.7% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-21T22:09:30.297 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32760.log 2024-09-21T22:09:30.322 INFO:teuthology.orchestra.run.smithi064.stderr: 91.6% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-09-21T22:09:30.336 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.36871.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36871.log.gz 2024-09-21T22:09:30.336 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32390.log 2024-09-21T22:09:30.336 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30593.log 2024-09-21T22:09:30.336 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.32760.log: /var/log/ceph/ceph-client.admin.32390.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32760.log.gz 2024-09-21T22:09:30.336 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37587.log 2024-09-21T22:09:30.336 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32390.log.gz 2024-09-21T22:09:30.336 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30905.log 2024-09-21T22:09:30.336 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.30593.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30593.log.gz 2024-09-21T22:09:30.336 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.37587.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37587.log.gz 2024-09-21T22:09:30.336 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28648.log 2024-09-21T22:09:30.336 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.30905.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30028.log 2024-09-21T22:09:30.337 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30905.log.gz 2024-09-21T22:09:30.337 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.28648.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29814.log 2024-09-21T22:09:30.337 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28648.log.gz 2024-09-21T22:09:30.337 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.30028.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31505.log 2024-09-21T22:09:30.337 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30028.log.gz 2024-09-21T22:09:30.337 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.29814.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29814.log.gz 2024-09-21T22:09:30.337 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32931.log 2024-09-21T22:09:30.337 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.31505.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.31505.log.gz 2024-09-21T22:09:30.337 INFO:teuthology.orchestra.run.smithi039.stderr: /var/log/ceph/ceph-client.admin.32656.log 2024-09-21T22:09:30.337 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.32931.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28038.log 2024-09-21T22:09:30.337 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32931.log.gz 2024-09-21T22:09:30.337 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.32656.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34564.log 2024-09-21T22:09:30.337 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32656.log.gz 2024-09-21T22:09:30.337 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36050.log 2024-09-21T22:09:30.337 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.28038.log: /var/log/ceph/ceph-client.admin.34564.log: gzip 0.0% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.28038.log.gz -- 2024-09-21T22:09:30.338 INFO:teuthology.orchestra.run.smithi039.stderr: /var/log/ceph/ceph-client.admin.31343.log 2024-09-21T22:09:30.338 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34564.log.gz 2024-09-21T22:09:30.338 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.36050.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36050.log.gz 2024-09-21T22:09:30.338 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36008.log 2024-09-21T22:09:30.338 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.31343.log: gzip -5 --verbose 0.0% -- /var/log/ceph/ceph-client.admin.36703.log -- replaced with /var/log/ceph/ceph-client.admin.31343.log.gz 2024-09-21T22:09:30.338 INFO:teuthology.orchestra.run.smithi039.stderr: 2024-09-21T22:09:30.338 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.36008.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30492.log 2024-09-21T22:09:30.338 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36008.log.gz 2024-09-21T22:09:30.338 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.36703.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32558.log 2024-09-21T22:09:30.338 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36703.log.gz 2024-09-21T22:09:30.338 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29083.log 2024-09-21T22:09:30.338 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.30492.log: /var/log/ceph/ceph-client.admin.32558.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30492.log.gz 2024-09-21T22:09:30.338 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28042.log 2024-09-21T22:09:30.338 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32558.log.gz 2024-09-21T22:09:30.338 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31188.log 2024-09-21T22:09:30.339 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.29083.log: /var/log/ceph/ceph-client.admin.28042.log: gzip 0.0% -5 --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.29083.log.gz /var/log/ceph/ceph-client.admin.30447.log 2024-09-21T22:09:30.339 INFO:teuthology.orchestra.run.smithi039.stderr: 2024-09-21T22:09:30.339 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28042.log.gz 2024-09-21T22:09:30.339 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.31188.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31188.log.gz 2024-09-21T22:09:30.339 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28372.log 2024-09-21T22:09:30.339 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.30447.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31667.log 2024-09-21T22:09:30.339 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30447.log.gz 2024-09-21T22:09:30.339 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.28372.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36878.log 2024-09-21T22:09:30.339 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28372.log.gz 2024-09-21T22:09:30.339 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31764.log 2024-09-21T22:09:30.339 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.31667.log: /var/log/ceph/ceph-client.admin.36878.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31667.log.gz 2024-09-21T22:09:30.339 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35393.log 2024-09-21T22:09:30.339 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36878.log.gz 2024-09-21T22:09:30.339 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33409.log 2024-09-21T22:09:30.340 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.31764.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31764.log.gz/var/log/ceph/ceph-client.admin.35393.log: 2024-09-21T22:09:30.340 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31803.log 2024-09-21T22:09:30.340 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35393.log.gz 2024-09-21T22:09:30.340 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.33409.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27889.log 2024-09-21T22:09:30.340 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33409.log.gz 2024-09-21T22:09:30.340 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.31803.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31803.log.gz 2024-09-21T22:09:30.340 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37027.log 2024-09-21T22:09:30.340 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.27889.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.27889.log.gz --verbose 2024-09-21T22:09:30.340 INFO:teuthology.orchestra.run.smithi039.stderr: -- /var/log/ceph/ceph-client.admin.37561.log 2024-09-21T22:09:30.340 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.37027.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28722.log 2024-09-21T22:09:30.340 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37027.log.gz 2024-09-21T22:09:30.340 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.37561.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28344.log 2024-09-21T22:09:30.340 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37561.log.gz 2024-09-21T22:09:30.340 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30877.log 2024-09-21T22:09:30.340 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.28722.log: /var/log/ceph/ceph-client.admin.28344.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28722.log.gz 2024-09-21T22:09:30.340 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28344.log.gz 2024-09-21T22:09:30.341 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32609.log 2024-09-21T22:09:30.341 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.30877.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28044.log 2024-09-21T22:09:30.341 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30877.log.gz 2024-09-21T22:09:30.341 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.32609.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29447.log 2024-09-21T22:09:30.341 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32609.log.gz 2024-09-21T22:09:30.341 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36215.log 2024-09-21T22:09:30.341 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.28044.log: /var/log/ceph/ceph-client.admin.29447.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28596.log 2024-09-21T22:09:30.341 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28044.log.gz 2024-09-21T22:09:30.341 INFO:teuthology.orchestra.run.smithi039.stderr: -- replaced with /var/log/ceph/ceph-client.admin.29447.log.gz 2024-09-21T22:09:30.341 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.36215.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.36215.log.gz 2024-09-21T22:09:30.341 INFO:teuthology.orchestra.run.smithi039.stderr: --verbose -- /var/log/ceph/ceph-client.admin.32480.log 2024-09-21T22:09:30.341 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.28596.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28810.log 2024-09-21T22:09:30.341 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28596.log.gz 2024-09-21T22:09:30.341 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.32480.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32480.log.gz 2024-09-21T22:09:30.341 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27863.log 2024-09-21T22:09:30.341 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.28810.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30002.log 2024-09-21T22:09:30.342 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28810.log.gz 2024-09-21T22:09:30.342 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36781.log/var/log/ceph/ceph-client.admin.27863.log: 2024-09-21T22:09:30.342 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27863.log.gz 2024-09-21T22:09:30.342 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.30002.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30002.log.gzgzip 2024-09-21T22:09:30.342 INFO:teuthology.orchestra.run.smithi039.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.37717.log 2024-09-21T22:09:30.342 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.36781.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28037.log 2024-09-21T22:09:30.342 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36781.log.gz 2024-09-21T22:09:30.342 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.37717.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31227.log 2024-09-21T22:09:30.342 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37717.log.gz 2024-09-21T22:09:30.342 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.28037.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28037.log.gz 2024-09-21T22:09:30.342 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29568.log 2024-09-21T22:09:30.342 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.31227.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28913.log 2024-09-21T22:09:30.342 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31227.log.gz 2024-09-21T22:09:30.342 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.29568.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29538.log 2024-09-21T22:09:30.343 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29568.log.gz 2024-09-21T22:09:30.343 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.28913.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31934.log 2024-09-21T22:09:30.343 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28913.log.gz 2024-09-21T22:09:30.343 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36137.log 2024-09-21T22:09:30.343 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.29538.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29538.log.gz 2024-09-21T22:09:30.343 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33266.log 2024-09-21T22:09:30.343 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.31934.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31934.log.gz 2024-09-21T22:09:30.343 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.36137.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36137.log.gz 2024-09-21T22:09:30.343 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36876.log 2024-09-21T22:09:30.343 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.33266.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36729.log 2024-09-21T22:09:30.343 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33266.log.gz 2024-09-21T22:09:30.343 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.36876.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31022.log 2024-09-21T22:09:30.343 INFO:teuthology.orchestra.run.smithi039.stderr: -- replaced with /var/log/ceph/ceph-client.admin.36876.log.gz 2024-09-21T22:09:30.343 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.36729.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37353.log 2024-09-21T22:09:30.344 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36729.log.gz 2024-09-21T22:09:30.344 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.31022.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29120.log 2024-09-21T22:09:30.344 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31022.log.gz 2024-09-21T22:09:30.344 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28678.log 2024-09-21T22:09:30.344 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.37353.log: /var/log/ceph/ceph-client.admin.29120.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37353.log.gz 2024-09-21T22:09:30.344 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29120.log.gz 2024-09-21T22:09:30.344 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/objectstore_tool.33403.log 2024-09-21T22:09:30.344 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.28678.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34757.log 2024-09-21T22:09:30.344 INFO:teuthology.orchestra.run.smithi039.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28678.log.gz 2024-09-21T22:09:30.344 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/objectstore_tool.33403.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32786.log 2024-09-21T22:09:30.344 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.34757.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34757.log.gz 2024-09-21T22:09:30.344 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.32786.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32786.log.gz 2024-09-21T22:09:30.344 INFO:teuthology.orchestra.run.smithi039.stderr: 95.2% -- replaced with /var/log/ceph/objectstore_tool.33403.log.gz 2024-09-21T22:09:30.351 INFO:teuthology.orchestra.run.smithi039.stderr: 92.4% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-09-21T22:09:30.357 INFO:teuthology.orchestra.run.smithi064.stderr: 92.8% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-09-21T22:09:30.375 INFO:teuthology.orchestra.run.smithi064.stderr: 91.1% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-09-21T22:09:30.400 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21944.log 2024-09-21T22:09:30.400 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23699.log 2024-09-21T22:09:30.400 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.21944.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25091.log 2024-09-21T22:09:30.400 INFO:teuthology.orchestra.run.smithi157.stderr: 19.4% -- replaced with /var/log/ceph/ceph-client.admin.21944.log.gz 2024-09-21T22:09:30.401 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.23699.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23551.log 2024-09-21T22:09:30.401 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23699.log.gz 2024-09-21T22:09:30.401 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.25091.log: 18.2% -- replaced with /var/log/ceph/ceph-client.admin.25091.log.gz 2024-09-21T22:09:30.401 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23348.log 2024-09-21T22:09:30.401 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22565.log 2024-09-21T22:09:30.401 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.23551.log: /var/log/ceph/ceph-client.admin.23348.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25360.log 2024-09-21T22:09:30.401 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23348.log.gz 2024-09-21T22:09:30.401 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23551.log.gz 2024-09-21T22:09:30.401 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.22565.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22775.log 2024-09-21T22:09:30.401 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22565.log.gz 2024-09-21T22:09:30.401 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.25360.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24783.log 2024-09-21T22:09:30.401 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25360.log.gz 2024-09-21T22:09:30.401 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.22775.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21978.log 2024-09-21T22:09:30.401 INFO:teuthology.orchestra.run.smithi157.stderr: 19.4% -- replaced with /var/log/ceph/ceph-client.admin.22775.log.gz 2024-09-21T22:09:30.402 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.24783.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23239.log 2024-09-21T22:09:30.402 INFO:teuthology.orchestra.run.smithi157.stderr: 5.5% -- replaced with /var/log/ceph/ceph-client.admin.24783.log.gz 2024-09-21T22:09:30.402 INFO:teuthology.orchestra.run.smithi157.stderr:gzip/var/log/ceph/ceph-client.admin.21978.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.21928.log 2024-09-21T22:09:30.402 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.21978.log.gz 2024-09-21T22:09:30.402 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.23239.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23335.log 2024-09-21T22:09:30.402 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23239.log.gz 2024-09-21T22:09:30.402 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.21928.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23055.log 2024-09-21T22:09:30.402 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.21928.log.gz 2024-09-21T22:09:30.402 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.23335.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23416.log 2024-09-21T22:09:30.402 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23335.log.gz 2024-09-21T22:09:30.402 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.23055.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24145.log 2024-09-21T22:09:30.402 INFO:teuthology.orchestra.run.smithi157.stderr: 18.2% -- replaced with /var/log/ceph/ceph-client.admin.23055.log.gz 2024-09-21T22:09:30.402 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.23416.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23469.log 2024-09-21T22:09:30.402 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23416.log.gz 2024-09-21T22:09:30.403 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.24145.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22614.log 2024-09-21T22:09:30.403 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24145.log.gz 2024-09-21T22:09:30.403 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.23469.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23898.log 2024-09-21T22:09:30.403 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23469.log.gz 2024-09-21T22:09:30.403 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.22614.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24031.log 2024-09-21T22:09:30.403 INFO:teuthology.orchestra.run.smithi157.stderr: 41.2% -- replaced with /var/log/ceph/ceph-client.admin.22614.log.gz 2024-09-21T22:09:30.403 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.23898.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23829.log 2024-09-21T22:09:30.403 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23898.log.gz 2024-09-21T22:09:30.403 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.24031.log: 19.4% -- replaced with /var/log/ceph/ceph-client.admin.24031.log.gz 2024-09-21T22:09:30.403 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23793.log 2024-09-21T22:09:30.403 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.23829.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22066.log 2024-09-21T22:09:30.403 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23829.log.gz 2024-09-21T22:09:30.403 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.23793.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21839.log 2024-09-21T22:09:30.403 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23793.log.gz 2024-09-21T22:09:30.403 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.22066.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24950.log 2024-09-21T22:09:30.404 INFO:teuthology.orchestra.run.smithi157.stderr: 56.3% -- replaced with /var/log/ceph/ceph-client.admin.22066.log.gz 2024-09-21T22:09:30.404 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.21839.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25148.log 2024-09-21T22:09:30.404 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.21839.log.gz 2024-09-21T22:09:30.404 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.24950.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22148.log 2024-09-21T22:09:30.404 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24950.log.gz 2024-09-21T22:09:30.404 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.25148.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25148.log.gz 2024-09-21T22:09:30.404 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22248.log 2024-09-21T22:09:30.404 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.22148.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24616.log 2024-09-21T22:09:30.404 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22148.log.gz 2024-09-21T22:09:30.404 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.22248.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22248.log.gz 2024-09-21T22:09:30.404 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24309.log 2024-09-21T22:09:30.404 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.24616.log: gzip 4.5% -- replaced with /var/log/ceph/ceph-client.admin.24616.log.gz -5 2024-09-21T22:09:30.404 INFO:teuthology.orchestra.run.smithi157.stderr: --verbose -- /var/log/ceph/ceph-client.admin.23223.log 2024-09-21T22:09:30.404 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.24309.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22195.log 2024-09-21T22:09:30.404 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24309.log.gz 2024-09-21T22:09:30.404 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.23223.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22969.log 2024-09-21T22:09:30.405 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23223.log.gz 2024-09-21T22:09:30.405 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.22195.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22195.log.gz 2024-09-21T22:09:30.405 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22676.log 2024-09-21T22:09:30.405 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.22969.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24066.log 2024-09-21T22:09:30.405 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22969.log.gz 2024-09-21T22:09:30.405 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.22676.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22222.log 2024-09-21T22:09:30.405 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22676.log.gz 2024-09-21T22:09:30.405 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.24066.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24410.log 2024-09-21T22:09:30.405 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24066.log.gz 2024-09-21T22:09:30.405 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.22222.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22477.log 2024-09-21T22:09:30.405 INFO:teuthology.orchestra.run.smithi157.stderr: 54.2% -- replaced with /var/log/ceph/ceph-client.admin.22222.log.gz 2024-09-21T22:09:30.405 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.24410.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22092.log 2024-09-21T22:09:30.405 INFO:teuthology.orchestra.run.smithi157.stderr: 19.4% -- replaced with /var/log/ceph/ceph-client.admin.24410.log.gz 2024-09-21T22:09:30.405 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.22477.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.22477.log.gz 2024-09-21T22:09:30.406 INFO:teuthology.orchestra.run.smithi157.stderr: /var/log/ceph/ceph-client.admin.22883.log 2024-09-21T22:09:30.406 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.22092.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21685.log 2024-09-21T22:09:30.406 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22092.log.gz 2024-09-21T22:09:30.406 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.22883.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22504.log 2024-09-21T22:09:30.406 INFO:teuthology.orchestra.run.smithi157.stderr: 35.3% -- replaced with /var/log/ceph/ceph-client.admin.22883.log.gz 2024-09-21T22:09:30.406 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.21685.log: gzip -5 --verbose -- 18.8% /var/log/ceph/ceph-client.admin.24282.log -- replaced with /var/log/ceph/ceph-client.admin.21685.log.gz 2024-09-21T22:09:30.406 INFO:teuthology.orchestra.run.smithi157.stderr: 2024-09-21T22:09:30.406 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.22504.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22120.log 2024-09-21T22:09:30.406 INFO:teuthology.orchestra.run.smithi157.stderr: 52.6% -- replaced with /var/log/ceph/ceph-client.admin.22504.log.gz 2024-09-21T22:09:30.406 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.24282.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22549.log 2024-09-21T22:09:30.406 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24282.log.gz 2024-09-21T22:09:30.406 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.22120.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24198.log 2024-09-21T22:09:30.406 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22120.log.gz 2024-09-21T22:09:30.406 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23959.log 2024-09-21T22:09:30.406 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.22549.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22549.log.gz 2024-09-21T22:09:30.407 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.24198.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24198.log.gz 2024-09-21T22:09:30.407 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23099.log 2024-09-21T22:09:30.407 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.23959.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25343.log 2024-09-21T22:09:30.407 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23959.log.gz 2024-09-21T22:09:30.407 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.23099.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23099.log.gz 2024-09-21T22:09:30.407 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24172.log 2024-09-21T22:09:30.407 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24015.log 2024-09-21T22:09:30.407 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.25343.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25343.log.gz 2024-09-21T22:09:30.407 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.24172.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24256.log 2024-09-21T22:09:30.407 INFO:teuthology.orchestra.run.smithi157.stderr: 24.1% -- replaced with /var/log/ceph/ceph-client.admin.24172.log.gz 2024-09-21T22:09:30.407 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.24015.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23241.log 2024-09-21T22:09:30.407 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24015.log.gz 2024-09-21T22:09:30.407 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.24256.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23364.log 2024-09-21T22:09:30.408 INFO:teuthology.orchestra.run.smithi157.stderr: 45.0% -- replaced with /var/log/ceph/ceph-client.admin.24256.log.gz 2024-09-21T22:09:30.408 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.23241.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24163.log 2024-09-21T22:09:30.408 INFO:teuthology.orchestra.run.smithi157.stderr: 19.4% -- replaced with /var/log/ceph/ceph-client.admin.23241.log.gz 2024-09-21T22:09:30.408 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.23364.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22847.log 2024-09-21T22:09:30.408 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23364.log.gz 2024-09-21T22:09:30.408 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.24163.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21846.log 2024-09-21T22:09:30.408 INFO:teuthology.orchestra.run.smithi157.stderr: 20.0% -- replaced with /var/log/ceph/ceph-client.admin.24163.log.gz 2024-09-21T22:09:30.408 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.22847.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23188.log 2024-09-21T22:09:30.408 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22847.log.gz 2024-09-21T22:09:30.408 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.21846.log: gzip -5 0.0% --verbose -- replaced with /var/log/ceph/ceph-client.admin.21846.log.gz -- 2024-09-21T22:09:30.408 INFO:teuthology.orchestra.run.smithi157.stderr: /var/log/ceph/ceph-client.admin.22713.log 2024-09-21T22:09:30.408 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.23188.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25160.log 2024-09-21T22:09:30.408 INFO:teuthology.orchestra.run.smithi157.stderr: 25.3% -- replaced with /var/log/ceph/ceph-client.admin.23188.log.gz 2024-09-21T22:09:30.408 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.22713.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22014.log 2024-09-21T22:09:30.408 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22713.log.gz 2024-09-21T22:09:30.408 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.25160.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25160.log.gz -5 2024-09-21T22:09:30.409 INFO:teuthology.orchestra.run.smithi157.stderr: --verbose -- /var/log/ceph/ceph-client.admin.22369.log 2024-09-21T22:09:30.409 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.22014.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24247.log 2024-09-21T22:09:30.409 INFO:teuthology.orchestra.run.smithi157.stderr: 57.9% -- replaced with /var/log/ceph/ceph-client.admin.22014.log.gz 2024-09-21T22:09:30.409 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.22369.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23090.log 2024-09-21T22:09:30.409 INFO:teuthology.orchestra.run.smithi157.stderr: 19.4% -- replaced with /var/log/ceph/ceph-client.admin.22369.log.gz 2024-09-21T22:09:30.409 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.24247.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21668.log 2024-09-21T22:09:30.409 INFO:teuthology.orchestra.run.smithi157.stderr: 19.4% -- replaced with /var/log/ceph/ceph-client.admin.24247.log.gz 2024-09-21T22:09:30.409 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.23090.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22042.log 2024-09-21T22:09:30.409 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23090.log.gz 2024-09-21T22:09:30.409 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.21668.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23663.log 2024-09-21T22:09:30.409 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.21668.log.gz 2024-09-21T22:09:30.409 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.22042.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23276.log 2024-09-21T22:09:30.409 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22042.log.gz 2024-09-21T22:09:30.409 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.23663.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24408.log 2024-09-21T22:09:30.409 INFO:teuthology.orchestra.run.smithi157.stderr: 19.4% -- replaced with /var/log/ceph/ceph-client.admin.23663.log.gz 2024-09-21T22:09:30.410 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.23276.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24124.log 2024-09-21T22:09:30.410 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23276.log.gz 2024-09-21T22:09:30.410 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.24408.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24220.log 2024-09-21T22:09:30.410 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24408.log.gz 2024-09-21T22:09:30.410 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.24124.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23332.log 2024-09-21T22:09:30.410 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24124.log.gz 2024-09-21T22:09:30.410 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.24220.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24857.log 2024-09-21T22:09:30.410 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24220.log.gz 2024-09-21T22:09:30.410 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.23332.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24913.log 2024-09-21T22:09:30.410 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23332.log.gz 2024-09-21T22:09:30.410 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.24857.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22639.log 2024-09-21T22:09:30.410 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24857.log.gz 2024-09-21T22:09:30.410 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.24913.log: -- /var/log/ceph/ceph-client.admin.23177.log 2024-09-21T22:09:30.410 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24913.log.gz 2024-09-21T22:09:30.410 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.22639.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24359.log 2024-09-21T22:09:30.410 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22639.log.gz 2024-09-21T22:09:30.411 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.23177.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22170.log 2024-09-21T22:09:30.411 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23177.log.gz 2024-09-21T22:09:30.411 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.24359.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24079.log 2024-09-21T22:09:30.411 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24359.log.gz 2024-09-21T22:09:30.411 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.22170.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22170.log.gz 2024-09-21T22:09:30.411 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25073.log 2024-09-21T22:09:30.411 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.24079.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24346.log 2024-09-21T22:09:30.411 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24079.log.gz 2024-09-21T22:09:30.411 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.25073.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25073.log.gz 2024-09-21T22:09:30.411 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23006.log 2024-09-21T22:09:30.411 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.24346.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24346.log.gz 2024-09-21T22:09:30.411 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21834.log 2024-09-21T22:09:30.411 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.23006.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24363.log 2024-09-21T22:09:30.412 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23006.log.gz 2024-09-21T22:09:30.412 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.21834.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22326.log 2024-09-21T22:09:30.412 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.21834.log.gz 2024-09-21T22:09:30.412 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.24363.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24683.log 2024-09-21T22:09:30.412 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24363.log.gz 2024-09-21T22:09:30.412 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.22326.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24817.log 2024-09-21T22:09:30.412 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22326.log.gz 2024-09-21T22:09:30.412 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.24683.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24040.log 2024-09-21T22:09:30.412 INFO:teuthology.orchestra.run.smithi157.stderr: 4.5% -- replaced with /var/log/ceph/ceph-client.admin.24683.log.gz 2024-09-21T22:09:30.412 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.24817.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25181.log 2024-09-21T22:09:30.412 INFO:teuthology.orchestra.run.smithi157.stderr: 5.5% -- replaced with /var/log/ceph/ceph-client.admin.24817.log.gz 2024-09-21T22:09:30.412 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.24040.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22055.log 2024-09-21T22:09:30.412 INFO:teuthology.orchestra.run.smithi157.stderr: 45.8% -- replaced with /var/log/ceph/ceph-client.admin.24040.log.gz 2024-09-21T22:09:30.412 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.25181.log: -- /var/log/ceph/ceph-client.admin.24320.log 2024-09-21T22:09:30.412 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25181.log.gz 2024-09-21T22:09:30.413 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.22055.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22198.log 2024-09-21T22:09:30.413 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22055.log.gz 2024-09-21T22:09:30.413 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.24320.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22351.log 2024-09-21T22:09:30.413 INFO:teuthology.orchestra.run.smithi157.stderr: 45.8% -- replaced with /var/log/ceph/ceph-client.admin.24320.log.gz 2024-09-21T22:09:30.413 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.22198.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23610.log 2024-09-21T22:09:30.413 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22198.log.gz 2024-09-21T22:09:30.413 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.22351.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25076.log 2024-09-21T22:09:30.413 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22351.log.gz 2024-09-21T22:09:30.413 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.23610.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23117.log 2024-09-21T22:09:30.413 INFO:teuthology.orchestra.run.smithi157.stderr: 45.9% -- replaced with /var/log/ceph/ceph-client.admin.23610.log.gz 2024-09-21T22:09:30.413 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.25076.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22378.log 2024-09-21T22:09:30.413 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25076.log.gz 2024-09-21T22:09:30.413 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.23117.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22276.log 2024-09-21T22:09:30.413 INFO:teuthology.orchestra.run.smithi157.stderr: 20.0% -- replaced with /var/log/ceph/ceph-client.admin.23117.log.gz 2024-09-21T22:09:30.413 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.22378.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24582.log 2024-09-21T22:09:30.414 INFO:teuthology.orchestra.run.smithi157.stderr: 53.4% -- replaced with /var/log/ceph/ceph-client.admin.22378.log.gz 2024-09-21T22:09:30.414 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.22276.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24924.log 2024-09-21T22:09:30.414 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22276.log.gz 2024-09-21T22:09:30.414 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.24582.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25061.log 2024-09-21T22:09:30.414 INFO:teuthology.orchestra.run.smithi157.stderr: 5.5% -- replaced with /var/log/ceph/ceph-client.admin.24582.log.gz 2024-09-21T22:09:30.414 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.24924.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24245.log 2024-09-21T22:09:30.414 INFO:teuthology.orchestra.run.smithi157.stderr: 53.9% -- replaced with /var/log/ceph/ceph-client.admin.24924.log.gz 2024-09-21T22:09:30.414 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.25061.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25133.log 2024-09-21T22:09:30.414 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25061.log.gz 2024-09-21T22:09:30.414 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.24245.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23226.log 2024-09-21T22:09:30.414 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24245.log.gz 2024-09-21T22:09:30.414 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.25133.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22404.log 2024-09-21T22:09:30.414 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25133.log.gz 2024-09-21T22:09:30.414 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.23226.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23226.log.gz 2024-09-21T22:09:30.415 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21901.log 2024-09-21T22:09:30.415 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.22404.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22642.log 2024-09-21T22:09:30.415 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22404.log.gz 2024-09-21T22:09:30.415 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.21901.log: 26.3% -- replaced with /var/log/ceph/ceph-client.admin.21901.log.gz 2024-09-21T22:09:30.415 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25264.log 2024-09-21T22:09:30.415 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.22642.log: /var/log/ceph/ceph-client.admin.24295.log 2024-09-21T22:09:30.415 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22642.log.gz 2024-09-21T22:09:30.415 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.25264.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25264.log.gz 2024-09-21T22:09:30.415 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22819.log 2024-09-21T22:09:30.415 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.24295.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21931.log 2024-09-21T22:09:30.415 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24295.log.gz 2024-09-21T22:09:30.415 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.22819.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22819.log.gz 2024-09-21T22:09:30.415 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23516.log 2024-09-21T22:09:30.415 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.21931.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22687.log 2024-09-21T22:09:30.415 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.21931.log.gz 2024-09-21T22:09:30.415 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.23516.log: gzip -5 24.1% --verbose -- replaced with /var/log/ceph/ceph-client.admin.23516.log.gz -- 2024-09-21T22:09:30.416 INFO:teuthology.orchestra.run.smithi157.stderr: /var/log/ceph/ceph-client.admin.21567.log 2024-09-21T22:09:30.416 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.22687.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25100.log 2024-09-21T22:09:30.416 INFO:teuthology.orchestra.run.smithi157.stderr: 25.3% -- replaced with /var/log/ceph/ceph-client.admin.22687.log.gz 2024-09-21T22:09:30.416 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24207.log 2024-09-21T22:09:30.416 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.21567.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.21567.log.gz 2024-09-21T22:09:30.416 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.25100.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24161.log 2024-09-21T22:09:30.416 INFO:teuthology.orchestra.run.smithi157.stderr: 25.3% -- replaced with /var/log/ceph/ceph-client.admin.25100.log.gz 2024-09-21T22:09:30.416 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.24207.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23652.log 2024-09-21T22:09:30.416 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24207.log.gz 2024-09-21T22:09:30.416 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.24161.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24161.log.gz 2024-09-21T22:09:30.416 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24716.log 2024-09-21T22:09:30.416 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.23652.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24303.log 2024-09-21T22:09:30.416 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23652.log.gz 2024-09-21T22:09:30.416 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.24716.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23961.log 2024-09-21T22:09:30.416 INFO:teuthology.orchestra.run.smithi157.stderr: 4.5% -- replaced with /var/log/ceph/ceph-client.admin.24716.log.gz 2024-09-21T22:09:30.417 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.24303.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24303.log.gz 2024-09-21T22:09:30.417 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24750.log 2024-09-21T22:09:30.417 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23028.log 2024-09-21T22:09:30.417 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.23961.log: 19.4% -- replaced with /var/log/ceph/ceph-client.admin.23961.log.gz 2024-09-21T22:09:30.417 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.24750.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22211.log 2024-09-21T22:09:30.417 INFO:teuthology.orchestra.run.smithi157.stderr: 4.5% -- replaced with /var/log/ceph/ceph-client.admin.24750.log.gz 2024-09-21T22:09:30.417 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.23028.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21648.log 2024-09-21T22:09:30.417 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23028.log.gz 2024-09-21T22:09:30.417 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.22211.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22749.log 2024-09-21T22:09:30.417 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22211.log.gz 2024-09-21T22:09:30.417 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.21648.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23941.log 2024-09-21T22:09:30.417 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.21648.log.gz 2024-09-21T22:09:30.417 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.22749.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22057.log 2024-09-21T22:09:30.417 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22749.log.gz 2024-09-21T22:09:30.418 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.23941.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24549.log 2024-09-21T22:09:30.418 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23941.log.gz 2024-09-21T22:09:30.418 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.22057.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24854.log 2024-09-21T22:09:30.418 INFO:teuthology.orchestra.run.smithi157.stderr: 18.2% -- replaced with /var/log/ceph/ceph-client.admin.22057.log.gz 2024-09-21T22:09:30.418 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.24549.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24210.log 2024-09-21T22:09:30.418 INFO:teuthology.orchestra.run.smithi157.stderr: 3.6% -- replaced with /var/log/ceph/ceph-client.admin.24549.log.gz 2024-09-21T22:09:30.418 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.24854.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23647.log 2024-09-21T22:09:30.418 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24854.log.gz 2024-09-21T22:09:30.418 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.24210.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23490.log 2024-09-21T22:09:30.418 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24210.log.gz 2024-09-21T22:09:30.418 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.23647.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23647.log.gz 2024-09-21T22:09:30.418 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24075.log 2024-09-21T22:09:30.418 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.23490.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25024.log 2024-09-21T22:09:30.419 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23490.log.gz 2024-09-21T22:09:30.419 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.24075.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25035.log 2024-09-21T22:09:30.419 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24075.log.gz 2024-09-21T22:09:30.419 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.25024.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25024.log.gz 2024-09-21T22:09:30.419 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23569.log 2024-09-21T22:09:30.419 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22480.log 2024-09-21T22:09:30.419 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.25035.log: 24.7% -- replaced with /var/log/ceph/ceph-client.admin.25035.log.gz 2024-09-21T22:09:30.419 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.23569.log: gzip -5 --verbose 0.0% -- -- replaced with /var/log/ceph/ceph-client.admin.23569.log.gz 2024-09-21T22:09:30.419 INFO:teuthology.orchestra.run.smithi157.stderr: /var/log/ceph/ceph-client.admin.25222.log 2024-09-21T22:09:30.419 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.22480.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23636.log 2024-09-21T22:09:30.419 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22480.log.gz 2024-09-21T22:09:30.419 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23161.log 2024-09-21T22:09:30.419 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.25222.log: 24.9% -- replaced with /var/log/ceph/ceph-client.admin.25222.log.gz 2024-09-21T22:09:30.419 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.23636.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.23636.log.gz 2024-09-21T22:09:30.419 INFO:teuthology.orchestra.run.smithi157.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.24482.log 2024-09-21T22:09:30.419 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.23161.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23102.log 2024-09-21T22:09:30.420 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23161.log.gz 2024-09-21T22:09:30.420 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22254.log 2024-09-21T22:09:30.420 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.24482.log: 5.5% -- replaced with /var/log/ceph/ceph-client.admin.24482.log.gz 2024-09-21T22:09:30.420 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.23102.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21694.log 2024-09-21T22:09:30.420 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23102.log.gz 2024-09-21T22:09:30.420 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.22254.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25089.log 2024-09-21T22:09:30.420 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22254.log.gz 2024-09-21T22:09:30.420 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22098.log 2024-09-21T22:09:30.420 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.21694.log: 26.9% -- replaced with /var/log/ceph/ceph-client.admin.21694.log.gz 2024-09-21T22:09:30.420 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.25089.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25089.log.gz 2024-09-21T22:09:30.420 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25145.log 2024-09-21T22:09:30.420 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.22098.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24089.log 2024-09-21T22:09:30.420 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22098.log.gz 2024-09-21T22:09:30.420 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.25145.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23672.log 2024-09-21T22:09:30.420 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25145.log.gz 2024-09-21T22:09:30.421 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.24089.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24915.log 2024-09-21T22:09:30.421 INFO:teuthology.orchestra.run.smithi157.stderr: 20.0% -- replaced with /var/log/ceph/ceph-client.admin.24089.log.gz 2024-09-21T22:09:30.421 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.23672.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23250.log 2024-09-21T22:09:30.421 INFO:teuthology.orchestra.run.smithi157.stderr: 61.2% -- replaced with /var/log/ceph/ceph-client.admin.23672.log.gz 2024-09-21T22:09:30.421 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.24915.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22909.log 2024-09-21T22:09:30.421 INFO:teuthology.orchestra.run.smithi157.stderr: 19.4% -- replaced with /var/log/ceph/ceph-client.admin.24915.log.gz 2024-09-21T22:09:30.421 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.23250.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23425.log 2024-09-21T22:09:30.421 INFO:teuthology.orchestra.run.smithi157.stderr: 24.1% -- replaced with /var/log/ceph/ceph-client.admin.23250.log.gz 2024-09-21T22:09:30.421 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.22909.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21892.log 2024-09-21T22:09:30.421 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22909.log.gz 2024-09-21T22:09:30.421 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.23425.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22945.log 2024-09-21T22:09:30.421 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23425.log.gz 2024-09-21T22:09:30.421 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.21892.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23164.log 2024-09-21T22:09:30.421 INFO:teuthology.orchestra.run.smithi157.stderr: 19.4% -- replaced with /var/log/ceph/ceph-client.admin.21892.log.gz 2024-09-21T22:09:30.421 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.22945.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24291.log 2024-09-21T22:09:30.422 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22945.log.gz 2024-09-21T22:09:30.422 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.23164.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22495.log 2024-09-21T22:09:30.422 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23164.log.gz 2024-09-21T22:09:30.422 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.24291.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23126.log 2024-09-21T22:09:30.422 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24291.log.gz 2024-09-21T22:09:30.422 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.22495.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23351.log 2024-09-21T22:09:30.422 INFO:teuthology.orchestra.run.smithi157.stderr: 19.4% -- replaced with /var/log/ceph/ceph-client.admin.22495.log.gz 2024-09-21T22:09:30.422 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.23126.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23443.log 2024-09-21T22:09:30.422 INFO:teuthology.orchestra.run.smithi157.stderr: 24.7% -- replaced with /var/log/ceph/ceph-client.admin.23126.log.gz 2024-09-21T22:09:30.422 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.23351.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.23351.log.gz 2024-09-21T22:09:30.422 INFO:teuthology.orchestra.run.smithi157.stderr: --verbose -- /var/log/ceph/ceph-client.admin.23944.log 2024-09-21T22:09:30.422 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.23443.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23214.log 2024-09-21T22:09:30.422 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23443.log.gz 2024-09-21T22:09:30.423 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.23944.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23507.log 2024-09-21T22:09:30.423 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23944.log.gz 2024-09-21T22:09:30.423 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.23214.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23214.log.gz 2024-09-21T22:09:30.423 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22936.log 2024-09-21T22:09:30.423 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.23507.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23323.log 2024-09-21T22:09:30.423 INFO:teuthology.orchestra.run.smithi157.stderr: 19.4% -- replaced with /var/log/ceph/ceph-client.admin.23507.log.gz 2024-09-21T22:09:30.423 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.22936.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22936.log.gz 2024-09-21T22:09:30.423 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23053.log 2024-09-21T22:09:30.423 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.23323.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23970.log 2024-09-21T22:09:30.423 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23323.log.gz 2024-09-21T22:09:30.423 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.23053.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23053.log.gz 2024-09-21T22:09:30.423 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22810.log 2024-09-21T22:09:30.423 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.23970.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22039.log 2024-09-21T22:09:30.423 INFO:teuthology.orchestra.run.smithi157.stderr: 43.9% -- replaced with /var/log/ceph/ceph-client.admin.23970.log.gz 2024-09-21T22:09:30.423 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.22810.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22810.log.gz 2024-09-21T22:09:30.424 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24419.log 2024-09-21T22:09:30.424 INFO:teuthology.orchestra.run.smithi157.stderr:gzip/var/log/ceph/ceph-client.admin.22039.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.22530.log 2024-09-21T22:09:30.424 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22039.log.gz 2024-09-21T22:09:30.424 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.24419.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23765.log 2024-09-21T22:09:30.424 INFO:teuthology.orchestra.run.smithi157.stderr: 54.7% -- replaced with /var/log/ceph/ceph-client.admin.24419.log.gz 2024-09-21T22:09:30.424 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.22530.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21953.log 2024-09-21T22:09:30.424 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22530.log.gz 2024-09-21T22:09:30.424 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.23765.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21774.log 2024-09-21T22:09:30.424 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23765.log.gz 2024-09-21T22:09:30.424 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.21953.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24029.log 2024-09-21T22:09:30.424 INFO:teuthology.orchestra.run.smithi157.stderr: 56.3% -- replaced with /var/log/ceph/ceph-client.admin.21953.log.gz 2024-09-21T22:09:30.424 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.21774.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24098.log 2024-09-21T22:09:30.424 INFO:teuthology.orchestra.run.smithi157.stderr: 18.8% -- replaced with /var/log/ceph/ceph-client.admin.21774.log.gz 2024-09-21T22:09:30.424 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.24029.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23661.log 2024-09-21T22:09:30.424 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24029.log.gz 2024-09-21T22:09:30.425 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.24098.log: gzip -5 --verbose 24.7% -- -- replaced with /var/log/ceph/ceph-client.admin.24098.log.gz /var/log/ceph/ceph-client.admin.22602.log 2024-09-21T22:09:30.425 INFO:teuthology.orchestra.run.smithi157.stderr: 2024-09-21T22:09:30.425 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.23661.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22452.log 2024-09-21T22:09:30.425 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23661.log.gz 2024-09-21T22:09:30.425 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.22602.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22722.log 2024-09-21T22:09:30.425 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22602.log.gz 2024-09-21T22:09:30.425 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.22452.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23115.log 2024-09-21T22:09:30.425 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22452.log.gz 2024-09-21T22:09:30.425 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.22722.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22722.log.gz 2024-09-21T22:09:30.425 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25326.log 2024-09-21T22:09:30.425 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.23115.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24011.log 2024-09-21T22:09:30.425 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23115.log.gz 2024-09-21T22:09:30.425 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.25326.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24515.log 2024-09-21T22:09:30.425 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25326.log.gz 2024-09-21T22:09:30.425 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.24011.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24011.log.gz 2024-09-21T22:09:30.426 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25243.log 2024-09-21T22:09:30.426 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.24515.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23601.log 2024-09-21T22:09:30.426 INFO:teuthology.orchestra.run.smithi157.stderr: 4.5% -- replaced with /var/log/ceph/ceph-client.admin.24515.log.gz 2024-09-21T22:09:30.426 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.25243.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21983.log 2024-09-21T22:09:30.426 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25243.log.gz 2024-09-21T22:09:30.426 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.23601.log: 18.8% -- replaced with /var/log/ceph/ceph-client.admin.23601.log.gzgzip 2024-09-21T22:09:30.426 INFO:teuthology.orchestra.run.smithi157.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.21883.log 2024-09-21T22:09:30.426 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.21983.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25026.log 2024-09-21T22:09:30.426 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.21983.log.gz 2024-09-21T22:09:30.426 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.21883.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.21883.log.gz 2024-09-21T22:09:30.426 INFO:teuthology.orchestra.run.smithi157.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.24866.log 2024-09-21T22:09:30.426 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.25026.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22567.log 2024-09-21T22:09:30.426 INFO:teuthology.orchestra.run.smithi157.stderr: 20.0% -- replaced with /var/log/ceph/ceph-client.admin.25026.log.gz 2024-09-21T22:09:30.427 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.24866.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25305.log 2024-09-21T22:09:30.427 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24866.log.gz 2024-09-21T22:09:30.427 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.22567.log: 19.4% -- replaced with /var/log/ceph/ceph-client.admin.22567.log.gz 2024-09-21T22:09:30.427 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24355.log 2024-09-21T22:09:30.427 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.25305.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22773.log 2024-09-21T22:09:30.427 INFO:teuthology.orchestra.run.smithi157.stderr: 57.6% -- replaced with /var/log/ceph/ceph-client.admin.25305.log.gz 2024-09-21T22:09:30.427 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.24355.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24023.log 2024-09-21T22:09:30.427 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24355.log.gz 2024-09-21T22:09:30.427 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.22773.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22773.log.gz 2024-09-21T22:09:30.427 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23296.log 2024-09-21T22:09:30.427 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23578.log 2024-09-21T22:09:30.427 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.24023.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24023.log.gz 2024-09-21T22:09:30.427 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.23296.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22980.log 2024-09-21T22:09:30.427 INFO:teuthology.orchestra.run.smithi157.stderr: 24.7% -- replaced with /var/log/ceph/ceph-client.admin.23296.log.gz 2024-09-21T22:09:30.427 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.23578.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.23578.log.gz 2024-09-21T22:09:30.428 INFO:teuthology.orchestra.run.smithi157.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.22874.log 2024-09-21T22:09:30.428 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24649.log 2024-09-21T22:09:30.428 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.22980.log: 25.9% -- replaced with /var/log/ceph/ceph-client.admin.22980.log.gz 2024-09-21T22:09:30.428 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.22874.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24383.log 2024-09-21T22:09:30.428 INFO:teuthology.orchestra.run.smithi157.stderr: 19.4% -- replaced with /var/log/ceph/ceph-client.admin.22874.log.gz 2024-09-21T22:09:30.428 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.24649.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23381.log 2024-09-21T22:09:30.428 INFO:teuthology.orchestra.run.smithi157.stderr: 5.5% -- replaced with /var/log/ceph/ceph-client.admin.24649.log.gz 2024-09-21T22:09:30.428 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.24383.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22872.log 2024-09-21T22:09:30.428 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24383.log.gz 2024-09-21T22:09:30.428 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.23381.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21783.log 2024-09-21T22:09:30.428 INFO:teuthology.orchestra.run.smithi157.stderr: 19.4% -- replaced with /var/log/ceph/ceph-client.admin.23381.log.gz 2024-09-21T22:09:30.428 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.22872.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23704.log 2024-09-21T22:09:30.428 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22872.log.gz 2024-09-21T22:09:30.428 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22576.log 2024-09-21T22:09:30.429 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.21783.log: 26.3% -- replaced with /var/log/ceph/ceph-client.admin.21783.log.gz 2024-09-21T22:09:30.429 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.23704.log: gzip 45.1% -- replaced with /var/log/ceph/ceph-client.admin.23704.log.gz -5 2024-09-21T22:09:30.429 INFO:teuthology.orchestra.run.smithi157.stderr: --verbose -- /var/log/ceph/ceph-client.admin.23015.log 2024-09-21T22:09:30.429 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.22576.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23599.log 2024-09-21T22:09:30.429 INFO:teuthology.orchestra.run.smithi157.stderr: 55.9% -- replaced with /var/log/ceph/ceph-client.admin.22576.log.gz 2024-09-21T22:09:30.429 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24148.log 2024-09-21T22:09:30.429 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.23015.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23015.log.gz 2024-09-21T22:09:30.429 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.23599.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22367.log 2024-09-21T22:09:30.429 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23599.log.gz 2024-09-21T22:09:30.429 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.24148.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21848.log 2024-09-21T22:09:30.429 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24148.log.gz 2024-09-21T22:09:30.429 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.22367.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.22367.log.gz 2024-09-21T22:09:30.429 INFO:teuthology.orchestra.run.smithi157.stderr: --verbose -- /var/log/ceph/ceph-client.admin.23287.log 2024-09-21T22:09:30.429 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.21848.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24955.log 2024-09-21T22:09:30.430 INFO:teuthology.orchestra.run.smithi157.stderr: 18.8% -- replaced with /var/log/ceph/ceph-client.admin.21848.log.gz 2024-09-21T22:09:30.430 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.23287.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22651.log 2024-09-21T22:09:30.430 INFO:teuthology.orchestra.run.smithi157.stderr: 19.4% -- replaced with /var/log/ceph/ceph-client.admin.23287.log.gz 2024-09-21T22:09:30.430 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.24955.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21750.log 2024-09-21T22:09:30.430 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24955.log.gz 2024-09-21T22:09:30.430 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.22651.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23542.log 2024-09-21T22:09:30.430 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22651.log.gz 2024-09-21T22:09:30.430 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.21750.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22213.log 2024-09-21T22:09:30.430 INFO:teuthology.orchestra.run.smithi157.stderr: 3.6% -- replaced with /var/log/ceph/ceph-client.admin.21750.log.gz 2024-09-21T22:09:30.430 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.23542.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23826.log 2024-09-21T22:09:30.430 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23542.log.gz 2024-09-21T22:09:30.430 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.22213.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24982.log 2024-09-21T22:09:30.430 INFO:teuthology.orchestra.run.smithi157.stderr: 19.4% -- replaced with /var/log/ceph/ceph-client.admin.22213.log.gz 2024-09-21T22:09:30.430 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.23826.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22304.log 2024-09-21T22:09:30.430 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23826.log.gz 2024-09-21T22:09:30.431 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.24982.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21836.log 2024-09-21T22:09:30.431 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24982.log.gz 2024-09-21T22:09:30.431 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.22304.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22678.log 2024-09-21T22:09:30.431 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22304.log.gz 2024-09-21T22:09:30.431 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.21836.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21604.log 2024-09-21T22:09:30.431 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.21836.log.gz 2024-09-21T22:09:30.431 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.22678.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23179.log 2024-09-21T22:09:30.431 INFO:teuthology.orchestra.run.smithi157.stderr: 19.4% -- replaced with /var/log/ceph/ceph-client.admin.22678.log.gz 2024-09-21T22:09:30.431 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.21604.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23733.log 2024-09-21T22:09:30.431 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.21604.log.gz 2024-09-21T22:09:30.431 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.23179.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23452.log 2024-09-21T22:09:30.431 INFO:teuthology.orchestra.run.smithi157.stderr: 19.4% -- replaced with /var/log/ceph/ceph-client.admin.23179.log.gz 2024-09-21T22:09:30.431 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.23733.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22784.log 2024-09-21T22:09:30.431 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23733.log.gz 2024-09-21T22:09:30.431 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.23452.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24087.log 2024-09-21T22:09:30.432 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23452.log.gz 2024-09-21T22:09:30.432 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.22784.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22354.log 2024-09-21T22:09:30.432 INFO:teuthology.orchestra.run.smithi157.stderr: 25.3% -- replaced with /var/log/ceph/ceph-client.admin.22784.log.gz 2024-09-21T22:09:30.432 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.24087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23285.log 2024-09-21T22:09:30.432 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24087.log.gz 2024-09-21T22:09:30.432 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.22354.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23996.log 2024-09-21T22:09:30.432 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22354.log.gz 2024-09-21T22:09:30.432 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.23285.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22411.log 2024-09-21T22:09:30.432 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23285.log.gz 2024-09-21T22:09:30.432 INFO:teuthology.orchestra.run.smithi157.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21624.log 2024-09-21T22:09:30.432 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.23996.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23996.log.gz 2024-09-21T22:09:30.432 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.22411.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.22411.log.gz 2024-09-21T22:09:30.432 INFO:teuthology.orchestra.run.smithi157.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.25009.log 2024-09-21T22:09:30.432 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.21624.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21741.log 2024-09-21T22:09:30.433 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.21624.log.gz 2024-09-21T22:09:30.433 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.25009.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24446.log 2024-09-21T22:09:30.433 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25009.log.gz 2024-09-21T22:09:30.433 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.21741.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23018.log 2024-09-21T22:09:30.433 INFO:teuthology.orchestra.run.smithi157.stderr: 18.8% -- replaced with /var/log/ceph/ceph-client.admin.21741.log.gz 2024-09-21T22:09:30.433 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.24446.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25006.log 2024-09-21T22:09:30.433 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.24446.log.gz 2024-09-21T22:09:30.433 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.23018.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23390.log 2024-09-21T22:09:30.433 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23018.log.gz 2024-09-21T22:09:30.433 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.25006.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22837.log 2024-09-21T22:09:30.433 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25006.log.gz 2024-09-21T22:09:30.433 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.23390.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23064.log 2024-09-21T22:09:30.433 INFO:teuthology.orchestra.run.smithi157.stderr: 24.1% -- replaced with /var/log/ceph/ceph-client.admin.23390.log.gz 2024-09-21T22:09:30.433 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.22837.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22493.log 2024-09-21T22:09:30.433 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22837.log.gz 2024-09-21T22:09:30.434 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.23064.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22552.log 2024-09-21T22:09:30.434 INFO:teuthology.orchestra.run.smithi157.stderr: 25.9% -- replaced with /var/log/ceph/ceph-client.admin.23064.log.gz 2024-09-21T22:09:30.434 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.22493.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22740.log 2024-09-21T22:09:30.434 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22493.log.gz 2024-09-21T22:09:30.434 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.22552.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22431.log 2024-09-21T22:09:30.434 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22552.log.gz 2024-09-21T22:09:30.434 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.22740.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25285.log 2024-09-21T22:09:30.434 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22740.log.gz 2024-09-21T22:09:30.434 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.22431.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24887.log 2024-09-21T22:09:30.434 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22431.log.gz 2024-09-21T22:09:30.434 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.25285.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.23152.log 2024-09-21T22:09:30.434 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25285.log.gz 2024-09-21T22:09:30.434 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.24887.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.23880.log -- replaced with /var/log/ceph/ceph-client.admin.24887.log.gz 2024-09-21T22:09:30.434 INFO:teuthology.orchestra.run.smithi157.stderr: 2024-09-21T22:09:30.435 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.23152.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25202.log 2024-09-21T22:09:30.435 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23152.log.gz 2024-09-21T22:09:30.435 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.23880.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21857.log 2024-09-21T22:09:30.435 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.23880.log.gz 2024-09-21T22:09:30.435 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.25202.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22918.log 2024-09-21T22:09:30.435 INFO:teuthology.orchestra.run.smithi157.stderr: 24.2% -- replaced with /var/log/ceph/ceph-client.admin.25202.log.gz 2024-09-21T22:09:30.435 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.21857.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21584.log 2024-09-21T22:09:30.435 INFO:teuthology.orchestra.run.smithi157.stderr: 23.5% -- replaced with /var/log/ceph/ceph-client.admin.21857.log.gz 2024-09-21T22:09:30.435 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.22918.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.24311.log 2024-09-21T22:09:30.435 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22918.log.gz 2024-09-21T22:09:30.435 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.21584.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21724.log 2024-09-21T22:09:30.435 INFO:teuthology.orchestra.run.smithi157.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.21584.log.gz 2024-09-21T22:09:30.435 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.24311.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22971.log 2024-09-21T22:09:30.435 INFO:teuthology.orchestra.run.smithi157.stderr: 19.4% -- replaced with /var/log/ceph/ceph-client.admin.24311.log.gz 2024-09-21T22:09:30.435 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.21724.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.21724.log.gz 2024-09-21T22:09:30.436 INFO:teuthology.orchestra.run.smithi157.stderr:/var/log/ceph/ceph-client.admin.22971.log: 19.4% -- replaced with /var/log/ceph/ceph-client.admin.22971.log.gz 2024-09-21T22:09:30.436 INFO:teuthology.orchestra.run.smithi157.stderr: 2024-09-21T22:09:30.436 INFO:teuthology.orchestra.run.smithi157.stderr:real 0m0.167s 2024-09-21T22:09:30.436 INFO:teuthology.orchestra.run.smithi157.stderr:user 0m0.206s 2024-09-21T22:09:30.436 INFO:teuthology.orchestra.run.smithi157.stderr:sys 0m0.058s 2024-09-21T22:09:30.436 INFO:teuthology.orchestra.run.smithi039.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-09-21T22:09:30.436 INFO:teuthology.orchestra.run.smithi039.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-09-21T22:09:30.494 INFO:teuthology.orchestra.run.smithi039.stderr: 90.6% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-09-21T22:09:30.547 INFO:teuthology.orchestra.run.smithi064.stderr: 94.4% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-09-21T22:09:30.582 INFO:teuthology.orchestra.run.smithi039.stderr: 94.1% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-09-21T22:09:30.584 INFO:teuthology.orchestra.run.smithi039.stderr: 2024-09-21T22:09:30.584 INFO:teuthology.orchestra.run.smithi039.stderr:real 0m0.401s 2024-09-21T22:09:30.584 INFO:teuthology.orchestra.run.smithi039.stderr:user 0m1.238s 2024-09-21T22:09:30.584 INFO:teuthology.orchestra.run.smithi039.stderr:sys 0m0.081s 2024-09-21T22:09:30.705 INFO:teuthology.orchestra.run.smithi064.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-09-21T22:09:30.706 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-09-21T22:09:30.706 INFO:teuthology.orchestra.run.smithi064.stderr:real 0m0.499s 2024-09-21T22:09:30.706 INFO:teuthology.orchestra.run.smithi064.stderr:user 0m1.119s 2024-09-21T22:09:30.707 INFO:teuthology.orchestra.run.smithi064.stderr:sys 0m0.085s 2024-09-21T22:09:30.707 INFO:tasks.ceph:Archiving logs... 2024-09-21T22:09:30.708 DEBUG:teuthology.misc:Transferring archived files from smithi039:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-21_21:16:02-rbd-squid-distro-default-smithi/7914449/remote/smithi039/log 2024-09-21T22:09:30.709 DEBUG:teuthology.orchestra.run.smithi039:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-21T22:09:31.006 DEBUG:teuthology.misc:Transferring archived files from smithi064:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-21_21:16:02-rbd-squid-distro-default-smithi/7914449/remote/smithi064/log 2024-09-21T22:09:31.007 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-21T22:09:31.092 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 has been restored 2024-09-21T22:09:31.149 DEBUG:teuthology.misc:Transferring archived files from smithi157:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-21_21:16:02-rbd-squid-distro-default-smithi/7914449/remote/smithi157/log 2024-09-21T22:09:31.150 DEBUG:teuthology.orchestra.run.smithi157:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-21T22:09:31.345 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-21T22:09:31.368 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-21T22:09:31.368 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-21T22:09:31.371 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-21T22:09:31.373 DEBUG:teuthology.orchestra.run.smithi157:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-21T22:09:31.459 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-21T22:09:31.469 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-21T22:09:31.469 DEBUG:teuthology.orchestra.run.smithi039:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T22:09:31.471 DEBUG:teuthology.orchestra.run.smithi064:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T22:09:31.473 DEBUG:teuthology.orchestra.run.smithi157:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T22:09:31.481 INFO:teuthology.orchestra.run.smithi039.stdout: remote refid st t when poll reach delay offset jitter 2024-09-21T22:09:31.481 INFO:teuthology.orchestra.run.smithi039.stdout:============================================================================== 2024-09-21T22:09:31.481 INFO:teuthology.orchestra.run.smithi039.stdout:*hv01.front.sepi 67.205.162.81 3 u 43 64 77 0.070 -0.509 0.876 2024-09-21T22:09:31.481 INFO:teuthology.orchestra.run.smithi039.stdout:+hv02.front.sepi 63.231.80.2 3 u 38 64 77 0.088 +1.058 0.733 2024-09-21T22:09:31.481 INFO:teuthology.orchestra.run.smithi039.stdout:+hv03.front.sepi 74.6.168.72 3 u 43 64 77 0.121 -1.382 0.831 2024-09-21T22:09:31.481 INFO:teuthology.orchestra.run.smithi039.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-21T22:09:31.485 INFO:teuthology.orchestra.run.smithi064.stdout: remote refid st t when poll reach delay offset jitter 2024-09-21T22:09:31.485 INFO:teuthology.orchestra.run.smithi064.stdout:============================================================================== 2024-09-21T22:09:31.485 INFO:teuthology.orchestra.run.smithi064.stdout: hv01.front.sepi 67.205.162.81 3 u 42 64 77 0.085 -1.395 1.435 2024-09-21T22:09:31.485 INFO:teuthology.orchestra.run.smithi064.stdout:*hv02.front.sepi 63.231.80.2 3 u 49 64 77 0.078 +1.696 1.653 2024-09-21T22:09:31.485 INFO:teuthology.orchestra.run.smithi064.stdout: hv03.front.sepi 74.6.168.72 3 u 43 64 77 0.081 -0.768 1.880 2024-09-21T22:09:31.485 INFO:teuthology.orchestra.run.smithi064.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-21T22:09:31.487 INFO:teuthology.orchestra.run.smithi157.stdout: remote refid st t when poll reach delay offset jitter 2024-09-21T22:09:31.487 INFO:teuthology.orchestra.run.smithi157.stdout:============================================================================== 2024-09-21T22:09:31.488 INFO:teuthology.orchestra.run.smithi157.stdout:+hv01.front.sepi 67.205.162.81 3 u 39 64 77 0.057 -1.289 1.647 2024-09-21T22:09:31.488 INFO:teuthology.orchestra.run.smithi157.stdout:+hv02.front.sepi 63.231.80.2 3 u 37 64 77 0.065 -0.634 1.875 2024-09-21T22:09:31.488 INFO:teuthology.orchestra.run.smithi157.stdout:*hv03.front.sepi 74.6.168.72 3 u 45 64 77 0.089 -0.206 2.827 2024-09-21T22:09:31.488 INFO:teuthology.orchestra.run.smithi157.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-21T22:09:31.488 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-21T22:09:31.497 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-21T22:09:31.498 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-21T22:09:31.506 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-21T22:09:31.514 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-21T22:09:31.526 INFO:teuthology.task.internal:Duration was 832.713956 seconds 2024-09-21T22:09:31.527 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-21T22:09:31.534 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-21T22:09:31.534 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-21T22:09:31.537 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-21T22:09:31.539 DEBUG:teuthology.orchestra.run.smithi157:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-21T22:09:31.577 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-21T22:09:31.577 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi039.front.sepia.ceph.com 2024-09-21T22:09:31.577 DEBUG:teuthology.orchestra.run.smithi039:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-21T22:09:31.625 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi064.front.sepia.ceph.com 2024-09-21T22:09:31.625 DEBUG:teuthology.orchestra.run.smithi064:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-21T22:09:31.636 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi157.front.sepia.ceph.com 2024-09-21T22:09:31.636 DEBUG:teuthology.orchestra.run.smithi157:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-21T22:09:31.646 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-21T22:09:31.646 DEBUG:teuthology.orchestra.run.smithi039:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-21T22:09:31.668 DEBUG:teuthology.orchestra.run.smithi064:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-21T22:09:31.682 DEBUG:teuthology.orchestra.run.smithi157:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-21T22:09:31.720 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-21T22:09:31.720 DEBUG:teuthology.orchestra.run.smithi039:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-21T22:09:31.741 DEBUG:teuthology.orchestra.run.smithi064:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-21T22:09:31.754 DEBUG:teuthology.orchestra.run.smithi157:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-21T22:09:31.821 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-21T22:09:31.836 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-21T22:09:31.836 DEBUG:teuthology.orchestra.run.smithi039:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-21T22:09:31.849 DEBUG:teuthology.orchestra.run.smithi064:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-21T22:09:31.862 DEBUG:teuthology.orchestra.run.smithi157:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-21T22:09:31.873 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-21T22:09:31.928 DEBUG:teuthology.orchestra.run.smithi039:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-21T22:09:31.931 DEBUG:teuthology.orchestra.run.smithi064:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-21T22:09:31.934 DEBUG:teuthology.orchestra.run.smithi157:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-21T22:09:31.941 INFO:teuthology.orchestra.run.smithi039.stdout:kernel.core_pattern = core 2024-09-21T22:09:31.944 INFO:teuthology.orchestra.run.smithi064.stdout:kernel.core_pattern = core 2024-09-21T22:09:31.947 INFO:teuthology.orchestra.run.smithi157.stdout:kernel.core_pattern = core 2024-09-21T22:09:31.966 DEBUG:teuthology.orchestra.run.smithi039:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-21T22:09:32.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T22:09:32.011 DEBUG:teuthology.orchestra.run.smithi064:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-21T22:09:32.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T22:09:32.017 DEBUG:teuthology.orchestra.run.smithi157:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-21T22:09:32.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T22:09:32.021 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-21T22:09:32.030 INFO:teuthology.task.internal:Transferring archived files... 2024-09-21T22:09:32.034 DEBUG:teuthology.misc:Transferring archived files from smithi039:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-21_21:16:02-rbd-squid-distro-default-smithi/7914449/remote/smithi039 2024-09-21T22:09:32.035 DEBUG:teuthology.orchestra.run.smithi039:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-21T22:09:32.077 DEBUG:teuthology.misc:Transferring archived files from smithi064:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-21_21:16:02-rbd-squid-distro-default-smithi/7914449/remote/smithi064 2024-09-21T22:09:32.077 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-21T22:09:32.098 DEBUG:teuthology.misc:Transferring archived files from smithi157:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-21_21:16:02-rbd-squid-distro-default-smithi/7914449/remote/smithi157 2024-09-21T22:09:32.099 DEBUG:teuthology.orchestra.run.smithi157:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-21T22:09:32.118 INFO:teuthology.task.internal:Removing archive directory... 2024-09-21T22:09:32.118 DEBUG:teuthology.orchestra.run.smithi039:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-21T22:09:32.121 DEBUG:teuthology.orchestra.run.smithi064:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-21T22:09:32.142 DEBUG:teuthology.orchestra.run.smithi157:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-21T22:09:32.162 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-21T22:09:32.178 INFO:teuthology.task.internal:Not uploading archives. 2024-09-21T22:09:32.179 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-21T22:09:32.188 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-21T22:09:32.188 DEBUG:teuthology.orchestra.run.smithi039:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-21T22:09:32.191 DEBUG:teuthology.orchestra.run.smithi064:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-21T22:09:32.193 DEBUG:teuthology.orchestra.run.smithi157:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-21T22:09:32.195 INFO:teuthology.orchestra.run.smithi039.stdout: 658634 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 21 22:09 /home/ubuntu/cephtest 2024-09-21T22:09:32.196 INFO:teuthology.orchestra.run.smithi064.stdout: 658641 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 21 22:09 /home/ubuntu/cephtest 2024-09-21T22:09:32.210 INFO:teuthology.orchestra.run.smithi157.stdout: 658636 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 21 22:09 /home/ubuntu/cephtest 2024-09-21T22:09:32.212 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-21T22:09:32.227 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-21T22:09:32.251 INFO:teuthology.run:Summary data: description: rbd/device/{base/install cluster/{fixed-3 openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-zlib supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/rbd_nbd} duration: 832.7139558792114 flavor: default owner: scheduled_teuthology@teuthology success: true 2024-09-21T22:09:32.251 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-21T22:09:32.334 INFO:teuthology.run:pass